Realtime Stats
Readers: 0
Likes: 0
Let's D4Y |> using K8S
As promised in my last post, here is step 3 in my deployment adventure. The one where I use Kubernetes!
In the last post we explored a very straightforward and simple way to deploy a Phoenix application using the built-in Elixir releases on render.com.
Today I want to discuss a much more complicated setup, using Kubernetes. I know for most applications you will surely not need a setup using Kubernetes, but I've been having an urge to try it out for a while now and wanted to see if it is really that complicated as I seem to be hearing through the grapevine. Who knows, maybe it turns out it's actually not that hard at all. Also I really like the fact that I can have a cluster to spin up nodes (ie. applications) of all the experiments / ideas / apps that I'm currently playing with, so I can actually expose those into the Real World with a reproducable and easy deployment workflow. Let's see if we can make that dream come true!
The Kubernetes Cluster
If you are not aware of what Kubernetes is, head over to their website, as they have great documentation to get started on the basics.
To use Kubernetes for our deployment the first thing we'll need is a Kubernetes cluster. This is the part where you see most tutorials and guides grab for a local setup with minikube. While I like that it's possible to do this, it still seems a bit too far from real life for me, so I wanted to have a bit more of a production-like environment, like... for instance... a production environment :)
The beauty is that these days there are more and more managed Kubernetes services that do all the hard work like setting up the cluster, managing upgrades etc etc. I'll be using the service DigitalOcean is offering. Mainly because I already have a lot of content running at DigitalOcean, so it is known territory for me and they make it really easy to get started with Kubernetes. If you haven't used DigitalOcean before, you can use this referral link to create an account and get $100 credit to spend in the first 60 days. So more than enough to follow along with this tutorial.
Terraform + Helm
We are going to need to use 2 more tools to get this setup up and running. Terraform and Helm. We'll use terraform to setup most of the infrastructure. In this way I don't have to remember all the tweaks and install steps I took to get everything up and running. I can just automate the whole setup declaratively and put the whole thing under version control. This is often referred to these days as an infrastructure-as-code setup. I wanted to use terraform for everything, but ran into some trouble setting up a few of the tools needed, so reverted to Helm (the Kubernetes package manager) to install Traefik and our Gitlab Runner.
Our Terraform setup will include:
- Setting up a DigitalOcean Kubernetes Cluster with 2 worker nodes
- Connecting a gitlab project to the kubernetes cluster
- Creating a kubernetes_secret to be able to pull images from our private gitlab registry.
- Setup a managed database cluster service @ digitalocean
And we'll use Helm to:
- Install Traefik as an ingress controller
- Create a Gitlab Runner in our cluster
So let's get crackin'!
Install doctl to talk to your DigitalOcean account
If we want to automate any of this stuff we'll have to be able to talk to DigitalOcean programmatically, so installing doctl is step one. Please refer to this guide to get that setup: doctl up and running guide
After you have doctl
installed, make sure you create a personal access token in DigitalOcean and add that in an environment variable and also initialize your
account.
export DIGITALOCEAN_TOKEN=[token]
doctl auth init -t $DIGITALOCEAN_TOKEN
doctl account get
Install Terraform
Of course there is a handy install guide for terraform also. Actually just an executable, as it is written in go, so they have a handy packaged binary you can use. See: terraform install
Setup our project
If you want to follow along with this guide, create a folder that will hold your
terraform configs and create a main.tf
file that will hold the terraform declarations.
mkdir kube-terra
cd kube-terra
touch main.tf
Setup the DigitalOcean Kubernetes cluster
Terraform works in a declaritive manner, which means that we state what we want the world to be like and terraform figures out how to get to that state. In terraform you describe an item you want to exist as a resource. A Kubernetes cluster is a form of a resource that is provided by one of the many providers that exist in terraform. So let's create a DigitalOcean kubernetes cluster:
Creating a resource is always in the format: resource [kind] [reference] {}
We can use the reference we provide later on to refer back to this resource.
resource "digitalocean_tag" "kubernetes-cl01" {
name = "kubernetes-cl01"
}
data "digitalocean_kubernetes_versions" "versions" {}
resource "digitalocean_kubernetes_cluster" "cl01" {
name = "cl01"
region = "ams3"
version = data.digitalocean_kubernetes_versions.versions.latest_version
node_pool {
name = "default"
size = "s-1vcpu-2gb"
node_count = 2
tags = ["${digitalocean_tag.kubernetes-cl01.id}"]
}
}
Now you can get your cluster setup by issuing the following commands in the folder you created:
tarraform init
terraform apply
This will take about 5 minutes and then you'll have a fresh kubernetes cluster up and running. Now that was easy!
Install Traefik
Kubernetes needs an ingress controller to route traffic from the outside world
to the services that are running inside the cluster as these services are not
exposed to the outside world, which is a good thing! Nginx is often used, but I went with Traefik, a very nice alternative that has a lot of nice functinality out of the box, like automatic ssl certificates using Let's Encrypt
and auto discovery of services running in the cluster. This setup also takes advantage of a DigitalOcean load balancer, which will automatically be created by the serviceType set below to LoadBalancer
.
Create a file called traefik-values.yml
in the root of your kubernetes config directory and add the following content. If you manage dns settings with DigitalOcean, you can comment out those parts as well. This will provide automatic ssl certificate generation. Very cool! If not you can just leave it like this. It'll work, but without ssl enabled. A benefit of using the dns-challenge as opposed to the more standard acme-challenge is that you can also get a valid certificate if your production system is behind a firewall.
image: traefik
dashboard:
enabled: true
domain: domain.example.com # put a (sub)domain here where you want to access the traefik dashboard
serviceType: LoadBalancer
rbac:
enabled: true
# ssl:
# enabled: true # Enables SSL
# enforced: true # Redirects HTTP to HTTPS
# acme:
# enabled: true # Enables Let's Encrypt certificates
# staging: false # Use Lets Encrypt staging area for this example. For production purposes set this to false
# email: info@drumusician.com # Email address that Let's Encrypt uses to notify about certificate expiry etc.
# challengeType: "dns-01"
# dnsProvider:
# name: digitalocean # This is why you need your domain to be under Digital Ocean control
# digitalocean:
# DO_AUTH_TOKEN: $DIGITALOCEAN_TOKEN
# domains:
# enabled: true
# domainsList:
# - main: domain.example.com # domain that belongs to this certificate
Now we can go ahead and install traefik in our cluster:
Install helm:
brew install kubernetes-helm
Pull in our fresh cluster configuration locally (otherwise helm will fail):
doctl kubernetes cluster kubeconfig save cl01
And install traefik:
# get the helm/stable charts
helm repo add stable https://kubernetes-charts.storage.googleapis.com/
# install Traefik
helm install traefik --values traefik-values.yml stable/traefik
After Traefik is installed we have enough running to be able to deploy our application to the cluster. We are going to use gitlab-ci to deploy so we'll need to install a gitlab-runner in our cluster first so we can use that in our project.
Creating a kubernetes secret to access our registry
If we install the Gitlab runner in the next step, we'll need a kubernetes secret to be able to pull from our private registry in gitlab. Go to https://gitlab.com/profile/personal_access_tokens and create an access token to access the gitlab registry. Scopes: api, read_registry.
Then create a file at ~/.docker/docker-registry.json
with the credentials for
the access token:
{
"registry.gitlab.com": {
"username": "", # access-token-name
"password": "", # token
"email": "" # email of the gitlab account
}
}
And then in our main.tf we can create the kubernetes secret.
resource "kubernetes_secret" "docker_pull_secret" {
metadata {
name = "gitlab.com"
}
data = {
".dockercfg" = file("~/.docker/docker-registry.json")
}
type = "kubernetes.io/dockercfg"
}
With that in place, we'll have to initialize the new provider and then we can add the secret:
terraform init
terraform apply
Install Gitlab Runner
To be able to deploy into our cluster from Gitlab CI we will need our runner to run inside of our cluster and have enough access rights to actually spin-up pods and services in our cluster. So we'll install that using helm as well. Gitlab has an install script from their interface, but that doesn't provide you with customisation options that we'll need for our use case.
Create the following file in your kubernetes dir: gitlab-runner-values.yml
and
add this content:
gitlabUrl: https://gitlab.com/
runnerRegistrationToken: "" # copy this from your gitlab project settings:
rbac:
create: true
clusterWideAccess: true
runners:
privileged: true
imagePullSecrets: ["gitlab.com"]
Now let's install the runner:
helm install --namespace default gitlab-runner -f gitlab-runner-values.yml gitlab/gitlab-runner
Connect a gitlab project to the cluster
This is something you can do through the Gitlab interface, but I like automating
it here as well. We need to provide the Gitlab project in terraform like this.
If you don't have a project yet, you should create it on gitlab.com
and add the
ID here.
data "gitlab_project" "project-x" {
id = [your-project-id]
}
Then we can use this reference to create the settings in our project:
resource "gitlab_project_cluster" "gitlab-kubernetes" {
project = data.gitlab_project.project-x.id
name = "my-awesome-cluster"
domain = "[mydomain.com]"
enabled = true
kubernetes_api_url = digitalocean_kubernetes_cluster.cl01.endpoint
kubernetes_token = digitalocean_kubernetes_cluster.cl01.kube_config[0].token
kubernetes_ca_cert = base64decode(digitalocean_kubernetes_cluster.cl01.kube_config[0].cluster_ca_certificate)
kubernetes_namespace = ""
kubernetes_authorization_type = "rbac"
environment_scope = "*"
}
And lastly we are not going the skip the database, as that is too easy to do. You could potentially setup a database persistent volume in kubernetes and have db_pods spinup, but I think that it is much easier to have the database as a separate service outside of kubernetes. The DigitalOcean managed database service is a great option. You pay a little extra, but it takes care of backups etc. It is just one of those things that you don't want to worry about, right?
Of course, we can easily add the creation of a database cluster to our terraform setup. Note that I am using only 1 node for the cluster in this example. For actual failsafe production usage, you probably want to have at least 2 nodes.
resource "digitalocean_database_cluster" "postgres-db" {
name = "postgres-db-cluster"
engine = "pg"
version = "11"
size = "db-s-1vcpu-1gb"
region = "ams3"
node_count = 1
}
And while we are at it, let's create the database and firewall settings as well.
resource "digitalocean_database_db" "test-prod" {
cluster_id = digitalocean_database_cluster.postgres-db.id
name = "test_prod"
}
resource "digitalocean_database_firewall" "db-fw" {
cluster_id = digitalocean_database_cluster.postgres-db.id
rule {
type = "k8s"
value = digitalocean_kubernetes_cluster.cl01.id
}
}
This concludes our cluster setup. To get our app deployed we'll have to add some setup to our project. We'll need to dockerize our project, add kubectl deploy config files and add a gitlab-ci.yml that will trigger the gitlab-ci pipeline.
A project to deploy
Now we have our kubernetes cluster up-and-running it is time to see how we would deploy our app to the cluster.
I have a git tag prepared for you to use from my real_world_phoenix project. You can clone that checkout to use to deploy to the cluster. It has the necessary files to trigger deployment which I'll explain next.
Go ahead and clone the project from the tag I created:
git clone https://gitlab.com/drumusician/real_world_phoenix.git --branch kubernetes-deploy
Gitlab pipeline
To get our app deployed to kubernetes we'll need to package it up in a container and we'll use Gitlab CI system for all of this. That means we'll prepare our app and package it using a 2-step dockerfile. The advantage of the 2-step dockerfile method is that we can use a larger docker image to package our app that has all the packages for building and packaging our release and use a much slimmer container to just run our release. The release will be self-contained, so there is not a lot needed for it to run.
The steps we'll need in our CI file are:
- init # some compiling and pushing of artifacts
- build and push # build our container and push to our registry
- deploy # deploy to our kubernetes cluster
For pushing we'll use the gitlab container registry that is available in every gitlab project. The deployment will be a collection of kubernetes yaml files that we can version control in our project.
Dockerfile
# ---- Build Stage ----
FROM elixir:alpine AS app_builder
# Set environment variables for building the application
ENV MIX_ENV=prod \
TEST=1 \
LANG=C.UTF-8
RUN apk add --update git nodejs npm && \
rm -rf /var/cache/apk/*
# Install hex and rebar
RUN mix local.hex --force && \
mix local.rebar --force
# Create the application build directory
RUN mkdir /app
WORKDIR /app
# Copy over all the necessary application files and directories
COPY config ./config
COPY lib ./lib
COPY priv ./priv
COPY assets ./assets
COPY mix.exs .
COPY mix.lock .
# Fetch the application dependencies and build the application
RUN mix deps.get
RUN mix deps.compile
RUN npm run deploy --prefix ./assets
RUN mix phx.digest
RUN mix release
# ---- Application Stage ----
FROM alpine AS app
ENV LANG=C.UTF-8
# Install openssl
RUN apk add --update openssl ncurses-libs postgresql-client && \
rm -rf /var/cache/apk/*
# Copy over the build artifact from the previous step and create a non root user
RUN adduser -D -h /home/app app
WORKDIR /home/app
COPY --from=app_builder /app/_build .
RUN chown -R app: ./prod
USER app
COPY entrypoint.sh .
# Run the Phoenix app
CMD ["./entrypoint.sh"]
Entrypoint
The entrypoint.sh is basically just a bash script that verifies the db is up and running before starting and running any pending migrations before startup as well.
#!/bin/sh
# Docker entrypoint script.
# Wait until Postgres is ready
# while ! pg_isready -q -h $DB_HOST -p 5432 -U $DB_USER
# do
# echo "$(date) - waiting for database to start"
# sleep 2
# done
./prod/rel/real_world_phoenix/bin/real_world_phoenix eval RealWorldPhoenix.Release.migrate
./prod/rel/real_world_phoenix/bin/real_world_phoenix start
Gitlab CI Yaml file
To trigger the gitlab-ci pipeline we'll need to add a .gitlab-ci.yml
file in
the root of our project.
This is the content needed to initialize, build, push and deploy using the CI pipeline. We start off creating a few variables we'll need and then define the stages. I have added some comments in the yaml file below that should explain the steps in detail.
Note the use of Kaniko to build the container and push it to our registry. Kaniko doesn't depend on a Docker daemon and executes each command within a Dockerfile completely in userspace. This enables building container images in environments that can't easily or securely run a Docker daemon, such as a our Kubernetes cluster. Very nice!
variables:
REGISTRY: registry.gitlab.com
CONTAINER_RELEASE_IMAGE: $REGISTRY/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME:$CI_COMMIT_SHORT_SHA
stages:
- init
- build
- deploy
# the dot syntax makes this a hidden step that we can include in other places further down.
.elixir_default: &elixir_default
image: elixir:1.9
before_script:
- mix local.hex --force
- mix local.rebar --force
.javascript_default: &javascript_default
image: node:alpine
before_script:
- cd assets
# Compile our elixir artifacts
elixir_compile:
<<: *elixir_default
stage: init
script:
- mix deps.get --only test
- mix compile
- mix compile --warnings-as-errors
artifacts:
paths:
- mix.lock
- _build
- deps
# compile javascript artifacts
javascript_deps:
<<: *javascript_default
stage: init
script:
- npm install --progress=false
artifacts:
paths:
- assets/node_modules
# build our container image
build:
stage: build
image:
name: gcr.io/kaniko-project/executor:debug
entrypoint: [""]
script:
- echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
- /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CONTAINER_RELEASE_IMAGE
# and deploy our kubernetes cluster
deploy:
stage: deploy
image:
name: lwolf/kubectl_deployer:latest
script:
- cat deploy.yml | envsubst | kubectl apply -f -
- cat service.yml | envsubst | kubectl apply -f -
- cat ingress.yml | envsubst | kubectl apply -f -
only:
refs:
- master
Yaml deployment files: service, ingress and deploy
The last thing we need is the kubernetes yaml files that are used in the ci pipeline:
service.yml
The service pod running our application. This is basically our app running in a docker container.
apiVersion: v1
kind: Service
metadata:
name: real-world-phoenix-service
spec:
ports:
- name: web
port: 8001
protocol: TCP
selector:
app: real_world_phoenix
ingress.yml
The mapping of our domain name to our service
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: real-world-phoenix-ingress
spec:
rules:
- host: $DOMAIN
http:
paths:
- path: "/"
backend:
serviceName: real-world-phoenix-service
servicePort: web
deploy.yml
Our deployment configuration. This provides Kubernetes the info about which container image to use and how many replicas of our app we want to have running.
apiVersion: apps/v1
kind: Deployment
metadata:
name: real-world-phoenix
labels:
app: real_world_phoenix
spec:
revisionHistoryLimit: 5
replicas: 1
selector:
matchLabels:
app: real_world_phoenix
template:
metadata:
labels:
app: real_world_phoenix
spec:
imagePullSecrets:
- name: "gitlab.com"
containers:
- name: real-world-phoenix
image: registry.gitlab.com/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME:$CI_COMMIT_SHORT_SHA
imagePullPolicy: Always
name: real-world-phoenix-deployment
resources: {}
ports:
- containerPort: 8001
env:
- name: SECRET_KEY_BASE
value: $SECRET_KEY_BASE
- name: DB_USER
value: $DB_USER
- name: DB_PASSWORD
value: $DB_PASS
- name: DB_NAME
value: "test_prod"
- name: DB_HOST
value: $DB_CLUSTER
- name: DB_PORT
value: "25060"
- name: APP_PORT
value: "8001"
- name: APP_HOSTNAME
value: $DOMAIN
The above yaml file has references to a number of environment variables. The nice thing is that we can add all of these secrets inside of our Gitlab project and the deployment will use these values. In your project go to Settings -> CI/CD -> Variables
.
Now if we have all of this in our project all we have to do is commit and push it to our gitlab repo. Then Gitlab CI will do it's magic and our application will be up and running after the pipeline jobs succeed. Nice!
Conclusion
While it is not necessarily the easiest setup, I really like the fact that once I have this setup I can fairly easily put some configuration in any of my gitlab projects and CI will take care of the rest, even the ssl certificate is handled for me so I don't have to think about renewing them and all that stuff.
The reduction of manual work by using terraform is also a really nice benefit here. Terraform is a powerful tool, so use with care! If there is any interest in exploring terraform more, please leave a comment and I'll plan in some more in-depth post exploring the world of terraform.
I hope this post was easy to follow and I hope that for those who followed along everything worked as expected? If not, do let me know, because I want to make sure everything here works as expected.
Until next time!