One place for hosting & domains


      Webinar Series: Building Blocks for Doing CI/CD with Kubernetes

      Webinar Series

      This article supplements a webinar series on doing CI/CD with Kubernetes. The series discusses how to take a Cloud Native approach to building, testing, and deploying applications, covering release management, Cloud Native tools, Service Meshes, and CI/CD tools that can be used with Kubernetes. It is designed to help developers and businesses that are interested in integrating CI/CD best practices with Kubernetes into their workflows.

      This tutorial includes the concepts and commands from the first session of the series, Building Blocks for Doing CI/CD with Kubernetes.


      If you are getting started with containers, you will likely want to know how to automate building, testing, and deployment. By taking a Cloud Native approach to these processes, you can leverage the right infrastructure APIs to package and deploy applications in an automated way.

      Two building blocks for doing automation include container images and container orchestrators. Over the last year or so, Kubernetes has become the default choice for container orchestration. In this first article of the CI/CD with Kubernetes series, you will:

      • Build container images with Docker, Buildah, and Kaniko.
      • Set up a Kubernetes cluster with Terraform, and create Deployments and Services.
      • Extend the functionality of a Kubernetes cluster with Custom Resources.

      By the end of this tutorial, you will have container images built with Docker, Buildah, and Kaniko, and a Kubernetes cluster with Deployments, Services, and Custom Resources.

      Future articles in the series will cover related topics: package management for Kubernetes, CI/CD tools like Jenkins X and Spinnaker, Services Meshes, and GitOps.


      Step 1 — Building Container Images with Docker and Buildah

      A container image is a self-contained entity with its own application code, runtime, and dependencies that you can use to create and run containers. You can use different tools to create container images, and in this step you will build containers with two of them: Docker and Buildah.

      Building Container Images with Dockerfiles

      Docker builds your container images automatically by reading instructions from a Dockerfile, a text file that includes the commands required to assemble a container image. Using the docker image build command, you can create an automated build that will execute the command-line instructions provided in the Dockerfile. When building the image, you will also pass the build context with the Dockerfile, which contains the set of files required to create an environment and run an application in the container image.

      Typically, you will create a project folder for your Dockerfile and build context. Create a folder called demo to begin:

      Next, create a Dockerfile inside the demo folder:

      Add the following content to the file:


      FROM ubuntu:16.04
      RUN apt-get update 
          && apt-get install -y nginx 
          && apt-get clean 
          && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* 
          && echo "daemon off;" >> /etc/nginx/nginx.conf
      EXPOSE 80
      CMD ["nginx"]

      This Dockerfile consists of a set of instructions that will build an image to run Nginx. During the build process ubuntu:16.04 will function as the base image, and the nginx package will be installed. Using the CMD instruction, you've also configured nginx to be the default command when the container starts.

      Next, you'll build the container image with the docker image build command, using the current directory (.) as the build context. Passing the -t option to this command names the image nkhare/nginx:latest:

      • sudo docker image build -t nkhare/nginx:latest .

      You will see the following output:


      Sending build context to Docker daemon 49.25MB Step 1/5 : FROM ubuntu:16.04 ---> 7aa3602ab41e Step 2/5 : MAINTAINER ---> Using cache ---> 552b90c2ff8d Step 3/5 : RUN apt-get update && apt-get install -y nginx && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && echo "daemon off;" >> /etc/nginx/nginx.conf ---> Using cache ---> 6bea966278d8 Step 4/5 : EXPOSE 80 ---> Using cache ---> 8f1c4281309e Step 5/5 : CMD ["nginx"] ---> Using cache ---> f545da818f47 Successfully built f545da818f47 Successfully tagged nginx:latest

      Your image is now built. You can list your Docker images using the following command:


      REPOSITORY TAG IMAGE ID CREATED SIZE nkhare/nginx latest 4073540cbcec 3 seconds ago 171MB ubuntu 16.04 7aa3602ab41e 11 days ago

      You can now use the nkhare/nginx:latest image to create containers.

      Building Container Images with Project Atomic-Buildah

      Buildah is a CLI tool, developed by Project Atomic, for quickly building Open Container Initiative (OCI)-compliant images. OCI provides specifications for container runtimes and images in an effort to standardize industry best practices.

      Buildah can create an image either from a working container or from a Dockerfile. It can build images completely in user space without the Docker daemon, and can perform image operations like build, list, push, and tag. In this step, you'll compile Buildah from source and then use it to create a container image.

      To install Buildah you will need the required dependencies, including tools that will enable you to manage packages and package security, among other things. Run the following commands to install these packages:

      • cd
      • sudo apt-get install software-properties-common
      • sudo add-apt-repository ppa:alexlarsson/flatpak
      • sudo add-apt-repository ppa:gophers/archive
      • sudo apt-add-repository ppa:projectatomic/ppa
      • sudo apt-get update
      • sudo apt-get install bats btrfs-tools git libapparmor-dev libdevmapper-dev libglib2.0-dev libgpgme11-dev libostree-dev libseccomp-dev libselinux1-dev skopeo-containers go-md2man

      Because you will compile the buildah source code to create its package, you'll also need to install Go:

      • sudo apt-get update
      • sudo curl -O
      • sudo tar -xvf go1.8.linux-amd64.tar.gz
      • sudo mv go /usr/local
      • sudo echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.profile
      • source ~/.profile
      • go version

      You will see the following output, indicating a successful installation:


      go version go1.8 linux/amd64

      You can now get the buildah source code to create its package, along with the runc binary. runc is the implementation of the OCI container runtime, which you will use to run your Buildah containers.

      Run the following commands to install runc and buildah:

      • mkdir ~/buildah
      • cd ~/buildah
      • export GOPATH=`pwd`
      • git clone ./src/
      • cd ./src/
      • make runc all TAGS="apparmor seccomp"
      • sudo cp ~/buildah/src/ /usr/bin/.
      • sudo apt install buildah

      Next, create the /etc/containers/registries.conf file to configure your container registries:

      • sudo nano /etc/containers/registries.conf

      Add the following content to the file to specify your registries:


      # This is a system-wide configuration file used to
      # keep track of registries for various container backends.
      # It adheres to TOML format and does not support recursive
      # lists of registries.
      # The default location for this configuration file is /etc/containers/registries.conf.
      # The only valid categories are: '', 'registries.insecure',
      # and 'registries.block'.
      registries = ['', '', '', '', '']
      # If you need to access insecure registries, add the registry's fully-qualified name.
      # An insecure registry is one that does not have a valid SSL certificate or only does HTTP.
      registries = []
      # If you need to block pull access from a registry, uncomment the section below
      # and add the registries fully-qualified name.
      # Docker only
      registries = []

      The registries.conf configuration file specifies which registries should be consulted when completing image names that do not include a registry or domain portion.

      Now run the following command to build an image, using the repository as the build context. This repository also contains the relevant Dockerfile:

      • sudo buildah build-using-dockerfile -t rsvpapp:buildah

      This command creates an image named rsvpapp:buildah from the Dockerfille available in the repository.

      To list the images, use the following command:

      You will see the following output:


      IMAGE ID IMAGE NAME CREATED AT SIZE b0c552b8cf64 Sep 30, 2016 04:39 95.3 MB 22121fd251df localhost/rsvpapp:buildah Sep 11, 2018 14:34 114 MB

      One of these images is localhost/rsvpapp:buildah, which you just created. The other,, is the base image from the Dockerfile.

      Once you have built the image, you can push it to Docker Hub. This will allow you to store it for future use. You will first need to login to your Docker Hub account from the command line:

      • docker login -u your-dockerhub-username -p your-dockerhub-password

      Once the login is successful, you will get a file, ~/.docker/config.json, that will contain your Docker Hub credentials. You can then use that file with buildah to push images to Docker Hub.

      For example, if you wanted to push the image you just created, you could run the following command, citing the authfile and the image to push:

      • sudo buildah push --authfile ~/.docker/config.json rsvpapp:buildah docker://your-dockerhub-username/rsvpapp:buildah

      You can also push the resulting image to the local Docker daemon using the following command:

      • sudo buildah push rsvpapp:buildah docker-daemon:rsvpapp:buildah

      Finally, take a look at the Docker images you have created:


      REPOSITORY TAG IMAGE ID CREATED SIZE rsvpapp buildah 22121fd251df 4 minutes ago 108MB nkhare/nginx latest 01f0982d91b8 17 minutes ago 172MB ubuntu 16.04 b9e15a5d1e1a 5 days ago 115MB

      As expected, you should now see a new image, rsvpapp:buildah, that has been exported using buildah.

      You now have experience building container images with two different tools, Docker and Buildah. Let's move on to discussing how to set up a cluster of containers with Kubernetes.

      Step 2 — Setting Up a Kubernetes Cluster on DigitalOcean using kubeadm and Terraform

      There are different ways to set up Kubernetes on DigitalOcean. To learn more about how to set up Kubernetes with kubeadm, for example, you can look at How To Create a Kubernetes Cluster Using Kubeadm on Ubuntu 18.04.

      Since this tutorial series discusses taking a Cloud Native approach to application development, we'll apply this methodology when setting up our cluster. Specifically, we will automate our cluster creation using kubeadm and Terraform, a tool that simplifies creating and changing infrastructure.

      Using your personal access token, you will connect to DigitalOcean with Terraform to provision 3 servers. You will run the kubeadm commands inside of these VMs to create a 3-node Kubernetes cluster containing one master node and two workers.

      On your Ubuntu server, create a pair of SSH keys, which will allow password-less logins to your VMs:

      You will see the following output:


      Generating public/private rsa key pair. Enter file in which to save the key (~/.ssh/id_rsa):

      Press ENTER to save the key pair in the ~/.ssh directory in your home directory, or enter another destination.

      Next, you will see the following prompt:


      Enter passphrase (empty for no passphrase):

      In this case, press ENTER without a password to enable password-less logins to your nodes.

      You will see a confirmation that your key pair has been created:


      Your identification has been saved in ~/.ssh/id_rsa. Your public key has been saved in ~/.ssh/ The key fingerprint is: SHA256:lCVaexVBIwHo++NlIxccMW5b6QAJa+ZEr9ogAElUFyY root@3b9a273f18b5 The key's randomart image is: +---[RSA 2048]----+ |++.E ++o=o*o*o | |o +..=.B = o | |. .* = * o | | . =.o + * | | . . o.S + . | | . +. . | | . ... = | | o= . | | ... | +----[SHA256]-----+

      Get your public key by running the following command, which will display it in your terminal:

      Add this key to your DigitalOcean account by following these directions.

      Next, install Terraform:

      • sudo apt-get update
      • sudo apt-get install unzip
      • wget
      • unzip
      • sudo mv terraform /usr/bin/.
      • terraform version

      You will see output confirming your Terraform installation:


      Terraform v0.11.7

      Next, run the following commands to install kubectl, a CLI tool that will communicate with your Kubernetes cluster, and to create a ~/.kube directory in your user's home directory:

      • sudo apt-get install apt-transport-https
      • curl -s | sudo apt-key add -
      • sudo touch /etc/apt/sources.list.d/kubernetes.list
      • echo "deb kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
      • sudo apt-get update
      • sudo apt-get install kubectl
      • mkdir -p ~/.kube

      Creating the ~/.kube directory will enable you to copy the configuration file to this location. You’ll do that once you run the Kubernetes setup script later in this section. By default, the kubectl CLI looks for the configuration file in the ~/.kube directory to access the cluster.

      Next, clone the sample project repository for this tutorial, which contains the Terraform scripts for setting up the infrastructure:

      • git clone

      Go to the Terrafrom script directory:

      • cd k8s-cicd-webinars/webinar1/2-kubernetes/1-Terraform/

      Get a fingerprint of your SSH public key:

      • ssh-keygen -E md5 -lf ~/.ssh/ | awk '{print $2}'

      You will see output like the following, with the highlighted portion representing your key:



      Keep in mind that your key will differ from what's shown here.

      Save the fingerprint to an environmental variable so Terraform can use it:

      • export FINGERPRINT=dd:d1:b7:0f:6d:30:c0:be:ed:ae:c7:b9:b8:4a:df:5e

      Next, export your DO personal access token:

      • export TOKEN=your-do-access-token

      Now take a look at the ~/k8s-cicd-webinars/webinar1/2-kubernetes/1-Terraform/ project directory:

      Output files

      This folder contains the necessary scripts and configuration files for deploying your Kubernetes cluster with Terraform.

      Execute the script to trigger the Kubernetes cluster setup:

      When the script execution is complete, kubectl will be configured to use the Kubernetes cluster you've created.

      List the cluster nodes using kubectl get nodes:


      NAME STATUS ROLES AGE VERSION k8s-master-node Ready master 2m v1.10.0 k8s-worker-node-1 Ready <none> 1m v1.10.0 k8s-worker-node-2 Ready <none> 57s v1.10.0

      You now have one master and two worker nodes in the Ready state.

      With a Kubernetes cluster set up, you can now explore another option for building container images: Kaniko from Google.

      Step 3 — Building Container Images with Kaniko

      Earlier in this tutorial, you built container images with Dockerfiles and Buildah. But what if you could build container images directly on Kubernetes? There are ways to run the docker image build command inside of Kubernetes, but this isn't native Kubernetes tooling. You would have to depend on the Docker daemon to build images, and it would need to run on one of the Pods in the cluster.

      A tool called Kaniko allows you to build container images with a Dockerfile on an existing Kubernetes cluster. In this step, you will build a container image with a Dockerfile using Kaniko. You will then push this image to Docker Hub.

      In order to push your image to Docker Hub, you will need to pass your Docker Hub credentials to Kaniko. In the previous step, you logged into Docker Hub and created a ~/.docker/config.json file with your login credentials. Let's use this configuration file to create a Kubernetes ConfigMap object to store the credentials inside the Kubernetes cluster. The ConfigMap object is used to store configuration parameters, decoupling them from your application.

      To create a ConfigMap called docker-config using the ~/.docker/config.json file, run the following command:

      • sudo kubectl create configmap docker-config --from-file=$HOME/.docker/config.json

      Next, you can create a Pod definition file called pod-kaniko.yml in the ~/k8s-cicd-webinars/webinar1/2-kubernetes/1-Terraform/ directory (though it can go anywhere).

      First, make sure that you are in the ~/k8s-cicd-webinars/webinar1/2-kubernetes/1-Terraform/ directory:

      • cd ~/k8s-cicd-webinars/webinar1/2-kubernetes/1-Terraform/

      Create the pod-kaniko.yml file:

      Add the following content to the file to specify what will happen when you deploy your Pod. Be sure to replace your-dockerhub-username in the Pod's args field with your own Docker Hub username:


      apiVersion: v1
      kind: Pod
        name: kaniko
        - name: kaniko
          args: ["--dockerfile=./Dockerfile",
                  "--force" ]
            - name: docker-config
              mountPath: /root/.docker/
            - name: demo
              mountPath: /tmp/rsvpapp
        restartPolicy: Never
          - image: python
            name: demo
            command: ["/bin/sh"]
            args: ["-c", "git clone /tmp/rsvpapp"] 
            - name: demo
              mountPath: /tmp/rsvpapp
        restartPolicy: Never
          - name: docker-config
              name: docker-config
          - name: demo
            emptyDir: {}

      This configuration file describes what will happen when your Pod is deployed. First, the Init container will clone the Git repository with the Dockerfile,, into a shared volume called demo. Init containers run before application containers and can be used to run utilties or other tasks that are not desirable to run from your application containers. Your application container, kaniko, will then build the image using the Dockerfile and push the resulting image to Docker Hub, using the credentials you passed to the ConfigMap volume docker-config.

      To deploy the kaniko pod, run the following command:

      • kubectl apply -f pod-kaniko.yml

      You will see the following confirmation:


      pod/kaniko created

      Get the list of pods:

      You will see the following list:


      NAME READY STATUS RESTARTS AGE kaniko 0/1 Init:0/1 0 47s

      Wait a few seconds, and then run kubectl get pods again for a status update:

      You will see the following:


      NAME READY STATUS RESTARTS AGE kaniko 1/1 Running 0 1m

      Finally, run kubectl get pods once more for a final status update:


      NAME READY STATUS RESTARTS AGE kaniko 0/1 Completed 0 2m

      This sequence of output tells you that the Init container ran, cloning the GitHub repository inside of the demo volume. After that, the Kaniko build process ran and eventually finished.

      Check the logs of the pod:

      You will see the following output:


      time="2018-08-02T05:01:24Z" level=info msg="appending to multi args" time="2018-08-02T05:01:24Z" level=info msg="Downloading base image nkhare/python:alpine" . . . ime="2018-08-02T05:01:46Z" level=info msg="Taking snapshot of full filesystem..." time="2018-08-02T05:01:48Z" level=info msg="cmd: CMD" time="2018-08-02T05:01:48Z" level=info msg="Replacing CMD in config with [/bin/sh -c python]" time="2018-08-02T05:01:48Z" level=info msg="Taking snapshot of full filesystem..." time="2018-08-02T05:01:49Z" level=info msg="No files were changed, appending empty layer to config." 2018/08/02 05:01:51 mounted blob: sha256:bc4d09b6c77b25d6d3891095ef3b0f87fbe90621bff2a333f9b7f242299e0cfd 2018/08/02 05:01:51 mounted blob: sha256:809f49334738c14d17682456fd3629207124c4fad3c28f04618cc154d22e845b 2018/08/02 05:01:51 mounted blob: sha256:c0cb142e43453ebb1f82b905aa472e6e66017efd43872135bc5372e4fac04031 2018/08/02 05:01:51 mounted blob: sha256:606abda6711f8f4b91bbb139f8f0da67866c33378a6dcac958b2ddc54f0befd2 2018/08/02 05:01:52 pushed blob sha256:16d1686835faa5f81d67c0e87eb76eab316e1e9cd85167b292b9fa9434ad56bf 2018/08/02 05:01:53 pushed blob sha256:358d117a9400cee075514a286575d7d6ed86d118621e8b446cbb39cc5a07303b 2018/08/02 05:01:55 pushed blob sha256:5d171e492a9b691a49820bebfc25b29e53f5972ff7f14637975de9b385145e04 2018/08/02 05:01:56 digest: sha256:831b214cdb7f8231e55afbba40914402b6c915ef4a0a2b6cbfe9efb223522988 size: 1243

      From the logs, you can see that the kaniko container built the image from the Dockerfile and pushed it to your Docker Hub account.

      You can now pull the Docker image. Be sure again to replace your-dockerhub-username with your Docker Hub username:

      • docker pull your-dockerhub-username/rsvpapp:kaniko

      You will see a confirmation of the pull:


      kaniko: Pulling from your-dockerhub-username/rsvpapp c0cb142e4345: Pull complete bc4d09b6c77b: Pull complete 606abda6711f: Pull complete 809f49334738: Pull complete 358d117a9400: Pull complete 5d171e492a9b: Pull complete Digest: sha256:831b214cdb7f8231e55afbba40914402b6c915ef4a0a2b6cbfe9efb223522988 Status: Downloaded newer image for your-dockerhub-username/rsvpapp:kaniko

      You have now successfully built a Kubernetes cluster and created new images from within the cluster. Let's move on to discussing Deployments and Services.

      Step 4 — Create Kubernetes Deployments and Services

      Kubernetes Deployments allow you to run your applications. Deployments specify the desired state for your Pods, ensuring consistency across your rollouts. In this step, you will create an Nginx deployment file called deployment.yml in the ~/k8s-cicd-webinars/webinar1/2-kubernetes/1-Terraform/ directory to create an Nginx Deployment.

      First, open the file:

      Add the following configuration to the file to define your Nginx Deployment:


      apiVersion: apps/v1
      kind: Deployment
        name: nginx-deployment
          app: nginx
        replicas: 3
            app: nginx
              app: nginx
            - name: nginx
              image: nginx:1.7.9
              - containerPort: 80

      This file defines a Deployment named nginx-deployment that creates three pods, each running an nginx container on port 80.

      To deploy the Deployment, run the following command:

      • kubectl apply -f deployment.yml

      You will see a confirmation that the Deployment was created:


      deployment.apps/nginx-deployment created

      List your Deployments:


      NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE nginx-deployment 3 3 3 3 29s

      You can see that the nginx-deployment Deployment has been created and the desired and current count of the Pods are same: 3.

      To list the Pods that the Deployment created, run the following command:


      NAME READY STATUS RESTARTS AGE kaniko 0/1 Completed 0 9m nginx-deployment-75675f5897-nhwsp 1/1 Running 0 1m nginx-deployment-75675f5897-pxpl9 1/1 Running 0 1m nginx-deployment-75675f5897-xvf4f 1/1 Running 0 1m

      You can see from this output that the desired number of Pods are running.

      To expose an application deployment internally and externally, you will need to create a Kubernetes object called a Service. Each Service specifies a ServiceType, which defines how the service is exposed. In this example, we will use a NodePort ServiceType, which exposes the Service on a static port on each node.

      To do this, create a file, service.yml, in the ~/k8s-cicd-webinars/webinar1/2-kubernetes/1-Terrafrom/ directory:

      Add the following content to define your Service:


      kind: Service
      apiVersion: v1
        name: nginx-service
          app: nginx
        type: NodePort
        - protocol: TCP
          port: 80
          targetPort: 80
          nodePort: 30111

      These settings define the Service, nginx-service, and specify that it will target port 80 on your Pod. nodePort defines the port where the application will accept external traffic.

      To deploy the Service run the following command:

      • kubectl apply -f service.yml

      You will see a confirmation:


      service/nginx-service created

      List the Services:

      You will see the following list:


      NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP <none> 443/TCP 5h nginx-service NodePort <none> 80:30111/TCP 7s

      Your Service, nginx-service, is exposed on port 30111 and you can now access it on any of the node’s public IPs. For example, navigating to http://node_1_ip:30111 or http://node_2_ip:30111 should take you to Nginx's standard welcome page.

      Once you have tested the Deployment, you can clean up both the Deployment and Service:

      • kubectl delete deployment nginx-deployment
      • kubectl delete service nginx-service

      These commands will delete the Deployment and Service you have created.

      Now that you have worked with Deployments and Services, let's move on to creating Custom Resources.

      Step 5 — Creating Custom Resources in Kubernetes

      Kubernetes offers limited but production-ready functionalities and features. It is possible to extend Kubernetes' offerings, however, using its Custom Resources feature. In Kubernetes, a resource is an endpoint in the Kubernetes API that stores a collection of API objects. A Pod resource contains a collection of Pod objects, for instance. With Custom Resources, you can add custom offerings for networking, storage, and more. These additions can be created or removed at any point.

      In addition to creating custom objects, you can also employ sub-controllers of the Kubernetes Controller component in the control plane to make sure that the current state of your objects is equal to the desired state. The Kubernetes Controller has sub-controllers for specified objects. For example, ReplicaSet is a sub-controller that makes sure the desired Pod count remains consistent. When you combine a Custom Resource with a Controller, you get a true declarative API that allows you to specify the desired state of your resources.

      In this step, you will create a Custom Resource and related objects.

      To create a Custom Resource, first make a file called crd.yml in the ~/k8s-cicd-webinars/webinar1/2-kubernetes/1-Terrafrom/ directory:

      Add the following Custom Resource Definition (CRD):


      kind: CustomResourceDefinition
        version: v1
        scope: Namespaced
          plural: webinars
          singular: webinar
          kind: Webinar
          - wb

      To deploy the CRD defined in crd.yml, run the following command:

      • kubectl create -f crd.yml

      You will see a confirmation that the resource has been created:

      Output created

      The crd.yml file has created a new RESTful resource path: /apis/*/webinars. You can now refer to your objects using webinars, webinar, Webinar, and wb, as you listed them in the names section of the CustomResourceDefinition. You can check the RESTful resource with the following command:

      • kubectl proxy & curl

      Note: If you followed the initial server setup guide in the prerequisites, then you will need to allow traffic to port 8001 in order for this test to work. Enable traffic to this port with the following command:

      You will see the following output:


      HTTP/1.1 200 OK Content-Length: 238 Content-Type: application/json Date: Fri, 03 Aug 2018 06:10:12 GMT { "apiVersion": "v1", "kind": "APIGroup", "name": "", "preferredVersion": { "groupVersion": "", "version": "v1" }, "serverAddressByClientCIDRs": null, "versions": [ { "groupVersion": "", "version": "v1" } ] }

      Next, create the object for using new Custom Resources by opening a file called webinar.yml:

      Add the following content to create the object:


      apiVersion: ""
      kind: Webinar
        name: webinar1
        name: webinar
        image: nginx

      Run the following command to push these changes to the cluster:

      • kubectl apply -f webinar.yml

      You will see the following output:

      Output created

      You can now manage your webinar objects using kubectl. For example:


      NAME CREATED AT webinar1 21s

      You now have an object called webinar1. If there had been a Controller, it would have intercepted the object creation and performed any defined operations.

      Deleting a Custom Resource Definition

      To delete all of the objects for your Custom Resource, use the following command:

      • kubectl delete webinar --all

      You will see:

      Output "webinar1" deleted

      Remove the Custom Resource itself:

      • kubectl delete crd

      You will see a confirmation that it has been deleted:

      Output "" deleted

      After deletion you will not have access to the API endpoint that you tested earlier with the curl command.

      This sequence is an introduction to how you can extend Kubernetes functionalities without modifying your Kubernetes code.

      Step 6 — Deleting the Kubernetes Cluster

      To destroy the Kubernetes cluster itself, you can use the script from the ~/k8s-cicd-webinars/webinar1/2-kubernetes/1-Terrafrom folder. Make sure that you are in this directory:

      • cd ~/k8s-cicd-webinars/webinar1/2-kubernetes/1-Terrafrom

      Run the script:

      By running this script, you'll allow Terraform to communicate with the DigitalOcean API and delete the servers in your cluster.


      In this tutorial, you used different tools to create container images. With these images, you can create containers in any environment. You also set up a Kubernetes cluster using Terraform, and created Deployment and Service objects to deploy and expose your application. Additionally, you extended Kubernetes' functionality by defining a Custom Resource.

      You now have a solid foundation to build a CI/CD environment on Kubernetes, which we'll explore in future articles.

      Source link

      Building Optimized Containers for Kubernetes


      Container images are the primary packaging format for defining applications within Kubernetes. Used as the basis for pods and other objects, images play an important role in leveraging Kubernetes’ features to efficiently run applications on the platform. Well-designed images are secure, highly performant, and focused. They are able to react to configuration data or instructions provided by Kubernetes and also implement endpoints the orchestration system uses to understand internal application state.

      In this article, we’ll introduce some strategies for creating high quality images and discuss a few general goals to help guide your decisions when containerizing applications. We will focus on building images intended to be run on Kubernetes, but many of the suggestions apply equally to running containers on other orchestration platforms or in other contexts.

      Characteristics of Efficient Container Images

      Before we go over specific actions to take when building container images, we will talk about what makes a good container image. What should your goals be when designing new images? Which characteristics and what behavior are most important?

      Some qualities to aim for are:

      A single, well-defined purpose

      Container images should have a single discrete focus. Avoid thinking of container images as virtual machines, where it can make sense to package related functionality together. Instead, treat your container images like Unix utilities, maintaining a strict focus on doing one small thing well. Applications can be coordinated outside of the container scope to compose complex functionality.

      Generic design with the ability to inject configuration at runtime

      Container images should be designed with reuse in mind when possible. For instance, the ability to adjust configuration at runtime is often required to fulfill basic requirements like testing your images before deploying to production. Small, generic images can be combined in different configurations to modify behavior without creating new images.

      Small image size

      Smaller images have a number of benefits in clustered environments like Kubernetes. They download quickly to new nodes and often have a smaller set of installed packages, which can improve security. Pared down container images make it simpler to debug problems by minimizing the amount of software involved.

      Externally managed state

      Containers in clustered environments experience a very volatile life cycle including planned and unplanned shutdowns due to resource scarcity, scaling, or node failures. To maintain consistency, aid in recovery and availability of your services, and to avoid losing data, it is critical that you store application state in a stable location outside of the container.

      Easy to understand

      It is important to try to keep container images as simple and easy to understand as possible. When troubleshooting, being able to easily reason about the problem by viewing container image configuration or testing container behavior can help you reach a resolution faster. Thinking of container images as a packaging format for your application instead of a machine configuration can help you strike the right balance.

      Follow containerized software best practices

      Images should aim to work within the container model instead of acting against it. Avoid implementing conventional system administration practices, like including full init systems and daemonizing applications. Log to standard out so Kubernetes can expose the data to administrators instead of using an internal logging daemon. Each of these differs from best practices for full operating systems.

      Fully leverage Kubernetes features

      Beyond conforming to the container model, it’s important to understand and reconcile with the environment and tooling that Kubernetes provides. For example, providing endpoints for liveness and readiness checks or adjusting operation based on changes in the configuration or environment can help your applications use Kubernetes’ dynamic deployment environment to their advantage.

      Now that we’ve established some of the qualities that define highly functional container images, we can dive deeper into strategies that help you achieve these goals.

      Reuse Minimal, Shared Base Layers

      We can start off by examining the resources that container images are built from: base images. Each container image is built either from a parent image, an image used as a starting point, or from the abstract scratch layer, an empty image layer with no filesystem. A base image is a container image that serves as a foundation for future images by defining the basic operating system and providing core functionality. Images are comprised of one or more image layers built on top of one another to form a final image.

      No standard utilities or filesystem are available when working directly from scratch, which means that you only have access to extremely limited functionality. While images created directly from scratch can be very streamlined and minimal, their main purpose is in defining base images. Typically, you want to build your container images on top of a parent image that sets up a basic environment that your applications run in so that you do not have to construct a complete system for every image.

      While there are base images for a variety of Linux distributions, it’s best to be deliberate about which systems you choose. Each new machine will have to download the parent image and any additional layers you’ve added. For large images, this can consume a significant amount of bandwidth and noticeably lengthen the startup time of your containers on their first run. There is no way to pare down an image that’s used as a parent downstream in the container build process, so starting with a minimal parent is a good idea.

      Feature rich environments like Ubuntu allow your application to run in an environment you’re familiar with, but there are some tradeoffs to consider. Ubuntu images (and similar conventional distribution images) tend to be relatively large (over 100MB), meaning that any container images built from them will inherit that weight.

      Alpine Linux is a popular alternative for base images because it successfully packages a lot of functionality into a very small base image (~ 5MB). It includes a package manager with sizable repositories and has most of the standard utilities you would expect from a minimal Linux environment.

      When designing your applications, it’s a good idea to try to reuse the same parent for each image. When your images share a parent, machines running your containers will download the parent layer only once. Afterwards, they will only need to download the layers that differ between your images. This means that if you have common features or functionality you’d like to embed in each image, creating a common parent image to inherit from might be a good idea. Images that share a lineage help minimize the amount of extra data you need to download on fresh servers.

      Managing Container Layers

      Once you’ve selected a parent image, you can define your container image by adding additional software, copying files, exposing ports, and choosing processes to run. Certain instructions in the image configuration file (a Dockerfile if you are using Docker) will add additional layers to your image.

      For many of the same reasons mentioned in the previous section, it’s important to be mindful of how you add layers to your images due to the resulting size, inheritance, and runtime complexity. To avoid building large, unwieldy images, it’s important to develop a good understanding of how container layers interact, how the build engine caches layers, and how subtle differences in similar instructions can have a big impact on the images you create.

      Understanding Image Layers and Build Cache

      Docker creates a new image layer each time it executes a RUN, COPY, or ADD instruction. If you build the image again, the build engine will check each instruction to see if it has an image layer cached for the operation. If it finds a match in the cache, it uses the existing image layer rather than executing the instruction again and rebuilding the layer.

      This process can significantly shorten build times, but it is important to understand the mechanism used to avoid potential problems. For file copying instructions like COPY and ADD, Docker compares the checksums of the files to see if the operation needs to be performed again. For RUN instructions, Docker checks to see if it has an existing image layer cached for that particular command string.

      While it might not be immediately obvious, this behavior can cause unexpected results if you are not careful. A common example of this is updating the local package index and installing packages in two separate steps. We will be using Ubuntu for this example, but the basic premise applies equally well to base images for other distributions:

      Package installation example Dockerfile

      FROM ubuntu:18.04
      RUN apt -y update
      RUN apt -y install nginx
      . . .

      Here, the local package index is updated in one RUN instruction (apt -y update) and Nginx is installed in another operation. This works without issue when it is first used. However, if the Dockerfile is updated later to install an additional package, there may be problems:

      Package installation example Dockerfile

      FROM ubuntu:18.04
      RUN apt -y update
      RUN apt -y install nginx php-fpm
      . . .

      We’ve added a second package to the installation command run by the second instruction. If a significant amount of time has passed since the previous image build, the new build might fail. That’s because the package index update instruction (RUN apt -y update) has not changed, so Docker reuses the image layer associated with that instruction. Since we are using an old package index, the version of the php-fpm package we have in our local records may no longer be in the repositories, resulting in an error when the second instruction is run.

      To avoid this scenario, be sure to consolidate any steps that are interdependent into a single RUN instruction so that Docker will re-execute all of the necessary commands when a change occurs:

      Package installation example Dockerfile

      FROM ubuntu:18.04
      RUN apt -y update && apt -y install nginx php-fpm
      . . .

      The instruction now updates the local package cache whenever the package list changes.

      Reducing Image Layer Size by Tweaking RUN Instructions

      The previous example demonstrates how Docker’s caching behavior can subvert expectations, but there are some other things to keep in mind with how RUN instructions interact with Docker’s layering system. As mentioned earlier, at the end of each RUN instruction, Docker commits the changes as an additional image layer. In order to exert control over the scope of the image layers produced, you can clean up unnecessary files in the final environment that will be committed by paying attention to the artifacts introduced by the commands you run.

      In general, chaining commands together into a single RUN instruction offers a great deal of control over the layer that will be written. For each command, you can set up the state of the layer (apt -y update), perform the core command (apt install -y nginx php-fpm), and remove any unnecessary artifacts to clean up the environment before it’s committed. For example, many Dockerfiles chain rm -rf /var/lib/apt/lists/* to the end of apt commands, removing the downloaded package indexes, to reduce the final layer size:

      Package installation example Dockerfile

      FROM ubuntu:18.04
      RUN apt -y update && apt -y install nginx php-fpm && rm -rf /var/lib/apt/lists/*
      . . .

      To further reduce the size of the image layers you are creating, trying to limit other unintended side effects of the commands you’re running can be helpful. For instance, in addition to the explicitly declared packages, apt also installs “recommended” packages by default. You can include --no-install-recommends to your apt commands to remove this behavior. You may have to experiment to find out if you rely on any of the functionality provided by recommended packages.

      We’ve used package management commands in this section as an example, but these same principles apply to other scenarios. The general idea is to construct the prerequisite conditions, execute the minimum viable command, and then clean up any unnecessary artifacts in a single RUN command to reduce the overhead of the layer you’ll be producing.

      Using Multi-stage Builds

      Multi-stage builds were introduced in Docker 17.05, allowing developers to more tightly control the final runtime images they produce. Multi-stage builds allow you to divide your Dockerfile into multiple sections representing distinct stages, each with a FROM statement to specify separate parent images.

      Earlier sections define images that can be used to build your application and prepare assets. These often contain build tools and development files that are needed to produce the application, but are not necessary to run it. Each subsequent stage defined in the file will have access to artifacts produced by previous stages.

      The last FROM statement defines the image that will be used to run the application. Typically, this is a pared down image that installs only the necessary runtime requirements and then copies the application artifacts produced by previous stages.

      This system allows you worry less about optimizing RUN instructions in the build stages since those container layers will not be present in the final runtime image. You should still pay attention to how instructions interact with layer caching in the build stages, but your efforts can be directed towards minimizing build time rather than final image size. Paying attention to instructions in the final stage is still important in reducing image size, but by separating the different stages of your container build, it’s easier to to obtain streamlined images without as much Dockerfile complexity.

      Scoping Functionality at the Container and Pod Level

      While the choices you make regarding container build instructions are important, broader decisions about how to containerize your services often have a more direct impact on your success. In this section, we’ll talk a bit more about how to best transition your applications from a more conventional environment to running on a container platform.

      Containerizing by Function

      Generally, it is good practice to package each piece of independent functionality into a separate container image.

      This differs from common strategies employed in virtual machine environments where applications are frequently grouped together within the same image to reduce the size and minimize the resources required to run the VM. Since containers are lightweight abstractions that don’t virtualize the entire operating system stack, this tradeoff is less compelling on Kubernetes. So while a web stack virtual machine might bundle an Nginx web server with a Gunicorn application server on a single machine to serve a Django application, in Kubernetes these might be split into separate containers.

      Designing containers that implement one discrete piece of functionality for your services offers a number of advantages. Each container can be developed independently if standard interfaces between services are established. For instance, the Nginx container could potentially be used to proxy to a number of different backends or could be used as a load balancer if given a different configuration.

      Once deployed, each container image can be scaled independently to address varying resource and load constraints. By splitting your applications into multiple container images, you gain flexibility in development, organization, and deployment.

      Combining Container Images in Pods

      In Kubernetes, pods are the smallest unit that can be directly managed by the control plane. Pods consist of one or more containers along with additional configuration data to tell the platform how those components should be run. The containers within a pod are always scheduled on the same worker node in the cluster and the system automatically restarts failed containers. The pod abstraction is very useful, but it introduces another layer of decisions about how to bundle together the components of your applications.

      Like container images, pods also become less flexible when too much functionality is bundled into a single entity. Pods themselves can be scaled using other abstractions, but the containers within cannot be managed or scaled independently. So, to continue using our previous example, the separate Nginx and Gunicorn containers should probably not be bundled together into a single pod so that they can be controlled and deployed separately.

      However, there are scenarios where it does make sense to combine functionally different containers as a unit. In general, these can be categorized as situations where an additional container supports or enhances the core functionality of the main container or helps it adapt to its deployment environment. Some common patterns are:

      • Sidecar: The secondary container extends the main container’s core functionality by acting in a supporting utility role. For example, the sidecar container might forward logs or update the filesystem when a remote repository changes. The primary container remains focused on its core responsibility, but is enhanced by the features provided by the sidecar.
      • Ambassador: An ambassador container is responsible for discovering and connecting to (often complex) external resources. The primary container can connect to an ambassador container on well-known interfaces using the internal pod environment. The ambassador abstracts the backend resources and proxies traffic between the primary container and the resource pool.
      • Adaptor: An adaptor container is responsible for normalizing the primary containers interfaces, data, and protocols to align with the properties expected by other components. The primary container can operate using native formats and the adaptor container translates and normalizes the data to communicate with the outside world.

      As you might have noticed, each of these patterns support the strategy of building standard, generic primary container images that can then be deployed in a variety contexts and configurations. The secondary containers help bridge the gap between the primary container and the specific deployment environment being used. Some sidecar containers can also be reused to adapt multiple primary containers to the same environmental conditions. These patterns benefit from the shared filesystem and networking namespace provided by the pod abstraction while still allowing independent development and flexible deployment of standardized containers.

      Designing for Runtime Configuration

      There is some tension between the desire to build standardized, reusable components and the requirements involved in adapting applications to their runtime environment. Runtime configuration is one of the best methods to bridge the gap between these concerns. Components are built to be both general and flexible and the required behavior is outlined at runtime by providing the software with additional configuration information. This standard approach works for containers as well as it does for applications.

      Building with runtime configuration in mind requires you to think ahead during both the application development and containerization steps. Applications should be designed to read values from command line parameters, configuration files, or environment variables when they are launched or restarted. This configuration parsing and injection logic must be implemented in code prior to containerization.

      When writing a Dockerfile, the container must also be designed with runtime configuration in mind. Containers have a number of mechanisms for providing data at runtime. Users can mount files or directories from the host as volumes within the container to enable file-based configuration. Likewise, environment variables can be passed into the internal container runtime when the container is started. The CMD and ENTRYPOINT Dockerfile instructions can also be defined in a way that allows for runtime configuration information to be passed in as command parameters.

      Since Kubernetes manipulates higher level objects like pods instead of managing containers directly, there are mechanisms available to define configuration and inject it into the container environment at runtime. Kubernetes ConfigMaps and Secrets allow you to define configuration data separately and then project the values into the container environment as environment variables or files at runtime. ConfigMaps are general purpose objects intended to store configuration data that might vary based on environment, testing stage, etc. Secrets offer a similar interface but are specifically designed for sensitive data, like account passwords or API credentials.

      By understanding and correctly using the runtime configuration options available throughout each layer of abstraction, you can build flexible components that take their cues from environment-provided values. This makes it possible to reuse the same container images in very different scenarios, reducing development overhead by improving application flexibility.

      Implementing Process Management with Containers

      When transitioning to container-based environments, users often start by shifting existing workloads, with few or no changes, to the new system. They package applications in containers by wrapping the tools they are already using in the new abstraction. While it is helpful to use your usual patterns to get migrated applications up and running, dropping in previous implementations within containers can sometimes lead to ineffective design.

      Treating Containers like Applications, Not Services

      Problems frequently arise when developers implement significant service management functionality within containers. For example, running systemd services within the container or daemonizing web servers may be considered best practices in a normal computing environment, but they often conflict with assumptions inherent in the container model.

      Hosts manage container life cycle events by sending signals to the process operating as PID (process ID) 1 inside the container. PID 1 is the first process started, which would be the init system in traditional computing environments. However, because the host can only manage PID 1, using a conventional init system to manage processes within the container sometimes means there is no way to control the primary application. The host can start, stop, or kill the internal init system, but can’t manage the primary application directly. The signals sometimes propagate the intended behavior to the running application, but this adds complexity and isn’t always necessary.

      Most of the time, it is better to simplify the running environment within the container so that PID 1 is running the primary application in the foreground. In cases where multiple processes must be run, PID 1 is responsible for managing the life cycle of subsequent processes. Certain applications, like Apache, handle this natively by spawning and managing workers that handle connections. For other applications, a wrapper script or a very simple init system like dumb-init or the included tini init system can be used in some cases. Regardless of the implementation you choose, the process running as PID 1 within the container should respond appropriately to TERM signals sent by Kubernetes to behave as expected.

      Managing Container Health in Kubernetes

      Kubernetes deployments and services offer life cycle management for long-running processes and reliable, persistent access to applications, even when underlying containers need to be restarted or the implementations themselves change. By extracting the responsibility of monitoring and maintaining service health out of the container, you can leverage the platform’s tools for managing healthy workloads.

      In order for Kubernetes to manage containers properly, it has to understand whether the applications running within containers are healthy and capable of performing work. To enable this, containers can implement liveness probes: network endpoints or commands that can be used to report application health. Kubernetes will periodically check defined liveness probes to determine if the container is operating as expected. If the container does not respond appropriately, Kubernetes restarts the container in an attempt to reestablish functionality.

      Kubernetes also provides readiness probes, a similar construct. Rather than indicating whether the application within a container is healthy, readiness probes determine whether the application is ready to receive traffic. This can be useful when a containerized application has an initialization routine that must complete before it is ready to receive connections. Kubernetes uses readiness probes to determine whether to add a pod to or remove a pod from a service.

      Defining endpoints for these two probe types can help Kubernetes manage your containers efficiently and can prevent container life cycle problems from affecting service availability. The mechanisms to respond to these types of health requests must be built into the application itself and must be exposed in the Docker image configuration.


      In this guide, we’ve covered some important considerations to keep in mind when
      running containerized applications in Kubernetes. To reiterate, some of the
      suggestions we went over were:

      • Use minimal, shareable parent images to build images with minimal bloat and reduce startup time
      • Use multi-stage builds to separate the container build and runtime environments
      • Combine Dockerfile instructions to create clean image layers and avoid image caching mistakes
      • Containerize by isolating discrete functionality to enable flexible scaling and management
      • Design pods to have a single, focused responsibility
      • Bundle helper containers to enhance the main container’s functionality or to adapt it to the deployment environment
      • Build applications and containers to respond to runtime configuration to allow greater flexibility when deploying
      • Run applications as the primary processes in containers so Kubernetes can manage life cycle events
      • Develop health and liveness endpoints within the application or container so that Kubernetes can monitor the health of the container

      Throughout the development and implementation process, you will need to make decisions that can affect your service’s robustness and effectiveness. Understanding the ways that containerized applications differ from conventional applications, and learning how they operate in a managed cluster environment can help you avoid some common pitfalls and allow you to take advantage of all of the capabilities Kubernetes provides.

      Source link