One place for hosting & domains


      How To Create a Kubernetes Cluster Using Kubeadm on Debian 9

      The author selected the Free and Open Source Fund to receive a donation as part of the Write for DOnations program.


      Kubernetes is a container orchestration system that manages containers at scale. Initially developed by Google based on its experience running containers in production, Kubernetes is open source and actively developed by a community around the world.

      Note: This tutorial uses version 1.14 of Kubernetes, the official supported version at the time of this article’s publication. For up-to-date information on the latest version, please see the current release notes in the official Kubernetes documentation.

      Kubeadm automates the installation and configuration of Kubernetes components such as the API server, Controller Manager, and Kube DNS. It does not, however, create users or handle the installation of operating-system-level dependencies and their configuration. For these preliminary tasks, it is possible to use a configuration management tool like Ansible or SaltStack. Using these tools makes creating additional clusters or recreating existing clusters much simpler and less error prone.

      In this guide, you will set up a Kubernetes cluster from scratch using Ansible and Kubeadm, and then deploy a containerized Nginx application to it.


      Your cluster will include the following physical resources:

      The master node (a node in Kubernetes refers to a server) is responsible for managing the state of the cluster. It runs Etcd, which stores cluster data among components that schedule workloads to worker nodes.

      Worker nodes are the servers where your workloads (i.e. containerized applications and services) will run. A worker will continue to run your workload once they’re assigned to it, even if the master goes down once scheduling is complete. A cluster’s capacity can be increased by adding workers.

      After completing this guide, you will have a cluster ready to run containerized applications, provided that the servers in the cluster have sufficient CPU and RAM resources for your applications to consume. Almost any traditional Unix application including web applications, databases, daemons, and command line tools can be containerized and made to run on the cluster. The cluster itself will consume around 300-500MB of memory and 10% of CPU on each node.

      Once the cluster is set up, you will deploy the web server Nginx to it to ensure that it is running workloads correctly.


      Step 1 — Setting Up the Workspace Directory and Ansible Inventory File

      In this section, you will create a directory on your local machine that will serve as your workspace. You will configure Ansible locally so that it can communicate with and execute commands on your remote servers. Once that’s done, you will create a hosts file containing inventory information such as the IP addresses of your servers and the groups that each server belongs to.

      Out of your three servers, one will be the master with an IP displayed as master_ip. The other two servers will be workers and will have the IPs worker_1_ip and worker_2_ip.

      Create a directory named ~/kube-cluster in the home directory of your local machine and cd into it:

      • mkdir ~/kube-cluster
      • cd ~/kube-cluster

      This directory will be your workspace for the rest of the tutorial and will contain all of your Ansible playbooks. It will also be the directory inside which you will run all local commands.

      Create a file named ~/kube-cluster/hosts using nano or your favorite text editor:

      • nano ~/kube-cluster/hosts

      Add the following text to the file, which will specify information about the logical structure of your cluster:


      master ansible_host=master_ip ansible_user=root
      worker1 ansible_host=worker_1_ip ansible_user=root
      worker2 ansible_host=worker_2_ip ansible_user=root

      You may recall that inventory files in Ansible are used to specify server information such as IP addresses, remote users, and groupings of servers to target as a single unit for executing commands. ~/kube-cluster/hosts will be your inventory file and you’ve added two Ansible groups (masters and workers) to it specifying the logical structure of your cluster.

      In the masters group, there is a server entry named “master” that lists the master node’s IP (master_ip) and specifies that Ansible should run remote commands as the root user.

      Similarly, in the workers group, there are two entries for the worker servers (worker_1_ip and worker_2_ip) that also specify the ansible_user as root.

      The last line of the file tells Ansible to use the remote servers’ Python 3 interpreters for its management operations.

      Save and close the file after you’ve added the text.

      Having set up the server inventory with groups, let’s move on to installing operating system level dependencies and creating configuration settings.

      Step 2 — Creating a Non-Root User on All Remote Servers

      In this section you will create a non-root user with sudo privileges on all servers so that you can SSH into them manually as an unprivileged user. This can be useful if, for example, you would like to see system information with commands such as top/htop, view a list of running containers, or change configuration files owned by root. These operations are routinely performed during the maintenance of a cluster, and using a non-root user for such tasks minimizes the risk of modifying or deleting important files or unintentionally performing other dangerous operations.

      Create a file named ~/kube-cluster/initial.yml in the workspace:

      • nano ~/kube-cluster/initial.yml

      Next, add the following play to the file to create a non-root user with sudo privileges on all of the servers. A play in Ansible is a collection of steps to be performed that target specific servers and groups. The following play will create a non-root sudo user:


      - hosts: all
        become: yes
          - name: create the 'sammy' user
            user: name=sammy append=yes state=present createhome=yes shell=/bin/bash
          - name: allow 'sammy' to have passwordless sudo
              dest: /etc/sudoers
              line: 'sammy ALL=(ALL) NOPASSWD: ALL'
              validate: 'visudo -cf %s'
          - name: set up authorized keys for the sammy user
            authorized_key: user=sammy key="{{item}}"
              - ~/.ssh/

      Here’s a breakdown of what this playbook does:

      • Creates the non-root user sammy.

      • Configures the sudoers file to allow the sammy user to run sudo commands without a password prompt.

      • Adds the public key in your local machine (usually ~/.ssh/ to the remote sammy user’s authorized key list. This will allow you to SSH into each server as the sammy user.

      Save and close the file after you’ve added the text.

      Next, execute the playbook by locally running:

      • ansible-playbook -i hosts ~/kube-cluster/initial.yml

      The command will complete within two to five minutes. On completion, you will see output similar to the following:


      PLAY [all] **** TASK [Gathering Facts] **** ok: [master] ok: [worker1] ok: [worker2] TASK [create the 'sammy' user] **** changed: [master] changed: [worker1] changed: [worker2] TASK [allow 'sammy' user to have passwordless sudo] **** changed: [master] changed: [worker1] changed: [worker2] TASK [set up authorized keys for the sammy user] **** changed: [worker1] => (item=ssh-rsa AAAAB3...) changed: [worker2] => (item=ssh-rsa AAAAB3...) changed: [master] => (item=ssh-rsa AAAAB3...) PLAY RECAP **** master : ok=5 changed=4 unreachable=0 failed=0 worker1 : ok=5 changed=4 unreachable=0 failed=0 worker2 : ok=5 changed=4 unreachable=0 failed=0

      Now that the preliminary setup is complete, you can move on to installing Kubernetes-specific dependencies.

      Step 3 — Installing Kubernetes Dependencies

      In this section, you will install the operating-system-level packages required by Kubernetes with Debian’s package manager. These packages are:

      • Docker – a container runtime. It is the component that runs your containers. Support for other runtimes such as rkt is under active development in Kubernetes.

      • kubeadm – a CLI tool that will install and configure the various components of a cluster in a standard way.

      • kubelet – a system service/program that runs on all nodes and handles node-level operations.

      • kubectl – a CLI tool used for issuing commands to the cluster through its API Server.

      Create a file named ~/kube-cluster/kube-dependencies.yml in the workspace:

      • nano ~/kube-cluster/kube-dependencies.yml

      Add the following plays to the file to install these packages to your servers:


      - hosts: all
        become: yes
         - name: install remote apt deps
             name: "{{ item }}"
             state: present
             - apt-transport-https
             - ca-certificates
             - gnupg2
             - software-properties-common
         - name: add Docker apt-key
             state: present
         - name: add Docker's APT repository
            repo: deb stretch stable
            state: present
            filename: 'docker'
         - name: install Docker
             name: docker-ce
             state: present
             update_cache: true
         - name: add Kubernetes apt-key
             state: present
         - name: add Kubernetes' APT repository
            repo: deb kubernetes-xenial main
            state: present
            filename: 'kubernetes'
         - name: install kubelet
             name: kubelet=1.14.0-00
             state: present
             update_cache: true
         - name: install kubeadm
             name: kubeadm=1.14.0-00
             state: present
      - hosts: master
        become: yes
         - name: install kubectl
             name: kubectl=1.14.0-00
             state: present
             force: yes

      The first play in the playbook does the following:

      • Add dependencies for adding, verifying and installing packages from remote repositories.

      • Adds the Docker APT repository’s apt-key for key verification.

      • Installs Docker, the container runtime.

      • Adds the Kubernetes APT repository’s apt-key for key verification.

      • Adds the Kubernetes APT repository to your remote servers’ APT sources list.

      • Installs kubelet and kubeadm.

      The second play consists of a single task that installs kubectl on your master node.

      Note: While the Kubernetes documentation recommends you use the latest stable release of Kubernetes for your environment, this tutorial uses a specific version. This will ensure that you can follow the steps successfully, as Kubernetes changes rapidly and the latest version may not work with this tutorial.

      Save and close the file when you are finished.

      Next, execute the playbook by locally running:

      • ansible-playbook -i hosts ~/kube-cluster/kube-dependencies.yml

      On completion, you will see output similar to the following:


      PLAY [all] **** TASK [Gathering Facts] **** ok: [worker1] ok: [worker2] ok: [master] TASK [install Docker] **** changed: [master] changed: [worker1] changed: [worker2] TASK [install APT Transport HTTPS] ***** ok: [master] ok: [worker1] changed: [worker2] TASK [add Kubernetes apt-key] ***** changed: [master] changed: [worker1] changed: [worker2] TASK [add Kubernetes' APT repository] ***** changed: [master] changed: [worker1] changed: [worker2] TASK [install kubelet] ***** changed: [master] changed: [worker1] changed: [worker2] TASK [install kubeadm] ***** changed: [master] changed: [worker1] changed: [worker2] PLAY [master] ***** TASK [Gathering Facts] ***** ok: [master] TASK [install kubectl] ****** ok: [master] PLAY RECAP **** master : ok=9 changed=5 unreachable=0 failed=0 worker1 : ok=7 changed=5 unreachable=0 failed=0 worker2 : ok=7 changed=5 unreachable=0 failed=0

      After execution, Docker, kubeadm, and kubelet will be installed on all of the remote servers. kubectl is not a required component and is only needed for executing cluster commands. Installing it only on the master node makes sense in this context, since you will run kubectl commands only from the master. Note, however, that kubectl commands can be run from any of the worker nodes or from any machine where it can be installed and configured to point to a cluster.

      All system dependencies are now installed. Let’s set up the master node and initialize the cluster.

      Step 4 — Setting Up the Master Node

      In this section, you will set up the master node. Before creating any playbooks, however, it’s worth covering a few concepts such as Pods and Pod Network Plugins, since your cluster will include both.

      A pod is an atomic unit that runs one or more containers. These containers share resources such as file volumes and network interfaces in common. Pods are the basic unit of scheduling in Kubernetes: all containers in a pod are guaranteed to run on the same node that the pod is scheduled on.

      Each pod has its own IP address, and a pod on one node should be able to access a pod on another node using the pod’s IP. Containers on a single node can communicate easily through a local interface. Communication between pods is more complicated, however, and requires a separate networking component that can transparently route traffic from a pod on one node to a pod on another.

      This functionality is provided by pod network plugins. For this cluster, you will use Flannel, a stable and performant option.

      Create an Ansible playbook named master.yml on your local machine:

      • nano ~/kube-cluster/master.yml

      Add the following play to the file to initialize the cluster and install Flannel:


      - hosts: master
        become: yes
          - name: initialize the cluster
            shell: kubeadm init --pod-network-cidr= >> cluster_initialized.txt
              chdir: $HOME
              creates: cluster_initialized.txt
          - name: create .kube directory
            become: yes
            become_user: sammy
              path: $HOME/.kube
              state: directory
              mode: 0755
          - name: copy admin.conf to user's kube config
              src: /etc/kubernetes/admin.conf
              dest: /home/sammy/.kube/config
              remote_src: yes
              owner: sammy
          - name: install Pod network
            become: yes
            become_user: sammy
            shell: kubectl apply -f >> pod_network_setup.txt
              chdir: $HOME
              creates: pod_network_setup.txt

      Here’s a breakdown of this play:

      • The first task initializes the cluster by running kubeadm init. Passing the argument --pod-network-cidr= specifies the private subnet that the pod IPs will be assigned from. Flannel uses the above subnet by default; we’re telling kubeadm to use the same subnet.

      • The second task creates a .kube directory at /home/sammy. This directory will hold configuration information such as the admin key files, which are required to connect to the cluster, and the cluster’s API address.

      • The third task copies the /etc/kubernetes/admin.conf file that was generated from kubeadm init to your non-root user’s home directory. This will allow you to use kubectl to access the newly-created cluster.

      • The last task runs kubectl apply to install Flannel. kubectl apply -f descriptor.[yml|json] is the syntax for telling kubectl to create the objects described in the descriptor.[yml|json] file. The kube-flannel.yml file contains the descriptions of objects required for setting up Flannel in the cluster.

      Save and close the file when you are finished.

      Execute the playbook locally by running:

      • ansible-playbook -i hosts ~/kube-cluster/master.yml

      On completion, you will see output similar to the following:


      PLAY [master] **** TASK [Gathering Facts] **** ok: [master] TASK [initialize the cluster] **** changed: [master] TASK [create .kube directory] **** changed: [master] TASK [copy admin.conf to user's kube config] ***** changed: [master] TASK [install Pod network] ***** changed: [master] PLAY RECAP **** master : ok=5 changed=4 unreachable=0 failed=0

      To check the status of the master node, SSH into it with the following command:

      Once inside the master node, execute:

      You will now see the following output:


      NAME STATUS ROLES AGE VERSION master Ready master 1d v1.14.0

      The output states that the master node has completed all initialization tasks and is in a Ready state from which it can start accepting worker nodes and executing tasks sent to the API Server. You can now add the workers from your local machine.

      Step 5 — Setting Up the Worker Nodes

      Adding workers to the cluster involves executing a single command on each. This command includes the necessary cluster information, such as the IP address and port of the master's API Server, and a secure token. Only nodes that pass in the secure token will be able join the cluster.

      Navigate back to your workspace and create a playbook named workers.yml:

      • nano ~/kube-cluster/workers.yml

      Add the following text to the file to add the workers to the cluster:


      - hosts: master
        become: yes
        gather_facts: false
          - name: get join command
            shell: kubeadm token create --print-join-command
            register: join_command_raw
          - name: set join command
              join_command: "{{ join_command_raw.stdout_lines[0] }}"
      - hosts: workers
        become: yes
          - name: join cluster
            shell: "{{ hostvars['master'].join_command }} >> node_joined.txt"
              chdir: $HOME
              creates: node_joined.txt

      Here's what the playbook does:

      • The first play gets the join command that needs to be run on the worker nodes. This command will be in the following format:kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>. Once it gets the actual command with the proper token and hash values, the task sets it as a fact so that the next play will be able to access that info.

      • The second play has a single task that runs the join command on all worker nodes. On completion of this task, the two worker nodes will be part of the cluster.

      Save and close the file when you are finished.

      Execute the playbook by locally running:

      • ansible-playbook -i hosts ~/kube-cluster/workers.yml

      On completion, you will see output similar to the following:


      PLAY [master] **** TASK [get join command] **** changed: [master] TASK [set join command] ***** ok: [master] PLAY [workers] ***** TASK [Gathering Facts] ***** ok: [worker1] ok: [worker2] TASK [join cluster] ***** changed: [worker1] changed: [worker2] PLAY RECAP ***** master : ok=2 changed=1 unreachable=0 failed=0 worker1 : ok=2 changed=1 unreachable=0 failed=0 worker2 : ok=2 changed=1 unreachable=0 failed=0

      With the addition of the worker nodes, your cluster is now fully set up and functional, with workers ready to run workloads. Before scheduling applications, let's verify that the cluster is working as intended.

      Step 6 — Verifying the Cluster

      A cluster can sometimes fail during setup because a node is down or network connectivity between the master and worker is not working correctly. Let's verify the cluster and ensure that the nodes are operating correctly.

      You will need to check the current state of the cluster from the master node to ensure that the nodes are ready. If you disconnected from the master node, you can SSH back into it with the following command:

      Then execute the following command to get the status of the cluster:

      You will see output similar to the following:


      NAME STATUS ROLES AGE VERSION master Ready master 1d v1.14.0 worker1 Ready <none> 1d v1.14.0 worker2 Ready <none> 1d v1.14.0

      If all of your nodes have the value Ready for STATUS, it means that they're part of the cluster and ready to run workloads.

      If, however, a few of the nodes have NotReady as the STATUS, it could mean that the worker nodes haven't finished their setup yet. Wait for around five to ten minutes before re-running kubectl get nodes and inspecting the new output. If a few nodes still have NotReady as the status, you might have to verify and re-run the commands in the previous steps.

      Now that your cluster is verified successfully, let's schedule an example Nginx application on the cluster.

      Step 7 — Running An Application on the Cluster

      You can now deploy any containerized application to your cluster. To keep things familiar, let's deploy Nginx using Deployments and Services to see how this application can be deployed to the cluster. You can use the commands below for other containerized applications as well, provided you change the Docker image name and any relevant flags (such as ports and volumes).

      Still within the master node, execute the following command to create a deployment named nginx:

      • kubectl create deployment nginx --image=nginx

      A deployment is a type of Kubernetes object that ensures there's always a specified number of pods running based on a defined template, even if the pod crashes during the cluster's lifetime. The above deployment will create a pod with one container from the Docker registry's Nginx Docker Image.

      Next, run the following command to create a service named nginx that will expose the app publicly. It will do so through a NodePort, a scheme that will make the pod accessible through an arbitrary port opened on each node of the cluster:

      • kubectl expose deploy nginx --port 80 --target-port 80 --type NodePort

      Services are another type of Kubernetes object that expose cluster internal services to clients, both internal and external. They are also capable of load balancing requests to multiple pods, and are an integral component in Kubernetes, frequently interacting with other components.

      Run the following command:

      This will output text similar to the following:


      NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP <none> 443/TCP 1d nginx NodePort <none> 80:nginx_port/TCP 40m

      From the third line of the above output, you can retrieve the port that Nginx is running on. Kubernetes will assign a random port that is greater than 30000 automatically, while ensuring that the port is not already bound by another service.

      To test that everything is working, visit http://worker_1_ip:nginx_port or http://worker_2_ip:nginx_port through a browser on your local machine. You will see Nginx's familiar welcome page.

      If you would like to remove the Nginx application, first delete the nginx service from the master node:

      • kubectl delete service nginx

      Run the following to ensure that the service has been deleted:

      You will see the following output:


      NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP <none> 443/TCP 1d

      Then delete the deployment:

      • kubectl delete deployment nginx

      Run the following to confirm that this worked:


      No resources found.


      In this guide, you've successfully set up a Kubernetes cluster on Debian 9 using Kubeadm and Ansible for automation.

      If you're wondering what to do with the cluster now that it's set up, a good next step would be to get comfortable deploying your own applications and services onto the cluster. Here's a list of links with further information that can guide you in the process:

      • Dockerizing applications - lists examples that detail how to containerize applications using Docker.

      • Pod Overview - describes in detail how Pods work and their relationship with other Kubernetes objects. Pods are ubiquitous in Kubernetes, so understanding them will facilitate your work.

      • Deployments Overview - provides an overview of deployments. It is useful to understand how controllers such as deployments work since they are used frequently in stateless applications for scaling and the automated healing of unhealthy applications.

      • Services Overview - covers services, another frequently used object in Kubernetes clusters. Understanding the types of services and the options they have is essential for running both stateless and stateful applications.

      Other important concepts that you can look into are Volumes, Ingresses and Secrets, all of which come in handy when deploying production applications.

      Kubernetes has a lot of functionality and features to offer. The Kubernetes Official Documentation is the best place to learn about concepts, find task-specific guides, and look up API references for various objects.

      Source link

      How To Install and Configure an Apache ZooKeeper Cluster on Ubuntu 18.04

      The author selected Wikimedia Foundation Inc. to receive a donation as part of the Write for DOnations program.


      Apache ZooKeeper is open-source software that enables resilient and highly reliable distributed coordination. It is commonly used in distributed systems to manage configuration information, naming services, distributed synchronization, quorum, and state. In addition, distributed systems rely on ZooKeeper to implement consensus, leader election, and group management.

      In this guide, you will install and configure Apache ZooKeeper 3.4.13 on Ubuntu 18.04. To achieve resilience and high availability, ZooKeeper is intended to be replicated over a set of hosts, called an ensemble. First, you will create a standalone installation of a single-node ZooKeeper server and then add in details for setting up a multi-node cluster. The standalone installation is useful in development and testing environments, but a cluster is the most practical solution for production environments.


      Before you begin this installation and configuration guide, you’ll need the following:

      • The standalone installation needs one Ubuntu 18.04 server with a minimum of 4GB of RAM set up by following the Ubuntu 18.04 initial server setup guide, including a non-root user with sudo privileges and a firewall. You need two additional servers, set up by following the same steps, for the multi-node cluster.
      • OpenJDK 8 installed on your server, as ZooKeeper requires Java to run. To do this, follow the “Install Specific Versions of OpenJDK” step from the How To Install Java with `apt` on Ubuntu 18.04 guide.

      Because ZooKeeper keeps data in memory to achieve high throughput and low latency, production systems work best with 8GB of RAM. Lower amounts of RAM may lead to JVM swapping, which could cause ZooKeeper server latency. High ZooKeeper server latency could result in issues like client session timeouts that would have an adverse impact on system functionality.

      Step 1 — Creating a User for ZooKeeper

      A dedicated user should run services that handle requests over a network and consume resources. This practice creates segregation and control that will improve your environment’s security and manageability. In this step, you’ll create a non-root sudo user, named zk in this tutorial, to run the ZooKeeper service.

      First, log in as the non-root sudo user that you created in the prerequisites.

      ssh sammy@your_server_ip

      Create the user that will run the ZooKeeper service:

      Passing the -m flag to the useradd command will create a home directory for this user. The home directory for zk will be /home/zk by default.

      Set bash as the default shell for the zk user:

      • sudo usermod --shell /bin/bash zk

      Set a password for this user:

      Next, you will add the zk user to the sudo group so it can run commands in a privileged mode:

      In terms of security, it is recommended that you allow SSH access to as few users as possible. Logging in remotely as sammy and then using su to switch to the desired user creates a level of separation between credentials for accessing the system and running processes. You will disable SSH access for both your zk and root user in this step.

      Open your sshd_config file:

      • sudo nano /etc/ssh/sshd_config

      Locate the PermitRootLogin line and set the value to no to disable SSH access for the root user:


      PermitRootLogin no

      Under the PermitRootLogin value, add a DenyUsers line and set the value as any user who should have SSH access disabled:


      DenyUsers zk

      Save and exit the file and then restart the SSH daemon to activate the changes.

      • sudo systemctl restart sshd

      Switch to the zk user:

      The -l flag invokes a login shell after switching users. A login shell resets environment variables and provides a clean start for the user.

      Enter the password at the prompt to authenticate the user.

      Now that you have created, configured, and logged in as the zk user, you will create a directory to store your ZooKeeper data.

      Step 2 — Creating a Data Directory for ZooKeeper

      ZooKeeper persists all configuration and state data to disk so it can survive a reboot. In this step, you will create a data directory that ZooKeeper will use to read and write data. You can create the data directory on the local filesystem or on a remote storage drive. This tutorial will focus on creating the data directory on your local filesystem.

      Create a directory for ZooKeeper to use:

      • sudo mkdir -p /data/zookeeper

      Grant your zk user ownership to the directory:

      • sudo chown zk:zk /data/zookeeper

      chown changes the ownership and group of the /data/zookeeper directory so that the user zk, who belongs to the group zk, owns the data directory.

      You have successfully created and configured the data directory. When you move on to configure ZooKeeper, you will specify this path as the data directory that ZooKeeper will use to store its files.

      Step 3 — Downloading and Extracting the ZooKeeper Binaries

      In this step, you will manually download and extract the ZooKeeper binaries to the /opt directory. You can use the Advanced Packaging Tool, apt, to download ZooKeeper, but it may install an older version with different features. Installing ZooKeeper manually will give you full control to choose which version you would like to use.

      Since you are downloading these files manually, start by changing to the /opt directory:

      From your local machine, navigate to the Apache download page. This page will automatically provide you with the mirror closest to you for the fastest download. Click the link to the suggested mirror site, then scroll down and click zookeeper/ to view the available releases. Select the version of ZooKeeper that you would like to install. This tutorial will focus on using 3.4.13. Once you select the version, right click the binary file ending with .tar.gz and copy the link address.

      From your server, use the wget command along with the copied link to download the ZooKeeper binaries:

      • sudo wget

      Extract the binaries from the compressed archive:

      • sudo tar -xvf zookeeper-3.4.13.tar.gz

      The .tar.gz extension represents a combination of TAR packaging followed by a GNU zip (gzip) compression. You will notice that you passed the flag -xvf to the command to extract the archive. The flag x stands for extract, v enables verbose mode to show the extraction progress, and f allows specifying the input, in our case zookeeper-3.4.13.tar.gz, as opposed to STDIN.

      Next, give the zk user ownership of the extracted binaries so that it can run the executables. You can change ownership like so:

      • sudo chown zk:zk -R zookeeper-3.4.13

      Next, you will configure a symbolic link to ensure that your ZooKeeper directory will remain relevant across updates. You can also use symbolic links to shorten directory names, which can lessen the time it takes to set up your configuration files.

      Create a symbolic link using the ln command.

      • sudo ln -s zookeeper-3.4.13 zookeeper

      Change the ownership of that link to zk:zk. Notice that you have passed a -h flag to change the ownership of the link itself. Not specifying -h changes the ownership of the target of the link, which you explicitly did in the previous step.

      • sudo chown -h zk:zk zookeeper

      With the symbolic links created, your directory paths in the configurations will remain relevant and unchanged through future upgrades. You can now configure ZooKeeper.

      Step 4 — Configuring ZooKeeper

      Now that you've set up your environment, you are ready to configure ZooKeeper.

      The configuration file will live in the /opt/zookeeper/conf directory. This directory contains a sample configuration file that comes with the ZooKeeper distribution. This sample file, named zoo_sample.cfg, contains the most common configuration parameter definitions and sample values for these parameters. Some of the common parameters are as follows:

      • tickTime: Sets the length of a tick in milliseconds. A tick is a time unit used by ZooKeeper to measure the length between heartbeats. Minimum session timeouts are twice the tickTime.
      • dataDir: Specifies the directory used to store snapshots of the in-memory database and the transaction log for updates. You could choose to specify a separate directory for transaction logs.
      • clientPort: The port used to listen for client connections.
      • maxClientCnxns: Limits the maximum number of client connections.

      Create a configuration file named zoo.cfg at /opt/zookeeper/conf. You can create and open a file using nano or your favorite editor:

      • nano /opt/zookeeper/conf/zoo.cfg

      Add the following set of properties and values to that file:



      A tickTime of 2000 milliseconds is the suggested interval between heartbeats. A shorter interval could lead to system overhead with limited benefits. The dataDir parameter points to the path defined by the symbolic link you created in the previous section. Conventionally, ZooKeeper uses port 2181 to listen for client connections. In most situations, 60 allowed client connections are plenty for development and testing.

      Save the file and exit the editor.

      You have configured ZooKeeper and are ready to start the server.

      Step 5 — Starting ZooKeeper and Testing the Standalone Installation

      You've configured all the components needed to run ZooKeeper. In this step, you will start the ZooKeeper service and test your configuration by connecting to the service locally.

      Navigate back to the /opt/zookeeper directory.

      Start ZooKeeper with the command.

      You will see the following on your standard output:


      ZooKeeper JMX enabled by default Using config: /opt/zookeeper/bin/../conf/zoo.cfg Starting zookeeper ... STARTED

      Connect to the local ZooKeeper server with the following command:

      • bin/ -server

      You will get a prompt with the label CONNECTED. This confirms that you have a successful local, standalone ZooKeeper installation. If you encounter errors, you will want to verify that the configuration is correct.


      Connecting to ... ... [zk: 0]

      Type help on this prompt to get a list of commands that you can execute from the client. The output will be as follows:


      [zk: 0] help ZooKeeper -server host:port cmd args stat path [watch] set path data [version] ls path [watch] delquota [-n|-b] path ls2 path [watch] setAcl path acl setquota -n|-b val path history redo cmdno printwatches on|off delete path [version] sync path listquota path rmr path get path [watch] create [-s] [-e] path data acl addauth scheme auth quit getAcl path close connect host:port

      After you've done some testing, you will close the client session by typing quit on the prompt. The ZooKeeper service will continue running after you closed the client session. Shut down the ZooKeeper service, as you'll configure it as a systemd service in the next step:

      You have now installed, configured, and tested a standalone ZooKeeper service. This setup is useful to familiarize yourself with ZooKeeper, but is also helpful for developmental and testing environments. Now that you know the configuration works, you will configure systemd to simplify the management of your ZooKeeper service.

      Step 6 — Creating and Using a Systemd Unit File

      The systemd, system and service manager, is an init system used to bootstrap the user space and to manage system processes after boot. You can create a daemon for starting and checking the status of ZooKeeper using systemd.

      Systemd Essentials is a great introductory resource for learning more about systemd and its constituent components.

      Use your editor to create a .service file named zk.service at /etc/systemd/system/.

      • sudo nano /etc/systemd/system/zk.service

      Add the following lines to the file to define the ZooKeeper Service:


      Description=Zookeeper Daemon
      ExecStart=/opt/zookeeper/bin/ start /opt/zookeeper/conf/zoo.cfg
      ExecStop=/opt/zookeeper/bin/ stop /opt/zookeeper/conf/zoo.cfg
      ExecReload=/opt/zookeeper/bin/ restart /opt/zookeeper/conf/zoo.cfg

      The Service section in the unit file configuration specifies the working directory, the user under which the service would run, and the executable commands to start, stop, and restart the ZooKeeper service. For additional information on all the unit file configuration options, you can read the Understanding Systemd Units and Unit Files article.

      Save the file and exit the editor.

      Now that your systemd configuration is in place, you can start the service:

      Once you've confirmed that your systemd file can successfully start the service, you will enable the service to start on boot.

      This output confirms the creation of the symbolic link:


      Created symlink /etc/systemd/system/ → /etc/systemd/system/zk.service.

      Check the status of the ZooKeeper service using:

      Stop the ZooKeeper service using systemctl.

      Finally, to restart the daemon, use the following command:

      • sudo systemctl restart zk

      The systemd mechanism is becoming the init system of choice on many Linux distributions. Now that you've configured systemd to manage ZooKeeper, you can leverage this fast and flexible init model to start, stop, and restart the ZooKeeper service.

      Step 7 — Configuring a Multi-Node ZooKeeper Cluster

      While the standalone ZooKeeper server is useful for development and testing, every production environment should have a replicated multi-node cluster.

      Nodes in the ZooKeeper cluster that work together as an application form a quorum. Quorum refers to the minimum number of nodes that need to agree on a transaction before it's committed. A quorum needs an odd number of nodes so that it can establish a majority. An even number of nodes may result in a tie, which would mean the nodes would not reach a majority or consensus.

      In a production environment, you should run each ZooKeeper node on a separate host. This prevents service disruption due to host hardware failure or reboots. This is an important and necessary architectural consideration for building a resilient and highly available distributed system.

      In this tutorial, you will install and configure three nodes in the quorum to demonstrate a multi-node setup. Before you configure a three-node cluster, you will spin up two additional servers with the same configuration as your standalone ZooKeeper installation. Ensure that the two additional nodes meet the prerequisites, and then follow steps one through six to set up a running ZooKeeper instance.

      Once you've followed steps one through six for the new nodes, open zoo.cfg in the editor on each node.

      • sudo nano /opt/zookeeper/conf/zoo.cfg

      All nodes in a quorum will need the same configuration file. In your zoo.cfg file on each of the three nodes, add the additional configuration parameters and values for initLimit, syncLimit, and the servers in the quorum, at the end of the file.



      initLimit specifies the time that the initial synchronization phase can take. This is the time within which each of the nodes in the quorum needs to connect to the leader. syncLimit specifies the time that can pass between sending a request and receiving an acknowledgment. This is the maximum time nodes can be out of sync from the leader. ZooKeeper nodes use a pair of ports, :2888 and :3888, for follower nodes to connect to the leader node and for leader election, respectively.

      Once you've updated the file on each node, you will save and exit the editor.

      To complete your multi-node configuration, you will specify a node ID on each of the servers. To do this, you will create a myid file on each node. Each file will contain a number that correlates to the server number assigned in the configuration file.

      On your_zookeeper_node_1, create the myid file that will specify the node ID:

      • sudo nano /data/zookeeper/myid

      Since your_zookeeper_node_1 is identified as server.1, you will enter 1 to define the node ID. After adding the value, your file will look like this:

      your_zookeeper_node_1 /data/zookeeper/myid


      Follow the same steps for the remaining nodes. The myid file on each node should be as follows:

      your_zookeeper_node_1 /data/zookeeper/myid


      your_zookeeper_node_2 /data/zookeeper/myid


      your_zookeeper_node_3 /data/zookeeper/myid


      You have now configured a three-node ZooKeeper cluster. Next, you will run the cluster and test your installation.

      Step 8 — Running and Testing the Multi-Node Installation

      With each node configured to work as a cluster, you are ready to start a quorum. In this step, you will start the quorum on each node and then test your cluster by creating sample data in ZooKeeper.

      To start a quorum node, first change to the /opt/zookeeper directory on each node:

      Start each node with the following command:

      • java -cp zookeeper-3.4.13.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.25.jar:lib/slf4j-api-1.7.25.jar:conf org.apache.zookeeper.server.quorum.QuorumPeerMain conf/zoo.cfg

      As nodes start up, you will intermittently see some connection errors followed by a stage where they join the quorum and elect a leader among themselves. After a few seconds of initialization, you can start testing your installation.

      Log in via SSH to your_zookeeper_node_3 as the non-root user you configured in the prerequisites:

      • ssh sammy@your_zookeeper_node_3

      Once logged in, switch to your zk user:

      your_zookeeper_node_3 /data/zookeeper/myid

      Enter the password for the zk user. Once logged in, change the directory to /opt/zookeeper:

      your_zookeeper_node_3 /data/zookeeper/myid

      You will now start a ZooKeeper command line client and connect to ZooKeeper on your_zookeeper_node_1:

      your_zookeeper_node_3 /data/zookeeper/myid

      • bin/ -server your_zookeeper_node_1:2181

      In the standalone installation, both the client and server were running on the same host. This allowed you to establish a client connection with the ZooKeeper server using localhost. Since the client and server are running on different nodes in your multi-node cluster, in the previous step you needed to specify the IP address of your_zookeeper_node_1 to connect to it.

      You will see the familiar prompt with the CONNECTED label, similar to what you saw in Step 5.

      Next, you will create, list, and then delete a znode. The znodes are the fundamental abstractions in ZooKeeper that are analogous to files and directories on a file system. ZooKeeper maintains its data in a hierarchical namespace, and znodes are the data registers of this namespace.

      Testing that you can successfully create, list, and then delete a znode is essential to establishing that your ZooKeeper cluster is installed and configured correctly.

      Create a znode named zk_znode_1 and associate the string sample_data with it.

      • create /zk_znode_1 sample_data

      You will see the following output once created:


      Created /zk_znode_1

      List the newly created znode:

      Get the data associated with it:

      ZooKeeper will respond like so:


      [zk: your_zookeeper_node_1:2181(CONNECTED)] ls / [zk_znode_1, zookeeper] [zk: your_zookeeper_node_1:2181(CONNECTED)] get /zk_znode_1 sample_data cZxid = 0x100000002 ctime = Tue Nov 06 19:47:41 UTC 2018 mZxid = 0x100000002 mtime = Tue Nov 06 19:47:41 UTC 2018 pZxid = 0x100000002 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 11 numChildren = 0

      The output confirms the value, sample_data, that you associated with zk_node_1. ZooKeeper also provides additional information about creation time, ctime, and modification time, mtime. ZooKeeper is a versioned data store, so it also presents you with metadata about the data version.

      Delete the zk_znode_1 znode:

      In this step, you successfully tested connectivity between two of your ZooKeeper nodes. You also learned basic znode management by creating, listing, and deleting znodes. Your multi-node configuration is complete, and you are ready to start using ZooKeeper.


      In this tutorial, you configured and tested both a standalone and multi-node ZooKeeper environment. Now that your multi-node ZooKeeper deployment is ready to use, you can review the official ZooKeeper documentation for additional information and projects.

      Source link

      How to Back Up and Restore a Kubernetes Cluster on DigitalOcean Using Heptio Ark


      Heptio Ark is a convenient backup tool for Kubernetes clusters that compresses and backs up Kubernetes objects to object storage. It also takes snapshots of your cluster’s Persistent Volumes using your cloud provider’s block storage snapshot features, and can then restore your cluster’s objects and Persistent Volumes to a previous state.

      StackPointCloud’s DigitalOcean Ark Plugin allows you to use DigitalOcean block storage to snapshot your Persistent Volumes, and Spaces to back up your Kubernetes objects. When running a Kubernetes cluster on DigitalOcean, this allows you to quickly back up your cluster’s state and restore it should disaster strike.

      In this tutorial we’ll set up and configure the Ark client on a local machine, and deploy the Ark server into our Kubernetes cluster. We’ll then deploy a sample Nginx app that uses a Persistent Volume for logging, and simulate a disaster recovery scenario.


      Before you begin this tutorial, you should have the following available to you:

      On your local computer:

      In your DigitalOcean account:

      • A DigitalOcean Kubernetes cluster, or a Kubernetes cluster (version 1.7.5 or later) on DigitalOcean Droplets
      • A DNS server running inside of your cluster. If you are using DigitalOcean Kubernetes, this is running by default. To learn more about configuring a Kubernetes DNS service, consult Customizing DNS Service from the official Kuberentes documentation.
      • A DigitalOcean Space that will store your backed-up Kubernetes objects. To learn how to create a Space, consult the Spaces product documentation.
      • An access key pair for your DigitalOcean Space. To learn how to create a set of access keys, consult How to Manage Administrative Access to Spaces.
      • A personal access token for use with the DigitalOcean API. To learn how to create a personal access token, consult How to Create a Personal Access Token.

      Once you have all of this set up, you’re ready to begin with this guide.

      Step 1 — Installing the Ark Client

      The Heptio Ark backup tool consists of a client installed on your local computer and a server that runs in your Kubernetes cluster. To begin, we’ll install the local Ark client.

      In your web browser, navigate to the Ark GitHub repo releases page, find the latest release corresponding to your OS and system architecture, and copy the link address. For the purposes of this guide, we’ll use an Ubuntu 18.04 server on an x86-64 (or AMD64) processor as our local machine.

      Then, from the command line on your local computer, navigate to the temporary /tmp directory and cd into it:

      Use wget and the link you copied earlier to download the release tarball:

      • wget https://link_copied_from_release_page

      Once the download completes, extract the tarball using tar (note the filename may differ depending on the current release version and your OS):

      • tar -xvzf ark-v0.9.6-linux-amd64.tar.gz

      The /tmp directory should now contain the extracted ark binary as well as the tarball you just downloaded.

      Verify that you can run the ark client by executing the binary:

      You should see the following help output:


      Heptio Ark is a tool for managing disaster recovery, specifically for Kubernetes cluster resources. It provides a simple, configurable, and operationally robust way to back up your application state and associated data. If you're familiar with kubectl, Ark supports a similar model, allowing you to execute commands such as 'ark get backup' and 'ark create schedule'. The same operations can also be performed as 'ark backup get' and 'ark schedule create'. Usage: ark [command] Available Commands: backup Work with backups client Ark client related commands completion Output shell completion code for the specified shell (bash or zsh) create Create ark resources delete Delete ark resources describe Describe ark resources get Get ark resources help Help about any command plugin Work with plugins restic Work with restic restore Work with restores schedule Work with schedules server Run the ark server version Print the ark version and associated image . . .

      At this point you should move the ark executable out of the temporary /tmp directory and add it to your PATH. To add it to your PATH on an Ubuntu system, simply copy it to /usr/local/bin:

      • sudo mv ark /usr/local/bin/ark

      You're now ready to configure the Ark server and deploy it to your Kubernetes cluster.

      Step 2 — Installing and Configuring the Ark Server

      Before we deploy Ark into our Kubernetes cluster, we'll first create Ark's prerequisite objects. Ark's prerequisites consist of:

      • A heptio-ark Namespace

      • The ark Service Account

      • Role-based access control (RBAC) rules to grant permissions to the ark Service Account

      • Custom Resources (CRDs) for the Ark-specific resources: Backup, Schedule, Restore, Config

      A YAML file containing the specs for the above Kubernetes objects can be found in the official Ark Git repository. While still in the /tmp directory, download the Ark repo using git:

      • git clone

      Once downloaded, navigate into the ark directory:

      The prerequisite resources listed above can be found in the examples/common/00-prereqs.yaml YAML file. We'll create these resources in our Kubernetes cluster by using kubectl apply and passing in the file:

      • kubectl apply -f examples/common/00-prereqs.yaml

      You should see the following output:

      Output created created created created created created created created created created namespace/heptio-ark created serviceaccount/ark created created

      Now that we've created the necessary Ark Kubernetes objects in our cluster, we can download and install the Ark DigitalOcean Plugin, which will allow us to use DigitalOcean Spaces as a backupStorageProvider (for Kubernetes objects), and DigitalOcean Block Storage as a persistentVolumeProvider (for Persistent Volume backups).

      Move back out of the ark directory and fetch the plugin from StackPointCloud's repo using git:

      • cd ..
      • git clone

      Move into the plugin directory:

      • cd ark-plugin-digitalocean

      We'll now save the access keys for our DigitalOcean Space as a Kubernetes Secret. First, open up the examples/credentials-ark file using your favorite editor:

      • nano examples/credentials-ark

      Replace <AWS_ACCESS_KEY_ID> and <AWS_SECRET_ACCESS_KEY> with your Spaces access key and secret key:



      Save and close the file.

      Now, create the cloud-credentials Secret using kubectl, inserting your Personal Access Token using the digitalocean_token data item:

      • kubectl create secret generic cloud-credentials
      • --namespace heptio-ark
      • --from-file cloud=examples/credentials-ark
      • --from-literal digitalocean_token=your_personal_access_token

      You should see the following output:


      secret/cloud-credentials created

      To confirm that the cloud-credentials Secret was created successfully, you can describe it using kubectl:

      • kubectl describe secrets/cloud-credentials --namespace heptio-ark

      You should see the following output describing the cloud-credentials secret:


      Name: cloud-credentials Namespace: heptio-ark Labels: <none> Annotations: <none> Type: Opaque Data ==== cloud: 115 bytes digitalocean_token: 64 bytes

      We can now move on to creating an Ark Config object named default. To do this, we'll edit a YAML configuration file and then create the object in our Kubernetes cluster.

      Open examples/10-ark-config.yaml in your favorite editor:

      • nano examples/10-ark-config.yaml

      Insert your Space's name and region in the highlighted fields:


      kind: Config
        namespace: heptio-ark
        name: default
        name: digitalocean
        name: aws
        bucket: space_name_here
          region: space_region_here
          s3ForcePathStyle: "true"
      backupSyncPeriod: 30m
      gcSyncPeriod: 30m
      scheduleSyncPeriod: 1m
      restoreOnlyMode: false

      persistentVolumeProvider sets DigitalOcean Block Storage as the the provider for Persistent Volume backups. These will be Block Storage Volume Snapshots.

      backupStorageProvider sets DigitalOcean Spaces as the provider for Kubernetes object backups. Ark will create a tarball of all your Kubernetes objects (or some, depending on how you execute it), and upload this tarball to Spaces.

      When you're done, save and close the file.

      Create the object in your cluster using kubectl apply:

      • kubectl apply -f examples/10-ark-config.yaml

      You should see the following output:

      Output created

      At this point, we've finished configuring the Ark server and can create its Kubernetes deployment, found in the examples/20-deployment.yaml configuration file. Let's take a quick look at this file:

      • cat examples/20-deployment.yaml

      You should see the following text:


      apiVersion: apps/v1beta1
      kind: Deployment
        namespace: heptio-ark
        name: ark
        replicas: 1
              component: ark
            restartPolicy: Always
            serviceAccountName: ark
              - name: ark
                  - /ark
                  - server
                  - name: cloud-credentials
                    mountPath: /credentials
                  - name: plugins
                    mountPath: /plugins
                  - name: scratch
                    mountPath: /scratch
                  - name: AWS_SHARED_CREDENTIALS_FILE
                    value: /credentials/cloud
                  - name: ARK_SCRATCH_DIR
                    value: /scratch
                  - name: DIGITALOCEAN_TOKEN
                        key: digitalocean_token
                        name: cloud-credentials
              - name: cloud-credentials
                  secretName: cloud-credentials
              - name: plugins
                emptyDir: {}
              - name: scratch
                emptyDir: {}

      We observe here that we're creating a Deployment called ark that consists of a single replica of the container. The Pod is configured using the cloud-credentials secret we created earlier.

      Create the Deployment using kubectl apply:

      • kubectl apply -f examples/20-deployment.yaml

      You should see the following output:


      deployment.apps/ark created

      We can double check that the Deployment has been successfully created using kubectl get on the heptio-ark Namespace :

      • kubectl get deployments --namespace=heptio-ark

      You should see the following output:



      The Ark server Pod may not start correctly until you install the Ark DigitalOcean plugin. To install the ark-blockstore-digitalocean plugin, use the ark client we installed earlier:

      • ark plugin add

      You can specify the kubeconfig to use with the --kubeconfig flag. If you don't use this flag, ark will check the KUBECONFIG environment variable and then fall back to the kubectl default (~/.kube/config).

      At this point Ark is running and fully configured, and ready to back up and restore your Kubernetes cluster objects and Persistent Volumes to DigitalOcean Spaces and Block Storage.

      In the next section, we'll run a quick test to make sure that the backup and restore functionality works as expected.

      Step 3 — Testing Backup and Restore Procedure

      Now that we've successfully installed and configured Ark, we can create a test Nginx Deployment and Persistent Volume, and run through a backup and restore drill to ensure that everything is working properly.

      The ark-plugin-digitalocean repository contains a sample Nginx deployment called nginx-pv.yaml.

      Let's take a quick look:

      • cat examples/nginx-pv.yaml

      You should see the following text:


      --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: nginx-logs namespace: nginx-example labels: app: nginx spec: storageClassName: do-block-storage accessModes: - ReadWriteOnce resources: requests: storage: 5Gi --- apiVersion: apps/v1beta1 kind: Deployment metadata: name: nginx-deployment namespace: nginx-example spec: replicas: 1 template: metadata: labels: app: nginx spec: volumes: - name: nginx-logs persistentVolumeClaim: claimName: nginx-logs containers: - image: nginx:1.7.9 name: nginx ports: - containerPort: 80 volumeMounts: - mountPath: "/var/log/nginx" name: nginx-logs readOnly: false --- apiVersion: v1 kind: Service metadata: labels: app: nginx name: my-nginx namespace: nginx-example spec: ports: - port: 80 targetPort: 80 selector: app: nginx type: LoadBalancer

      In this file, we observe specs for:

      • An Nginx Deployment consisting of a single replica of the nginx:1.7.9 container image
      • A 5Gi Persistent Volume Claim (called nginx-logs), using the do-block-storage StorageClass
      • A LoadBalancer Service that exposes port 80

      Create the deployment using kubectl apply:

      • kubectl apply -f examples/nginx-pv.yml

      You should see the following output:


      namespace/nginx-example created persistentvolumeclaim/nginx-logs created deployment.apps/nginx-deployment created service/my-nginx created

      Check that the Deployment succeeded:

      • kubectl get deployments --namespace=nginx-example

      You should see the following output:


      NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE nginx-deployment 1 1 1 1 1h

      Once Available reaches 1, fetch the Nginx load balancer’s external IP using kubectl get:

      • kubectl get services --namespace=nginx-example

      You should see both the internal CLUSTER-IP and EXTERNAL-IP for the my-nginx Service:


      NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE my-nginx LoadBalancer 80:30754/TCP 3m

      Note the EXTERNAL-IP and navigate to it using your web browser.

      You should see the following NGINX welcome page:

      Nginx Welcome Page

      This indicates that your Nginx Deployment and Service are up and running.

      Before we simulate our disaster scenario, let’s first check the Nginx access logs (stored on a Persistent Volume attached to the Nginx Pod):

      Fetch the Pod’s name using kubectl get:

      • kubectl get pods --namespace nginx-example


      NAME READY STATUS RESTARTS AGE nginx-deployment-77d8f78fcb-zt4wr 1/1 Running 0 29m

      Now, exec into the running Nginx container to get a shell inside of it:

      • kubectl exec -it nginx-deployment-77d8f78fcb-zt4wr --namespace nginx-example -- /bin/bash

      Once inside the Nginx container, cat the Nginx access logs:

      • cat /var/log/nginx/access.log

      You should see some Nginx access entries:

      Output - - [01/Oct/2018:21:47:01 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ Safari/537.36" "-" - - [01/Oct/2018:21:47:01 +0000] "GET /favicon.ico HTTP/1.1" 404 570 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ Safari/537.36" "-"

      Note these down (especially the timestamps), as we will use them to confirm the success of the restore procedure.

      We can now perform the backup procedure to copy all nginx Kubernetes objects to Spaces and take a Snapshot of the Persistent Volume we created when deploying Nginx.

      We'll create a backup called nginx-backup using the ark client:

      • ark backup create nginx-backup --selector app=nginx

      The --selector app=nginx instructs the Ark server to only back up Kubernetes objects with the app=nginx Label Selector.

      You should see the following output:


      Backup request "nginx-backup" submitted successfully. Run `ark backup describe nginx-backup` for more details.

      Running ark backup describe nginx-backup should provide the following output after a short delay:


      Name: nginx-backup Namespace: heptio-ark Labels: <none> Annotations: <none> Phase: Completed Namespaces: Included: * Excluded: <none> Resources: Included: * Excluded: <none> Cluster-scoped: auto Label selector: app=nginx Snapshot PVs: auto TTL: 720h0m0s Hooks: <none> Backup Format Version: 1 Started: 2018-09-26 00:14:30 -0400 EDT Completed: 2018-09-26 00:14:34 -0400 EDT Expiration: 2018-10-26 00:14:30 -0400 EDT Validation errors: <none> Persistent Volumes: pvc-e4862eac-c2d2-11e8-920b-92c754237aeb: Snapshot ID: 2eb66366-c2d3-11e8-963b-0a58ac14428b Type: ext4 Availability Zone: IOPS: <N/A>

      This output indicates that nginx-backup completed successfully.

      From the DigitalOcean Cloud Control Panel, navigate to the Space containing your Kubernetes backup files.

      You should see a new directory called nginx-backup containing the Ark backup files.

      Using the left-hand navigation bar, go to Images and then Snapshots. Within Snapshots, navigate to Volumes. You should see a Snapshot corresponding to the PVC listed in the above output.

      We can now test the restore procedure.

      Let's first delete the nginx-example Namespace. This will delete everything in the Namespace, including the Load Balancer and Persistent Volume:

      • kubectl delete namespace nginx-example

      Verify that you can no longer access Nginx at the Load Balancer endpoint, and that the nginx-example Deployment is no longer running:

      • kubectl get deployments --namespace=nginx-example


      No resources found.

      We can now perform the restore procedure, once again using the ark client:

      • ark restore create --from-backup nginx-backup

      Here we use create to create an Ark Restore object from the nginx-backup object.

      You should see the following output:


      • Restore request "nginx-backup-20180926143828" submitted successfully.
      • Run `ark restore describe nginx-backup-20180926143828` for more details.

      Check the status of the restored Deployment:

      • kubectl get deployments --namespace=nginx-example


      NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE nginx-deployment 1 1 1 1 1m

      Check for the creation of a Persistent Volume:

      • kubectl get pvc --namespace=nginx-example


      NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE nginx-logs Bound pvc-e4862eac-c2d2-11e8-920b-92c754237aeb 5Gi RWO do-block-storage 3m

      Navigate to the Nginx Service’s external IP once again to confirm that Nginx is up and running.

      Finally, check the logs on the restored Persistent Volume to confirm that the log history has been preserved post-restore.

      To do this, once again fetch the Pod’s name using kubectl get:

      • kubectl get pods --namespace nginx-example


      NAME READY STATUS RESTARTS AGE nginx-deployment-77d8f78fcb-zt4wr 1/1 Running 0 29m

      Then exec into it:

      • kubectl exec -it nginx-deployment-77d8f78fcb-zt4wr --namespace nginx-example -- /bin/bash

      Once inside the Nginx container, cat the Nginx access logs:

      • cat /var/log/nginx/access.log

      Output - - [01/Oct/2018:21:47:01 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ Safari/537.36" "-" - - [01/Oct/2018:21:47:01 +0000] "GET /favicon.ico HTTP/1.1" 404 570 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ Safari/537.36" "-"

      You should see the same pre-backup access attempts (note the timestamps), confirming that the Persistent Volume restore was successful. Note that there may be additional attempts in the logs if you visited the Nginx landing page after you performed the restore.

      At this point, we've successfully backed up our Kubernetes objects to DigitalOcean Spaces, and our Persistent Volumes using Block Storage Volume Snapshots. We simulated a disaster scenario, and restored service to the test Nginx application.


      In this guide we installed and configured the Ark Kubernetes backup tool on a DigitalOcean-based Kubernetes cluster. We configured the tool to back up Kubernetes objects to DigitalOcean Spaces, and back up Persistent Volumes using Block Storage Volume Snapshots.

      Ark can also be used to schedule regular backups of your Kubernetes cluster. To do this, you can use the ark schedule command. It can also be used to migrate resources from one cluster to another. To learn more about these two use cases, consult the official Ark documentation.

      To learn more about DigitalOcean Spaces, consult the official Spaces documentation. To learn more about Block Storage Volumes, consult the Block Storage Volume documentation.

      This tutorial builds on the README found in StackPointCloud's ark-plugin-digitalocean GitHub repo.

      Source link