One place for hosting & domains

      Software

      Use Buildbot for Software Testing on Ubuntu 18.04


      Updated by Linode Written by Tyler Langlois

      Use promo code DOCS10 for $10 credit on a new account.

      Buildbot is an open source system for testing software projects. In this guide, you will set up a Linode as a Buildbot server to use as a continuous integration platform to test code. Similarly to hosted solutions like Travis CI, Buildbot is an automated testing platform that can watch for code changes, test a project’s code, and send notifications regarding build failures.

      Before you Begin

      1. Familiarize yourself with Linode’s Getting Started guide and complete the steps for deploying and setting up a Linode running Ubuntu 18.04, including setting the hostname and timezone.

      2. This guide uses sudo wherever possible. Complete the sections of our Securing Your Server guide to create a standard user account, harden SSH access and remove unnecessary network services.

      3. Ensure your system is up to date:

        sudo apt update && sudo apt upgrade
        
      4. Complete the Add DNS Records steps to register a domain name that will point to your Linode instance hosting Buildbot.

        Note

        Replace each instance of example.com in this guide with your Buildbot site’s domain name.

      5. Your Buildbot site will serve its content over HTTPS, so you will need to obtain an SSL/TLS certificate. Use Certbot to request and download a free certificate from Let’s Encrypt.

        sudo apt install software-properties-common
        sudo add-apt-repository ppa:certbot/certbot
        sudo apt update
        sudo apt install certbot
        sudo certbot certonly --standalone -d example.com
        

        These commands will download a certificate to /etc/letsencrypt/live/example.com/ on your Linode.

        Note

      Install Buildbot

      Install the Buildbot Master

      Since Buildbot is provided as an Ubuntu package, install the software from the official Ubuntu repositories.

      1. Install the buildbot package along with pip3, which will be used to install additional python packages:

        sudo apt-get install -y buildbot python3-pip
        
      2. Install the required Buildbot Python packages:

        sudo pip3 install buildbot-www buildbot-waterfall-view buildbot-console-view buildbot-grid-view
        
      3. The buildbot package sets up several file paths and services to run persistently on your host. In order to create a new configuration for a Buildbot master, enter the directory for Buildbot master configurations and create a new master called ci (for “continuous integration”).

        cd /var/lib/buildbot/masters
        sudo -u buildbot -- buildbot create-master ci
        

        The generated master configuration file’s location is /var/lib/buildbot/masters/ci/master.cfg.sample.

      4. Make a copy of the default configuration to the path that Buildbot expects for its configuration file:

        sudo cp ci/master.cfg.sample ci/master.cfg
        
      5. Change the permissions for this configuration file so that the buildbot user has rights for the configuration file:

        sudo chown buildbot:buildbot ci/master.cfg
        

      Configure the Buildbot Master

      In order to secure and customize Buildbot, you will change a few settings in the master configuration file before using the application. The master configuration file’s location is /var/lib/buildbot/masters/ci/master.cfg.

      Buildbot has a number of concepts that are represented in the master build configuration file. Open this file in your preferred text editor and browse the Buildbot configuration. The Buildbot configuration is written in Python instead of a markup language like Yaml.

      1. Generate a random string to serve as the password that workers will use to authenticate against the Buildbot master. This is accomplished by using openssl to create a random sequence of characters.

        openssl rand -hex 16
        <a random string>
        
      2. Update the following line in the master.cfg file and replace pass with the randomly-generated password:

        /var/lib/buildbot/masters/ci/master.cfg
        1
        2
        3
        4
        5
        6
        7
        
        ...
        # The 'workers' list defines the set of recognized workers. Each element is
        # a Worker object, specifying a unique worker name and password.  The same
        # worker name and password must be configured on the worker.
        c['workers'] = [worker.Worker("example-worker", "pass")]
        ...
            
      3. Uncomment the cUse Buildbot for Software Testing on Ubuntu 18.04 and the c[titleURL] lines. If desired, change the name of the Buildbot installation by updating the value of cUse Buildbot for Software Testing on Ubuntu 18.04. Replace the c[titleURL] value with the URL of your Buildbot instance. In the example, the URL value is replaced with example.com.

        /var/lib/buildbot/masters/ci/master.cfg
        1
        2
        3
        4
        5
        
        ...
        c['title'] = "My CI"
        c['titleURL'] = "https://example.com"
        ...
            
      4. Uncomment the c['buildbotURL'] line and replace the URL value with the your Buildbot instance’s URL:

        /var/lib/buildbot/masters/ci/master.cfg
        1
        2
        3
        4
        
        ...
        c['buildbotURL'] = "https://example.com/"
        ...
            

        These options assume that you will use a custom domain secured with Let’s Encrypt certificates from certbot as outlined in the Before You Begin section of this guide.

      5. Uncomment the web interface configuration lines and keep the default options:

        /var/lib/buildbot/masters/ci/master.cfg
        1
        2
        3
        4
        5
        
        ...
        c['www'] = dict(port=8010,
                        plugins=dict(waterfall_view={}, console_view={}, grid_view={}))
        ...
            
      6. By default, Buildbot does not require people to authenticate in order to access control features in the web UI. To secure Buildbot, you will need to configure an authentication plugin.

        Configure users for the Buildbot master web interface. Add the following lines below the web interface configuration lines and replace the myusername and password values with the ones you would like to use.

        /var/lib/buildbot/masters/ci/master.cfg
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        16
        
        ...
        c['www'] = dict(port=8010,
                        plugins=dict(waterfall_view={}, console_view={}, grid_view={}))
        
        # user configurations
        c['www']['authz'] = util.Authz(
                allowRules = [
                    util.AnyEndpointMatcher(role="admins")
                ],
                roleMatchers = [
                    util.RolesFromUsername(roles=['admins'], usernames=['myusername'])
                ]
        )
        c['www']['auth'] = util.UserPasswordAuth([('myusername','password')])
        ...
            
      7. Buildbot supports building repositories based on GitHub activity. This is done with a GitHub webhook. Generate a random string to serve as a webhook secret token to validate payloads.

        openssl rand -hex 16
        <a random string>
        
      8. Configure Buildbot to recognize GitHub webhooks as a change source. Add the following snippet to the end of the master.cfg file and replace webhook secret with the random string generated in the previous step.

        /var/lib/buildbot/masters/ci/master.cfg
        1
        2
        3
        4
        5
        6
        
        c['www']['change_hook_dialects'] = {
            'github': {
                'secret': 'webhook_secret',
            }
        }
            
      9. Finally, start the Buildbot master. This command will start the Buildbot process and persist it across reboots.

        sudo systemctl enable --now buildmaster@ci.service
        

      Set up the Buildbot Master Web Interface

      Buildbot is now running and listening on HTTP without encryption. To secure the connection, install NGINX to terminate SSL and reverse proxy traffic to the Buildbot master process.

      These steps install NGINX Mainline on Ubuntu from NGINX Inc’s official repository. For other distributions, see the NGINX admin guide. For information on configuring NGINX for production environments, see our Getting Started with NGINX series.

      1. Open /etc/apt/sources.list in a text editor and add the following line to the bottom. Replace CODENAME in this example with the codename of your Ubuntu release. For example, for Ubuntu 18.04, named Bionic Beaver, insert bionic in place of CODENAME below:

        /etc/apt/sources.list
        1
        
        deb http://nginx.org/packages/mainline/ubuntu/ CODENAME nginx
      2. Import the repository’s package signing key and add it to apt:

        sudo wget http://nginx.org/keys/nginx_signing.key
        sudo apt-key add nginx_signing.key
        
      3. Install NGINX:

        sudo apt update
        sudo apt install nginx
        
      4. Ensure NGINX is running and and enabled to start automatically on reboot:

        sudo systemctl start nginx
        sudo systemctl enable nginx
        

      Now that NGINX is installed, configure NGINX to talk to the local Buildbot port. NGINX will listen for SSL traffic using the Let’s Encrypt certificate for your domain.

      1. Create your site’s NGINX configuration file. Ensure that you replace the configuration file’s name example.com.conf with your domain name. Replace all instances of example.com with your Buildbot instance’s URL.

        /etc/nginx/conf.d/example.com.conf
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        16
        17
        18
        19
        20
        21
        22
        23
        24
        25
        26
        27
        28
        29
        30
        31
        32
        33
        34
        35
        36
        37
        38
        39
        40
        41
        42
        43
        44
        45
        46
        47
        48
        
        server {
          # Enable SSL and http2
          listen 443 ssl http2 default_server;
        
          server_name example.com;
        
          root html;
          index index.html index.htm;
        
          ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
          ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
        
          # put a one day session timeout for websockets to stay longer
          ssl_session_cache      shared:SSL:10m;
          ssl_session_timeout  1440m;
        
          ssl_protocols TLSv1.2 TLSv1.3;
          ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
          ssl_prefer_server_ciphers   on;
        
          # force https
          add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";
          spdy_headers_comp 5;
        
          proxy_set_header HOST $host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto  $scheme;
          proxy_set_header X-Forwarded-Server  $host;
          proxy_set_header X-Forwarded-Host  $host;
        
          location / {
              proxy_pass http://127.0.0.1:8010/;
          }
          location /sse/ {
              # proxy buffering will prevent sse to work
              proxy_buffering off;
              proxy_pass http://127.0.0.1:8010/sse/;
          }
          location /ws {
              proxy_http_version 1.1;
              proxy_set_header Upgrade $http_upgrade;
              proxy_set_header Connection "upgrade";
              proxy_pass http://127.0.0.1:8010/ws;
              # raise the proxy timeout for the websocket
              proxy_read_timeout 6000s;
          }
        }
      2. Disable NGINX’s default configuration file:

        mv /etc/nginx/conf.d/default.conf /etc/nginx/conf.d/default.conf.disabled
        
      3. Restart NGINX to apply the Buildbot reverse proxy configuration:

        sudo systemctl restart nginx
        
      4. Navigate to your Buildbot instance’s URL over HTTPS. You will see the Buildbot homepage:

        Buildbot Landing Page

        Your continuous integration test server is now up and running.

      5. Ensure that you can log into your Buildbot instance with the admin credentials you created in the Configure Buildbot Master section. Click on the top right hand dropdown menu entitled Anonymous and then, click on Login. A Sign In modal will appear. Enter your credentials to log in to Buildbot as the admin user.

      Install the Buildbot Worker

      In order for Buildbot to execute test builds, the Buildbot master will require a worker. The following steps will setup a worker on the same host as the master.

      1. Install the buildbot-slave Ubuntu package:

        sudo apt-get install -y buildbot-slave
        
      2. Navigate to the directory which will store the Buildbot worker configurations:

        cd /var/lib/buildbot/workers
        
      3. Create the configuration directory for the Buildbot worker. Replace example-worker and my-worker-password with the values used for the c[worker] configuration in the master.cfg file.

        sudo -u buildbot -- buildbot-worker create-worker default localhost example-worker my-worker-password
        
      4. The Buildbot worker is ready to connect to the Buildbot master. Enable the worker process.

        sudo systemctl enable --now buildbot-worker@default.service
        

        Confirm that the worker has connected by going to your Buildbot site and navigating to Builds -> Workers in the sidebar menu:

        Buildbot Workers Page

      Configuring Builds

      Now that Buildbot is installed, you can configure it to run builds. In this tutorial, we will use a forked GitHub repository for the Linode Guides and Tutorials repository to illustrate how to use Buildbot as a system to run tests against a repository.

      Configuring GitHub

      Before creating the build configuration, fork the linode/docs repository into your GitHub account. This is the repository that will be used to run tests against. The repository will also require webhooks to be configured to send push or PR events to Buildbot.

      Note

      The actions you take to fork, add webhook, and push changes to your fork of linode/docs will not affect the parent (or upstream), so you can safely experiment with it. Any changes you make to branches of your fork will remain separate until you submit a pull request to the original linode/docs repository.

      Forking and Configuring the Repository

      1. Log in to your GitHub account and navigate to https://github.com/linode/docs. Click the Fork button:

        GitHub Fork Button

      2. Choose the account to fork the repository into (typically just your username). GitHub will bring you to the page for your own fork of the linode/docs repository.

        Select Settings to browse your fork’s settings:

        GitHub Fork Settings

        Then, select Webhooks from the sidebar:

        GitHub Webhook Settings

      3. Click on the Add webhook button. There are several fields to populate:

        • Under Payload URL enter the domain name for your Buildbot server with the change hook URL path appended to it: https://example.com/change_hook/github.
        • Leave the default value for Content type: application/x-www-form-urlencoded.
        • Under the Secret field, enter the secret value for the c['www']['change_hook_dialects'] option you configure in the master.cfg file.
        • Leave Enable SSL Verification selected.
        • For the Which events would you like to trigger this webhook?, select Let me select individual events and ensure that only the following boxes are checked:
        • Leave Active selected to indicate that GitHub should be configured to send webhooks to Buildbot.
      4. Click on the Add webhook button to save your settings.

        GitHub will return your browser to the list of webhooks for your repository. After configuring a new webhook, GitHub will send a test webhook to the configured payload URL. To indicate whether GitHub was able to send a webhook without errors, it adds a checkmark to the webhook item:

        GitHub Webhook Success

        Github will now send any new pushes made to your fork to your instance of Buildbot for testing.

      Build Prerequisites

      This guide runs builds as a simple process on the Buildbot worker, however, it is possible to execute builds within a Docker container, if desired. Consult the official Buildbot documentation for more information on configuring a Docker set up.

      Most software projects will define several prerequisites and tests for a project build. The Linode Guides and Tutorials repository defines several different tests to run for each build. This example will use one test defined in a python script named blueberry.py. This test checks for broken links, missing images, and more. This test’s dependencies can be installed via pip in a virtualenv.

      On your Linode, install the packages necessary to permit the worker to use a Python virtualenv to create a sandbox during the build.

      sudo apt-get install -y build-essential python3-dev python3-venv
      

      Writing Builds

      The /var/lib/buildbot/masters/ci/master.cfg file contains options to configure builds. The specific sections in the file that include these configurations are the following:

      • WORKERS, define the worker executors the master will connect to in order to run builds.
      • SCHEDULERS, specify how to react to incoming changes.
      • BUILDERS, outline the steps and build tests to run.

      Because the worker has already been configured and connected to the Buildbot master, the only settings necessary to define a custom build are the SCHEDULERS and BUILDERS.

      1. Add the following lines to the end of the /var/lib/buildbot/masters/ci/master.cfg file to define the custom build. Ensure you replace my-username and my-git-repo-name with the values for your own GitHub fork of the linode/docs repository and example-worker with the name of your Buildbot instance’s worker:

        /var/lib/buildbot/masters/ci/master.cfg
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        16
        17
        18
        19
        20
        21
        22
        23
        24
        
        docs_blueberry_test = util.BuildFactory()
        # Clone the repository
        docs_blueberry_test.addStep(
            steps.Git(
                repourl='git://github.com/my-username/my-git-repo-name.git',
                mode='incremental'))
        # Create virtualenv
        docs_blueberry_test.addStep(
            steps.ShellCommand(
                command=["python3", "-m", "venv", ".venv"]))
        # Install test dependencies
        docs_blueberry_test.addStep(
            steps.ShellCommand(
                command=["./.venv/bin/pip", "install", "-r", "ci/requirements.txt"]))
        # Run tests
        docs_blueberry_test.addStep(
            steps.ShellCommand(
                command=["./.venv/bin/python3", "ci/blueberry.py"]))
        # Add the BuildFactory configuration to the master
        c['builders'].append(
            util.BuilderConfig(name="linode-docs",
              workernames=["example-worker"],
              factory=docs_blueberry_test))
            

        The configuration code does the following:

        • A new Build Factory is instantiated. Build Factories define how builds are run.
        • Then, instructions are added to the Build Factory. The Build Factory clones the GitHub fork of the linode/docs repository.
        • Next, a Python virtualenv is setup. This ensures that the dependencies and libraries used for testing are kept separate, in a dedicated sandbox, from the Python libraries on the worker machine.
        • The necessary Python packages used in testing are then installed into the build’s virtualenv.
        • Finally, the blueberry.py testing script is run using the python3 executable from the virtualenv sandbox.
        • The defined Build Factory is then added to the configuration for the master.
      2. Define a simple scheduler to build any branch that is pushed to the GitHub repository. Add the following lines to the end of the master.cfg file:

        ~/buildbox-master/master/master.cfg
        1
        2
        3
        4
        5
        
            ...
        c['schedulers'].append(schedulers.AnyBranchScheduler(
            name="build-docs",
            builderNames=["linode-docs"]))
            

        This code instructs the Buildbot master to create a scheduler that builds any branch for the linode-docs builder. This scheduler will be invoked by the change hook defined for GitHub, which is triggered by the GitHub webhook configured in the GitHub interface.

      3. Restart the Buildbot master now that the custom scheduler and builder have been defined:

        sudo systemctl restart buildmaster@ci.service
        

      Running Builds

      Navigate to your Buildbot site to view the Builder and Scheduler created in the previous section. In the sidebar click on Build -> Builders. You will see linode-docs listed under the Builder Name heading:

      Buildbot Custom Builder

      A new build can be started for the linode-docs builder. Recall that the GitHub webhook configuration for your fork of linode/docs is set to call Buildbot upon any push or pull request event. To demonstrate how this works:

      1. Clone your fork of the linode/docs repository on your local machine (do not run the following commands on your Buildbot server) and navigate into the cloned repository. Replace username and repository with your own fork’s values:

        git clone https://github.com/username/repository.git
        cd repository
        
      2. Like many git repositories, the linode-docs repository changes often. To ensure that the remaining instructions work as expected, start at a specific revision in the code that is in a known state. Check out revision 76cd31a5271b41ff5a80dee2137dcb5e76296b93:

        git checkout 76cd31a5271b41ff5a80dee2137dcb5e76296b93
        
      3. Create a branch starting at this revision, which is where you will create dummy commits to test your Buildbot master:

        git checkout -b linode-tutorial-demo
        
      4. Create an empty commit so that you have something to push to your fork:

        git commit --allow-empty -m 'Buildbot test'
        
      5. Push your branch to your forked remote GitHub repository:

        git push --set-upstream origin linode-tutorial-demo
        
      6. Navigate to your Buildbot site and go to your running builds. The Home button on the sidebar displays currently executing builds.

        Buildbot running Builds

      7. Click on the running build to view more details. The build will display each step along with logging output:

        Buildbot Build Page

        Each step of the build process can be followed as the build progresses. While the build is running, click on a step to view standard output logs. A successful build will complete each step with an exit code of 0.

        Your Buildbot host will now actively build pushes to any branch or any pull requests to your repository.

      Features to Explore

      Now that you have a simple build configuration for your Buildbot instance, you can continue to add features to your CI server. Some useful functions that Buildbot supports include:

      • Reporters, which can notify you about build failures over IRC, GitHub comments, or email.
      • Workers that execute builds in Docker containers or in temporary cloud instances instead of static hosts.
      • Web server features, including the ability to generate badges for your repository indicating the current build status of the project.

      More Information

      You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

      Join our Community

      Find answers, ask questions, and help others.

      This guide is published under a CC BY-ND 4.0 license.



      Source link

      How To Install Software on Kubernetes Clusters with the Helm Package Manager


      Introduction

      Helm is a package manager for Kubernetes that allows developers and operators to more easily configure and deploy applications on Kubernetes clusters.

      In this tutorial we will set up Helm and use it to install, reconfigure, rollback, then delete an instance of the Kubernetes Dashboard application. The dashboard is an official web-based Kubernetes GUI.

      For a conceptual overview of Helm and its packaging ecosystem, please read our article An Introduction to Helm.

      Prerequisites

      For this tutorial you will need:

      • A Kubernetes 1.8+ cluster with role-based access control (RBAC) enabled.
      • The kubectl command-line tool installed on your local machine, configured to connect to your cluster. You can read more about installing kubectl in the official documentation.

        You can test your connectivity with the following command:

        If you see no errors, you're connected to the cluster. If you access multiple clusters with kubectl, be sure to verify that you've selected the correct cluster context:

        • kubectl config get-contexts

        Output

        CURRENT NAME CLUSTER AUTHINFO NAMESPACE * do-nyc1-k8s-example do-nyc1-k8s-example do-nyc1-k8s-example-admin docker-for-desktop docker-for-desktop-cluster docker-for-desktop

        In this example the asterisk (*) indicates that we are connected to the do-nyc1-k8s-example cluster. To switch clusters run:

        • kubectl config use-context context-name

      When you are connected to the correct cluster, continue to Step 1 to begin installing Helm.

      Step 1 — Installing Helm

      First we'll install the helm command-line utility on our local machine. Helm provides a script that handles the installation process on MacOS, Windows, or Linux.

      Change to a writable directory and download the script from Helm's GitHub repository:

      • cd /tmp
      • curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > install-helm.sh

      Make the script executable with chmod:

      • chmod u+x install-helm.sh

      At this point you can use your favorite text editor to open the script and inspect it to make sure it’s safe. When you are satisfied, run it:

      You may be prompted for your password. Provide it and press ENTER.

      Output

      helm installed into /usr/local/bin/helm Run 'helm init' to configure helm.

      Next we will finish the installation by installing some Helm components on our cluster.

      Step 2 — Installing Tiller

      Tiller is a companion to the helm command that runs on your cluster, receiving commands from helm and communicating directly with the Kubernetes API to do the actual work of creating and deleting resources. To give Tiller the permissions it needs to run on the cluster, we are going to make a Kubernetes serviceaccount resource.

      Note: We will bind this serviceaccount to the cluster-admin cluster role. This will give the tiller service superuser access to the cluster and allow it to install all resource types in all namespaces. This is fine for exploring Helm, but you may want a more locked-down configuration for a production Kubernetes cluster.

      Please refer to the official Helm RBAC documentation for more information on setting up different RBAC scenarios for Tiller.

      Create the tiller serviceaccount:

      • kubectl -n kube-system create serviceaccount tiller

      Next, bind the tiller serviceaccount to the cluster-admin role:

      • kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller

      Now we can run helm init, which installs Tiller on our cluster, along with some local housekeeping tasks such as downloading the stable repo details:

      • helm init --service-account tiller

      Output

      . . . Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster. Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy. For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation Happy Helming!

      To verify that Tiller is running, list the pods in thekube-system namespace:

      • kubectl get pods --namespace kube-system

      Output

      NAME READY STATUS RESTARTS AGE . . . kube-dns-64f766c69c-rm9tz 3/3 Running 0 22m kube-proxy-worker-5884 1/1 Running 1 21m kube-proxy-worker-5885 1/1 Running 1 21m kubernetes-dashboard-7dd4fc69c8-c4gwk 1/1 Running 0 22m tiller-deploy-5c688d5f9b-lccsk 1/1 Running 0 40s

      The Tiller pod name begins with the prefix tiller-deploy-.

      Now that we've installed both Helm components, we're ready to use helm to install our first application.

      Step 3 — Installing a Helm Chart

      Helm software packages are called charts. Helm comes preconfigured with a curated chart repository called stable. You can browse the available charts in their GitHub repo. We are going to install the Kubernetes Dashboard as an example.

      Use helm to install the kubernetes-dashboard package from the stable repo:

      • helm install stable/kubernetes-dashboard --name dashboard-demo

      Output

      NAME: dashboard-demo LAST DEPLOYED: Wed Aug 8 20:11:07 2018 NAMESPACE: default STATUS: DEPLOYED . . .

      Notice the NAME line, highlighted in the above example output. In this case we specified the name dashboard-demo. This is the name of our release. A Helm release is a single deployment of one chart with a specific configuration. You can deploy multiple releases of the same chart with, each with its own configuration.

      If you don't specify your own release name using --name, Helm will create a random name for you.

      We can ask Helm for a list of releases on this cluster:

      Output

      NAME REVISION UPDATED STATUS CHART NAMESPACE dashboard-demo 1 Wed Aug 8 20:11:11 2018 DEPLOYED kubernetes-dashboard-0.7.1 default

      We can now use kubectl to verify that a new service has been deployed on the cluster:

      Output

      NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE dashboard-demo-kubernetes-dashboard ClusterIP 10.32.104.73 <none> 443/TCP 51s kubernetes ClusterIP 10.32.0.1 <none> 443/TCP 34m

      Notice that by default the service name corresponding to our release is a combination of the Helm release name and the chart name.

      Now that we've deployed the application, let's use Helm to change its configuration and update the deployment.

      Step 4 — Updating a Release

      The helm upgrade command can be used to upgrade a release with a new or updated chart, or update the it’s configuration options.

      We're going to make a simple change to our dashboard-demo release to demonstrate the update and rollback process: we'll update the name of the dashboard service to just dashboard, instead of dashboard-demo-kubernetes-dashboard.

      The kubernetes-dashboard chart provides a fullnameOverride configuration option to control the service name. Let's run helm upgrade with this option set:

      • helm upgrade dashboard-demo stable/kubernetes-dashboard --set fullnameOverride="dashboard"

      You'll see output similar to the initial helm install step.

      Check if your Kubernetes services reflect the updated values:

      Output

      NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.32.0.1 <none> 443/TCP 36m dashboard ClusterIP 10.32.198.148 <none> 443/TCP 40s

      Our service name has been updated to the new value.

      Note: At this point you may want to actually load the Kubernetes Dashboard in your browser and check it out. To do so, first run the following command:

      This creates a proxy that lets you access remote cluster resources from your local computer. Based on the previous instructions your dashboard service is named kubernetes-dashboard and it's running in the default namespace. You may now access the dashboard at the following url:

      http://localhost:8001/api/v1/namespaces/default/services/https:dashboard:/proxy/
      

      If necessary, substitute your own service name and namespace for the highlighted portions. Instructions for actually using the dashboard are out of scope for this tutorial, but you can read the official Kubernetes Dashboard docs for more information.

      Next we'll look at Helm's ability to roll back releases.

      Step 5 — Rolling Back a Release

      When we updated our dashboard-demo release in the previous step, we created a second revision of the release. Helm retains all the details of previous releases in case you need to roll back to a prior configuration or chart.

      Use helm list to inspect the release again:

      Output

      NAME REVISION UPDATED STATUS CHART NAMESPACE dashboard-demo 2 Wed Aug 8 20:13:15 2018 DEPLOYED kubernetes-dashboard-0.7.1 default

      The REVISION column tells us that this is now the second revision.

      Use helm rollback to roll back to the first revision:

      • helm rollback dashboard-demo 1

      You should see the following output, indicating that the rollback succeeded:

      Output

      Rollback was a success! Happy Helming!

      At this point, if you run kubectl get services again, you will notice that the service name has changed back to its previous value. Helm has re-deployed the application with revision 1's configuration.

      Next we'll look into deleting releases with Helm.

      Step 6 — Deleting a Release

      Helm releases can be deleted with the helm delete command:

      • helm delete dashboard-demo

      Output

      release "dashboard-demo" deleted

      Though the release has been deleted and the dashboard application is no longer running, Helm saves all the revision information in case you want to re-deploy the release. If you tried to helm install a new dashboard-demo release right now, you'd get an error:

      Error: a release named dashboard-demo already exists.
      

      If you use the --deleted flag to list your deleted releases, you'll see that the release is still around:

      Output

      NAME REVISION UPDATED STATUS CHART NAMESPACE dashboard-demo 3 Wed Aug 8 20:15:21 2018 DELETED kubernetes-dashboard-0.7.1 default

      To really delete the release and purge all old revisions, use the --purge flag with the helm delete command:

      • helm delete dashboard-demo --purge

      Now the release has been truly deleted, and you can reuse the release name.

      Conclusion

      In this tutorial we installed the helm command-line tool and its tiller companion service. We also explored installing, upgrading, rolling back, and deleting Helm charts and releases.

      For more information about Helm and Helm charts, please see the official Helm documentation.



      Source link

      Use Buildbot for Software Testing on Ubuntu 18.04


      Updated by Linode Written by Tyler Langlois

      Use promo code DOCS10 for $10 credit on a new account.

      Buildbot is an open source system for testing software projects. In this guide, you will set up a Linode as a Buildbot server to use as a continuous integration platform to test code. Similarly to hosted solutions like Travis CI, Buildbot is an automated testing platform that can watch for code changes, test a project’s code, and send notifications regarding build failures.

      Before you Begin

      1. Familiarize yourself with Linode’s Getting Started guide and complete the steps for deploying and setting up a Linode running Ubuntu 18.04, including setting the hostname and timezone.

      2. This guide uses sudo wherever possible. Complete the sections of our Securing Your Server guide to create a standard user account, harden SSH access and remove unnecessary network services.

      3. Ensure your system is up to date:

        sudo apt update && sudo apt upgrade
        
      4. Complete the Add DNS Records steps to register a domain name that will point to your Linode instance hosting Buildbot.

        Note

        Replace each instance of example.com in this guide with your Buildbot site’s domain name.

      5. Your Buildbot site will serve its content over HTTPS, so you will need to obtain an SSL/TLS certificate. Use Certbot to request and download a free certificate from Let’s Encrypt.

        sudo apt install software-properties-common
        sudo add-apt-repository ppa:certbot/certbot
        sudo apt update
        sudo apt install certbot
        sudo certbot certonly --standalone -d example.com
        

        These commands will download a certificate to /etc/letsencrypt/live/example.com/ on your Linode.

        Note

      Install Buildbot

      Install the Buildbot Master

      Since Buildbot is provided as an Ubuntu package, install the software from the official Ubuntu repositories.

      1. Install the buildbot package along with pip3, which will be used to install additional python packages:

        sudo apt-get install -y buildbot python3-pip
        
      2. Install the required Buildbot Python packages:

        sudo pip3 install buildbot-www buildbot-waterfall-view buildbot-console-view buildbot-grid-view
        
      3. The buildbot package sets up several file paths and services to run persistently on your host. In order to create a new configuration for a Buildbot master, enter the directory for Buildbot master configurations and create a new master called ci (for “continuous integration”).

        cd /var/lib/buildbot/masters
        sudo -u buildbot -- buildbot create-master ci
        

        The generated master configuration file’s location is /var/lib/buildbot/masters/ci/master.cfg.sample.

      4. Make a copy of the default configuration to the path that Buildbot expects for its configuration file:

        sudo cp ci/master.cfg.sample ci/master.cfg
        
      5. Change the permissions for this configuration file so that the buildbot user has rights for the configuration file:

        sudo chown buildbot:buildbot ci/master.cfg
        

      Configure the Buildbot Master

      In order to secure and customize Buildbot, you will change a few settings in the master configuration file before using the application. The master configuration file’s location is /var/lib/buildbot/masters/ci/master.cfg.

      Buildbot has a number of concepts that are represented in the master build configuration file. Open this file in your preferred text editor and browse the Buildbot configuration. The Buildbot configuration is written in Python instead of a markup language like Yaml.

      1. Generate a random string to serve as the password that workers will use to authenticate against the Buildbot master. This is accomplished by using openssl to create a random sequence of characters.

        openssl rand -hex 16
        <a random string>
        
      2. Update the following line in the master.cfg file and replace pass with the randomly-generated password:

        /var/lib/buildbot/masters/ci/master.cfg
        1
        2
        3
        4
        5
        6
        7
        
        ...
        # The 'workers' list defines the set of recognized workers. Each element is
        # a Worker object, specifying a unique worker name and password.  The same
        # worker name and password must be configured on the worker.
        c['workers'] = [worker.Worker("example-worker", "pass")]
        ...
            
      3. Uncomment the cUse Buildbot for Software Testing on Ubuntu 18.04 and the c[titleURL] lines. If desired, change the name of the Buildbot installation by updating the value of cUse Buildbot for Software Testing on Ubuntu 18.04. Replace the c[titleURL] value with the URL of your Buildbot instance. In the example, the URL value is replaced with example.com.

        /var/lib/buildbot/masters/ci/master.cfg
        1
        2
        3
        4
        5
        
        ...
        c['title'] = "My CI"
        c['titleURL'] = "https://example.com"
        ...
            
      4. Uncomment the c['buildbotURL'] line and replace the URL value with the your Buildbot instance’s URL:

        /var/lib/buildbot/masters/ci/master.cfg
        1
        2
        3
        4
        
        ...
        c['buildbotURL'] = "https://example.com/"
        ...
            

        These options assume that you will use a custom domain secured with Let’s Encrypt certificates from certbot as outlined in the Before You Begin section of this guide.

      5. Uncomment the web interface configuration lines and keep the default options:

        /var/lib/buildbot/masters/ci/master.cfg
        1
        2
        3
        4
        5
        
        ...
        c['www'] = dict(port=8010,
                        plugins=dict(waterfall_view={}, console_view={}, grid_view={}))
        ...
            
      6. By default, Buildbot does not require people to authenticate in order to access control features in the web UI. To secure Buildbot, you will need to configure an authentication plugin.

        Configure users for the Buildbot master web interface. Add the following lines below the web interface configuration lines and replace the myusername and password values with the ones you would like to use.

        /var/lib/buildbot/masters/ci/master.cfg
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        16
        
        ...
        c['www'] = dict(port=8010,
                        plugins=dict(waterfall_view={}, console_view={}, grid_view={}))
        
        # user configurations
        c['www']['authz'] = util.Authz(
                allowRules = [
                    util.AnyEndpointMatcher(role="admins")
                ],
                roleMatchers = [
                    util.RolesFromUsername(roles=['admins'], usernames=['myusername'])
                ]
        )
        c['www']['auth'] = util.UserPasswordAuth([('myusername','password')])
        ...
            
      7. Buildbot supports building repositories based on GitHub activity. This is done with a GitHub webhook. Generate a random string to serve as a webhook secret token to validate payloads.

        openssl rand -hex 16
        <a random string>
        
      8. Configure Buildbot to recognize GitHub webhooks as a change source. Add the following snippet to the end of the master.cfg file and replace webhook secret with the random string generated in the previous step.

        /var/lib/buildbot/masters/ci/master.cfg
        1
        2
        3
        4
        5
        6
        
        c['www']['change_hook_dialects'] = {
            'github': {
                'secret': 'webhook_secret',
            }
        }
            
      9. Finally, start the Buildbot master. This command will start the Buildbot process and persist it across reboots.

        sudo systemctl enable --now buildmaster@ci.service
        

      Set up the Buildbot Master Web Interface

      Buildbot is now running and listening on HTTP without encryption. To secure the connection, install NGINX to terminate SSL and reverse proxy traffic to the Buildbot master process.

      These steps install NGINX Mainline on Ubuntu from NGINX Inc’s official repository. For other distributions, see the NGINX admin guide. For information on configuring NGINX for production environments, see our Getting Started with NGINX series.

      1. Open /etc/apt/sources.list in a text editor and add the following line to the bottom. Replace CODENAME in this example with the codename of your Ubuntu release. For example, for Ubuntu 18.04, named Bionic Beaver, insert bionic in place of CODENAME below:

        /etc/apt/sources.list
        1
        
        deb http://nginx.org/packages/mainline/ubuntu/ CODENAME nginx
      2. Import the repository’s package signing key and add it to apt:

        sudo wget http://nginx.org/keys/nginx_signing.key
        sudo apt-key add nginx_signing.key
        
      3. Install NGINX:

        sudo apt update
        sudo apt install nginx
        
      4. Ensure NGINX is running and and enabled to start automatically on reboot:

        sudo systemctl start nginx
        sudo systemctl enable nginx
        

      Now that NGINX is installed, configure NGINX to talk to the local Buildbot port. NGINX will listen for SSL traffic using the Let’s Encrypt certificate for your domain.

      1. Create your site’s NGINX configuration file. Ensure that you replace the configuration file’s name example.com.conf with your domain name. Replace all instances of example.com with your Buildbot instance’s URL.

        /etc/nginx/conf.d/example.com.conf
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        16
        17
        18
        19
        20
        21
        22
        23
        24
        25
        26
        27
        28
        29
        30
        31
        32
        33
        34
        35
        36
        37
        38
        39
        40
        41
        42
        43
        44
        45
        46
        47
        48
        
        server {
          # Enable SSL and http2
          listen 443 ssl http2 default_server;
        
          server_name example.com;
        
          root html;
          index index.html index.htm;
        
          ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
          ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
        
          # put a one day session timeout for websockets to stay longer
          ssl_session_cache      shared:SSL:10m;
          ssl_session_timeout  1440m;
        
          ssl_protocols TLSv1.2 TLSv1.3;
          ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
          ssl_prefer_server_ciphers   on;
        
          # force https
          add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";
          spdy_headers_comp 5;
        
          proxy_set_header HOST $host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto  $scheme;
          proxy_set_header X-Forwarded-Server  $host;
          proxy_set_header X-Forwarded-Host  $host;
        
          location / {
              proxy_pass http://127.0.0.1:8010/;
          }
          location /sse/ {
              # proxy buffering will prevent sse to work
              proxy_buffering off;
              proxy_pass http://127.0.0.1:8010/sse/;
          }
          location /ws {
              proxy_http_version 1.1;
              proxy_set_header Upgrade $http_upgrade;
              proxy_set_header Connection "upgrade";
              proxy_pass http://127.0.0.1:8010/ws;
              # raise the proxy timeout for the websocket
              proxy_read_timeout 6000s;
          }
        }
      2. Disable NGINX’s default configuration file:

        mv /etc/nginx/conf.d/default.conf /etc/nginx/conf.d/default.conf.disabled
        
      3. Restart NGINX to apply the Buildbot reverse proxy configuration:

        sudo systemctl restart nginx
        
      4. Navigate to your Buildbot instance’s URL over HTTPS. You will see the Buildbot homepage:

        Buildbot Landing Page

        Your continuous integration test server is now up and running.

      5. Ensure that you can log into your Buildbot instance with the admin credentials you created in the Configure Buildbot Master section. Click on the top right hand dropdown menu entitled Anonymous and then, click on Login. A Sign In modal will appear. Enter your credentials to log in to Buildbot as the admin user.

      Install the Buildbot Worker

      In order for Buildbot to execute test builds, the Buildbot master will require a worker. The following steps will setup a worker on the same host as the master.

      1. Install the buildbot-slave Ubuntu package:

        sudo apt-get install -y buildbot-slave
        
      2. Navigate to the directory which will store the Buildbot worker configurations:

        cd /var/lib/buildbot/workers
        
      3. Create the configuration directory for the Buildbot worker. Replace example-worker and my-worker-password with the values used for the c[worker] configuration in the master.cfg file.

        sudo -u buildbot -- buildbot-worker create-worker default localhost example-worker my-worker-password
        
      4. The Buildbot worker is ready to connect to the Buildbot master. Enable the worker process.

        sudo systemctl enable --now buildbot-worker@default.service
        

        Confirm that the worker has connected by going to your Buildbot site and navigating to Builds -> Workers in the sidebar menu:

        Buildbot Workers Page

      Configuring Builds

      Now that Buildbot is installed, you can configure it to run builds. In this tutorial, we will use a forked GitHub repository for the Linode Guides and Tutorials repository to illustrate how to use Buildbot as a system to run tests against a repository.

      Configuring GitHub

      Before creating the build configuration, fork the linode/docs repository into your GitHub account. This is the repository that will be used to run tests against. The repository will also require webhooks to be configured to send push or PR events to Buildbot.

      Note

      The actions you take to fork, add webhook, and push changes to your fork of linode/docs will not affect the parent (or upstream), so you can safely experiment with it. Any changes you make to branches of your fork will remain separate until you submit a pull request to the original linode/docs repository.

      Forking and Configuring the Repository

      1. Log in to your GitHub account and navigate to https://github.com/linode/docs. Click the Fork button:

        GitHub Fork Button

      2. Choose the account to fork the repository into (typically just your username). GitHub will bring you to the page for your own fork of the linode/docs repository.

        Select Settings to browse your fork’s settings:

        GitHub Fork Settings

        Then, select Webhooks from the sidebar:

        GitHub Webhook Settings

      3. Click on the Add webhook button. There are several fields to populate:

        • Under Payload URL enter the domain name for your Buildbot server with the change hook URL path appended to it: https://example.com/change_hook/github.
        • Leave the default value for Content type: application/x-www-form-urlencoded.
        • Under the Secret field, enter the secret value for the c['www']['change_hook_dialects'] option you configure in the master.cfg file.
        • Leave Enable SSL Verification selected.
        • For the Which events would you like to trigger this webhook?, select Let me select individual events and ensure that only the following boxes are checked:
        • Leave Active selected to indicate that GitHub should be configured to send webhooks to Buildbot.
      4. Click on the Add webhook button to save your settings.

        GitHub will return your browser to the list of webhooks for your repository. After configuring a new webhook, GitHub will send a test webhook to the configured payload URL. To indicate whether GitHub was able to send a webhook without errors, it adds a checkmark to the webhook item:

        GitHub Webhook Success

        Github will now send any new pushes made to your fork to your instance of Buildbot for testing.

      Build Prerequisites

      This guide runs builds as a simple process on the Buildbot worker, however, it is possible to execute builds within a Docker container, if desired. Consult the official Buildbot documentation for more information on configuring a Docker set up.

      Most software projects will define several prerequisites and tests for a project build. The Linode Guides and Tutorials repository defines several different tests to run for each build. This example will use one test defined in a python script named blueberry.py. This test checks for broken links, missing images, and more. This test’s dependencies can be installed via pip in a virtualenv.

      On your Linode, install the packages necessary to permit the worker to use a Python virtualenv to create a sandbox during the build.

      sudo apt-get install -y build-essential python3-dev python3-venv
      

      Writing Builds

      The /var/lib/buildbot/masters/ci/master.cfg file contains options to configure builds. The specific sections in the file that include these configurations are the following:

      • WORKERS, define the worker executors the master will connect to in order to run builds.
      • SCHEDULERS, specify how to react to incoming changes.
      • BUILDERS, outline the steps and build tests to run.

      Because the worker has already been configured and connected to the Buildbot master, the only settings necessary to define a custom build are the SCHEDULERS and BUILDERS.

      1. Add the following lines to the end of the /var/lib/buildbot/masters/ci/master.cfg file to define the custom build. Ensure you replace my-username and my-git-repo-name with the values for your own GitHub fork of the linode/docs repository and example-worker with the name of your Buildbot instance’s worker:

        /var/lib/buildbot/masters/ci/master.cfg
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        16
        17
        18
        19
        20
        21
        22
        23
        24
        
        docs_blueberry_test = util.BuildFactory()
        # Clone the repository
        docs_blueberry_test.addStep(
            steps.Git(
                repourl='git://github.com/my-username/my-git-repo-name.git',
                mode='incremental'))
        # Create virtualenv
        docs_blueberry_test.addStep(
            steps.ShellCommand(
                command=["python3", "-m", "venv", ".venv"]))
        # Install test dependencies
        docs_blueberry_test.addStep(
            steps.ShellCommand(
                command=["./.venv/bin/pip", "install", "-r", "ci/requirements.txt"]))
        # Run tests
        docs_blueberry_test.addStep(
            steps.ShellCommand(
                command=["./.venv/bin/python3", "ci/blueberry.py"]))
        # Add the BuildFactory configuration to the master
        c['builders'].append(
            util.BuilderConfig(name="linode-docs",
              workernames=["example-worker"],
              factory=docs_blueberry_test))
            

        The configuration code does the following:

        • A new Build Factory is instantiated. Build Factories define how builds are run.
        • Then, instructions are added to the Build Factory. The Build Factory clones the GitHub fork of the linode/docs repository.
        • Next, a Python virtualenv is setup. This ensures that the dependencies and libraries used for testing are kept separate, in a dedicated sandbox, from the Python libraries on the worker machine.
        • The necessary Python packages used in testing are then installed into the build’s virtualenv.
        • Finally, the blueberry.py testing script is run using the python3 executable from the virtualenv sandbox.
        • The defined Build Factory is then added to the configuration for the master.
      2. Define a simple scheduler to build any branch that is pushed to the GitHub repository. Add the following lines to the end of the master.cfg file:

        ~/buildbox-master/master/master.cfg
        1
        2
        3
        4
        5
        
            ...
        c['schedulers'].append(schedulers.AnyBranchScheduler(
            name="build-docs",
            builderNames=["linode-docs"]))
            

        This code instructs the Buildbot master to create a scheduler that builds any branch for the linode-docs builder. This scheduler will be invoked by the change hook defined for GitHub, which is triggered by the GitHub webhook configured in the GitHub interface.

      3. Restart the Buildbot master now that the custom scheduler and builder have been defined:

        sudo systemctl restart buildmaster@ci.service
        

      Running Builds

      Navigate to your Buildbot site to view the Builder and Scheduler created in the previous section. In the sidebar click on Build -> Builders. You will see linode-docs listed under the Builder Name heading:

      Buildbot Custom Builder

      A new build can be started for the linode-docs builder. Recall that the GitHub webhook configuration for your fork of linode/docs is set to call Buildbot upon any push or pull request event. To demonstrate how this works:

      1. Clone your fork of the linode/docs repository on your local machine (do not run the following commands on your Buildbot server) and navigate into the cloned repository. Replace username and repository with your own fork’s values:

        git clone https://github.com/username/repository.git
        cd repository
        
      2. Like many git repositories, the linode-docs repository changes often. To ensure that the remaining instructions work as expected, start at a specific revision in the code that is in a known state. Check out revision 76cd31a5271b41ff5a80dee2137dcb5e76296b93:

        git checkout 76cd31a5271b41ff5a80dee2137dcb5e76296b93
        
      3. Create a branch starting at this revision, which is where you will create dummy commits to test your Buildbot master:

        git checkout -b linode-tutorial-demo
        
      4. Create an empty commit so that you have something to push to your fork:

        git commit --allow-empty -m 'Buildbot test'
        
      5. Push your branch to your forked remote GitHub repository:

        git push --set-upstream origin linode-tutorial-demo
        
      6. Navigate to your Buildbot site and go to your running builds. The Home button on the sidebar displays currently executing builds.

        Buildbot running Builds

      7. Click on the running build to view more details. The build will display each step along with logging output:

        Buildbot Build Page

        Each step of the build process can be followed as the build progresses. While the build is running, click on a step to view standard output logs. A successful build will complete each step with an exit code of 0.

        Your Buildbot host will now actively build pushes to any branch or any pull requests to your repository.

      Features to Explore

      Now that you have a simple build configuration for your Buildbot instance, you can continue to add features to your CI server. Some useful functions that Buildbot supports include:

      • Reporters, which can notify you about build failures over IRC, GitHub comments, or email.
      • Workers that execute builds in Docker containers or in temporary cloud instances instead of static hosts.
      • Web server features, including the ability to generate badges for your repository indicating the current build status of the project.

      More Information

      You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

      Join our Community

      Find answers, ask questions, and help others.

      This guide is published under a CC BY-ND 4.0 license.



      Source link