One place for hosting & domains

      Buildbot

      How To Build and Deploy Packages for Your FreeBSD Servers Using Buildbot and Poudriere


      The author selected the Free and Open Source Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      The FreeBSD ports and packages collection, hereafter called ports tree, is FreeBSD’s build system for external software. It offers a Makefile-based, consistent way of building packages. The port refers to the build recipe, that is the Makefile and related files; while package is the output of building one port into a binary (compressed) archive of the package files and its meta information.

      Manually building and installing a subset or all of the over 30,000 ports is possible with make install. However, the builds would run on one of your servers—not a clean environment. For production use cases, manual builds would also mean that each host needs the same revision of the ports tree, and needs to compile all packages for itself. This means repeated, error-prone work by humans and the servers. It is preferable to retrieve and use identical, pre-built binary packages on each host and serve them from a central, secure package repository.

      To achieve this, Poudriere is the standard tool on FreeBSD to build, test, and audit packages as well as maintain the package repositories. Each build is run isolated in a fresh jail, running the desired version of FreeBSD, and starting with no packages installed. Only the base system, plus any explicitly specified dependencies, are available to the clean build. Poudriere takes care of rebuilding packages when necessary as well as updating the package repository after a build has finished. The poudriere command line tool is central to administering different ports trees, FreeBSD versions, port build options, and lastly, running the builds.

      In this tutorial you’ll configure Poudriere, build a set of desired packages, set up HTTP-based package hosting, and automate the build using Buildbot as a continuous integration platform. Finally, you will securely access the packages from a client machine.

      Note: To cover production-like use cases, the tutorial examples use the quarterly stable branches of the ports tree. Staying on one such branch protects you from breaking changes and provides security and build fixes where necessary—if you regularly update the tree from upstream (Subversion, or its GitHub mirror). You can choose to stay on one branch for an extended period of time, depending on the pace at which your system updates can be handled by developer/infrastructure teams. The ports collection supports FreeBSD releases until they become end-of-life (EOL)—see Supported FreeBSD releases—so that OS and package updates can be handled independently. Alternatively, you could consider a local version control repository cloned from the upstream tree. That way, you can manage patches and only merge upstream changes at the time you desire.

      Prerequisites

      Before you begin this guide, you will need:

      • A server running FreeBSD 11.2. If you’re new to working with FreeBSD, you may find it helpful to customize this server by following our guide on How to Get Started with FreeBSD.
        Note: FreeBSD 12.0 currently has an issue with nested jails, which first needs to be fixed before 12.x can be used for this tutorial.
      • 10 GB free disk space or more to have enough capacity to store packages and logs.
      • A basic Buildbot setup by completing the How To Set Up Buildbot on FreeBSD tutorial.
      • Another server running FreeBSD, the same version, which you are going to use as a client to fetch and install the packages that you’re going to automatically build and host in a HTTP/HTTPS-based package repository.

      Step 1 — Installing Poudriere for Use in Buildbot Worker

      After completing the prerequisite tutorial, you’ll have a working Buildbot master and worker jail plus Nginx setup. You will build upon this existing setup in the following steps. In this first step, you’re going to install the build tool Poudriere inside the worker jail, since that is where the Buildbot worker process will trigger builds later on.

      Connect to your server hosting Buildbot and open a root shell in the worker jail with the following command:

      • sudo jexec buildbot-worker0 csh

      Install Poudriere as a package:

      Then confirm installation by pressing y and then ENTER.

      Note: It is preferable to use the official FreeBSD package repository for installing Buildbot, Poudriere, and so on. If you build those tool packages yourself, you start off in a chicken-and-egg situation: wanting to install external software, but requiring Poudriere installed to get cleanly built packages. Since Poudriere is a very stable and backward-compatible tool, nothing speaks against updating it regularly and independently from your production packages.

      If you followed the prerequisite tutorial, this is already the case and you can continue without following this note.

      You’ve successfully installed the latest Poudriere tool and dependencies. In the next several steps, you will go through preparations to configure Poudriere.

      Step 2 — Creating a Package Signing Key (Optional)

      It’s recommended to set up digital signatures for built packages in order to provide more security. Skip this step if you want to secure your installation later, or in a different way. Otherwise, let’s go ahead and create a key pair used to sign packages (using the private key) and verify packages (using the public part).

      Packages, by default, are built as .txz files, which are strongly compressed tarballs of the package contents. The compressed files’ checksums, together with serving the files via HTTP/HTTPS (TCP checksums), already provide some protection against corrupted data. Package contents typically comprise files and directories plus meta information such as the package name, version, and miscellaneous options. Files may even include setuid-able programs (as seen in the sudo package—though sudo is not built into FreeBSD), and the installation-time scripts run as root user. Installing from unverified sources therefore poses a security risk.

      By serving the packages over HTTPS, you cannot detect whether someone tampered with the packages on-disk. Integrity and authenticity of your packages can be added by configuring Poudriere to sign the package repository with an RSA private key. Signed digests and the corresponding public key are thereby stored in the package repository’s digests.txz file. The required key pair (RSA private and public key) can be kept unchanged for a long time unless the private key was lost or compromised.

      In this step you’ll create the key pair where the builds run (worker jail) and download the public part for later use on package clients (discussed in a later step).

      Ensure you’re still in the worker jail root shell.

      Create a new RSA private key:

      • openssl genrsa -out /usr/local/etc/poudriere.key 4096

      The private key file only needs to be accessible by root—the user that runs Poudriere. Protect its access permissions:

      • chmod 0600 /usr/local/etc/poudriere.key

      Later, you’ll need the public key part available on clients for verifying package signatures. Let’s extract the public key now:

      • openssl rsa -in /usr/local/etc/poudriere.key -pubout -out /tmp/poudriere.pub

      Lastly, download the public key file from your own computer:

      • scp your-server:/usr/jails/buildbot-worker0/tmp/poudriere.pub /tmp/poudriere.pub

      This concludes the optional creation of a key pair for package signing. You will later configure the actual signing with Poudriere and use the downloaded public key file on clients for the verification.

      Another optional step follows: if you use the ZFS filesystem, Poudriere can make use of it to speed up builds. Otherwise, you can skip to Step 4 to configure Poudriere in order to get ready for running the first build.

      Step 3 — Setting Up ZFS (Optional)

      This step only applies if you run a FreeBSD system on top of the ZFS filesystem. For instance if you’re using a DigitalOcean Droplet the image is labeled 11.2 x64 zfs (for FreeBSD 11.2). In this step, you’re going to create the filesystems that Poudriere can use to create and manage jails faster, potentially speeding up your builds.

      You can find out whether you’re using ZFS by listing pools. Make sure you’re on the server’s shell, not inside a jail.

      Run the following command to list the zpools:

      If any pool is available, it will print information about it:

      Output

      NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT zroot 148G 94.4G 54.1G - - 66% 63% 1.00x ONLINE -

      Otherwise if ZFS support is not available the tool will print no pools available, or failed to initialize ZFS library. This means that none of your system is using ZFS; in this case, skip to the next step. If you have decided to use another disk or storage type, such as the UFS filesystem, you can also move on to the next step.

      If you plan to use ZFS, remember the printed pool name on which you want to store build-related data. You should plan for several gigabytes of storage.

      ZFS is helpful to separate the various datasets of Poudriere, like build jails, ports trees, logs, packages, and other data. These are stored independently and as a result can be deleted quickly with the certainty of not leaving free space or traces behind.

      For Poudriere to make use of ZFS, you need to do three things: create a parent ZFS dataset, allow for the creation and deletion of ZFS datasets (which the Buildbot worker jail, or any other jail, by default cannot do), and edit Poudriere’s configuration accordingly.

      In the prerequisite tutorial, you configured the Buildbot worker jail in /etc/jail.buildbot-worker0.conf. Open this file with your preferred text editor and add the following highlighted lines to delegate a parent dataset to allow the jail to administer ZFS datasets beneath the parent. Remember to replace zroot with your desired pool name:

      • sudo ee /etc/jail.buildbot-worker0.conf

      /etc/jail.buildbot-worker0.conf

      buildbot-worker0 {
          host.hostname = buildbot-worker0.localdomain;
          ip4.addr = "lo1|10.0.0.3/24";
          path = "/usr/jails/buildbot-worker0";
          exec.start = "/bin/sh /etc/rc";
          exec.stop = "/bin/sh /etc/rc.shutdown";
          mount.devfs; # need /dev/*random for Python
          persist;
      
          exec.poststart = "/sbin/zfs jail buildbot-worker0 zroot/pdr/w0";
      }
      

      In this article we will store build-related data on the ZFS pool zroot—please adapt this ZFS-related configuration here and throughout the rest of the article if you chose a pool of a different name.

      After adding this content, save and exit the editor. If you’re using ee, do this by pressing CTRL+C, typing exit, and pressing ENTER.

      Create the parent ZFS dataset mentioned in the configuration file:

      • sudo zfs create zroot/pdr
      • sudo zfs create zroot/pdr/w0

      This deliberately assumes that you may want to add more workers in the future and therefore creates a sub-dataset for your first worker. The dataset name is short on purpose, since older versions of FreeBSD (before 12.0) had a mount name limit of 88 characters.

      In order for a jail to take control of a parent dataset and administer any children, the dataset must be marked with the following flag:

      • sudo zfs set jailed=on zroot/pdr/w0

      With the preconditions now met, the jail will start correctly with the new configuration:

      • sudo service jail restart buildbot-worker0

      With these instructions, you successfully created the required filesystems—ZFS datasets—and allowed the jail to manage the parent dataset. In the next step, you will configure Poudriere, which involves specifying the chosen zpool and dataset used to store build-related data.

      Step 4 — Configuring Poudriere, the Build Jail, and the Ports Tree

      Until this point, you’ve installed Poudriere and optionally covered requirements for package signing and ZFS. For Poudriere to be able to run in a “jailed” fashion—that is, functioning correctly from within the Buildbot worker jail—you need to provide certain permissions to the jail. For example, if you use ZFS, you have already delegated a parent dataset for use and administration by the jail.

      Let’s first configure the loopback IP and all of the permissions, and then step through the respective meaning following the changes.

      Poudriere wants to start two build jails per build: one with loopback-only networking and one with internet access. Only build stages that are supposed to reach the internet will use the latter. For example, the fetch may download source tarballs, but the build phase is not allowed internet access. The existing configuration of the worker jail has ip4.addr = "lo1|10.0.0.3/24" that allows internet access. In order to allow Poudriere to assign a loopback address to freshly started build jails, the IP must also be passed to its parent (the worker jail). For this to work, please ensure you have applied the latest version of the firewall configuration file /usr/local/etc/ipfw.rules from the prerequisite tutorial, which will block the loopback interface lo0 from opening outgoing connections through NAT.

      Add the highlighted lines to your worker jail configuration:

      • sudo ee /etc/jail.buildbot-worker0.conf

      /etc/jail.buildbot-worker0.conf

      buildbot-worker0 {
          host.hostname = buildbot-worker0.localdomain;
          ip4.addr = "lo1|10.0.0.3/24";
          ip4.addr += "lo0|127.0.0.3";
          path = "/usr/jails/buildbot-worker0";
          exec.start = "/bin/sh /etc/rc";
          exec.stop = "/bin/sh /etc/rc.shutdown";
          mount.devfs; # need /dev/*random for Python
          persist;
      
          # If you followed the ZFS setup step, you have this line
          # already (keep it). For non-ZFS setup, this line must be absent.
          exec.poststart = "/sbin/zfs jail buildbot-worker0 zroot/pdr/w0";
      
          allow.chflags;
          allow.mount;
          allow.mount.devfs;
          allow.mount.nullfs;
          allow.mount.procfs;
          allow.mount.tmpfs;
          allow.mount.zfs; # only needed if you use ZFS
          allow.raw_sockets; # optional
          allow.socket_af; # optional
          allow.sysvipc; # optional
          children.max=16;
          enforce_statfs=1;
      }
      

      Here you’ve added the following (also see the jail(8) manpage):

      • ip4.addr += "lo0|127.0.0.3" adds another IPv4 address to the jail. You will later configure Poudriere’s LOIP4 variable in order to assign this loopback address to build jails that are not supposed to talk to the internet or other machines in your network, such as during the build phase. If you ever have a build that requires internet access during build, Poudriere supports a variable ALLOW_NETWORKING_PACKAGES as a workaround. However, it is preferable to follow best practice and perform downloads, and other internet-facing tasks earlier, in the fetch phase for which Poudriere permits internet access.
      • allow.chflags allows Poudriere to render certain system files like /bin/sh immutable in the build jail.
      • allow.mount and the other allow.mount.* options enable Poudriere to mount certain required filesystems into the build jails.
      • allow.raw_sockets which permits use of raw sockets, and allow.socket_af which permits use of any socket address family, are both applied to the internet-capable build jails. This is helpful so that you can run tools like ping in interactive mode, like when entering a build jail to debug problems.
      • allow.sysvipc is deprecated in favor of three separate settings sysvmsg/sysvsem/sysvshm to restrict jails to only see their own shared memory objects (via “SYS V” IPC primitives). However, Poudriere can only pass on allow.sysvipc to build jails because it cannot read the relevant sysctl information for the three separate parameters (as of FreeBSD 11.2). With this deprecated configuration, the jail could read shared memory of processes outside the jail. This is only relevant for certain software that depends on IPC features, like PostgreSQL, so chances are small for this to affect security. You can remove this configuration unless you depend on a port that requires it during build.
      • children.max=16 allows 16 sub-jails below the worker jail. You can raise this number later if you have a lot of CPUs and Poudriere tries to create more build jails than permitted. Each Poudriere build will try to create a reference jail and two build jails per “job”, and its default is to use the number of CPUs (as output by sysctl -n hw.ncpu) as the job count.
      • enforce_statfs=1 is required together with allow.mount in order to mount certain filesystems.

      Save and exit the configuration file.

      Restart the jail for its configuration to take affect immediately:

      • sudo service jail restart buildbot-worker0

      The respective kernel modules must be loaded so that Poudriere can perform mounts. Run the following commands to load the modules at boot time and immediately:

      • sudo sysrc -f /boot/loader.conf nullfs_load=YES
      • sudo kldload -n nullfs
      • sudo sysrc -f /boot/loader.conf tmpfs_load=YES
      • sudo kldload -n tmpfs

      You already installed the Poudriere package earlier, which has copied the sample file /usr/local/etc/poudriere.conf.sample to /usr/local/etc/poudriere.conf. Next, you will make edits to the configuration file. All possible configuration variables already exist in the sample, so uncomment or adapt the respective line in the file to set a variable to a certain value.

      For the following commands, please ensure you are still in a root shell in the worker jail:

      • sudo jexec buildbot-worker0 csh

      Open the file with the following command:

      • ee /usr/local/etc/poudriere.conf

      If you have decided to use ZFS, please fill in your desired zpool and parent dataset:

      /usr/local/etc/poudriere.conf (snippet)

      . . .
      # Poudriere can optionally use ZFS for its ports/jail storage. For
      # ZFS define ZPOOL, otherwise set NO_ZFS=yes
      #
      #### ZFS
      # The pool where poudriere will create all the filesystems it needs
      # poudriere will use ${ZPOOL}/${ZROOTFS} as its root
      #
      # You need at least 7GB of free space in this pool to have a working
      # poudriere.
      #
      ZPOOL=zroot
      
      ### NO ZFS
      # To not use ZFS, define NO_ZFS=yes
      #NO_ZFS=yes
      
      # root of the poudriere zfs filesystem, by default /poudriere
      ZROOTFS=/pdr/w0
      . . .
      

      Otherwise, if you decided against ZFS, please disable ZFS support:

      /usr/local/etc/poudriere.conf (snippet)

      . . .
      # Poudriere can optionally use ZFS for its ports/jail storage. For
      # ZFS define ZPOOL, otherwise set NO_ZFS=yes
      #
      #### ZFS
      # The pool where poudriere will create all the filesystems it needs
      # poudriere will use ${ZPOOL}/${ZROOTFS} as its root
      #
      # You need at least 7GB of free space in this pool to have a working
      # poudriere.
      #
      #ZPOOL=zroot
      
      ### NO ZFS
      # To not use ZFS, define NO_ZFS=yes
      NO_ZFS=yes
      
      # root of the poudriere zfs filesystem, by default /poudriere
      # ZROOTFS=/poudriere
      . . .
      

      You will later instruct Poudriere to download a FreeBSD base system and thereby bootstrap the first build jail. This requires specifying a download host, add the following highlighted line:

      /usr/local/etc/poudriere.conf (snippet)

      . . .
      # the host where to download sets for the jails setup
      # You can specify here a host or an IP
      # replace _PROTO_ by http or ftp
      # replace _CHANGE_THIS_ by the hostname of the mirrors where you want to fetch
      # by default: ftp://ftp.freebsd.org
      #
      # Also note that every protocols supported by fetch(1) are supported here, even
      # file:///
      # Suggested: https://download.FreeBSD.org
      FREEBSD_HOST=https://download.FreeBSD.org
      

      Since Poudriere will run jailed, the mount name limit of 88 characters of FreeBSD versions before 12.0 is especially harmful, as the full path of the jail /usr/jails/buildbot-worker0 is part of each mount path. Exceeding the limit would fatally break the builds, so let’s take good care to reduce path lengths. Instead of the typical directory /usr/local/poudriere, you can use /pdr like the following:

      /usr/local/etc/poudriere.conf (snippet)

      . . .
      # The directory where poudriere will store jails and ports
      BASEFS=/pdr
      

      Now, create that directory:

      Switch again to your editor of poudriere.conf:

      • ee /usr/local/etc/poudriere.conf

      Poudriere will mount a central directory for dist files (the source code tarballs for each port) while running builds so that all builders share the same cache. The default directory is:

      /usr/local/etc/poudriere.conf (snippet)

      . . .
      # If set the given directory will be used for the distfiles
      # This allows to share the distfiles between jails and ports tree
      # If this is "no", poudriere must be supplied a ports tree that already has
      # the required distfiles.
      DISTFILES_CACHE=/usr/ports/distfiles
      

      Now, create that directory:

      • mkdir -p /usr/ports/distfiles

      If you followed Step 2 and created a package repository signing key, please enter the editor again and specify it:

      • ee /usr/local/etc/poudriere.conf

      /usr/local/etc/poudriere.conf (snippet)

      . . .
      # Path to the RSA key to sign the PKG repo with. See pkg-repo(8)
      PKG_REPO_SIGNING_KEY=/usr/local/etc/poudriere.key
      

      Builds will run much faster if you cache C/C++ compiler and linker outputs for next time. The ports tree supports this directly by leveraging the tool ccache. Please enable it and create the respective cache directory if you can spare at least 5GB more space (the default cache size):

      /usr/local/etc/poudriere.conf (snippet)

      . . .
      # ccache support. Supply the path to your ccache cache directory.
      # It will be mounted into the jail and be shared among all jails.
      # It is recommended that extra ccache configuration be done with
      # ccache -o rather than from the environment.
      CCACHE_DIR=/var/cache/ccache
      

      Building and running Linux software is uncommon, so disable it until needed:

      • ee /usr/local/etc/poudriere.conf

      /usr/local/etc/poudriere.conf (snippet)

      . . .
      # Disable linux support
      NOLINUX=yes
      

      The jails should get a loopback address assigned, or Poudriere will warn about it. We can inherit the jail’s IP because it is on a loopback-only network interface (lo1). For this, please add the following line to the end of the configuration file:

      /usr/local/etc/poudriere.conf (snippet)

      LOIP4=127.0.0.3
      

      Save and exit the configuration file.

      For working builds, we need two more resources: a FreeBSD base system to use as the build jail template and an up-to-date ports tree. Choose the FreeBSD version you are targeting. In this tutorial, we will tell Poudriere to download FreeBSD 11.2 for amd64 architecture. You can name the jail how you like, but a consistent naming scheme like 112amd64 is recommended. Also keep in mind the choice between quarterly, stable ports tree branches (here, we use 2019Q2) and the bleeding edge “head” branch that might lead to breaking builds after updates every now and then. FreeBSD versions newer than that on the server can’t be used in the build jail.

      Download and create the build jail:

      • poudriere jail -c -j 112amd64 -v 11.2-RELEASE -a amd64

      Lastly, let’s download the ports tree. The default download method is portsnap, which uses compressed snapshots of the tree without history information. Either Subversion or Git are preferable to merge upstream changes or contribute back. This is also important if you want to use a custom, self-hosted tree in a version control system. In the following command, please fill in the current year and quarter.

      If you want to start with the upstream, official ports tree:

      • poudriere ports -c -p 2019Q2 -m svn+https -B branches/2019Q2

      The method svn+https would sync from the FreeBSD Subversion host (viewable online here). If you plan to use an alternative source, read the following note, otherwise skip it.

      Note: As an alternative, the method git clones the tree from the mirror on GitHub by default.

      To use the “head” branch, replace the last parameter with -B head (for Subversion) or -B master (for Git).

      If you prefer to use your own Git repository, you will have to explicitly specify your repository URL and branch name. Let’s assume you want to name your tree customtree and use the branch custom:

      • poudriere ports -c -p customtree -m git -B custom -U https://github.com/AndiDog/freebsd-ports.git

      The example URL points to a fork of freebsd-ports on GitHub, but could be any Git or other supported type of repository to which the CI server has access.

      Available trees can be listed with poudriere ports -l, which outputs a listing like:

      Output

      PORTSTREE METHOD TIMESTAMP PATH 2019Q2 svn+https 2019-04-20 19:23:19 /pdr/ports/2019Q2

      You’re now done setting up Poudriere’s configuration and resources. You’ve configured Poudriere with the required data to trigger the first builds and enabled the jail to create subjails. Next, you’re going to run the first build manually to verify that the setup is working.

      Step 5 — Running a Manual Test Build

      You can use the command poudriere bulk to build one or more packages and all its dependencies. After the first build of a package, Poudriere also automatically detects if a rebuild is necessary, or otherwise leaves the existing package file untouched. While the bulk subcommand only builds packages, running a build using poudriere testport would also test the specified ports using the definition of “testing” given in the port’s Makefile. For the scope of this article, we’re only interested in providing packages for installation on clients, so we are using bulk builds.

      Ensure you’re still in a root shell of the worker jail where you have installed Poudriere. Later on, this will also be where the Buildbot worker process will run builds automatically.

      Run the build, filling in the placeholders with the build jail name and ports tree name you chose earlier:

      • poudriere bulk -j 112amd64 -p 2019Q2 ports-mgmt/pkg

      This builds the port ports-mgmt/pkg. Ports in the official tree are stored in a <category>/<name> hierarchy, and those paths (called package origin) are used to tell Poudriere which packages should be built. For the start, we have chosen to only build the package manager pkg, which does not have any third-party dependencies and is therefore a good, quick check on the configuration. If everything runs fine, you’ll see output like this:

      Output

      [00:00:00] Creating the reference jail... done [00:00:06] Mounting system devices for 112amd64-2019Q2 [00:00:06] Mounting ports/packages/distfiles [00:00:06] Using packages from previously failed build [00:00:06] Mounting ccache from: /var/cache/ccache [00:00:06] Mounting packages from: /pdr/data/packages/112amd64-2019Q2 /etc/resolv.conf -> /pdr/data/.m/112amd64-2019Q2/ref/etc/resolv.conf [00:00:06] Starting jail 112amd64-2019Q2 [00:00:07] Logs: /pdr/data/logs/bulk/112amd64-2019Q2/2019-04-20_19h35m00s [00:00:07] Loading MOVED for /pdr/data/.m/112amd64-2019Q2/ref/usr/ports [00:00:08] Ports supports: FLAVORS SELECTED_OPTIONS [00:00:08] Gathering ports metadata [00:00:08] Calculating ports order and dependencies [00:00:08] pkg package missing, skipping sanity [00:00:08] Skipping incremental rebuild and repository sanity checks [00:00:08] Cleaning the build queue [00:00:08] Sanity checking build queue [00:00:08] Processing PRIORITY_BOOST [00:00:08] Balancing pool [00:00:08] Recording filesystem state for prepkg... done [00:00:08] Building 1 packages using 1 builders [00:00:08] Starting/Cloning builders [00:00:14] Hit CTRL+t at any time to see build progress and stats [00:00:14] [01] [00:00:00] Building ports-mgmt/pkg | pkg-1.10.5_5 [00:03:24] [01] [00:03:10] Finished ports-mgmt/pkg | pkg-1.10.5_5: Success [00:03:25] Stopping 1 builders [00:03:25] Creating pkg repository Creating repository in /tmp/packages: 100% Packing files for repository: 100% [00:03:25] Committing packages to repository [00:03:25] Removing old packages [00:03:25] Built ports: ports-mgmt/pkg [112amd64-2019Q2] [2019-04-20_19h35m00s] [committing:] Queued: 1 Built: 1 Failed: 0 Skipped: 0 Ignored: 0 Tobuild: 0 Time: 00:03:18 [00:03:25] Logs: /pdr/data/logs/bulk/112amd64-2019Q2/2019-04-20_19h35m00s [00:03:25] Cleaning up [00:03:25] Unmounting file systems

      This output shows where packages will go after build, and from where existing packages are taken in case they don’t need rebuild (here: /pdr/data/packages/112amd64-2019Q2). Also, the output shows an overview of running builds while Poudriere runs (you can press CTRL+T in an interactive shell to print the progress). In the final summary you’ll see one package was built. You can view verbose build output in the log directory (/pdr/data/logs/bulk/112amd64-2019Q2/*).

      This output confirms a successful build. If Poudriere has built at least one package successfully, it will automatically commit it to the package repository. This means that packages are only available after all builds have finished, even if other packages failed to build. You now have a working package repository at /pdr/data/packages/112amd64-2019Q2 within the Buildbot worker jail.

      You’ve completed all the configuration needed to return working Poudriere builds, and you’ve successfully verified with a manual build. You’ll see this same output later in the tutorial once you’ve automated the bulk build in Buildbot. In addition, a link to view the detailed logs shall be accessible from the web interface. To achieve this, and to serve the package repository to clients, you’ll set up a web server next.

      Step 6 — Configuring Nginx to Serve the Poudriere Web Interface and Package Repository

      Poudriere provides several output artifacts that we want to host using a web server:

      • Package repositories are made available to clients so they can access them with the regular pkg update and pkg install commands, using HTTPS or HTTP as transport.
      • Detailed build logs are helpful for developers to debug problematic builds or to investigate build output. They are stored per package and per build—in the Poudriere output from the last step, you saw that logs are stored in one directory per build, labeled with date and time.
      • Poudriere’s built-in web interface is a small, single HTML page per build that uses WebSockets to regularly update the status shown on the page. This is helpful to get a better overview of how far a build is, which dependencies triggered other package builds to fail, and lastly as a replacement for the command line output, which only shows a summary at the end unless you specifically make it print the current build progress.

      The configuration change in Nginx is short, as only static files need to be served. Since you’ll serve them to the outside world, you’re now going to configure the existing Nginx instance on the server, outside the jails, to serve the mentioned files from paths within the worker jail.

      Please exit the jail shell since you’re now going to work on the server:

      Open an editor with the Nginx configuration /usr/local/etc/nginx/nginx.conf:

      • sudo ee /usr/local/etc/nginx/nginx.conf

      Add the following locations inside the server { block:

      /usr/local/etc/nginx/nginx.conf

      . . .
      http {
          . . .
          server {
              . . .
              location / {
                  root /usr/local/www/nginx;
                  index index.html index.htm;
              }
      
              # poudriere logs
              location ~ ^/logs(/(.*))?$ {
                  include mime.types;
                  types {
                      text/plain log;
                  }
      
                  alias /usr/jails/buildbot-worker0/pdr/data/logs/bulk$1;
                  index index.html index.htm;
                  autoindex on;
              }
      
              # poudriere packages
              location ~ ^/packages(/(.*))?$ {
                  alias /usr/jails/buildbot-worker0/pdr/data/packages$1;
                  index no-index-file-but-required-directive-to-list-dir-contents;
                  autoindex on;
              }
      
              location /buildbot/ {
                  proxy_pass http://10.0.0.2:8010/;
              }
      
              . . .
          }
      }
      . . .
      

      Save and close the Nginx configuration file. Then, reload the Nginx service:

      • sudo service nginx reload

      Let’s now check out the artifacts created by the first manual build. Open up your preferred web browser on your local machine to access the resources.

      The package repository is below https://your-domain/packages/ (or http://your-server-ip/). You will find meta information in the root directory, e.g. 112amd64-2019Q2, and all built packages in the subdirectory All:

      Package repository listing

      Detailed build logs and Poudriere’s built-in web interface can be found below https://your-domain/logs/. Click through the directory hierarchy to reach the data of your previous manual build. In this example, you might end up on a URL like https://your-domain/logs/112amd64-2019Q2/latest/build.html.

      Poudriere web interface

      If you did not set up a domain name for your server, you will need to enter your server’s public IP address for these examples, e.g. http://your-server-ip/logs/.

      This concludes all manual setup to get working builds and have visibility into the output (packages and logs). Going forward, you will automate builds to achieve continuous integration.

      Step 7 — Setting Up a Buildbot Builder for Your Packages

      Your goal in this step is to automate bulk-package builds by executing Poudriere in the same way you already have manually—by adding to the existing Buildbot sample configuration. By the end of this step, Buildbot will trigger the package build whenever the chosen branch of the ports tree changes. In this tutorial’s examples, that would be the quarterly branch 2019Q2.

      All necessary changes are done in the Buildbot master configuration, so please open a root shell in the master jail:

      • sudo jexec buildbot-master csh

      First, a builder must be defined that describes the commands and actions performed to run a build. In the existing configuration /var/buildbot-master/master.cfg, you will find a section ####### BUILDERS—open an editor and replace the whole section until the next heading starting with ####### ..., with the following configuration:

      • ee /var/buildbot-master/master.cfg

      /var/buildbot-master/master.cfg (snippet)

      . . .
      ####### BUILDERS
      
      c['builders'] = []
      
      PORTS_TO_BUILD = {
          'security/sudo',
          'shells/bash',
          'sysutils/tmux',
      }
      
      
      # Custom classes
      class PoudriereLogLineObserver(util.LogLineObserver):
          _logsRe = re.compile(r'Logs: /pdr/data/logs/bulk(/[-_/0-9A-Za-z]+)$')
      
          def __init__(self):
              super().__init__()
              self._hadUrls = False
      
          def outLineReceived(self, line):
              if not self._hadUrls:
                  m = self._logsRe.search(line.strip())
                  if m:
                      poudriereUiUrl = f'''{re.sub('/buildbot/$', '', c['buildbotURL'])}/logs{m.group(1)}'''
                      self.step.addURL('Poudriere build', poudriereUiUrl)
                      self.step.addURL('Poudriere logs', poudriereUiUrl + '/logs/')
                      self._hadUrls = True
      
      
      class PoudriereCompileStep(steps.Compile):
          def __init__(self, *args, **kwargs):
              super().__init__(*args, **kwargs)
              self.addLogObserver('stdio', PoudriereLogLineObserver())
      
      
      # Poudriere bulk build
      bulkBuildFactory = util.BuildFactory()
      bulkBuildFactory.addSteps([
          steps.ShellCommand(
              name='update ports tree',
              command=['sudo', 'poudriere', 'ports', '-u', '-p', '2019Q2', '-v'],
              haltOnFailure=True,
          ),
          PoudriereCompileStep(
              name='make bulk',
              command=['sudo', 'poudriere', 'bulk', '-j', '112amd64', '-p', '2019Q2'] + list(sorted(PORTS_TO_BUILD)),
              haltOnFailure=True,
          ),
      ])
      c['builders'].append(util.BuilderConfig(name='bulk-112amd64-2019Q2',
                                              workernames=['worker0'],
                                              factory=bulkBuildFactory))
      . . .
      

      Note how this makes use of Buildbot’s extensibility: custom classes are used to observe and parse information from Poudriere’s log output. Namely, PoudriereLogLineObserver is added as “log observer”, i.e. gets called whenever a new log line is printed during the build. The class searches the logs for the log directory and converts that into hyperlinks. Those links will be displayed alongside the build step and take the user directly to Poudriere’s web interface and logs.

      In the first build step “update ports tree”, we use Poudriere’s built-in update command (ports -u) to pull the latest version of the ports tree. This will use the previously configured method automatically (for example SVN/Git). This way, you can be sure the packages are always built against the latest committed tree, which is especially helpful if you have your own versioned repository where you maintain software versions and patches.

      At the top, the list PORTS_TO_BUILD specifies which ports should be built. It is used in the steps of the build factory specified at the bottom of the block. The build factory is a template used to instantiate a build. Buildbot creates a unique build whenever one is triggered, and the build uses a copy of the steps that were defined for the build factory at the time. In this case, we configured exactly two steps:

      • Update the ports tree. Since this example uses the quarterly branch 2019Q2, it will not receive changes very often (typically only security and build fixes).
      • Run the bulk build using the same tree.

      To make the added code block work, please add a required import to the top of the file:

      /var/buildbot-master/master.cfg (snippet)

      # -*- python -*-
      # ex: set filetype=python:
      
      import re
      
      from buildbot.plugins import *
      

      The re library in Python implements regular expressions, a feature to search or replace parts of a string—the PoudriereLogLineObserver class uses it to search for a line Logs: /pdr/data/logs/... that mentions the log directory.

      The build commands use sudo to run certain commands. This is required because Poudriere needs superuser privileges when running a build—in order to create, manage, and destroy the build jails—and also the ports trees managed by Poudriere are created with the root user as owner. In the previous tutorial, we configured the user that runs the Buildbot worker process with sysrc buildbot_worker_uid=buildbot-worker. Hence, we want to allow the buildbot-worker user to run exactly the necessary commands as root, but not other commands (for security reasons). Let’s install the sudo program and configure it accordingly.

      This needs to be done on the worker jail, not the master. Please exit the master jail shell and enter the worker jail:

      • sudo jexec buildbot-worker0 csh

      Install the sudo package:

      Confirm installation with y and ENTER.

      On FreeBSD, the sudo package by default reads configuration files from /usr/local/etc/sudoers.d/. Open an editor to create a new configuration file:

      • env EDITOR=ee visudo /usr/local/etc/sudoers.d/buildbot-worker

      The use of visudo is intentional, since it will warn on syntax errors and allow fixing them instead of committing a bad configuration.

      Specify which commands the buildbot-worker user can run as root without requiring any password:

      /usr/local/etc/sudoers.d/buildbot-worker

      buildbot-worker ALL=(ALL) NOPASSWD: /usr/local/bin/poudriere bulk *
      buildbot-worker ALL=(ALL) NOPASSWD: /usr/local/bin/poudriere ports -u *
      

      Save the file and switch back to the master jail for further required configuration of the Buildbot master:

      • sudo jexec buildbot-master csh

      You just fulfilled the requirements to get the bulk build to work. But as mentioned, each build must be triggered to run. Buildbot uses the term scheduler for an object that defines when a build is triggered, and with which extra information, such as which branch has been changed. Please remove the existing section SCHEDULERS from the configuration file, and place the following content after the BUILDERS section, so that the code can use all existing builder names:

      • ee /var/buildbot-master/master.cfg

      /var/buildbot-master/master.cfg (snippet)

      . . .
      ####### SCHEDULERS
      
      c['schedulers'] = []
      
      # Forceful scheduler allowed for all builders
      c['schedulers'].append(schedulers.ForceScheduler(
          name='force',
          builderNames=[builder.name for builder in c['builders']]))
      
      # Watch ports tree for changes on given branch
      c['schedulers'].append(schedulers.SingleBranchScheduler(
          name='sched-bulk-112amd64-2019Q2',
          change_filter=util.ChangeFilter(project='freebsd-ports', branch='branches/2019Q2'),
          builderNames=['bulk-112amd64-2019Q2']))
      . . .
      

      This replaces the sample configuration so that a force button appears on every builder. And most importantly, it creates a scheduler that watches all changes pertaining to the given project/branch and triggers a build for each change. Yet, no such change events can occur—you first have to create a change source. Typically, those are version control systems like SVN or Git on which one can detect changes on a branch. Buildbot supports the most popular ones, so we can use its functionality to add our chosen upstream ports tree repository as source. Completely replace the section CHANGESOURCES with the following configuration:

      /var/buildbot-master/master.cfg (snippet)

      . . .
      ####### CHANGESOURCES
      
      c['change_source'] = []
      
      c['change_source'].append(changes.SVNPoller(
          'svn://svn.freebsd.org/ports/',
          project='freebsd-ports',
          split_file=util.svn.split_file_branches,
          svnbin='svnlite',
          pollInterval=4 * 3600))
      
      # Example for Git:
      # c['change_source'].append(changes.GitPoller(
      #     repourl='https://github.com/AndiDog/freebsd-ports.git',
      #     project='freebsd-ports',
      #     branches=['custom'],
      #     pollInterval=4 * 3600))
      . . .
      

      This polls the SVN repository every four hours on the Buildbot master, and any new (not seen before) changes are forwarded to matching schedulers which in turn would trigger builds that are eventually dispatched to run on our single Buildbot worker. The ports tree is very large, and at first run these pollers will download the full history (for Git, only the specified branches), which can take a few minutes and require significant space (several gigabytes).

      Apply the new configuration file by restarting Buildbot:

      In this example, you have used the upstream ports collection from svn://svn.freebsd.org/ports/ and builds are scheduled whenever the branch 2019Q2 changes. As noted before, quarterly branches are mostly stable and do not receive updates very often. Since you probably do not want to wait for such a change to come in before the build is triggered the first time, let’s run it once by hand.

      Open your Buildbot web interface (https://your-domain/buildbot/), navigate to Builds > Builders > bulk-112amd64-2019Q2. It will not show any builds yet.

      Bulk builder page – no builds yet

      Click the force button at the top-right and then Start Build. That will trigger the build using its default settings, i.e. reason, branch, and other values are not overridden. The “update ports tree” step might take a minute to run, and eventually the Poudriere build should also run through successfully. The web interface will show the build as successful.

      Successful build

      Clicking one of the links (Poudriere build and Poudriere logs) will take you to the Poudriere web interface and build logs for this specific build, respectively (as shown in Step 6). Expand by clicking the arrow next to make bulk and then stdio > view all … lines to show the full output of the poudriere bulk ... command.

      Having completed the first build, the packages are now available, as configured in Nginx in Step 6. Head to https://your-domain/packages/ (or http://your-server-ip/packages/) in a browser and click through the package repository created by Poudriere. You can find the actual package files (*.txz) once you enter one of the repositories and navigate to the All/ subdirectory.

      List of package repositories

      Now that packages are available over HTTPS (or HTTP if you decided so) and built automatically on ports tree changes, you can configure one or more hosts to use those packages.

      Step 8 — Configuring Package Clients

      In this step, you need a second FreeBSD server and will set it up such that it can fetch and install the packages built on the CI server. We will call this second server the package client.

      SSH into the client host. Most remaining instructions in this section will be done on the client:

      Create the directory for custom package repository configurations:

      • sudo mkdir -p /usr/local/etc/pkg/repos

      As root user, open an editor to create the file /usr/local/etc/pkg/repos/ci.conf, and specify how and from where to retrieve packages:

      • sudo ee /usr/local/etc/pkg/repos/ci.conf

      In case you chose package signing, use this content:

      /usr/local/etc/pkg/repos/ci.conf

      ci: {
          url: "https://your-domain/packages/112amd64-2019Q2",
          signature_type: "pubkey",
          pubkey: "/usr/local/etc/pkg/repos/ci.pub",
          enabled: yes
      }
      

      Alternatively, if you decided to go without package signing, disable signature checks as follows:

      /usr/local/etc/pkg/repos/ci.conf

      ci: {
          url: "https://your-domain/packages/112amd64-2019Q2",
          signature_type: "none",
          enabled: yes
      }
      

      Note: This note applies only if you followed Step 2 to create a package repository signing key. Please skip it otherwise.

      From your local machine, upload the public key to the package client:

      • scp /tmp/poudriere.pub package-client:/tmp/ci.pub

      Using the client shell again, move the key into place so it can verify the authenticity of packages:

      • sudo mv /tmp/ci.pub /usr/local/etc/pkg/repos/ci.pub

      You completed configuring the package repository and enabled it, but on a regular FreeBSD installation, the official package repository “FreeBSD” would be enabled as well. Mixing installed packages from different sources is a foolproof way to have your production software crash at some point due to incompatible software versions or differing ABI, API, or build options. All packages on a host should stem from the same source.

      The default configuration of the official repository is stored in /etc/pkg/FreeBSD.conf. This file belongs to the base system and should not be touched. However, you can override its settings—namely, we want to disable the repository altogether—by adding the respective flag in a configuration file under /usr/local/etc/pkg/repos, where also your own repository is configured. Please create a new file /usr/local/etc/pkg/repos/FreeBSD.conf with an editor, and use the following content to disable the FreeBSD repository:

      • sudo ee /usr/local/etc/pkg/repos/FreeBSD.conf

      /usr/local/etc/pkg/repos/FreeBSD.conf

      FreeBSD: {
          enabled: no
      }
      

      If you are on a fully pristine package client host, no packages are installed yet and you can immediately begin using your own package repository. However, if even only one package was installed from another source, you are recommended to uninstall those packages and begin from scratch using your own source. The package manager pkg itself is installed as a package—to solve the chicken-and-egg problem, FreeBSD’s base system ships with a small executable /usr/sbin/pkg, which can bootstrap the package manager. That is, download the pkg package and install it as the very first package on the system. From that point on, the executable /usr/local/sbin/pkg of that package supports you as full-blown package manager.

      Run the following command to bootstrap pkg:

      In the output of pkg bootstrap, you should see that packages are taken from your own package repository which we called ci in the configuration file. If you are using a package signing key, the output will also hint about the security verification.

      Output

      The package management tool is not yet installed on your system. Do you want to fetch and install it now? [y/N]: y Bootstrapping pkg from https://your-domain/packages/112amd64-2019Q2, please wait... Verifying signature with public key /usr/local/etc/pkg/repos/ci.pub... done Installing pkg-1.10.5_5... Extracting pkg-1.10.5_5: 100%

      If you see this successful output, please skip to the next note block. However, if the package manager or other packages had already been installed from another source, and you get this error:

      Output

      pkg already bootstrapped at /usr/local/sbin/pkg

      Then please follow the instructions in the note.

      Note – only if package manager was bootstrapped already:

      You can list installed packages with pkg info. In this case, you should uninstall all of them including pkg, and reinstall them later. To do that, please first list the manually installed packages with pkg query -e "%a==0" "%n". Remember which of them you want to install again later. If, for instance, you use a shell which is not part of the base system (e.g. bash is an external package), you will want to reinstall it later or you might not be able to log in again.

      The following commands will remove all existing packages and the package manager, bootstrap the package manager again from your own package repository, and give an example of reinstalling your desired packages such as bash. Note though that you will only be able to install packages that you have built through the CI, i.e. listed in the Buildbot master configuration (variable PORTS_TO_BUILD).

      First, open a root shell before uninstalling the sudo package, or else you may not be able to gain superuser privileges anymore. Keep it open until you have bootstrapped pkg through the course of the tutorial and successfully reinstalled sudo:

      Uninstall all packages, including pkg:

      Bootstrap the package manager:

      Confirm to bootstrap the package manager by pressing y, followed by ENTER.

      In the likely case that you set up your package host using a Let’s Encrypt certificate for HTTPS, you will run into the chicken-and-egg problem where your package host is not trusted but you would need to install the package ca_root_nss (containing trustworthy root certificate authorities) to trust the Let’s Encrypt CA and thereby also trust the server hosting your custom-built packages. The same problem would arise if you used an internal CA (self-signed by you or your company). Certificate verification errors would result in error output like this when bootstrapping the package manager:

      Output

      The package management tool is not yet installed on your system. Do you want to fetch and install it now? [y/N]: y Bootstrapping pkg from https://example.com/packages/112amd64-2019Q2, please wait... Certificate verification failed for /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3 34389740104:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/s3_clnt.c:1269: [...]

      If you see this error, please follow the instructions in the note below. Otherwise, you are all set and can skip this part and continue after the note.

      Note – only if using HTTPS and certificate verification failed:

      There is one straight workaround: trust the security of the package signing key, hence bootstrapping pkg and installing the ca_root_nss package via unencrypted HTTP. Since this is not always an option because of privacy concerns, blocked HTTP ports etc., we should prefer a more “best practice” way. The official FreeBSD repository is also signed by Let’s Encrypt, so we cannot simply install the ca_root_nss package from there. No matter which CA it is, you are recommended to set up your package clients with a fixed set of HTTPS CAs to trust. You can achieve exactly that within the next few instructions. We are going to assume this is for Let’s Encrypt, but the instructions will work the same way for your own, self-signed CA (you’ll need its certificate chain handy).

      In your web browser, visit Let’s Encrypt’s certificate listing at https://letsencrypt.org/certificates/. Make sure the web site is trusted by the browser. Download the certificates under Root Certificates > Active > ISRG Root X1 (self-signed) and Intermediate Certificates > Active > Let’s Encrypt Authority X3 (Signed by ISRG Root X1) in PEM format to /tmp/root.pem and /tmp/intermediate.pem on your local computer, respectively.

      After the download has succeeded, concatenate the files into a certificate chain:

      • cat /tmp/intermediate.pem /tmp/root.pem >/tmp/letsencrypt-chain.pem
      • scp /tmp/letsencrypt-chain.pem package-client:/tmp/.

      Back in the shell of the package client, you now need to specify this chain of trust in the package manager configuration /usr/local/etc/pkg.conf so it gets used for TLS verification. Add these lines using an editor, and create the file if it does not exist yet:

      • sudo ee /usr/local/etc/pkg.conf

      /usr/local/etc/pkg.conf (snippet)

      pkg_env: {
          SSL_CA_CERT_FILE: "/usr/local/etc/pkg/repos/letsencrypt-chain.pem",
      }
      

      Move the CA chain into place:

      • sudo mv /tmp/letsencrypt-chain.pem /usr/local/etc/pkg/repos/.

      If you stayed in a root shell until now because the sudo package was removed, this command must be run without sudo. The same applies to the next command within this note.

      With this setting, you can try bootstrapping once again and should not get any more TLS errors. There is one small twist: the FreeBSD built-in /usr/sbin/pkg, which bootstraps the full package manager, does not honor the configured pkg_env setting, so we have to override the respective environment variable for this one time only, using the same value as configured:

      • sudo env SSL_CA_CERT_FILE=/usr/local/etc/pkg/repos/letsencrypt-chain.pem pkg bootstrap

      If you previously deleted existing packages, it’s a good time to reinstall essential tools now (e.g. sudo), plus any other desired packages.

      And drop out of the root shell, if that is still the case:

      In order to test whether everything works, install packages from the list specified in the Buildbot master config (variable PORTS_TO_BUILD). For example, the Bash shell and sudo:

      • sudo pkg install bash sudo tmux

      Again, confirm installation by pressing y and then ENTER. The package installation should run through without any issues.

      You can use pkg info to list which packages are currently installed (including dependencies, if any). To verify that no packages from other sources are installed, possibly causing clashes or incompatibilities, you could list installed packages with these details using pkg query "%n: autoinstalled=%a from repo=%R". Mind that pkg will be shown as bootstrapped from unknown-repository—this is why previously, you verified the bootstrapping output to see that the package manager itself is also taken from your own package repository.

      In this last step, you configured access to the CI’s package repository on a client, optionally enabled package signature verification for security purposes, ensured that packages only come from a single source to avoid compatibility issues, bootstrapped the package manager pkg, and installed your desired packages as built by the CI.

      Conclusion

      In this tutorial, you have installed and configured Poudriere, automated running package builds, and configured secure access to the package repository from a client host, ending up with the latest built packages installed from a single, central source. The setup puts you in an excellent position to keep your servers consistent and up-to-date, and manage version upgrades of external software packages.

      To further enhance your current setup, you could consider select follow-up steps:

      • Private access only: By default, Droplets have a public IP address on the internet. Also, Buildbot supports authentication but is by default unprotected.
      • Alert on build problems: Check out how to set up Buildbot reporters to get started.
      • Keep ports tree up to date: In the examples from the tutorial, the quarterly branch 2019Q2 was used, but you should switch to a newer tree eventually or use your own version-controlled repository to apply desired patches.
      • Adding builds for own projects: The FreeBSD Porter’s Handbook explains how to write a build recipe (a port) if you want to build and install your internal software as FreeBSD packages.
      • Monitor outdated packages on clients: You can compare installed packages on a client with the latest available packages on the CI using the output of sudo pkg update -q && sudo pkg version -q --not-like "=" which prints all packages whose version does not exactly match. See the manpage of pkg-version for more details.
      • Add cleanup job: Over time, the Buildbot worker jail will run full of old build log files, source tarballs, and possibly deprecated packages. Use the commands poudriere {logclean,distclean,pkgclean} to clean up (see manpage of poudriere).



      Source link

      How To Set Up Buildbot on FreeBSD


      The author selected the Open Internet/Free Speech Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      Buildbot is a job scheduling system commonly used for the purpose of continuous integration (CI). CI is a software development practice that typically comprises automatically building and testing your software on a regular basis and for every change. While it’s commonly used as a CI platform, Buildbot can also be used for any automated task that runs on a computer. Buildbot’s task execution configuration includes four components:

      • Change sources: these detect changes – such as those in a Git repository – and notify schedulers about them
      • Schedulers: schedulers trigger builders according to incoming changes
      • Builders: these contain the actual build steps, such as compilation of a software project
      • Reporters: reporters use the build results to send failure emails or other notifications

      Buildbot functions through at least one Buildbot master which runs and checks all build configuration and other settings, and distributes the actual builds to its workers. Additionally, the master offers a browser-based user interface subcomponent which, if enabled, is used to trigger or view builds and check status reports and other settings. There are also one or more Buildbot workers which connect to the master and receive commands, namely to run builds.

      In this guide, you will use FreeBSD jails to install and run each Buildbot component in a separate, isolated environment. You’ll then serve Buildbot using the Nginx web server and access its web interface using a web browser on your local machine. After completing this guide, you will have a working setup with a sample project build, ready to extend for your own CI or other use cases.

      Prerequisites

      Before you begin this guide, you will need:

      • A server running FreeBSD 11.2, although newer and older supported versions of FreeBSD should work as well. If you’re new to working with FreeBSD, you may find it helpful to customize this server by following our guide on How to Get Started with FreeBSD.
      • Nginx installed on your server. Read how to set this up by following our guide on How to Install Nginx on FreeBSD 11.2.

      If you want to host the Buildbot web interface using secure HTTPS, you’ll also need the following:

      • A registered domain name that you own and control. If you do not already have a registered domain name, you may register one with one of the many domain name registrars out there (e.g. Namecheap, GoDaddy, etc.).
      • A DNS A Record that points your domain to the public IP address of your server. This is required because of how Let’s Encrypt validates that you own the domain it’s issuing a certificate for. For example, if you want to obtain a certificate for example.com, that domain must resolve to your server for the validation process to work. You can follow this DNS Quickstart guide for details on how to add this. Throughout this tutorial, we will use example.com as an example domain name.
      • An SSL/TLS certificate for your domain. Follow How To Secure Nginx with Let’s Encrypt on FreeBSD to set this up.

      Step 1 – Setting up Jails for the Buildbot Master and Worker

      Because Buildbot allows external contributers to run code on your system, it’s recommended that you isolate its various components to prevent arbitrary or malicious code from taking up your server’s resources. In this tutorial, you will do this using FreeBSD jails.

      Similar to LXC, Docker, and other container mechanisms, FreeBSD jails offer lightweight isolation from the host system. Processes running inside a jail can only access the resources to which the jail was already granted access; otherwise, they behave like any other FreeBSD environment. Jails share the same kernel but typically run on a filesystem that has a copy of the FreeBSD base system, which may be any version of FreeBSD compatible with the host kernel. For most workloads, performance differences between running a task on the host versus in a jail are not noticeable.

      Several external software packages exist to help with the creation and management of FreeBSD jails. Since none of them are the de facto standard, we will use the operating system’s built-in jail configuration mechanism.

      First, we want to create a separate network interface for the system’s jails. In jails, the kernel rewrites network connections to the first IPv4/IPv6 address assigned to the jail. For example, if the first assigned IP address is public and a service in the jail listens on 127.0.0.1:1234, port 1234 will be publicly accessible. The recommended practice is to have a separate network interface for jails. We will follow this recommendation of “cloning” the primary loopback interface (lo0) into a separate interface (lo1). We’ll use the network 10.0.0.0/24, but any other non-overlapping network will work as well.

      Begin by configuring a cloned interface to be created on boot. This sysrc command writes a rule to the /etc/rc.conf file but doesn’t create the interface itself:

      • sudo sysrc cloned_interfaces+=lo1

      Next, create the network interface with the following command:

      • sudo service netif cloneup

      You can check the interface state and IP with:

      Output

      lo1: flags=8008<LOOPBACK,MULTICAST> metric 0 mtu 16384 options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> groups: lo

      The output shows that the interface exists, but has no IP addresses listed and attached to it yet. Its flag LOOPBACK means that this interface is only available locally and does not represent an actual hardware device.

      Next, open up a new configuration file for the master jail using your preferred editor. Here, we will use ee:

      • sudo ee /etc/jail.buildbot-master.conf

      Then add the following content to the file, which will configure a master jail named buildbot-master:

      /etc/jail.buildbot-master.conf

      buildbot-master {
          host.hostname = buildbot-master.localdomain;
          ip4.addr = "lo1|10.0.0.2/24";
          path = "/usr/jails/buildbot-master";
          exec.start = "/bin/sh /etc/rc";
          exec.stop = "/bin/sh /etc/rc.shutdown";
          mount.devfs; # need /dev/*random for Python
          persist;
      }
      

      This code assigns a fixed hostname and IP address on the jail network interface, 10.0.0.2, and specifies the root filesystem, /usr/jails/buildbot-master. The exec.start and exec.stop values used here declare that the jail's start and stop services will behave like boot processes and use the startup and shutdown scripts found in the /etc/ directory. The persist option lets the jail keep running even if all of its processes are finished.

      To learn more about possible master jail settings, check out the jail(8) manpage.

      After adding this content, save and exit the editor. If you're using ee, do this by pressing CTRL+C, typing exit, and pressing ENTER.

      The configuration file for the master jail is separate from the global jail configuration file, /etc/jail.conf. Because of this, you will need to add the name of the master jail to the list of known jails:

      • sudo sysrc "jail_list+=buildbot-master"

      Then enable any jails listed in jail_list to start automatically on boot:

      • sudo sysrc jail_enable=YES

      If you already have jails on your system configured with the /etc/jail.conf global file but you haven't used jail_list before, enabling this setting will mean that only the jails in jail_list will auto-start and you may want to add your existing jails to the list.

      Note:
      If you use the ZFS filesystem, it's recommended that you create a separate dataset for the jail's files in order to easily back up, clone, or destroy it later. The following commands assume your zpool to have the standard name zroot. If you aren't sure of the name of your zpool, you can find it with the following command:

      First, create the parent dataset for all jails:

      • sudo zfs create zroot/usr/jails

      Next, create the master jail's dataset:

      • sudo zfs create zroot/usr/jails/buildbot-master

      Next, we will create the master jail's root directory and extract the FreeBSD system.

      Ensure the jail's root filesystem directory exists. If you ran the ZFS commands in the previous note, then this has already been done and you can skip this command:

      • sudo mkdir -p /usr/jails/buildbot-master

      Then download a FreeBSD 11.2 base system archive. We first install root certificates to trust the download server:

      • sudo pkg install ca_root_nss

      This command will prompt you to approve installation of the ca_root_nss package. Do so by pressing y and then ENTER.

      Next, download the archive:

      • fetch -o /tmp/base.txz "https://download.freebsd.org/ftp/releases/amd64/11.2-RELEASE/base.txz"

      Extract the contents of this file as the jail's root filesystem:

      • sudo tar -x -f /tmp/base.txz -C /usr/jails/buildbot-master

      This guide describes the process of installing exactly one worker – also contained in a jail – and you will configure it in the same way as you did the master, reusing the base system that you just downloaded. Open up another new configuration file for the worker jail using the ee command:

      • sudo ee /etc/jail.buildbot-worker0.conf

      Add the following content to this file:

      /etc/jail.buildbot-worker0.conf

      buildbot-worker0 {
          host.hostname = buildbot-worker0.localdomain;
          ip4.addr = "lo1|10.0.0.3/24";
          path = "/usr/jails/buildbot-worker0";
          exec.start = "/bin/sh /etc/rc";
          exec.stop = "/bin/sh /etc/rc.shutdown";
          mount.devfs; # need /dev/*random for Python
          persist;
      }
      

      Looking at these lines, notice how the worker jail has a different hostname, IP, and root filesystem directory from the master. Save and close this file.

      Again, because we are using a separate jail configuration file instead of the global /etc/jail.conf, add the name to the list of known jails:

      • sudo sysrc "jail_list+=buildbot-worker0"

      Note: As with the master jail, if you use the ZFS filesystem it's recommended that you create a separate dataset for the worker jail's files. Again, the following command creates the worker jail's dataset and assumes your zpool to have the standard name zroot:

      • sudo zfs create zroot/usr/jails/buildbot-worker0

      Extract the already-downloaded FreeBSD 11.2 base system as you did for the master:

      • sudo mkdir /usr/jails/buildbot-worker0
      • sudo tar -x -f /tmp/base.txz -C /usr/jails/buildbot-worker0

      At this point, both jails are configured and contain a FreeBSD base system with no extra packages installed. Let's start the jails:

      Check whether the startup was successful by listing all the running jails on the system with the following command:

      This will return an output similar to the following that shows the jails currently running on your server:

      Output

      JID IP Address Hostname Path 1 10.0.0.2 buildbot-master.localdomain /usr/jails/buildbot-master 2 10.0.0.3 buildbot-worker0.localdomain /usr/jails/buildbot-worker0

      This confirms that the jails are running as expected. At this point, however, they have no access to the internet, which means you won't be able to install the Buildbot packages within them. Continue reading to resolve this.

      Step 2 – Setting up Internet Access for the Jails

      Although the master and worker jails are running, both of them are closed off from the internet. Opening them up to the internet is necessary, as they must be able to install packages as well as communicate with one another.

      To resolve this, copy the host's DNS resolver configuration to both of the jails:

      • sudo cp /etc/resolv.conf /usr/jails/buildbot-master/etc/resolv.conf
      • sudo cp /etc/resolv.conf /usr/jails/buildbot-worker0/etc/resolv.conf

      Next, route the outgoing internet traffic from the jail. To do this, use IPFW – FreeBSD's built-in firewall – to set up NAT (network address translation) networking rules. When you complete this step, traffic going out of the jail network will be translated to the public IP address of your host.

      If you followed the Let's Encrypt tutorial from the prerequisites, you will already have the firewall configured to allow access to your web server. In that case, some of the below steps will be redundant but there's no harm in running through them again.

      Warning: Performing erroneous changes to your firewall configuration can render the remote host inaccessible over SSH, so it's important to ensure that you have an alternative method you can use to log into the machine. For example, if you're using a server obtained from DigitalOcean, you can access it through the "Console access" feature.

      To enable access through this feature, set a root password with the following command:

      Alternatively, you can set a password for your current user with by typing:

      Include the predefined workstation firewall rules in your rc.conf file with the following command. The workstation rules protect the server but still allow basic services through, such as pinging the host or Dynamic Host Configuration Protocol:

      • sudo sysrc firewall_type="workstation"

      Next, allow access to the web server ports from the outside world. The following command allows traffic over port 22, for SSH; port 80, allowing Buildbot to be served over HTTP; and port 443, allowing Buildbot to be served over HTTPS. If you've secured your server with Let's Encrypt, then all three of these ports are necessary, but if you haven't and don't plan to do so, then you can exclude port 443:

      • sudo sysrc firewall_myservices="22/tcp 80/tcp 443/tcp"

      Allow access from any IP address to the ports specified in the firewall_myservices directive:

      • sudo sysrc firewall_allowservices="any"

      Configure the firewall to start at boot:

      • sudo sysrc firewall_enable=YES

      Then start the firewall with basic rules. The following nohup command avoids interruption of the firewall start, and also redirects both stderr and stdout to a temporary log file. This is important so as not to leave firewall rules in an inconsistent state, which can potentially leave your remote host inaccessible over SSH:

      • sudo nohup service ipfw start >/tmp/ipfw.log 2>&1

      If you're using either the csh or tcsh shells, this redirect will cause Ambiguous output redirect. to appear in your output. If you're using either of these shells, run sudo nohup service ipfw start >&/tmp/ipfw.log the following instead to start ipfw:

      At this point, the firewall service will start and begin protecting the host from connections to unsecured ports.

      Note: If something went wrong or you used a different firewall type, the firewall may not yet know the state of your SSH connection, causing your connection to the server to stall. You can find out by typing something into the shell. During a stalled connection, the characters won't be printed on the remote side. If this is the case, you can either wait until SSH notices the timeout, or drop out of the hanging terminal by pressing these keys one after another: ENTER, ~, .

      After your SSH connection is closed, reconnect to your server from your local machine:

      • ssh freebsd@your_server_ip

      If the SSH connection can't be reestablished, you will need to connect to it using an alternative method. If, for example, you're using a DigitalOcean Droplet, you could use the its "Console access" feature to log as your root user with the password your set previously.

      Once you've regained access deactivate the firewall:

      With the firewall stopped, you'll be free to to debug the problem.

      Next, you need to determine the network interface of the host which connects to the internet. Find this by running:

      This command may output a couple different interfaces. The one used by the host to connect to the internet is the one that includes your server's public IP address. To illustrate, the following example output shows that vtnet0 is the network interface used by the host:

      Output

      vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> ether 9a:3e:fa:2a:5f:56 hwaddr 9a:3e:fa:2a:5f:56 inet6 fe80::983e:faff:fe2a:5f56%vtnet0 prefixlen 64 scopeid 0x1 inet public_server_ip netmask 0xffffffc0 broadcast broadcast_ip inet 10.10.0.23 netmask 0xffff0000 broadcast 10.10.255.255 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet 10Gbase-T <full-duplex> status: active lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> groups: lo lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6> inet 10.0.0.2 netmask 0xffffff00 inet 10.0.0.3 netmask 0xffffff00 inet6 fe80::1%lo1 prefixlen 64 scopeid 0x3 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> groups: lo

      Make a note of this interface, then configure its name globally:

      • sudo sysrc firewall_nat_interface=vtnet0

      Open up the new firewall configuration script file:

      • sudo ee /usr/local/etc/ipfw.rules

      Then add the following content to the file, defining the firewall rules for IPFW:

      /usr/local/etc/ipfw.rules

      #!/bin/sh
      set -e
      
      # Add basic rules as defined by firewall_type, firewall_myservices, etc.
      . /etc/rc.firewall
      
      # External network interface
      ext_if="$firewall_nat_interface"
      
      # The interface we chose for communication between jails
      jail_if="lo1"
      
      for interface in "$ext_if" "$jail_if"; do
          if [ -z "$interface" ]; then
              >&2 echo "Missing network interface"
              exit 1
          fi
          if ! ifconfig $interface >/dev/null 2>&1; then
              >2 echo "No such network interface: $interface"
              exit 1
          fi
      done
      
      ipfw nat 123 config if $ext_if
      ipfw add 1 allow all from any to any via $jail_if
      ipfw add 2 nat 123 ip4 from any to any in via $ext_if
      ipfw add 3 skipto 20000 udp from any to any 53 out via $ext_if keep-state
      ipfw add 4 skipto 20000 udp from any to any 67 out via $ext_if keep-state
      ipfw add 5 skipto 20000 tcp from any to any out via $ext_if setup keep-state
      ipfw add 6 skipto 20000 icmp from any to any out via $ext_if keep-state
      ipfw add 19999 deny all from any to any
      ipfw add 20000 nat 123 ip4 from any to any out via $ext_if
      ipfw add 20001 allow ip from any to any
      

      Here's what each part of the script does:

      • . /etc/rc.firewall includes the system's predefined IPFW rules script, which adds basic rules according to your configuration of the firewall_* variables in /etc/rc.conf.
      • The next block checks whether all configured interfaces exist. This is for your safety and quits the script early if there's a misconfiguration.
      • The directives starting with ipfw add the actual firewall configuration and rules. Each rule – added in the lines starting with ipfw add – has a number. The firewall uses these numbers to evaluate the rules in sequential order.
        • ipfw nat 123 config if $ext_if creates an in-kernel NAT facility with the ID "123" to translate traffic using the public-facing network interface.
        • ipfw add 1 allow all from any to any via $jail_if allows all traffic between the jails. Note that if an allow rule matches, rule processing stops and the packet is allowed to pass.
        • ipfw add 2 nat 123 ip4 from any to any in via $ext_if translates all incoming IPv4 packets on the external interface. This is needed as a counterpart to the translation of outgoing packets, as described in the explanation of ipfw add 20000....
        • ipfw add 3 skipto 20000 udp from any to any 53 out via $ext_if keep-state and the following skipto rules define which outbound traffic to allow and to consider for network address translation. If there is a match, processing continues by jumping to rule 20000, which performs NAT.
        • Between rules 6 and 19999, there is an intentional gap where the /etc/rc.firewall script inserts the basic rules. If none of the above skipto rules match, the basic rules will take care of allowing different types of traffic including loopback, incoming ICMP ping messages, and the ports specified by firewall_myservices.
        • ipfw add 19999 deny all from any to any comes after all the basic rules and ensures the end of non-NAT rule processing, essentially disallowing all traffic that was not matched by a previous allow rule.
        • ipfw add 20000 nat 123 ip4 from any to any out via $ext_if translates the address of all outbound IPv4 packets leaving on the external interface. You only need IPv4 here because, in this tutorial, jails are assigned IPv4 addresses exclusively.
        • ipfw add 20001 allow ip from any to any is only required if you have turned off one-pass mode for nat rules, in which case processing will continue after going through the rule 20000, requiring you to explicitly allow those packets through with a separate rule. For the default one-pass mode, the firewall will stop processing at the nat rule and therefore ignore rule 20001.

      Save the file and exit the editor.

      Since we want to amend the predefined, basic firewall rules with the rules defined in the ipfw.rules script, we must point to this script in the rc.conf file. The following command will configure the script to execute whenever the firewall starts:

      • sudo sysrc firewall_script="/usr/local/etc/ipfw.rules"

      This setup uses the in-kernel NAT support of IPFW, so you must tell the system to load the respective kernel module at boot. Additionally, load the module immediately without requiring a reboot:

      • sudo sysrc -f /boot/loader.conf ipfw_nat_load=YES
      • sudo kldload ipfw_nat

      Restart the firewall to put the extended firewall rules script in effect:

      • sudo nohup service ipfw restart >/tmp/ipfw.log 2>&1

      Again, if you're using the csh shell or one of its derivatives (like tcsh), run sudo nohup service ipfw restart >&/tmp/ipfw.lo instead of the previous command to restart the firewall:

      Check that the firewall rules have been loaded correctly:

      This lists the firewall rules, followed by a success message:

      Output

      Flushed all rules. 00100 allow ip from any to any via lo0 [...] 10001 allow ip from any to any Firewall rules loaded.

      You can also view installed firewall rules at any time using:

      Output

      00001 allow ip from any to any via lo1 00002 nat 123 ip from any to any in via em0 [...] 65535 deny ip from any to any

      With all firewall rules in place, your jails are now able to access the internet. You can check by trying to download a web page from within a jail:

      • sudo jexec buildbot-master fetch -q -o- http://example.com/

      Output

      <!doctype html> <html> <head> <title>Example Domain</title> [...]

      With that, you've successfully prepared both jails to run like a regular operating system, set up internet access for each jail, and started both of them. The next two steps in this tutorial will walk you through installing both the master and worker components and then running them as services.

      Step 3 – Installing and Running the Buildbot Master

      Buildbot's components are split into several packages. You only need to install the py36-buildbot package to run the master component, but in this guide we will also go over how to install the web interface package, py36-buildbot-www.

      Since we are using jails to segment the various components, start by opening a root shell in the master jail:

      • sudo jexec buildbot-master csh

      Please note that, in this guide, shell command blocks are marked with a different color if they must be executed within a jail shell. Additionally, the command prompt will reflect which of the jail's user profiles – either the root or unprivileged buildbot-master user – that commands must be run under.

      Install the packages:

      • pkg install py36-buildbot py36-buildbot-www

      If you haven't yet installed or used the pkg package manager in this jail, it will prompt you to confirm that you allow it to bootstrap itself. To do this, press y and then ENTER. Then, approve installation of the Buildbot packages by again entering y.

      Next, create a regular, unprivileged user to run the master service. The following command will assign a random password to this user, but you do not need to remember it since the root user of the host (outside the jail) can change it or become any user within the jail without a password:

      • pw useradd -n buildbot-master -m -w random

      Following this, create the master directory where you will store the configuration:

      • mkdir /var/buildbot-master

      And give ownership to the service user:

      • chown buildbot-master:buildbot-master /var/buildbot-master

      From this point on, all master-related setup and changes should be executed as the unprivileged user, as this will help to keep ownership and permissions consistent.

      Switch over to the the unprivileged user:

      Then use the built-in buildbot utility to create a directory and configuration structure in the specified directory:

      • buildbot create-master /var/buildbot-master

      Unlike other CI software like Jenkins, Buildbot's behavior is defined directly in its configuration file, which is interpreted with Python. This allows for streamlined versioning of your configuration, while the use of a scripting language allows for the freedom to write custom build configurations and extend existing Buildbot functionalities.

      The Buildbot package comes with a sample master configuration file which you can use as a template for your own configuration. Copy over the sample configuration and name it master.cfg:

      • cp /var/buildbot-master/master.cfg.sample /var/buildbot-master/master.cfg

      Then open the base configuration file with your preferred text editor. Here, we'll use ee:

      • ee /var/buildbot-master/master.cfg

      The configuration file contains a password required for workers to connect to the master. Replace the default pass with a secure password of your choosing. Also, our worker's name will be worker0, so also replace example-worker with worker0 in both the WORKERS and BUILDERS sections.

      When finished, the parts of the file you need to edit will look like this:

      /var/buildbot-master/master.cfg

      ####### WORKERS
      
      # ...
      c['workers'] = [worker.Worker("worker0", "your_secure_password")]
      # ...
      
      ####### BUILDERS
      
      # ...
      c['builders'] = []
      c['builders'].append(
          util.BuilderConfig(name="runtests",
            workernames=["worker0"],
            factory=factory))
      # ...
      

      Save and close this file, then run the exit command to switch back to the root user within the jail:

      Because the sample configuration watches the Git repository git://github.com/buildbot/hello-world.git as its change source, you also need to install Git:

      With that, you've created the master directory structure and configuration, but the service isn't running yet. To run Buildbot manually, one could run the command buildbot start from the master directory, /var/buildbot-master. However, this does not take care of boot-time startup or other system-wide configuration. Instead, we will use rc scripts, FreeBSD's standard method for running services. Specifically, we'll do so using the service utility.

      For the purposes of this tutorial, we want to enable the service to run at every boot. In the case of jails, this means the start event of the jail. Use the following command to define the location of the master directory:

      • sysrc buildbot_basedir=/var/buildbot-master

      Then specify that the service should run under the buildbot-master user:

      • sysrc buildbot_user=buildbot-master

      Next, enable the service to run at jail startup:

      • sysrc buildbot_enable=YES

      Then, start the service:

      The service should start with no errors. You can verify success by viewing the contents of the log file:

      • tail /var/buildbot-master/twistd.log

      Output

      2018-06-08 15:14:52+0000 [-] Starting BuildMaster -- buildbot.version: 0.9.11 2018-06-08 15:14:52+0000 [-] Loading configuration from '/var/buildbot-master/master.cfg' [...] 2018-06-08 15:14:52+0000 [-] BuildMaster is running

      To return to the host shell, run exit from the jail shell:

      You have successfully configured and started the Buildbot master service. The second component, the worker, is required to actually run builds. You will install one worker in the next section inside a second jail and then configure its connection to the master service.

      Step 4 – Installing and Running the Buildbot Worker

      Although the Buildbot master is running, no builds can occur since you need at least one worker running. This step is similar to the previous one in that we will set up a separate jail first and then install the service. This time, however, the Buildbot worker component will connect to the master to listen for commands and report results back.

      The instructions in this step are almost identical to the master setup, except that the worker component is part of another package and the only configuration changes you'll make involve adding details about connecting it to the master and some display information about the worker itself.

      Ensure you are in the host shell, not inside a jail. Then open a root shell in the worker jail:

      • sudo jexec buildbot-worker0 csh

      Remember that, in this guide, command blocks are marked with a different color if they must be executed within a jail shell and the command prompts will reflect which user profile the commands should be run under.

      Install the Buildbot worker package with the following command:

      • pkg install py36-buildbot-worker

      As this command runs, it will prompt you to confirm whether you want to bootstrap the pkg package management utility. Do so by entering y. It will also ask you to confirm that you approve the installation of the packages, so enter y again when prompted.

      Next, create a regular, unprivileged user to run the worker service:

      • pw useradd -n buildbot-worker -m -w random

      Then create the worker directory. This is the location where the worker's configuration, display information, and build directories will be stored:

      • mkdir /var/buildbot-worker

      Give ownership to the service user:

      • chown buildbot-worker:buildbot-worker /var/buildbot-worker

      From this point on, all worker-related setup and changes should be executed as the unprivileged user. To that end, switch over to the buildbot-worker user:

      Use the built-in buildbot-worker utility to create a directory and configuration structure in the /var/buildbot-worker directory. Specify the master jail's IP address — 10.0.0.2, which we chose in the previous step — so the worker can connect to it and replace pass with the password you defined in the master configuration file:

      • buildbot-worker create-worker /var/buildbot-worker 10.0.0.2 worker0 'pass'

      To finish the setup, fill in a few details about the system administrator and the worker's purpose:

      • echo 'Your Name <your.email.address@example.com>' >/var/buildbot-worker/info/admin
      • echo 'Description of this worker' >/var/buildbot-worker/info/host

      Following this, run the exit command to switch back to the root user within the jail:

      Because the sample configuration clones the Git repository git://github.com/buildbot/hello-world.git to build the sample project, you also need to install Git within this jail. Note how the Buildbot master also required Git because change sources run on the master. Additionally, the builder uses a test runner called trial which is part of the py27-twisted package, so install this along with git-lite:

      • pkg install git-lite py27-twisted

      The built-in mechanism for running a worker is buildbot-worker start, which should be run from the worker directory, /var/buildbot-worker. However, this does not take care of boot-time startup and does not ensure that it will run under the correct user. As you did with the master, leverage the packaged rc script by using the service utility, to manage the service.

      Use the following commands to define the worker directory as well as the user and group under which the service should run:

      • sysrc buildbot_worker_basedir=/var/buildbot-worker
      • sysrc buildbot_worker_uid=buildbot-worker
      • sysrc buildbot_worker_gid=buildbot-worker

      Next, enable the service to run at jail startup:

      • sysrc buildbot_worker_enable=YES

      At time of writing, the py36-buildbot-worker package has a bug which prevents the service from starting (see this bug report). Until this is fixed, you will need to manually patch the start script by running the following command from your buildbot-worker jail:

      • sed -i '' 's|command="/usr/local/bin/twistd"|command="/usr/local/bin/twistd-3.6"|' /usr/local/etc/rc.d/buildbot-worker

      Finally, start the worker component:

      • service buildbot-worker start

      The service should start with no errors. You can verify that it was successful by viewing the most recent entries to the log file:

      • tail /var/buildbot-worker/twistd.log

      If the service started successfully, a message like Connected to 10.0.0.2:9989; worker is ready will appear in the log file. If you forgot to specify a new password earlier in this step, the service will fail to connect to the master. In this case, edit the file /var/buildbot-worker/buildbot.tac and then run service buildbot-worker restart to rectify this issue.

      Once the service starts correctly, drop out to the host shell by running the exit command from the jail shell:

      With that, the second jail has been configured and you have all the basic components needed for operating Buildbot. To be readily usable for your users, it's recommended that you also set up the web-based user interface. Doing so will allow you to control Buildbot and see build results with greater convenience.

      Step 5 – Setting Up the Buildbot Web Interface

      Buildbot features a web-based user interface that shows build overviews and results, and allows you to trigger builds manually if a "force" scheduler is configured, as is the case in the sample configuration.

      Your master configuration already sets up the www component to serve HTTP over port 8010. In a production setting, you would not serve unencrypted HTTP or open up the non-standard port 8010 to the outside, as this would open your system up to security vulnerabilities. Also, the web interface can be served from any URL path, which means that it does not need to be the only application on your domain. For example, you could serve build outputs or logs to your users. Hence, we will serve the UI to users with a separate web server – Nginx – in order to support HTTPS, protect internal ports, and gain the ability to serve other content alongside the Buildbot web interface.

      Open up the Nginx configuration file for editing:

      • sudo ee /usr/local/etc/nginx/nginx.conf

      Add the following highlighted location blocks within the file's existing server block:

      /usr/local/etc/nginx/nginx.conf

       . . .
      http {
       . . .
          server {
      
       . . .
              location / {
                  root /usr/local/www/nginx;
                  index index.html index.htm;
              }
      
              location /buildbot/ {
                  proxy_pass http://10.0.0.2:8010/;
              }
              location /buildbot/sse/ {
                  # proxy buffering will prevent sse to work
                  proxy_buffering off;
                  proxy_pass http://10.0.0.2:8010/sse/;
              }
              # required for websocket
              location /buildbot/ws {
                  proxy_http_version 1.1;
                  proxy_set_header Upgrade $http_upgrade;
                  proxy_set_header Connection "upgrade";
                  proxy_pass http://10.0.0.2:8010/ws;
                  # raise the proxy timeout for the websocket
                  proxy_read_timeout 6000s;
              }
      
              error_page 500 502 503 504 /50x.html;
              location = /50x.html {
                  root /usr/local/www/nginx-dist;
              }
      
                      . . .
      
          }
      }
      

      This configuration forwards all requests below the URL path /buildbot/ to the web interface and enables WebSocket support, which is used by the interface to receive updates that it will display such as the log output of a running build.

      Save and close the Nginx configuration file. Then, reload the Nginx service:

      • sudo service nginx reload

      Open up your preferred web browser on your local machine and access the Buildbot web interface by going to the following URL:

      https://example.com/buildbot/
      

      Alternatively, if you did not set up a domain name for your server, you'll instead need to enter your server's public IP address, http://your_server_ip/buildbot/.

      When you arrive at the interface, you will see an overview similar to the following:

      Web interface overview

      The main page may show a warning that the Buildbot URL is misconfigured. This occurs if the hostname provided in the nginx.conf file mismatches what's listed in master Buildbot configuration. Since build result emails contain links to the Buildbot web interface by default, the master must know the correct URL where it can be reached.

      Note that, in our example configurations, we have not set up this email service. If you're interested in configuring this, see Buildbot's documentation about reporters for more information:

      With that said, to resolve the warning and send emails that contain the correct content, edit the Buildbot master configuration to point to your domain.

      • sudo ee /usr/jails/buildbot-master/var/buildbot-master/master.cfg

      Find the line that begins with c['buildbotURL'] and replace the default option with your domain name, followed by /buildbot/:

      /var/buildbot-master/master.cfg

      ####### PROJECT IDENTITY
      # ...
      c['buildbotURL'] = 'https://example.com/buildbot/'
      # ...
      

      Save and close the file. Then, to apply the new configuration, reload the buildbot service:

      • sudo jexec buildbot-master service buildbot reload

      Refresh the Buildbot web interface in your browser, and the warning will disappear.

      Continuous Integration servers often serve other purposes besides CI. For example, a CI server might serve build outputs for FreeBSD packages or logs over HTTPS. It's therefore recommended that you reserve the URL path /buildbot/ for the web interface. This allows you to host more applications under different paths. For the moment, we will create a simple home page that redirects to the web interface. You can add more links once you implement more use cases for the web server.

      Run the following command to open an index file in your web root – replacing example.com with your own domain – to create an automatic redirect to the Buildbot web interface:

      • sudo ee /usr/local/www/example.com/html/index.html

      Note: If you did not follow the prerequisite Nginx tutorial and create a new web root for your Nginx configuration, you will instead need to create an index file under the default Nginx web root by running sudo ee /usr/local/www/nginx/index.html.

      Replace any existing file content with the following lines:

      /usr/local/www/nginx/index.html

      <html>
      <body>
      <a href="http://www.digitalocean.com/buildbot/">buildbot</a>
      <script>
          // Auto-redirect while only the web interface should be served
          window.location.href = "/buildbot/";
      </script>
      </body>
      </html>
      

      Save and close this file, then enter your domain name or IP address in the URL bar of your browser. It should automatically redirect you to the Buildbot interface.

      You've finished the installation of all the Buildbot components, including its web-based control and viewing interface. With all of this in place, let's run an actual build as specified in the sample configuration which we have set up for the master.

      The builder has a "force" scheduler configured by default, which allows you to trigger your first build. In the web interface, click Builds > Builders > runtests > force > Start Build and see how the build runs. If you see any errors, check the server's internet connection and whether all the dependent packages were installed as described previously.

      Sample build success screenshot

      You can find the artifacts from this build (and others) by looking at the contents of the build directory:

      • ls /usr/jails/buildbot-worker0/var/buildbot-worker/runtests

      Output

      build

      You have successfully configured a permanently running and versatile CI system and can now begin implementing your own builds.

      Conclusion

      By completing this tutorial, you practiced creating FreeBSD jails and learned some of the basics of the Buildbot automation framework, resulting in a ready-to-use installation. To learn more about Buildbot and its configuration, we encourage you to read through the official Buildbot documentation.

      From here, you're free to implement your own Continuous Integration and automation practices. In order to have a secure, stable and performant setup for production use, you may want to take these optional configuration steps:

      • Use HTTPS only (as explained in this tutorial)
      • In the tutorial, you used a separate, host-internal network lo1 for your jails. In this guide, we used ipfw for NAT purposes, but other firewalls have this feature as well. Check out the FreeBSD documentation about available firewalls. Unless your use case requires otherwise, it is recommended to keep the jail network inaccessible from the outside through the use of NAT or other mechanisms.
      • Buildbot's web interface does not require a login or check user permissions by default. To implement these, you will have to enable user authentication.



      Source link

      Use Buildbot for Software Testing on Ubuntu 18.04


      Updated by Linode Written by Tyler Langlois

      Use promo code DOCS10 for $10 credit on a new account.

      Buildbot is an open source system for testing software projects. In this guide, you will set up a Linode as a Buildbot server to use as a continuous integration platform to test code. Similarly to hosted solutions like Travis CI, Buildbot is an automated testing platform that can watch for code changes, test a project’s code, and send notifications regarding build failures.

      Before you Begin

      1. Familiarize yourself with Linode’s Getting Started guide and complete the steps for deploying and setting up a Linode running Ubuntu 18.04, including setting the hostname and timezone.

      2. This guide uses sudo wherever possible. Complete the sections of our Securing Your Server guide to create a standard user account, harden SSH access and remove unnecessary network services.

      3. Ensure your system is up to date:

        sudo apt update && sudo apt upgrade
        
      4. Complete the Add DNS Records steps to register a domain name that will point to your Linode instance hosting Buildbot.

        Note

        Replace each instance of example.com in this guide with your Buildbot site’s domain name.

      5. Your Buildbot site will serve its content over HTTPS, so you will need to obtain an SSL/TLS certificate. Use Certbot to request and download a free certificate from Let’s Encrypt.

        sudo apt install software-properties-common
        sudo add-apt-repository ppa:certbot/certbot
        sudo apt update
        sudo apt install certbot
        sudo certbot certonly --standalone -d example.com
        

        These commands will download a certificate to /etc/letsencrypt/live/example.com/ on your Linode.

        Note

      Install Buildbot

      Install the Buildbot Master

      Since Buildbot is provided as an Ubuntu package, install the software from the official Ubuntu repositories.

      1. Install the buildbot package along with pip3, which will be used to install additional python packages:

        sudo apt-get install -y buildbot python3-pip
        
      2. Install the required Buildbot Python packages:

        sudo pip3 install buildbot-www buildbot-waterfall-view buildbot-console-view buildbot-grid-view
        
      3. The buildbot package sets up several file paths and services to run persistently on your host. In order to create a new configuration for a Buildbot master, enter the directory for Buildbot master configurations and create a new master called ci (for “continuous integration”).

        cd /var/lib/buildbot/masters
        sudo -u buildbot -- buildbot create-master ci
        

        The generated master configuration file’s location is /var/lib/buildbot/masters/ci/master.cfg.sample.

      4. Make a copy of the default configuration to the path that Buildbot expects for its configuration file:

        sudo cp ci/master.cfg.sample ci/master.cfg
        
      5. Change the permissions for this configuration file so that the buildbot user has rights for the configuration file:

        sudo chown buildbot:buildbot ci/master.cfg
        

      Configure the Buildbot Master

      In order to secure and customize Buildbot, you will change a few settings in the master configuration file before using the application. The master configuration file’s location is /var/lib/buildbot/masters/ci/master.cfg.

      Buildbot has a number of concepts that are represented in the master build configuration file. Open this file in your preferred text editor and browse the Buildbot configuration. The Buildbot configuration is written in Python instead of a markup language like Yaml.

      1. Generate a random string to serve as the password that workers will use to authenticate against the Buildbot master. This is accomplished by using openssl to create a random sequence of characters.

        openssl rand -hex 16
        <a random string>
        
      2. Update the following line in the master.cfg file and replace pass with the randomly-generated password:

        /var/lib/buildbot/masters/ci/master.cfg
        1
        2
        3
        4
        5
        6
        7
        
        ...
        # The 'workers' list defines the set of recognized workers. Each element is
        # a Worker object, specifying a unique worker name and password.  The same
        # worker name and password must be configured on the worker.
        c['workers'] = [worker.Worker("example-worker", "pass")]
        ...
            
      3. Uncomment the cUse Buildbot for Software Testing on Ubuntu 18.04 and the c[titleURL] lines. If desired, change the name of the Buildbot installation by updating the value of cUse Buildbot for Software Testing on Ubuntu 18.04. Replace the c[titleURL] value with the URL of your Buildbot instance. In the example, the URL value is replaced with example.com.

        /var/lib/buildbot/masters/ci/master.cfg
        1
        2
        3
        4
        5
        
        ...
        c['title'] = "My CI"
        c['titleURL'] = "https://example.com"
        ...
            
      4. Uncomment the c['buildbotURL'] line and replace the URL value with the your Buildbot instance’s URL:

        /var/lib/buildbot/masters/ci/master.cfg
        1
        2
        3
        4
        
        ...
        c['buildbotURL'] = "https://example.com/"
        ...
            

        These options assume that you will use a custom domain secured with Let’s Encrypt certificates from certbot as outlined in the Before You Begin section of this guide.

      5. Uncomment the web interface configuration lines and keep the default options:

        /var/lib/buildbot/masters/ci/master.cfg
        1
        2
        3
        4
        5
        
        ...
        c['www'] = dict(port=8010,
                        plugins=dict(waterfall_view={}, console_view={}, grid_view={}))
        ...
            
      6. By default, Buildbot does not require people to authenticate in order to access control features in the web UI. To secure Buildbot, you will need to configure an authentication plugin.

        Configure users for the Buildbot master web interface. Add the following lines below the web interface configuration lines and replace the myusername and password values with the ones you would like to use.

        /var/lib/buildbot/masters/ci/master.cfg
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        16
        
        ...
        c['www'] = dict(port=8010,
                        plugins=dict(waterfall_view={}, console_view={}, grid_view={}))
        
        # user configurations
        c['www']['authz'] = util.Authz(
                allowRules = [
                    util.AnyEndpointMatcher(role="admins")
                ],
                roleMatchers = [
                    util.RolesFromUsername(roles=['admins'], usernames=['myusername'])
                ]
        )
        c['www']['auth'] = util.UserPasswordAuth([('myusername','password')])
        ...
            
      7. Buildbot supports building repositories based on GitHub activity. This is done with a GitHub webhook. Generate a random string to serve as a webhook secret token to validate payloads.

        openssl rand -hex 16
        <a random string>
        
      8. Configure Buildbot to recognize GitHub webhooks as a change source. Add the following snippet to the end of the master.cfg file and replace webhook secret with the random string generated in the previous step.

        /var/lib/buildbot/masters/ci/master.cfg
        1
        2
        3
        4
        5
        6
        
        c['www']['change_hook_dialects'] = {
            'github': {
                'secret': 'webhook_secret',
            }
        }
            
      9. Finally, start the Buildbot master. This command will start the Buildbot process and persist it across reboots.

        sudo systemctl enable --now buildmaster@ci.service
        

      Set up the Buildbot Master Web Interface

      Buildbot is now running and listening on HTTP without encryption. To secure the connection, install NGINX to terminate SSL and reverse proxy traffic to the Buildbot master process.

      These steps install NGINX Mainline on Ubuntu from NGINX Inc’s official repository. For other distributions, see the NGINX admin guide. For information on configuring NGINX for production environments, see our Getting Started with NGINX series.

      1. Open /etc/apt/sources.list in a text editor and add the following line to the bottom. Replace CODENAME in this example with the codename of your Ubuntu release. For example, for Ubuntu 18.04, named Bionic Beaver, insert bionic in place of CODENAME below:

        /etc/apt/sources.list
        1
        
        deb http://nginx.org/packages/mainline/ubuntu/ CODENAME nginx
      2. Import the repository’s package signing key and add it to apt:

        sudo wget http://nginx.org/keys/nginx_signing.key
        sudo apt-key add nginx_signing.key
        
      3. Install NGINX:

        sudo apt update
        sudo apt install nginx
        
      4. Ensure NGINX is running and and enabled to start automatically on reboot:

        sudo systemctl start nginx
        sudo systemctl enable nginx
        

      Now that NGINX is installed, configure NGINX to talk to the local Buildbot port. NGINX will listen for SSL traffic using the Let’s Encrypt certificate for your domain.

      1. Create your site’s NGINX configuration file. Ensure that you replace the configuration file’s name example.com.conf with your domain name. Replace all instances of example.com with your Buildbot instance’s URL.

        /etc/nginx/conf.d/example.com.conf
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        16
        17
        18
        19
        20
        21
        22
        23
        24
        25
        26
        27
        28
        29
        30
        31
        32
        33
        34
        35
        36
        37
        38
        39
        40
        41
        42
        43
        44
        45
        46
        47
        48
        
        server {
          # Enable SSL and http2
          listen 443 ssl http2 default_server;
        
          server_name example.com;
        
          root html;
          index index.html index.htm;
        
          ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
          ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
        
          # put a one day session timeout for websockets to stay longer
          ssl_session_cache      shared:SSL:10m;
          ssl_session_timeout  1440m;
        
          ssl_protocols TLSv1.2 TLSv1.3;
          ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
          ssl_prefer_server_ciphers   on;
        
          # force https
          add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";
          spdy_headers_comp 5;
        
          proxy_set_header HOST $host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto  $scheme;
          proxy_set_header X-Forwarded-Server  $host;
          proxy_set_header X-Forwarded-Host  $host;
        
          location / {
              proxy_pass http://127.0.0.1:8010/;
          }
          location /sse/ {
              # proxy buffering will prevent sse to work
              proxy_buffering off;
              proxy_pass http://127.0.0.1:8010/sse/;
          }
          location /ws {
              proxy_http_version 1.1;
              proxy_set_header Upgrade $http_upgrade;
              proxy_set_header Connection "upgrade";
              proxy_pass http://127.0.0.1:8010/ws;
              # raise the proxy timeout for the websocket
              proxy_read_timeout 6000s;
          }
        }
      2. Disable NGINX’s default configuration file:

        mv /etc/nginx/conf.d/default.conf /etc/nginx/conf.d/default.conf.disabled
        
      3. Restart NGINX to apply the Buildbot reverse proxy configuration:

        sudo systemctl restart nginx
        
      4. Navigate to your Buildbot instance’s URL over HTTPS. You will see the Buildbot homepage:

        Buildbot Landing Page

        Your continuous integration test server is now up and running.

      5. Ensure that you can log into your Buildbot instance with the admin credentials you created in the Configure Buildbot Master section. Click on the top right hand dropdown menu entitled Anonymous and then, click on Login. A Sign In modal will appear. Enter your credentials to log in to Buildbot as the admin user.

      Install the Buildbot Worker

      In order for Buildbot to execute test builds, the Buildbot master will require a worker. The following steps will setup a worker on the same host as the master.

      1. Install the buildbot-slave Ubuntu package:

        sudo apt-get install -y buildbot-slave
        
      2. Navigate to the directory which will store the Buildbot worker configurations:

        cd /var/lib/buildbot/workers
        
      3. Create the configuration directory for the Buildbot worker. Replace example-worker and my-worker-password with the values used for the c[worker] configuration in the master.cfg file.

        sudo -u buildbot -- buildbot-worker create-worker default localhost example-worker my-worker-password
        
      4. The Buildbot worker is ready to connect to the Buildbot master. Enable the worker process.

        sudo systemctl enable --now buildbot-worker@default.service
        

        Confirm that the worker has connected by going to your Buildbot site and navigating to Builds -> Workers in the sidebar menu:

        Buildbot Workers Page

      Configuring Builds

      Now that Buildbot is installed, you can configure it to run builds. In this tutorial, we will use a forked GitHub repository for the Linode Guides and Tutorials repository to illustrate how to use Buildbot as a system to run tests against a repository.

      Configuring GitHub

      Before creating the build configuration, fork the linode/docs repository into your GitHub account. This is the repository that will be used to run tests against. The repository will also require webhooks to be configured to send push or PR events to Buildbot.

      Note

      The actions you take to fork, add webhook, and push changes to your fork of linode/docs will not affect the parent (or upstream), so you can safely experiment with it. Any changes you make to branches of your fork will remain separate until you submit a pull request to the original linode/docs repository.

      Forking and Configuring the Repository

      1. Log in to your GitHub account and navigate to https://github.com/linode/docs. Click the Fork button:

        GitHub Fork Button

      2. Choose the account to fork the repository into (typically just your username). GitHub will bring you to the page for your own fork of the linode/docs repository.

        Select Settings to browse your fork’s settings:

        GitHub Fork Settings

        Then, select Webhooks from the sidebar:

        GitHub Webhook Settings

      3. Click on the Add webhook button. There are several fields to populate:

        • Under Payload URL enter the domain name for your Buildbot server with the change hook URL path appended to it: https://example.com/change_hook/github.
        • Leave the default value for Content type: application/x-www-form-urlencoded.
        • Under the Secret field, enter the secret value for the c['www']['change_hook_dialects'] option you configure in the master.cfg file.
        • Leave Enable SSL Verification selected.
        • For the Which events would you like to trigger this webhook?, select Let me select individual events and ensure that only the following boxes are checked:
        • Leave Active selected to indicate that GitHub should be configured to send webhooks to Buildbot.
      4. Click on the Add webhook button to save your settings.

        GitHub will return your browser to the list of webhooks for your repository. After configuring a new webhook, GitHub will send a test webhook to the configured payload URL. To indicate whether GitHub was able to send a webhook without errors, it adds a checkmark to the webhook item:

        GitHub Webhook Success

        Github will now send any new pushes made to your fork to your instance of Buildbot for testing.

      Build Prerequisites

      This guide runs builds as a simple process on the Buildbot worker, however, it is possible to execute builds within a Docker container, if desired. Consult the official Buildbot documentation for more information on configuring a Docker set up.

      Most software projects will define several prerequisites and tests for a project build. The Linode Guides and Tutorials repository defines several different tests to run for each build. This example will use one test defined in a python script named blueberry.py. This test checks for broken links, missing images, and more. This test’s dependencies can be installed via pip in a virtualenv.

      On your Linode, install the packages necessary to permit the worker to use a Python virtualenv to create a sandbox during the build.

      sudo apt-get install -y build-essential python3-dev python3-venv
      

      Writing Builds

      The /var/lib/buildbot/masters/ci/master.cfg file contains options to configure builds. The specific sections in the file that include these configurations are the following:

      • WORKERS, define the worker executors the master will connect to in order to run builds.
      • SCHEDULERS, specify how to react to incoming changes.
      • BUILDERS, outline the steps and build tests to run.

      Because the worker has already been configured and connected to the Buildbot master, the only settings necessary to define a custom build are the SCHEDULERS and BUILDERS.

      1. Add the following lines to the end of the /var/lib/buildbot/masters/ci/master.cfg file to define the custom build. Ensure you replace my-username and my-git-repo-name with the values for your own GitHub fork of the linode/docs repository and example-worker with the name of your Buildbot instance’s worker:

        /var/lib/buildbot/masters/ci/master.cfg
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        16
        17
        18
        19
        20
        21
        22
        23
        24
        
        docs_blueberry_test = util.BuildFactory()
        # Clone the repository
        docs_blueberry_test.addStep(
            steps.Git(
                repourl='git://github.com/my-username/my-git-repo-name.git',
                mode='incremental'))
        # Create virtualenv
        docs_blueberry_test.addStep(
            steps.ShellCommand(
                command=["python3", "-m", "venv", ".venv"]))
        # Install test dependencies
        docs_blueberry_test.addStep(
            steps.ShellCommand(
                command=["./.venv/bin/pip", "install", "-r", "ci/requirements.txt"]))
        # Run tests
        docs_blueberry_test.addStep(
            steps.ShellCommand(
                command=["./.venv/bin/python3", "ci/blueberry.py"]))
        # Add the BuildFactory configuration to the master
        c['builders'].append(
            util.BuilderConfig(name="linode-docs",
              workernames=["example-worker"],
              factory=docs_blueberry_test))
            

        The configuration code does the following:

        • A new Build Factory is instantiated. Build Factories define how builds are run.
        • Then, instructions are added to the Build Factory. The Build Factory clones the GitHub fork of the linode/docs repository.
        • Next, a Python virtualenv is setup. This ensures that the dependencies and libraries used for testing are kept separate, in a dedicated sandbox, from the Python libraries on the worker machine.
        • The necessary Python packages used in testing are then installed into the build’s virtualenv.
        • Finally, the blueberry.py testing script is run using the python3 executable from the virtualenv sandbox.
        • The defined Build Factory is then added to the configuration for the master.
      2. Define a simple scheduler to build any branch that is pushed to the GitHub repository. Add the following lines to the end of the master.cfg file:

        ~/buildbox-master/master/master.cfg
        1
        2
        3
        4
        5
        
            ...
        c['schedulers'].append(schedulers.AnyBranchScheduler(
            name="build-docs",
            builderNames=["linode-docs"]))
            

        This code instructs the Buildbot master to create a scheduler that builds any branch for the linode-docs builder. This scheduler will be invoked by the change hook defined for GitHub, which is triggered by the GitHub webhook configured in the GitHub interface.

      3. Restart the Buildbot master now that the custom scheduler and builder have been defined:

        sudo systemctl restart buildmaster@ci.service
        

      Running Builds

      Navigate to your Buildbot site to view the Builder and Scheduler created in the previous section. In the sidebar click on Build -> Builders. You will see linode-docs listed under the Builder Name heading:

      Buildbot Custom Builder

      A new build can be started for the linode-docs builder. Recall that the GitHub webhook configuration for your fork of linode/docs is set to call Buildbot upon any push or pull request event. To demonstrate how this works:

      1. Clone your fork of the linode/docs repository on your local machine (do not run the following commands on your Buildbot server) and navigate into the cloned repository. Replace username and repository with your own fork’s values:

        git clone https://github.com/username/repository.git
        cd repository
        
      2. Like many git repositories, the linode-docs repository changes often. To ensure that the remaining instructions work as expected, start at a specific revision in the code that is in a known state. Check out revision 76cd31a5271b41ff5a80dee2137dcb5e76296b93:

        git checkout 76cd31a5271b41ff5a80dee2137dcb5e76296b93
        
      3. Create a branch starting at this revision, which is where you will create dummy commits to test your Buildbot master:

        git checkout -b linode-tutorial-demo
        
      4. Create an empty commit so that you have something to push to your fork:

        git commit --allow-empty -m 'Buildbot test'
        
      5. Push your branch to your forked remote GitHub repository:

        git push --set-upstream origin linode-tutorial-demo
        
      6. Navigate to your Buildbot site and go to your running builds. The Home button on the sidebar displays currently executing builds.

        Buildbot running Builds

      7. Click on the running build to view more details. The build will display each step along with logging output:

        Buildbot Build Page

        Each step of the build process can be followed as the build progresses. While the build is running, click on a step to view standard output logs. A successful build will complete each step with an exit code of 0.

        Your Buildbot host will now actively build pushes to any branch or any pull requests to your repository.

      Features to Explore

      Now that you have a simple build configuration for your Buildbot instance, you can continue to add features to your CI server. Some useful functions that Buildbot supports include:

      • Reporters, which can notify you about build failures over IRC, GitHub comments, or email.
      • Workers that execute builds in Docker containers or in temporary cloud instances instead of static hosts.
      • Web server features, including the ability to generate badges for your repository indicating the current build status of the project.

      More Information

      You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

      Join our Community

      Find answers, ask questions, and help others.

      This guide is published under a CC BY-ND 4.0 license.



      Source link