One place for hosting & domains

      Jenkins

      How To Set Up Continuous Integration Pipelines in Jenkins on Ubuntu 20.04


      Introduction

      Jenkins is an open source automation server intended to automate repetitive technical tasks involved in the continuous integration and delivery of software. With a robust ecosystem of plugins and broad support, Jenkins can handle a diverse set of workloads to build, test, and deploy applications.

      In previous guides, we installed Jenkins on an Ubuntu 20.04 server and configured Jenkins with SSL using an Nginx reverse proxy. In this guide, we will demonstrate how to set up Jenkins to automatically test an application when changes are pushed to a repository.

      For this tutorial, we will be integrating Jenkins with GitHub so that Jenkins is notified when new code is pushed to the repository. When Jenkins is notified, it will checkout the code and then test it within Docker containers to isolate the test environment from the Jenkins host machine. We will be using an example Node.js application to show how to define the CI/CD process for a project.

      Prerequisites

      To follow along with this guide, you will need an Ubuntu 20.04 server with at least 1G of RAM configured with a secure Jenkins installation. To properly secure the web interface, you will need to assign a domain name to the Jenkins server. Follow these guides to learn how to set up Jenkins in the expected format:

      To best control our testing environment, we will run our application’s tests within Docker containers. After Jenkins is up and running, install Docker on the server by following steps one and two of this guide:

      When you have completed the above guides, you can continue on with this article.

      Add the Jenkins User to the Docker Group

      After following the prerequisites, both Jenkins and Docker are installed on your server. However, by default, the Linux user responsible for running the Jenkins process cannot access Docker.

      To fix this, we need to add the jenkins user to the docker group using the usermod command:

      • sudo usermod -aG docker jenkins

      You can list the members of the docker group to confirm that the jenkins user has been added successfully:

      Output

      docker:x:999:sammy,jenkins

      In order for the Jenkins to use its new membership, you need to restart the process:

      • sudo systemctl restart jenkins

      If you installed Jenkins with the default plugins, you may need to check to ensure that the docker and docker-pipeline plugins are also enabled. To do so, click Manage Jenkins from the sidebar, and then Manage Plugins from the next menu. Click on the Available tab of the plugin menu to search for new plugins, and type docker into the search bar. If both Docker Pipeline and Docker plugin are returned as options, and they are unselected, select both, and when prompted, allow Jenkins to restart with the new plugins enabled.

      Jenkins Docker plugins

      This should take approximately a minute and the page will refresh afterward.

      Create a Personal Access Token in GitHub

      In order for Jenkins to watch your GitHub projects, you will need to create a Personal Access Token in our GitHub account.

      Begin by visiting GitHub and signing into your account if you haven’t already done so. Afterwards, click on your user icon in the upper-right hand corner and select Settings from the drop down menu:

      GitHub settings item

      On the page that follows, locate the Developer settings section of the left-hand menu and click Personal access tokens:

      GitHub personal access tokens link

      Click on Generate new token button on the next page:

      GitHub generate new token button

      You will be taken to a page where you can define the scope for your new token.

      In the Token description box, add a description that will allow you to recognize it later:

      GitHub token description

      In the Select scopes section, check the repo:status, repo:public_repo and admin:org_hook boxes. These will allow Jenkins to update commit statuses and to create webhooks for the project. If you are using a private repository, you will need to select the general repo permission instead of the repo subitems:

      GitHub token scope

      When you are finished, click Generate token at the bottom.

      You will be redirected back to the Personal access tokens index page and your new token will displayed:

      GitHub view new token

      Copy the token now so that we can reference it later. As the message indicates, there is no way to retrieve the token once you leave this page.

      Note: As mentioned in the screenshot above, for security reasons, there is no way to redisplay the token once you leave this page. If you lose your token, delete the current token from your GitHub account and then create a new one.

      Now that you have a personal access token for your GitHub account, we can configure Jenkins to watch your project’s repository.

      Add the GitHub Personal Access Token to Jenkins

      Now that we have a token, we need to add it to our Jenkins server so it can automatically set up webhooks. Log into your Jenkins web interface using the administrative account you configured during installation.

      Click on your username in the top-right corner to access your user settings, and from there, click Credentials in the left-hand menu. :

      Jenkins credentials item

      On the next page, click the arrow next to (global) within the Jenkins scope. In the box that appears, click Add credentials:

      Jenkins add credentials button

      You will be taken to a form to add new credentials.

      Under the Kind drop down menu, select Secret text. In the Secret field, paste your GitHub personal access token. Fill out the Description field so that you will be able to identify this entry at a later date. You can leave the Scope as Global and the ID field blank:

      Jenkins credentials form

      Click the OK button when you are finished.

      You will now be able to reference these credentials from other parts of Jenkins to aid in configuration.

      Set Up Jenkins Access to GitHub

      Back in the main Jenkins dashboard, click Manage Jenkins in the left hand menu:

      Jenkins sidebar

      In the list of links on the following page, click Configure System:

      Jenkins configure system link

      Scroll through the options on the next page until you find the GitHub section. Click the Add GitHub Server button and then select GitHub Server:

      Jenkins add GitHub server

      The section will expand to prompt for some additional information. In the Credentials drop down menu, select your GitHub personal access token that you added in the last section:

      Jenkins select GitHub credentials

      Click the Test connection button. Jenkins will make a test API call to your account and verify connectivity:

      Jenkins test GitHub credentials

      When you are finished, click the Save button to implement your changes.

      Set Up the Demonstration Application in your GitHub Account

      To demonstrate how to use Jenkins to test an application, we will be using a “hello world” program created with Hapi.js. Because we are setting up Jenkins to react to pushes to the repository, you need to have your own copy of the demonstration code.

      Visit the project repository and click the Fork button in the upper-right corner to make a copy of the repository in your account:

      Fork example project

      A copy of the repository will be added to your account.

      The repository contains a package.json file that defines the runtime and development dependencies, as well as how to run the included test suite. The dependencies can be installed by running npm install and the tests can be run using npm test.

      We’ve added a Jenkinsfile to the repo as well. Jenkins reads this file to determine the actions to run against the repository to build, test, or deploy. It is written using the declarative version of the Jenkins Pipeline DSL.

      The Jenkinsfile included in the hello-hapi repository looks like this:

      Jenkinsfile

      #!/usr/bin/env groovy
      
      pipeline {
      
          agent {
              docker {
                  image 'node'
                  args '-u root'
              }
          }
      
          stages {
              stage('Build') {
                  steps {
                      echo 'Building...'
                      sh 'npm install'
                  }
              }
              stage('Test') {
                  steps {
                      echo 'Testing...'
                      sh 'npm test'
                  }
              }
          }
      }
      

      The pipeline contains the entire definition that Jenkins will evaluate. Inside, we have an agent section that specifies where the actions in the pipeline will execute. To isolate our environments from the host system, we will be testing in Docker containers, specified by the docker agent.

      Since Hapi.js is a framework for Node.js, we will be using the node Docker image as our base. We specify the root user within the container so that the user can simultaneously write to both the attached volume containing the checked out code, and to the volume the script writes its output to.

      Next, the file defines two stages, i.e., logical divisions of work. We’ve named the first one “Build” and the second “Test”. The build step prints a diagnostic message and then runs npm install to obtain the required dependencies. The test step prints another message and then run the tests as defined in the package.json file.

      Now that you have a repository with a valid Jenkinsfile, we can set up Jenkins to watch this repository and run the file when changes are introduced.

      Create a New Pipeline in Jenkins

      Next, we can set up Jenkins to use the GitHub personal access token to watch our repository.

      Back in the main Jenkins dashboard, click New Item in the left hand menu:

      Jenkins side menu

      Enter a name for your new pipeline in the Enter an item name field. Afterwards, select Pipeline as the item type:

      Jenkins pipeline type

      Click the OK button at the bottom to move on.

      On the next screen, check the GitHub project box. In the Project url field that appears, enter your project’s GitHub repository URL.

      Note: Make sure to point to your fork of the Hello Hapi application so that Jenkins will have permission to configure webhooks.

      Jenkins add GitHub project

      Next, in the Build Triggers section, check the GitHub hook trigger for GITScm polling box:

      Jenkins GitHub hook box

      In the Pipeline section, we need to tell Jenkins to run the pipeline defined in the Jenkinsfile in our repository. Change the Definition type to Pipeline script from SCM.

      In the new section that appears, choose Git in the SCM menu. In the Repository URL field that appears, enter the URL to your fork of the repository again:

      Note: Again, make sure to point to your fork of the Hello Hapi application.

      Jenkins GitHub add pipeline repository

      Note: Our example references a Jenkinsfile available within a public repository. If your project is not publicly accessible, you will need to use the add credentials button to add additional access to the repository. You can add a personal access token as we did with the hooks configuration earlier.

      When you are finished, click the Save button at the bottom of the page.

      Performing an Initial Build and Configuring the Webhooks

      Jenkins does not automatically configure webhooks when you define the pipeline for the repository in the interface. In order to trigger Jenkins to set up the appropriate hooks, we need to perform a manual build the first time.

      In your pipeline’s main page, click Build Now in the left hand menu:

      Jenkins build pipeline now

      A new build will be scheduled. In the Build History box in the lower left corner, a new build should appear in a moment. Additionally, a Stage View will begin to be drawn in the main area of the interface. This will track the progress of your testing run as the different stages are completed:

      Jenkins build progress

      In the Build History box, click on the number associated with the build to go to the build detail page. From here, you can click the Console Output button in the left hand menu to see details of the steps that were run:

      Jenkins console output

      Click the Back to Project item in the left hand menu when you are finished in order to return to the main pipeline view.

      Now that we’ve built the project once, we can have Jenkins create the webhooks for our project. Click Configure in the left hand menu of the pipeline:

      Jenkins configure item

      No changes are necessary on this screen, just click the Save button at the bottom. Now that the Jenkins has information about the project from the initial build process, it will register a webhook with our GitHub project when you save the page.

      You can verify this by going to your GitHub repository and clicking the Settings button. On the next page, click Webhooks from the side menu. You should see your Jenkins server webhook in the main interface:

      Jenkins view webhooks

      If for any reason Jenkins failed to register the hook (for example, due to upstream API changes or outages between Jenkins and Github), you can quickly add one yourself by clicking Add webhook and ensuring that the Payload URL is set to https://my-jenkins-server:8080/github-webhook and the Content type is set to application/json, then clicking Add webhook again at the bottom of the prompt.

      Github add webhook

      Now, when you push new changes to your repository, Jenkins will be notified. It will then pull the new code and retest it using the same procedure.

      To approximate this, in our repository page on GitHub, you can click the Create new file button to the left of the green Clone or download button:

      Jenkins create new file button

      On the next page, choose a filename and some dummy contents:

      Jenkins new file contents

      Click the Commit new file button at the bottom when you are finished.

      If you return to your Jenkins interface, you will see a new build automatically started:

      Jenkins new build started

      You can kick off additional builds by making commits to a local copy of the repository and pushing it back up to GitHub.

      Conclusion

      In this guide, we configured Jenkins to watch a GitHub project and automatically test any new changes that are committed. Jenkins pulls code from the repository and then runs the build and testing procedures from within isolated Docker containers. The resulting code can be deployed or stored by adding additional instructions to the same Jenkinsfile.

      To learn more about GitHub Actions, refer to GitHub’s documentation.



      Source link

      How To Automate Jenkins Job Configuration Using Job DSL


      The author selected the Internet Archive to receive a donation as part of the Write for DOnations program.

      Introduction

      Jenkins is a popular automation server, often used to orchestrate continuous integration (CI) and continuous deployment (CD) workflows. However, the process of setting up Jenkins itself has traditionally been a manual, siloed process for the system administrator. The process typically involves installing dependencies, running the Jenkins server, configuring the server, defining pipelines, and configuring jobs.

      Then came the Everything as Code (EaC) paradigm, which allowed administrators to define these manual tasks as declarative code that can be version-controlled and automated. In previous tutorials, we covered how to define Jenkins pipelines as code using Jenkinsfiles, as well as how to install dependencies and define configuration of a Jenkins server as code using Docker and JCasC. But using only Docker, JCasC, and pipelines to set up your Jenkins instance would only get you so far—these servers would not come pre-loaded with any jobs, so someone would still have to configure them manually. The Job DSL plugin provides a solution, and allows you to configure Jenkins jobs as code.

      In this tutorial, you’ll use Job DSL to configure two demo jobs: one that prints a 'Hello World' message in the console, and one that runs a pipeline from a Git repository. If you follow the tutorial to the end, you will have a minimal Job DSL script that you can build on for your own use cases.

      Prerequisites

      To complete this tutorial, you will need:

      Step 1 — Installing the Job DSL Plugin

      The Job DSL plugin provides the Job DSL features you’ll use in this tutorial for your demo jobs. In this step, you will install the Job DSL plugin.

      First, navigate to your_jenkins_url/pluginManager/available. In the search box, type in Job DSL. Next, in the resulting plugins list, check the box next to Job DSL and click Install without restart.

      Plugin Manager page showing Job DSL checked

      Note: If searching for Job DSL returned no results, it either means the Job DSL plugin is already installed, or that your Jenkin server’s plugin list is not updated.

      You can check if the Job DSL plugin is already installed by navigating to your_jenkins_url/pluginManager/installed and searching for Job DSL.

      You can update your Jenkins server’s plugin list by navigating to your_jenkins_url/pluginManager/available and clicking on the Check Now button at the bottom of the (empty) plugins list.

      After initiating the installation process, you’ll be redirected to a page that shows the progress of the installation. Wait until you see Success next to both Job DSL and Loading plugin extensions before continuing to the next step.

      You’ve installed the Job DSL plugin. You are now ready to use Job DSL to configure jobs as code. In the next step, you will define a demo job inside a Job DSL script. You’ll then incorporate the script into a seed job, which, when executed, will create the jobs defined.

      Step 2 — Creating a Seed Job

      The seed job is a normal Jenkins job that runs the Job DSL script; in turn, the script contains instructions that create additional jobs. In short, the seed job is a job that creates more jobs. In this step, you will construct a Job DSL script and incorporate it into a seed job. The Job DSL script that you’ll define will create a single freestyle job that prints a 'Hello World!' message in the job’s console output.

      A Job DSL script consists of API methods provided by the Job DSL plugin; you can use these API methods to configure different aspects of a job, such as its type (freestyle versus pipeline jobs), build triggers, build parameters, post-build actions, and so on. You can find all supported methods on the API reference site.

      Jenkins Job DSL API Reference web page

      By default, the site shows the API methods for job configuration settings that are available as part of the core Jenkins installation, as well as settings that are enabled by 184 supported plugins (accurate as of v1.77). To get a clearer picture of what API methods the Job DSL plugin provides for only the core Jenkins installation, click on the funnel icon next to the search box, and then check and uncheck the Filter by Plugin checkbox to deselect all the plugins.

      Jenkins Job DSL API reference web page showing only the core APIs

      The list of API methods are now significantly reduced. The ones that remain would work even if the Jenkins installation had no plugins installed apart from the Job DSL plugin.

      For the ‘Hello World’ freestyle job, you need the job API method (freeStyleJob is an alias of job and would also work). Let’s navigate to the documentation for the job method.

      job API method reference

      Click the ellipsis icon () in job(String name) { … } to show the methods and blocks that are available within the job block.

      Expanded view of the job API method reference

      Let’s go over some of the most commonly used methods and blocks within the job block:

      • parameters: setting parameters for users to input when they create a new build of the job.
      • properties: static values that are to be used within the job.
      • scm: configuration for how to retrieve the source code from a source-control management provider like GitHub.
      • steps: definitions for each step of the build.
      • triggers: apart from manually creating a build, specifies in what situations the job should be run (for example, periodically like a cron job, or after some events like a push to a GitHub repository).

      You can further expand child blocks to see what methods and blocks are available within. Click on the ellipsis icon () in steps { … } to uncover the shell(String command) method, which you can use to run a shell script.

      Reference for the Job DSL steps block

      Putting the pieces together, you can write a Job DSL script like the following to create a freestyle job that, when run, will print 'Hello World!' in the output console.

      job('demo') {
          steps {
              shell('echo Hello World!')
          }
      }
      

      To run the Job DSL script, we must first incorporate it into a seed job.

      To create the seed job, go to your_jenkins_url, log in (if necessary), click the New Item link on the left of the dashboard. On the screen that follows, type in seed, select Freestyle project, and click OK.

      Part of the New Item screen where you give the item the name of 'seed' and with the 'Freestyle project' option selected

      In the screen that follows, scroll down to the Build section and click on the Add build step dropdown. Next select Process Job DSLs.

      Screen showing the Add build step dropdown expanded and the Process Job DSLs option selected

      Then, click on the radio button next to Use the provided DSL script, and paste the Job DSL script you wrote into the DSL Script text area.

      Job DSL script added to the Process Job DSLs build step

      Click Save to create the job. This will take you to the seed job page.

      Seed job page

      Then, navigate to your_jenkins_url and confirm that the seed job is there.

      Jenkins jobs list showing the seed job

      You’ve successfully created a seed job that incorporates your Job DSL script. In the next step, you will run the seed job so that new jobs are created based on your Job DSL script.

      Step 3 — Running the Seed Job

      In this step, you will run the seed job and confirm that the jobs defined within the Job DSL script are indeed created.

      First, click back into the seed job page and click on the Build Now button on the left to run the seed job.

      Refresh the page and you’ll see a new section that says Generated Items; it lists the demo job that you’ve specified in your Job DSL script.

      Seed job page showing a list of generated items from running the seed job

      Navigate to your_server_ip and you will find the demo job that you specified in the Job DSL script.

      Jenkins jobs list showing the demo and seed jobs

      Click the demo link to go to the demo job page. You’ll see Seed job: seed, indicating that this job is created by the seed job. Now, click the Build Now link to run the demo job once.

      Demo job page showing a section on seed job

      This creates an entry inside the Build History box. Hover over the date of the entry to reveal a little arrow; click on it to reveal the dropdown. From the dropdown, choose Console Output.

      Screen showing the Console Output option selected in the dropdown for Build #1 inside the Build History box

      This will bring you the logs and console output from this build. In it, you will find the line + echo Hello World! followed by Hello World!, which corresponds to the shell('echo Hello World!') step in your Job DSL script.

      Console output of build #1 showing the echo Hello World! command and output

      You’ve run the demo job and confirmed that the echo step specified in the Job DSL script was executed. In the next and final step, you will be modifying and re-applying the Job DSL script to include an additional pipeline job.

      Step 4 — Defining Pipeline Jobs

      In line with the Everything as Code paradigm, more and more developers are choosing to define their builds as pipeline jobs—those that use a pipeline script (typically named Jenkinsfile)—instead of freestyle jobs. The demo job you’ve defined so far is a small demonstration. In this step, you will define a more realistic job that pulls down a Git repository from GitHub and run a pipeline defined in one of its pipeline scripts.

      For Jenkins to pull a Git repository and build using pipeline scripts, you’ll need to install additional plugins. So, before you make any changes to the Job DSL script, first make sure that the required plugins are installed.

      Navigate to your_jenkins_url/pluginManager/installed and check the plugins lists for the presence of the Git, Pipeline: Job, and Pipeline: Groovy plugins. If any of them are not installed, go to your_jenkins_url/pluginManager/available and search for and select the plugins, then click Install without restart.

      Now that the required plugins are installed, let’s shift our focus to modifying your Job DSL script to include an additional pipeline job.

      We will be defining a pipeline job that pulls the code from the public jenkinsci/pipeline-examples Git repository and run the environmentInStage.groovy declarative pipeline script found in it.

      Once again, navigate to the Jenkins Job DSL API Reference, click the funnel icon to bring up the Filter by Plugin menu, then deselect all the plugins except Git, Pipeline: Job, and Pipeline: Groovy.

      The Jenkins Job DSL API Reference page with all plugins deselected except for Pipeline: Job, and (not shown) Git and Pipeline: Groovy

      Click on pipelineJob on the left-hand side menu and expand the pipelineJob(String name) { … } block, then, in order, the definition { … }, cpsScm { … }, and scm { … } blocks.

      Expanded view of the pipelineJob API method block

      There are comments above each API method that explain their roles. For our use case, you’d want to define your pipeline job using a pipeline script found inside a GitHub repository. So you’d need to modify your Job DSL script as follows:

      job('demo') {
          steps {
              shell('echo Hello World!')
          }
      }
      
      pipelineJob('github-demo') {
          definition {
              cpsScm {
                  scm {
                      git {
                          remote {
                              github('jenkinsci/pipeline-examples')
                          }
                      }
                  }
                  scriptPath('declarative-examples/simple-examples/environmentInStage.groovy')
              }
          }
      }
      

      To make the change, go to your_jenkins_url/job/seed/configure and find the DSL Script text area, and replace the contents with your new Job DSL script. Then press Save. In the next screen, click on Build Now to re-run the seed job.

      Then, go to the Console Output page of the new build and you’ll find Added items: GeneratedJob{name="github-demo"}, which means you’ve successfully added the new pipeline job, whilst the existing job remains unchanged.

      Console output for the modified seed job, showing that the github-demo job has been added

      You can confirm this by going to your_jenkins_url; you will find the github-demo job appear in the list of jobs.

      Job list showing the github-demo job

      Finally, confirm that your job is working as intended by navigating to your_jenkins_url/job/github-demo/ and clicking Build Now. After the build has finished, navigate to your_jenkins_url/job/github-demo/1/console and you will find the Console Output page showing that Jenkins has successfully cloned the repository and executed the pipeline script.

      Conclusion

      In this tutorial, you’ve used the Job DSL plugin to configure jobs on Jenkins servers in a consistent and repeatable way.

      But Job DSL is not the only tool in the Jenkins ecosystem that follows the Everything as Code (EaC) paradigm. You can also deploy Jenkins as Docker containers and set it up using Jenkins Configuration as Code (JCasC). Together, Docker, JCasC, Job DSL, and pipelines allow developers and administrators to deploy and configure Jenkins completely automatically, without any manual involvement.



      Source link

      How To Automate Jenkins Setup with Docker and Jenkins Configuration as Code


      The author selected the Wikimedia Foundation to receive a donation as part of the Write for DOnations program.

      Introduction

      Jenkins is one of the most popular open-source automation servers, often used to orchestrate continuous integration (CI) and/or continuous deployment (CD) workflows.

      Configuring Jenkins is typically done manually through a web-based setup wizard; this can be a slow, error-prone, and non-scalable process. You can see the steps involved by following Step 4 — Setting Up Jenkins of the How To Install Jenkins on Ubuntu 18.04 guide. Furthermore, configurations cannot be tracked in a version control system (VCS) like Git, nor be under the scrutiny of any code review process.

      In this tutorial, you will automate the installation and configuration of Jenkins using Docker and the Jenkins Configuration as Code (JCasC) method.

      Jenkins uses a pluggable architecture to provide most of its functionality. JCasC makes use of the Configuration as Code plugin, which allows you to define the desired state of your Jenkins configuration as one or more YAML file(s), eliminating the need for the setup wizard. On initialization, the Configuration as Code plugin would configure Jenkins according to the configuration file(s), greatly reducing the configuration time and eliminating human errors.

      Docker is the de facto standard for creating and running containers, which is a virtualization technology that allows you to run isolated, self-contained applications consistently across different operation systems (OSes) and hardware architectures. You will run your Jenkins instance using Docker to take advantage of this consistency and cross-platform capability.

      This tutorial starts by guiding you through setting up JCasC. You will then incrementally add to the JCasC configuration file to set up users, configuration authentication and authorization, and finally to secure your Jenkins instance. After you’ve completed this tutorial, you’ll have created a custom Docker image that is set up to use the Configuration as Code plugin on startup to automatically configure and secure your Jenkins instance.

      Prerequisites

      To complete this tutorial, you will need:

      • Access to a server with at least 2GB of RAM and Docker installed. This can be your local development machine, a Droplet, or any kind of server. Follow Step 1 — Installing Docker from one of the tutorials in the How to Install and Use Docker collection to set up Docker.

      Note: This tutorial is tested on Ubuntu 18.04; however, because Docker images are self-contained, the steps outlined here would work for any OSes with Docker installed.

      Step 1 — Disabling the Setup Wizard

      Using JCasC eliminates the need to show the setup wizard; therefore, in this first step, you’ll create a modified version of the official jenkins/jenkins image that has the setup wizard disabled. You will do this by creating a Dockerfile and building a custom Jenkins image from it.

      The jenkins/jenkins image allows you to enable or disable the setup wizard by passing in a system property named jenkins.install.runSetupWizard via the JAVA_OPTS environment variable. Users of the image can pass in the JAVA_OPTS environment variable at runtime using the --env flag to docker run. However, this approach would put the onus of disabling the setup wizard on the user of the image. Instead, you should disable the setup wizard at build time, so that the setup wizard is disabled by default.

      You can achieve this by creating a Dockerfile and using the ENV instruction to set the JAVA_OPTS environment variable.

      First, create a new directory inside your server to store the files you will be creating in this tutorial:

      • mkdir -p $HOME/playground/jcasc

      Then, navigate inside that directory:

      • cd $HOME/playground/jcasc

      Next, using your editor, create a new file named Dockerfile:

      • nano $HOME/playground/jcasc/Dockerfile

      Then, copy the following content into the Dockerfile:

      ~/playground/jcasc/

      FROM jenkins/jenkins:latest
      ENV JAVA_OPTS -Djenkins.install.runSetupWizard=false
      

      Here, you’re using the FROM instruction to specify jenkins/jenkins:latest as the base image, and the ENV instruction to set the JAVA_OPTS environment variable.

      Save the file and exit the editor by pressing CTRL+X followed by Y.

      With these modifications in place, build a new custom Docker image and assign it a unique tag (we’ll use jcasc here):

      • docker build -t jenkins:jcasc .

      You will see output similar to the following:

      Output

      Sending build context to Docker daemon 2.048kB Step 1/2 : FROM jenkins/jenkins:latest ---> 1f4b0aaa986e Step 2/2 : ENV JAVA_OPTS -Djenkins.install.runSetupWizard=false ---> 7566b15547af Successfully built 7566b15547af Successfully tagged jenkins:jcasc

      Once built, run your custom image by running docker run:

      • docker run --name jenkins --rm -p 8080:8080 jenkins:jcasc

      You used the --name jenkins option to give your container an easy-to-remember name; otherwise a random hexadecimal ID would be used instead (e.g. f1d701324553). You also specified the --rm flag so the container will automatically be removed after you’ve stopped the container process. Lastly, you’ve configured your server host’s port 8080 to proxy to the container’s port 8080 using the -p flag; 8080 is the default port where the Jenkins web UI is served from.

      Jenkins will take a short period of time to initiate. When Jenkins is ready, you will see the following line in the output:

      Output

      ... hudson.WebAppMain$3#run: Jenkins is fully up and running

      Now, open up your browser to server_ip:8080. You’re immediately shown the dashboard without the setup wizard.

      The Jenkins dashboard

      You have just confirmed that the setup wizard has been disabled. To clean up, stop the container by pressing CTRL+C. If you’ve specified the --rm flag earlier, the stopped container would automatically be removed.

      In this step, you’ve created a custom Jenkins image that has the setup wizard disabled. However, the top right of the web interface now shows a red notification icon indicating there are issues with the setup. Click on the icon to see the details.

      The Jenkins dashboard showing issues

      The first warning informs you that you have not configured the Jenkins URL. The second tells you that you haven’t configured any authentication and authorization schemes, and that anonymous users have full permissions to perform all actions on your Jenkins instance. Previously, the setup wizard guided you through addressing these issues. Now that you’ve disabled it, you need to replicate the same functions using JCasC. The rest of this tutorial will involve modifying your Dockerfile and JCasC configuration until no more issues remain (that is, until the red notification icon disappears).

      In the next step, you will begin that process by pre-installing a selection of Jenkins plugins, including the Configuration as Code plugin, into your custom Jenkins image.

      Step 2 — Installing Jenkins Plugins

      To use JCasC, you need to install the Configuration as Code plugin. Currently, no plugins are installed. You can confirm this by navigating to http://server_ip:8080/pluginManager/installed.

      Jenkins dashboard showing no plugins are installed

      In this step, you’re going to modify your Dockerfile to pre-install a selection of plugins, including the Configuration as Code plugin.

      To automate the plugin installation process, you can make use of an installation script that comes with the jenkins/jenkins Docker image. You can find it inside the container at /usr/local/bin/install-plugins.sh. To use it, you would need to:

      • Create a text file containing a list of plugins to install
      • Copy it into the Docker image
      • Run the install-plugins.sh script to install the plugins

      First, using your editor, create a new file named plugins.txt:

      • nano $HOME/playground/jcasc/plugins.txt

      Then, add in the following newline-separated list of plugin names and versions (using the format <id>:<version>):

      ~/playground/jcasc/plugins.txt

      ant:latest
      antisamy-markup-formatter:latest
      build-timeout:latest
      cloudbees-folder:latest
      configuration-as-code:latest
      credentials-binding:latest
      email-ext:latest
      git:latest
      github-branch-source:latest
      gradle:latest
      ldap:latest
      mailer:latest
      matrix-auth:latest
      pam-auth:latest
      pipeline-github-lib:latest
      pipeline-stage-view:latest
      ssh-slaves:latest
      timestamper:latest
      workflow-aggregator:latest
      ws-cleanup:latest
      

      Save the file and exit your editor.

      The list contains the Configuration as Code plugin, as well as all the plugins suggested by the setup wizard (correct as of Jenkins v2.251). For example, you have the Git plugin, which allows Jenkins to work with Git repositories; you also have the Pipeline plugin, which is actually a suite of plugins that allows you to define Jenkins jobs as code.

      Note: The most up-to-date list of suggested plugins can be inferred from the source code. You can also find a list of the most popular community-contributed plugins at plugins.jenkins.io. Feel free to include any other plugins you want into the list.

      Next, open up the Dockerfile file:

      • nano $HOME/playground/jcasc/Dockerfile

      In it, add a COPY instruction to copy the plugins.txt file into the /usr/share/jenkins/ref/ directory inside the image; this is where Jenkins normally looks for plugins. Then, include an additional RUN instruction to run the install-plugins.sh script:

      ~/playground/jcasc/Dockerfile

      FROM jenkins/jenkins
      ENV JAVA_OPTS -Djenkins.install.runSetupWizard=false
      COPY plugins.txt /usr/share/jenkins/ref/plugins.txt
      RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt
      

      Save the file and exit the editor. Then, build a new image using the revised Dockerfile:

      • docker build -t jenkins:jcasc .

      This step involves downloading and installing many plugins into the image, and may take some time to run depending on your internet connection. Once the plugins have finished installing, run the new Jenkins image:

      • docker run --name jenkins --rm -p 8080:8080 jenkins:jcasc

      After the Jenkins is fully up and running message appears on stdout, navigate to server_ip:8080/pluginManager/installed to see a list of installed plugins. You will see a solid checkbox next to all the plugins you’ve specified inside plugins.txt, as well as a faded checkbox next to plugins, which are dependencies of those plugins.

      A list of installed plugins

      Once you’ve confirmed that the Configuration As Code plugin is installed, terminate the container process by pressing CTRL+C.

      In this step, you’ve installed all the suggested Jenkins plugins and the Configuration as Code plugin. You’re now ready to use JCasC to tackle the issues listed in the notification box. In the next step, you will fix the first issue, which warns you that the Jenkins root URL is empty.

      Step 3 — Specifying the Jenkins URL

      The Jenkins URL is a URL for the Jenkins instance that is routable from the devices that need to access it. For example, if you’re deploying Jenkins as a node inside a private network, the Jenkins URL may be a private IP address, or a DNS name that is resolvable using a private DNS server. For this tutorial, it is sufficient to use the server’s IP address (or 127.0.0.1 for local hosts) to form the Jenkins URL.

      You can set the Jenkins URL on the web interface by navigating to server_ip:8080/configure and entering the value in the Jenkins URL field under the Jenkins Location heading. Here’s how to achieve the same using the Configuration as Code plugin:

      1. Define the desired configuration of your Jenkins instance inside a declarative configuration file (which we’ll call casc.yaml).
      2. Copy the configuration file into the Docker image (just as you did for your plugins.txt file).
      3. Set the CASC_JENKINS_CONFIG environment variable to the path of the configuration file to instruct the Configuration as Code plugin to read it.

      First, create a new file named casc.yaml:

      • nano $HOME/playground/jcasc/casc.yaml

      Then, add in the following lines:

      ~/playground/jcasc/casc.yaml

      unclassified:
        location:
          url: http://server_ip:8080/
      

      unclassified.location.url is the path for setting the Jenkins URL. It is just one of a myriad of properties that can be set with JCasC. Valid properties are determined by the plugins that are installed. For example, the jenkins.authorizationStrategy.globalMatrix.permissions property would only be valid if the Matrix Authorization Strategy plugin is installed. To see what properties are available, navigate to server_ip:8080/configuration-as-code/reference, and you’ll find a page of documentation that is customized to your particular Jenkins installation.

      Save the casc.yaml file, exit your editor, and open the Dockerfile file:

      • nano $HOME/playground/jcasc/Dockerfile

      Add a COPY instruction to the end of your Dockerfile that copies the casc.yaml file into the image at /var/jenkins_home/casc.yaml. You’ve chosen /var/jenkins_home/ because that’s the default directory where Jenkins stores all of its data:

      ~/playground/jcasc/Dockerfile

      FROM jenkins/jenkins:latest
      ENV JAVA_OPTS -Djenkins.install.runSetupWizard=false
      COPY plugins.txt /usr/share/jenkins/ref/plugins.txt
      RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt
      COPY casc.yaml /var/jenkins_home/casc.yaml
      

      Then, add a further ENV instruction that sets the CASC_JENKINS_CONFIG environment variable:

      ~/playground/jcasc/Dockerfile

      FROM jenkins/jenkins:latest
      ENV JAVA_OPTS -Djenkins.install.runSetupWizard=false
      ENV CASC_JENKINS_CONFIG /var/jenkins_home/casc.yaml
      COPY plugins.txt /usr/share/jenkins/ref/plugins.txt
      RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt
      COPY casc.yaml /var/jenkins_home/casc.yaml
      

      Note: You’ve put the ENV instruction near the top because it’s something that you are unlikely to change. By placing it before the COPY and RUN instructions, you can avoid invalidating the cached layer if you were to update the casc.yaml or plugins.txt.

      Save the file and exit the editor. Next, build the image:

      • docker build -t jenkins:jcasc .

      And run the updated Jenkins image:

      • docker run --name jenkins --rm -p 8080:8080 jenkins:jcasc

      As soon as the Jenkins is fully up and running log line appears, navigate to server_ip:8080 to view the dashboard. This time, you may have noticed that the notification count is reduced by one, and the warning about the Jenkins URL has disappeared.

      Jenkins Dashboard showing the notification counter has a count of 1

      Now, navigate to server_ip:8080/configure and scroll down to the Jenkins URL field. Confirm that the Jenkins URL has been set to the same value specified in the casc.yaml file.

      Lastly, stop the container process by pressing CTRL+C.

      In this step, you used the Configuration as Code plugin to set the Jenkins URL. In the next step, you will tackle the second issue from the notifications list (the Jenkins is currently unsecured message).

      Step 4 — Creating a User

      So far, your setup has not implemented any authentication and authorization mechanisms. In this step, you will set up a basic, password-based authentication scheme and create a new user named admin.

      Start by opening your casc.yaml file:

      • nano $HOME/playground/jcasc/casc.yaml

      Then, add in the highlighted snippet:

      ~/playground/jcasc/casc.yaml

      jenkins:
        securityRealm:
          local:
            allowsSignup: false
            users:
             - id: ${JENKINS_ADMIN_ID}
               password: ${JENKINS_ADMIN_PASSWORD}
      unclassified:
        ...
      

      In the context of Jenkins, a security realm is simply an authentication mechanism; the local security realm means to use basic authentication where users must specify their ID/username and password. Other security realms exist and are provided by plugins. For instance, the LDAP plugin allows you to use an existing LDAP directory service as the authentication mechanism. The GitHub Authentication plugin allows you to use your GitHub credentials to authenticate via OAuth.

      Note that you’ve also specified allowsSignup: false, which prevents anonymous users from creating an account through the web interface.

      Finally, instead of hard-coding the user ID and password, you are using variables whose values can be filled in at runtime. This is important because one of the benefits of using JCasC is that the casc.yaml file can be committed into source control; if you were to store user passwords in plaintext inside the configuration file, you would have effectively compromised the credentials. Instead, variables are defined using the ${VARIABLE_NAME} syntax, and its value can be filled in using an environment variable of the same name, or a file of the same name that’s placed inside the /run/secrets/ directory within the container image.

      Next, build a new image to incorporate the changes made to the casc.yaml file:

      • docker build -t jenkins:jcasc .

      Then, run the updated Jenkins image whilst passing in the JENKINS_ADMIN_ID and JENKINS_ADMIN_PASSWORD environment variables via the --env option (replace <password> with a password of your choice):

      • docker run --name jenkins --rm -p 8080:8080 --env JENKINS_ADMIN_ID=admin --env JENKINS_ADMIN_PASSWORD=password jenkins:jcasc

      You can now go to server_ip:8080/login and log in using the specified credentials.

      Jenkins Login Screen with the user ID and password fields populated

      Once you’ve logged in successfully, you will be redirected to the dashboard.

      Jenkins Dashboard for authenticated user, showing the user ID and a 'log out' link near the top right corner of the page

      Finish this step by pressing CTRL+C to stop the container.

      In this step, you used JCasC to create a new user named admin. You’ve also learned how to keep sensitive data, like passwords, out of files tracked by VCSs. However, so far you’ve only configured user authentication; you haven’t implemented any authorization mechanisms. In the next step, you will use JCasC to grant your admin user with administrative privileges.

      Step 5 — Setting Up Authorization

      After setting up the security realm, you must now configure the authorization strategy. In this step, you will use the Matrix Authorization Strategy plugin to configure permissions for your admin user.

      By default, the Jenkins core installation provides us with three authorization strategies:

      • unsecured: every user, including anonymous users, have full permissions to do everything
      • legacy: emulates legacy Jenkins (prior to v1.164), where any users with the role admin is given full permissions, whilst other users, including anonymous users, are given read access.

      Note: A role in Jenkins can be a user (for example, daniel) or a group (for example, developers)

      • loggedInUsersCanDoAnything: anonymous users are given either no access or read-only access. Authenticated users have full permissions to do everything. By allowing actions only for authenticated users, you are able to have an audit trail of which users performed which actions.

      Note: You can explore other authorization strategies and their related plugins in the documentation; these include plugins that handle both authentication and authorization.

      All of these authorization strategies are very crude, and does not afford granular control over how permissions are set for different users. Instead, you can use the Matrix Authorization Strategy plugin that was already included in your plugins.txt list. This plugin affords you a more granular authorization strategy, and allows you to set user permissions globally, as well as per project/job.

      The Matrix Authorization Strategy plugin allows you to use the jenkins.authorizationStrategy.globalMatrix.permissions JCasC property to set global permissions. To use it, open your casc.yaml file:

      • nano $HOME/playground/jcasc/casc.yaml

      And add in the highlighted snippet:

      ~/playground/jcasc/casc.yaml

      ...
             - id: ${JENKINS_ADMIN_ID}
               password: ${JENKINS_ADMIN_PASSWORD}
        authorizationStrategy:
          globalMatrix:
            permissions:
              - "Overall/Administer:admin"
              - "Overall/Read:authenticated"
      unclassified:
      ...
      

      The globalMatrix property sets global permissions (as opposed to per-project permissions). The permissions property is a list of strings with the format <permission-group>/<permission-name>:<role>. Here, you are granting the Overall/Administer permissions to the admin user. You’re also granting Overall/Read permissions to authenticated, which is a special role that represents all authenticated users. There’s another special role called anonymous, which groups all non-authenticated users together. But since permissions are denied by default, if you don’t want to give anonymous users any permissions, you don’t need to explicitly include an entry for it.

      Save the casc.yaml file, exit your editor, and build a new image:

      • docker build -t jenkins:jcasc .

      Then, run the updated Jenkins image:

      • docker run --name jenkins --rm -p 8080:8080 --env JENKINS_ADMIN_ID=admin --env JENKINS_ADMIN_PASSWORD=password jenkins:jcasc

      Wait for the Jenkins is fully up and running log line, and then navigate to server_ip:8080. You will be redirected to the login page. Fill in your credentials and you will be redirected to the main dashboard.

      In this step, you have set up global permissions for your admin user. However, resolving the authorization issue uncovered additional issues that are now shown in the notification menu.

      Jenkins Dashboard showing the notifications menu with two issues

      Therefore, in the next step, you will continue to modify your Docker image, to resolve each issue one by one until none remains.

      Before you continue, stop the container by pressing CTRL+C.

      Step 6 — Setting Up Build Authorization

      The first issue in the notifications list relates to build authentication. By default, all jobs are run as the system user, which has a lot of system privileges. Therefore, a Jenkins user can perform privilege escalation simply by defining and running a malicious job or pipeline; this is insecure.

      Instead, jobs should be ran using the same Jenkins user that configured or triggered it. To achieve this, you need to install an additional plugin called the Authorize Project plugin.

      Open plugins.txt:

      • nano $HOME/playground/jcasc/plugins.txt

      And add the highlighted line:

      ~/playground/jcasc/plugins.txt

      ant:latest
      antisamy-markup-formatter:latest
      authorize-project:latest
      build-timeout:latest
      ...
      

      The plugin provides a new build authorization strategy, which you would need to specify in your JCasC configuration. Exit out of the plugins.txt file and open the casc.yaml file:

      • nano $HOME/playground/jcasc/casc.yaml

      Add the highlighted block to your casc.yaml file:

      ~/playground/jcasc/casc.yaml

      ...
              - "Overall/Administer:admin"
              - "Overall/Read:authenticated"
      security:
        queueItemAuthenticator:
          authenticators:
          - global:
              strategy: triggeringUsersAuthorizationStrategy
      unclassified:
      ...
      

      Save the file and exit the editor. Then, build a new image using the modified plugins.txt and casc.yaml files:

      • docker build -t jenkins:jcasc .

      Then, run the updated Jenkins image:

      • docker run --name jenkins --rm -p 8080:8080 --env JENKINS_ADMIN_ID=admin --env JENKINS_ADMIN_PASSWORD=password jenkins:jcasc

      Wait for the Jenkins is fully up and running log line, then navigate to server_ip:8080/login, fill in your credentials, and arrive at the main dashboard. Open the notification menu, and you will see the issue related to build authentication no longer appears.

      Jenkins dashboard's notification menu showing a single issue related to agent to master security subsystem being turned off

      Stop the container by running CTRL+C before continuing.

      In this step, you have configured Jenkins to run builds using the user that triggered the build, instead of the system user. This eliminates one of the issues in the notifications list. In the next step, you will tackle the next issue related to the Agent to Controller Security Subsystem.

      Step 7 — Enabling Agent to Controller Access Control

      In this tutorial, you have deployed only a single instance of Jenkins, which runs all builds. However, Jenkins supports distributed builds using an agent/controller configuration. The controller is responsible for providing the web UI, exposing an API for clients to send requests to, and co-ordinating builds. The agents are the instances that execute the jobs.

      The benefit of this configuration is that it is more scalable and fault-tolerant. If one of the servers running Jenkins goes down, other instances can take up the extra load.

      However, there may be instances where the agents cannot be trusted by the controller. For example, the OPS team may manage the Jenkins controller, whilst an external contractor manages their own custom-configured Jenkins agent. Without the Agent to Controller Security Subsystem, the agent is able to instruct the controller to execute any actions it requests, which may be undesirable. By enabling Agent to Controller Access Control, you can control which commands and files the agents have access to.

      To enable Agent to Controller Access Control, open the casc.yaml file:

      • nano $HOME/playground/jcasc/casc.yaml

      Then, add the following highlighted lines:

      ~/playground/jcasc/casc.yaml

      ...
              - "Overall/Administer:admin"
              - "Overall/Read:authenticated"
        remotingSecurity:
          enabled: true
      security:
        queueItemAuthenticator:
      ...
      

      Save the file and build a new image:

      • docker build -t jenkins:jcasc .

      Run the updated Jenkins image:

      • docker run --name jenkins --rm -p 8080:8080 --env JENKINS_ADMIN_ID=admin --env JENKINS_ADMIN_PASSWORD=password jenkins:jcasc

      Navigate to server_ip:8080/login and authenticate as before. When you land on the main dashboard, the notifications menu will not show any more issues.

      Jenkins dashboard showing no issues

      Conclusion

      You’ve now successfully configured a simple Jenkins server using JCasC. Just as the Pipeline plugin enables developers to define their jobs inside a Jenkinsfile, the Configuration as Code plugin enables administrators to define the Jenkins configuration inside a YAML file. Both of these plugins bring Jenkins closer aligned with the Everything as Code (EaC) paradigm.

      However, getting the JCasC syntax correct can be difficult, and the documentation can be hard to decipher. If you’re stuck and need help, you may find it in the Gitter chat for the plugin.

      Although you have configured the basic settings of Jenkins using JCasC, the new instance does not contain any projects or jobs. To take this even further, explore the Job DSL plugin, which allows us to define projects and jobs as code. What’s more, you can include the Job DSL code inside your JCasC configuration file, and have the projects and jobs created as part of the configuration process.



      Source link