One place for hosting & domains

      How To Set Up Multi-Factor Authentication for SSH on Ubuntu 20.04


      Introduction

      SSH uses passwords for authentication by default, and most SSH hardening instructions recommend using an SSH key instead. However, an SSH key is still only a single factor, though a much more secure factor. The channel is the terminal on your computer sending the data via an encrypted tunnel to the remote machine. But like a hacker can guess a password, they can steal an SSH key, and then in either case, with that single piece of data, an attacker can gain access to your remote systems.

      In this tutorial, we’ll set up multi-factor authentication to combat that. Multi-factor authentication (MFA) or Two-factor authentication (2FA) requires more than one factor to authenticate or log in. This means a bad actor would have to compromise multiple things, like your computer and your phone, to get in. There are several types of factors used in authentication:

      1. Something you know, like a password or security question
      2. Something you have, like an authenticator app or security token
      3. Something you are, like your fingerprint or voice

      One common factor is an OATH-TOTP app, like Google Authenticator. OATH-TOTP (Open Authentication Time-Based One-Time Password) is an open protocol that generates a one-time use password, commonly a six-digit number recycled every 30 seconds.

      This article will go over how to enable SSH authentication using an OATH-TOTP app in addition to an SSH key. Logging into your server via SSH will require two factors across two channels, thereby making it more secure than a password or SSH key alone. Also, we’ll go over some additional use cases for MFA and some helpful tips and tricks.

      And if you are seeking further guidance on securing SSH connections, check out these tutorials on Hardening OpenSSH and Hardening OpenSSH Client.

      Prerequisites

      To follow this tutorial, you will need:

      • One Ubuntu 20.04 server with a sudo non-root user, SSH key, and firewall enabled, which you can set up by following this Initial Server Setup tutorial.
      • A smartphone or tablet with an OATH-TOTP app installed, like Google Authenticator (iOS, Android).
      • Alternatively, you can also use a Linux command line app called ‘oathtool’ to generate an OATH-TOTP code. It’s available in various distribution repos
      • If you want to secure your SSH connection, there are several good steps outlined in this SSH Essentials article, such as whitelisting users, disabling root login, and changing which port SSH uses.

      Step 1 — Installing Google’s PAM

      In this step, we’ll install and configure Google’s PAM.

      PAM, which stands for Pluggable Authentication Module, is an authentication infrastructure used on Linux systems to authenticate a user. Because Google made an OATH-TOTP app, they also made a PAM that generates TOTPs and is fully compatible with any OATH-TOTP app, like Google Authenticator or Authy.

      First, update Ubuntu’s repository cache:

      Next, install the PAM:

      • sudo apt-get install libpam-google-authenticator

      With the PAM installed, we’ll use a helper app that comes with the PAM to generate a TOTP key for the user that needs a second factor. This key is generated on a user-by-user basis, not system-wide. This means every user that wants to use a TOTP auth app will need to log in and run the helper app to get their key; you can’t just run it once to enable it for everyone (but there are some tips at the end of this tutorial to set up or require MFA for many users).

      Run the initialization app:

      After you run the command, the app will ask a few questions. The first one asks if authentication tokens should be time-based:

      Output

      Do you want authentication tokens to be time-based (y/n) y

      This PAM allows for time-based or sequential-based tokens. Using sequential-based tokens means the code starts at a certain point and then increments the code after every use. Using time-based tokens means the code changes after a certain time frame. We’ll stick with time-based because that is what apps like Google Authenticator anticipate, so answer y for yes.

      After answering this question, a lot of output will scroll past, including a large QR code. Use your authenticator app on your phone to scan the QR code or manually type in the secret key. If the QR code is too big to scan, you can use the URL above the QR code to get a smaller version. Once it’s added, you’ll see a six-digit code that changes every 30 seconds in your app.

      Note: Make sure you record the secret key, verification code, and the recovery codes in a safe place, like a password manager. The recovery codes are the only way to regain access if you, for example, lose access to your TOTP app.

      The remaining questions inform the PAM on how to function. We’ll go through them one by one:

      Output

      Do you want me to update your "~/.google_authenticator" file (y/n) y

      This writes the key and options to the .google_authenticator file. If you say no, the program quits and nothing is written, which means the authenticator won’t work:

      Output

      Do you want to disallow multiple uses of the same authentication token? This restricts you to one login about every 30s, but it increases your chances to notice or even prevent man-in-the-middle attacks (y/n) y

      By answering yes here, you prevent a replay attack by making each code expire immediately after use. This prevents an attacker from capturing a code you just used and logging in with it:

      Output

      By default, a new token is generated every 30 seconds by the mobile app. In order to compensate for possible time-skew between the client and the server, we allow an extra token before and after the current time. This allows for a time skew of up to 30 seconds between the authentication server and client. Suppose you experience problems with poor time synchronization. In that case, you can increase the window from its default size of 3 permitted codes (one previous code, the current code, the next code) to 17 permitted codes (the eight previous codes, the current code, and the eight next codes). This will permit a time skew of up to 4 minutes between client and server. Do you want to do so? (y/n) n

      Answering yes here allows up to 17 valid codes in a moving four-minute window. By answering no, you limit it to 3 valid codes in a 1:30 minute rolling window. Unless you find issues with the 1:30 minute window, answering no is the more secure choice. You can change this setting later in the .google_authenticator file stored at the root of your home directory:

      Output

      If the computer that you are logging into isn't hardened against brute-force login attempts, you can enable rate-limiting for the authentication module. By default, this limits attackers to no more than three login attempts every 30s. Do you want to enable rate-limiting (y/n) y

      Rate limiting means a remote attacker can only attempt a certain number of guesses before being forced to wait some time before being able to try again. If you haven’t previously configured rate limiting directly into SSH, doing so now is a great hardening technique.

      Note: Once you finish this setup, if you want to back up your secret key, you can copy the ~/.google-authenticator file to a trusted location. From there, you can deploy it on additional systems or redeploy it after a backup.

      Now that Google’s PAM is installed and configured, the next step is to configure SSH to use your TOTP key. We’ll need to tell SSH about the PAM and then configure SSH to use it.

      Step 2 — Configuring OpenSSH to Use MFA/2FA

      Because we’ll be making SSH changes over SSH, it’s important never to close your initial SSH connection. Instead, open a second SSH session to do testing. This is to avoid locking yourself out of your server if there was a mistake in your SSH configuration. Once everything works, then you can safely close any sessions. Another safety precaution is to create a backup of the system files you will be editing, so if something goes wrong, you cansrevert to the original file and start over again with a clean configuration.

      To begin, make a backup of the sshd configuration file:

      • sudo cp /etc/pam.d/sshd /etc/pam.d/sshd.bak

      Now open the file using nano or your favorite text editor:

      • sudo nano /etc/pam.d/sshd

      Add the following line to the bottom of the file:

      /etc/pam.d/sshd

      . . .
      # Standard Un*x password updating.
      @include common-password
      auth required pam_google_authenticator.so nullok
      auth required pam_permit.so
      

      The nullok word at the end of the last line tells the PAM that this authentication method is optional. This allows users without an OATH-TOTP token to still log in just using their SSH key. Once all users have an OATH-TOTP token, you can remove nullok from this line to make MFA mandatory. The second line with pam_permit.so is required to allow authentication if a user doesn’t use an MFA token to log in. When logging in, each method needs a SUCCESS to allow authentication. If a user doesn’t use the MFA auth tool, utilizing the nullok option returns an IGNORE for the interactive keyboard authentication. pam_permit.so then returns SUCCESS and allows authentication to proceed.

      Save and close the file.

      Next, we’ll configure SSH to support this kind of authentication.

      First make a backup of the file:

      • sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak

      Now open the SSH configuration file for editing:

      • sudo nano /etc/ssh/sshd_config

      Look for ChallengeResponseAuthentication and set its value to yes:

      /etc/ssh/sshd_config

      . . .
      # Change to yes to enable challenge-response passwords (beware issues with
      # some PAM modules and threads)
      ChallengeResponseAuthentication yes
      . . .
      

      Save and close the file, then restart SSH to reload the configuration files. Restarting the sshd service won’t close our current open connections, meaning you won’t risk locking yourself out with this command:

      • sudo systemctl restart sshd.service

      To test that everything’s working so far, open ANOTHER terminal and try logging in over SSH. It is very important that you keep your current SSH session open and test with an additional session, or you will lock yourself out at some point and will need to use the web console to get yourself back in.

      Note: If you’ve previously created an SSH key and are using it, you’ll notice you didn’t have to type in your user’s password or the MFA verification code. This is because an SSH key overrides all other authentication options by default. Otherwise, you should have gotten a password and verification code prompt.

      Next, to enable an SSH key as one factor and the verification code as a second, we need to tell SSH which factors to use and prevent the SSH key from overriding all other types.

      Step 3 — Making SSH Aware of MFA

      MFA is still not working if you are using and SSH key. To make SSH aware of MFA, reopen the sshd configuration file:

      • sudo nano /etc/ssh/sshd_config

      Add the following line at the bottom of the file. This tells SSH which authentication methods are required. We tell SSH that users need an SSH key and either a password or a verification code (or all three):

      /etc/ssh/sshd_config

      . . .
      AuthenticationMethods publickey,password publickey,keyboard-interactive
      

      Save and close the file.

      Next, open the PAM sshd configuration file again:

      • sudo nano /etc/pam.d/sshd

      Find the line @include common-auth and comment it out by adding a # character as the first character on the line. This tells PAM not to prompt for a password:

      /etc/pam.d/sshd

      . . .
      # Standard Un*x authentication.
      #@include common-auth
      . . .
      

      Save and close the file, then restart SSH:

      • sudo systemctl restart sshd.service

      Now try logging into the server again with a different terminal session/window. Unlike last time, SSH should ask for your verification code. Enter it, and you will complete the login. Even though there is no indication that your SSH key was used, your login attempt used two factors. If you want to verify this, you can add -v (for verbose) after the SSH command.

      The -v switch will produce an output like this:

      Example SSH output

      . . . debug1: Authentications that can continue: publickey debug1: Next authentication method: publickey debug1: Offering RSA public key: /Users/sammy/.ssh/id_rsa debug1: Server accepts key: pkalg rsa-sha2-512 blen 279 Authenticated with partial success. debug1: Authentications that can continue: password,keyboard-interactive debug1: Next authentication method: keyboard-interactive Verification code:

      Towards the end of the output, you’ll see where SSH uses your SSH key and then asks for the verification code. You can now log in over SSH with an SSH key and a one-time password. If you want to enforce all three authentication types, you can follow the next step.

      Congratulations, you’ve successfully added a second factor when logging in remotely to your server over SSH. If this is what you wanted — to use your SSH key and a TOTP token to enable MFA for SSH (for most people, this is the optimal configuration) — then you’re done.

      What follows are some tips and tricks for recovery, automated usage, and more.

      Step 4 — Adding a Third Factor (Optional)

      In Step 3, we listed the approved types of authentication in the sshd_config file:

      1. publickey (SSH key)
      2. password publickey (password)
      3. keyboard-interactive (verification code)

      Although we listed three different factors, the options we’ve chosen so far, they only allow for an SSH key and the verification code. If you’d like to have all three factors (SSH key, password, and verification code), one quick change will enable all three.

      Open the PAM sshd configuration file:

      • sudo nano /etc/pam.d/sshd

      Locate the line you commented out previously, #@include common-auth, and uncomment the line by removing the # character. Save and close the file. Now once again, restart SSH:

      • sudo systemctl restart sshd.service

      By enabling the option @include common-auth, PAM will now prompt for a password in addition to checking for an SSH key and asking for a verification code, which we had working previously. Now we can use something we know (password) and two different types of things we have (SSH key and verification code) over two different channels (your computer for the SSH key and your phone for the TOTP token).

      Step 5 — Recovering Access to Google MFA (optional)

      As with any system that you harden and secure, you become responsible for managing that security. In this case, that means not losing your SSH key or your TOTP secret key and &making sure you have access to your TOTP app. However, sometimes things happen, and you can lose control of the keys or apps you need to get in.

      Losing Your TOTP Secret Key

      If you lose your TOTP secret key, you can break the recovery process into a couple of steps. The first is getting back in without knowing the verification code, and the second is finding the secret key or regenerating it for normal MFA login. This often can happen if you get a new phone and don’t transfer over your secrets to a new authenticator app.

      To get in after losing the TOTP secret key on a DigitalOcean Droplet, you cansuse the virtual console from your dashboard to log in using your username and password. This works because we only protected your user account with MFA for SSH connections. Non-ssh connections, such as a console login, don’t use the Google Authenticator PAM module.

      If you’re on a non-Droplet system, then you have two options for regaining access:

      1. Console (local/non-ssh) access to the system (typically physical or via something like iDrac)
      2. Have a different user that doesn’t have MFA enabled

      The second option is the less secure option since the point in using MFA is to harden all SSH connections, but it’s one fail-safe if you lose access to your MFA authenticator app.

      Once you’ve logged in, there are two ways to help get the TOTP secret:

      1. Recover the existing key
      2. Generate a new key

      In each user’s home directory, the secret key and Google Authenticator settings are saved in a ~/.google-authenticator file. The very first line of this file is a secret key. A quick way to get the key is to execute the following command, which displays the first line of the google-authenticator file (i.e., the secret key). Then, take that secret key and manually type it into a TOTP app:

      • head -n 1 /home/sammy/.google_authenticator

      Once you’ve recovered your existing key, you can either manually type it into your authenticator app or fill in the relevant details in the URL below and have Google generate a QR code for you to scan. You’ll need to add your username, hostname, the secret key from the .google-authenticator file, and then any name of your choosing for 'entry-name-in-auth-app’ to easily identify this key versus a different TOTP token:

      https://www.google.com/chart?chs=200x200&chld=M|0&cht=qr&chl=otpauth://totp/username@hostname%3Fsecret%3D16-char-secret%26issuer%3Dentry-name-in-auth-app
      

      If there is a reason not to use the existing key (for example, being unable to easily share the secret key with the impacted user securely), you can remove the ~/.google-authenticator file outright. This will allow the user to log in again using only a single factor, assuming you haven’t enforced MFA by removing the nullok option. They can then run google-authenticator to generate a new key.

      Losing Access to the TOTP App

      If you need to log in to your server but don’t have access to your TOTP app to get your verification code, you can still log in using the recovery codes that were displayed when you first created your secret key, and are the last five lines of the .google-authenticator file. Note that these recovery codes are one-time use. However, for this to work, you need to make your recovery codes available when you don’t have access to the TOTP app.

      Step 6 — Changing Authentication Settings (optional)

      If you want to change your MFA settings after the initial configuration, instead of generating a new configuration with the updated settings, you can just edit the ~/.google-authenticator file. Options in this file appear in the following manner:

      .google-authenticator layout

      <secret key>
      <options>
      <recovery codes>
      

      Options set in this file have a line in the options section; if you answered “no” to a particular option during the initial setup, the program excludes the corresponding.

      Here are some changes you can make to this file:

      • To enable sequential codes instead of time-based codes, change the line " TOTP_AUTH to " HOTP_COUNTER 1.
      • To allow multiple uses of a single code, remove the line " DISALLOW_REUSE.
      • To extend the code expiration window to 4 minutes, add the line " WINDOW_SIZE 17.
      • To disable multiple failed logins (rate-limiting), remove the line " RATE_LIMIT 3 30.
      • To change the rate-limiting threshold, find the line " RATE_LIMIT 3 30 and adjust the numbers. The 3 in the original indicates the number of attempts over a period, and the 30 indicates the time in seconds.
      • To disable the use of recovery codes, remove the five eight-digit codes at the bottom of the file.

      Step 7 — Avoiding MFA for Some Accounts (optional)

      There may be a situation in which a single user or a few service accounts (i.e., accounts used by applications, not humans) need SSH access without MFA enabled. For example, some applications that use SSH, like some FTP clients, may not support MFA. If an application doesn’t have a way to request the verification code, the request may get stuck until the SSH connection times out.

      To control which factors to use for a user, you can edit the /etc/pam.d/sshd file.

      To allow MFA for some accounts and SSH only for others, make sure the following settings in /etc/pam.d/sshd are active:

      /etc/pam.d/sshd

      # PAM configuration for the Secure Shell service
      
      # Standard Un*x authentication.
      #@include common-auth
      
      . . .
      
      # Standard Un*x password updating.
      @include common-password
      auth required pam_google_authenticator.so nullok
      

      Here, @include common-auth is commented out because passwords need to be disabled. You cannot force MFA if some accounts have MFA disabled, so leave the nullok option on the final line.

      After setting this configuration, run google-authenticator as any users that need MFA, and don’t run it for users that will only use SSH keys.

      Step 8 — Automating Setup with Configuration Management (optional)

      Many system administrators use configuration management tools, like Puppet, Chef, or Ansible, to manage their systems. You can use a system like this to install and set up a secret key whenever a new user creates an account.

      google-authenticator supports command-line switches to set all the options in a single, non-interactive command. To see all the options, you can type google-authenticator --help. Below is the command that would set everything up as outlined in Step 1:

      • google-authenticator -t -d -f -r 3 -R 30 -w 3

      The options referenced above are as follows:

      • -t => Time based counter
      • -d => Disallow token reuse
      • -f => Force writing the settings to file without prompting the user
      • -r => How many attempts to enter the correct code
      • -R => How long in seconds a user can attempt to enter the correct code
      • -w => How many codes can are valid at a time (this references the 1:30 min - 4 min window of valid codes)

      This fully configures the authenticator, saves it to a file, and then outputs the secret key, QR code, and recovery codes. (If you add the flag -q, then there won’t be any output.) If you use this command in an automated fashion, make sure your script captures the secret key and/or recovery codes and makes them available to the user.

      Step 9 — Forcing MFA for All Users (optional)

      If you want to force MFA for all users, even on the first login, or prefer not to rely on your users to generate their keys, there’s a quick way to handle this. You can use the same .google-authenticator file for each user, as there’s no user-specific data stored in the file.

      To do this, after creating the configuration file, a privileged user needs to copy the file to the root of every home directory and change its permissions to the appropriate user. You can also copy the file to /etc/skel/, automatically copying the file to every new user’s home directory upon creation.

      Warning: This can be a security risk because everyone is sharing the same second factor. This means that if it’s leaked, it’s as if every user had only one factor. Take this into consideration if you want to use this approach.

      Another method to force the creation of a user’s secret key is to use a bash script that:

      1. Creates a TOTP token,
      2. Prompts them to download the Google Authenticator app and scan the QR code that will be displayed, and
      3. Runs the google-authenticator application for them after checking if the .google-authenticator file already exists.

      To make sure the script runs when a user logs in, you can name it .bash_login and place it at the root of their home directory.

      Conclusion

      In this tutorial, you added two factors (an SSH key + MFA token) across two channels (your computer + your phone) to your server. You’ve made it very difficult for an outside agent to brute force their way into your machine via SSH and greatly increased the security of your machine.

      And remember, if you are seeking further guidance on securing SSH connections, check out these tutorials on Hardening OpenSSH and Hardening OpenSSH Client.



      Source link

      How To Set Up Physical Streaming Replication with PostgreSQL 12 on Ubuntu 20.04


      The author selected the COVID-19 Relief Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      Streaming replication is a popular method you can use to horizontally scale your relational databases. It uses two or more copies of the same database cluster running on separate machines. One database cluster is referred to as the primary and serves both read and write operations; the others, referred to as the replicas, serve only read operations. You can also use streaming replication to provide high availability of a system. If the primary database cluster or server were to unexpectedly fail, the replicas are able to continue serving read operations, or (one of the replicas) become the new primary cluster.

      PostgreSQL is a widely used relational database that supports both logical and physical replication. Logical replication streams high-level changes from the primary database cluster to the replica databases. Using logical replication, you can stream changes to just a single database or table in a database. However, in physical replication, changes to the WAL (Write-Ahead-Logging) log file are streamed and replicated in the replica clusters. As a result, you can’t replicate specific areas of a primary database cluster, but instead all changes to the primary are replicated.

      In this tutorial, you will set up physical streaming replication with PostgreSQL 12 on Ubuntu 20.04 using two separate machines running two separate PostgreSQL 12 clusters. One machine will be the primary and the other, the replica.

      Prerequisites

      To complete this tutorial, you will need the following:

      • Two separate machines Ubuntu 20.04 machines; one referred to as the primary and the other referred to as the replica. You can set these up with our Initial Server Setup Guide, including non-root users with sudo permissions and a firewall.
      • Your firewalls configured to allow HTTP/HTTPS and traffic on port 5432—the default port used by PostgreSQL 12. You can follow How To Set Up a Firewall with ufw on Ubuntu 20.04 to configure these firewall settings.
      • PostgreSQL 12 running on both Ubuntu 20.04 Servers. Follow Step 1 of the How To Install and Use PostgreSQL on Ubuntu 20.04 tutorial that covers the installation and basic usage of PostgreSQL on Ubuntu 20.04.

      Step 1 — Configuring the Primary Database to Accept Connections

      In this first step, you’ll configure the primary database to allow your replica database(s) to connect. By default, PostgreSQL only listens to the localhost (127.0.0.1) for connections. To change this, you’ll first edit the listen_addresses configuration parameter on the primary database.

      On your primary server, run the following command to connect to the PostgreSQL cluster as the default postgres user:

      Once you have connected to the database, you’ll modify the listen_addresses parameter using the ALTER SYSTEM command:

      • ALTER SYSTEM SET listen_addresses TO 'your_replica_IP_addr';

      Replace 'your_replica_IP_addr' with the IP address of your replica machine.

      You will receive the following output:

      Output

      ALTER SYSTEM

      The command you just entered instructs the PostgreSQL database cluster to allow connections only from your replica machine. If you were using more than one replica machine, you would list the IP addresses of all your replicas separated by commas. You could also use '*' to allow connections from all IP addresses, however, this isn’t recommended for security reasons.

      Note: You can also run the command on the database from the terminal using psql -c as follows:

      • sudo -u postgres psql -c "ALTER SYSTEM SET listen_addresses TO 'your_replica_IP_adder';"

      Alternatively, you can change the value for listen_addresses by manually editing the postgresql.conf configuration file, which you can find in the /etc/postgresql/12/main/ directory by default. You can also get the location of the configuration file by running SHOW config_file; on the database cluster.

      To open the file using nano use:

      • sudo nano /etc/postgresql/12/main/postgresql.conf

      Once you’re done, your primary database will now accept connections from other machines. Next, you’ll create a role with the appropriate permissions that the replica will use when connecting to the primary.

      Step 2 — Creating a Special Role with Replication Permissions

      Now, you need to create a role in the primary database that has permission to replicate the database. Your replica will use this role when connecting to the primary. Creating a separate role just for replication also has security benefits. Your replica won’t be able to manipulate any data on the primary; it will only be able to replicate the data.

      To create a role, you need to run the following command on the primary cluster:

      • CREATE ROLE test WITH REPLICATION PASSWORD 'testpassword' LOGIN;

      You’ll receive the following output:

      Output

      CREATE ROLE

      This command creates a role named test with the password 'testpassword', which has permission to replicate the database cluster.

      PostgreSQL has a special replication pseudo-database that the replica connects to, but you first need to edit the /etc/postgresql/12/main/pg_hba.conf configuration file to allow your replica to access it. So, exit the PostgreSQL command prompt by running:

      Now that you’re back at your terminal command prompt, open the /etc/postgresql/12/main/pg_hba.conf configuration file using nano:

      • sudo nano /etc/postgresql/12/main/pg_hba.conf

      Append the following line to the end of the pg_hba.conf file:

      /etc/postgresql/12/main/pg_hba.conf

      . . .
      host    replication     test    your-replica-IP/32   md5
      

      This ensures that your primary allows your replica to connect to the replication pseudo-database using the role, test, you created earlier. The host value means to accept non-local connections via plain or SSL-encrypted TCP/IP sockets. replication is the name of the special pseudo-database that PostgreSQL uses for replication. Finally, the value md5 is the type of authentication used. If you want to have more than one replica, just add the same line again to the end of the file with the IP address of your other replica.

      To ensure these changes to the configuration file are implemented, you need to restart the primary cluster using:

      • sudo systemctl restart postgresql@12-main

      If your primary cluster restarted successfully, it is correctly set up and ready to start streaming once your replica connects. Next, you’ll move on to setting up your replica cluster.

      Step 3 — Backing Up the Primary Cluster on the Replica

      As you are setting up physical replication with PostgreSQL in this tutorial, you need to perform a physical backup of the primary cluster’s data files into the replica’s data directory. To do this, you’ll first clear out all the files in the replica’s data directory. The default data directory for PostgreSQL on Ubuntu is /var/lib/postgresql/12/main/.

      You can also find PostgreSQL’s data directory by running the following command on the replica’s database:

      Once you have the location of the data directory, run the following command to remove everything:

      • sudo -u postgres rm -r /var/lib/postgresql/12/main/*

      Since the default owner of the files in the directory is the postgres user, you will need to run the command as postgres using sudo -u postgres.

      Note:
      If in the exceedingly rare case a file in the directory is corrupted and the command does not work, remove the main directory all together and recreate it with the appropriate permissions as follows:

      • sudo -u postgres rm -r /var/lib/postgresql/12/main
      • sudo -u postgres mkdir /var/lib/postgresql/12/main
      • sudo -u postgres chmod 700 /var/lib/postgresql/12/main

      Now that the replica’s data directory is empty, you can perform a physical backup of the primary’s data files. PostgreSQL conveniently has the utility pg_basebackup that simplifies the process. It even allows you to put the server into standby mode using the -R option.

      Execute the pg_basebackup command on the replica as follows:

      • sudo -u postgres pg_basebackup -h primary-ip-addr -p 5432 -U test -D /var/lib/postgresql/12/main/ -Fp -Xs -R
      • The -h option specifies a non-local host. Here, you need to enter the IP address of your server with the primary cluster.

      • The -p option specifies the port number it connects to on the primary server. By default, PostgreSQL uses port :5432.

      • The -U option allows you to specify the user you connect to the primary cluster as. This is the role you created in the previous step.

      • The -D flag is the output directory of the backup. This is your replica’s data directory that you emptied just before.

      • The -Fp specifies the data to be outputted in the plain format instead of as a tar file.

      • -Xs streams the contents of the WAL log as the backup of the primary is performed.

      • Lastly, -R creates an empty file, named standby.signal, in the replica’s data directory. This file lets your replica cluster know that it should operate as a standby server. The -R option also adds the connection information about the primary server to the postgresql.auto.conf file. This is a special configuration file that is read whenever the regular postgresql.conf file is read, but the values in the .auto file override the values in the regular configuration file.

      When the pg_basebackup command connects to the primary, you will be prompted to enter the password for the role you created in the previous step. Depending on the size of your primary database cluster, it may take some time to copy all the files.

      Your replica will now have all the data files from the primary that it requires to begin replication. Next, you’ll be putting the replica into standby mode and start replicating.

      Step 4 — Restarting and Testing the Clusters

      Now that the primary cluster’s data files have been successfully backed up on the replica, the next step is to restart the replica database cluster to put it into standby mode. To restart the replica database, run the following command:

      • sudo systemctl restart postgresql@12-main

      If your replica cluster restarted in standby mode successfully, it should have already connected to the primary database cluster on your other machine. To check if the replica has connected to the primary and the primary is streaming, connect to the primary database cluster by running:

      Now query the pg_stat_replication table on the primary database cluster as follows:

      • SELECT client_addr, state FROM pg_stat_replication;

      Running this query on the primary cluster will output something similar to the following:

      Output

      client_addr | state ------------------+----------- your_replica_IP | streaming

      If you have similar output, then the primary is correctly streaming to the replica.

      Conclusion

      You now have two Ubuntu 20.04 servers each with a PostgreSQL 12 database cluster running with physical streaming between them. Any changes now made to the primary database cluster will also appear in the replica cluster.

      You can also add more replicas to your setup if your databases need to handle more traffic.

      If you wish to learn more about physical streaming replication including how to set up synchronous replication to ensure zero chance of losing any mission-critical data, you can read the entry in the official PostgreSQL docs.

      You can check out our PostgreSQL topic page for more tutorials and content.



      Source link

      How To Set Up a Continuous Deployment Pipeline with GitLab CI/CD on Ubuntu 18.04


      The author selected the Free and Open Source Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      GitLab is an open source collaboration platform that provides powerful features beyond hosting a code repository. You can track issues, host packages and registries, maintain Wikis, set up continuous integration (CI) and continuous deployment (CD) pipelines, and more.

      In this tutorial you’ll build a continuous deployment pipeline with GitLab. You will configure the pipeline to build a Docker image, push it to the GitLab container registry, and deploy it to your server using SSH. The pipeline will run for each commit pushed to the repository.

      You will deploy a small, static web page, but the focus of this tutorial is configuring the CD pipeline. The static web page is only for demonstration purposes; you can apply the same pipeline configuration using other Docker images for the deployment as well.

      When you have finished this tutorial, you can visit http://your_server_IP in a browser for the results of the automatic deployment.

      Prerequisites

      To complete this tutorial, you will need:

      Step 1 — Creating the GitLab Repository

      Let’s start by creating a GitLab project and adding an HTML file to it. You will later copy the HTML file into an Nginx Docker image, which in turn you’ll deploy to the server.

      Log in to your GitLab instance and click New project.

      The new project button in GitLab

      1. Give it a proper Project name.
      2. Optionally add a Project description.
      3. Make sure to set the Visibility Level to Private or Public depending on your requirements.
      4. Finally click Create project

      The new project form in GitLab

      You will be redirected to the Project’s overview page.

      Let’s create the HTML file. On your Project’s overview page, click New file.

      The new file button on the project overview page

      Set the File name to index.html and add the following HTML to the file body:

      index.html

      <html>
      <body>
      <h1>My Personal Website</h1>
      </body>
      </html>
      

      Click Commit changes at the bottom of the page to create the file.

      This HTML will produce a blank page with one headline showing My Personal Website when opened in a browser.

      Dockerfiles are recipes used by Docker to build Docker images. Let’s create a Dockerfile to copy the HTML file into an Nginx image.

      Go back to the Project’s overview page, click the + button and select the New file option.

      New file option in the project's overview page listed in the plus button

      Set the File name to Dockerfile and add these instructions to the file body:

      Dockerfile

      FROM nginx:1.18
      COPY index.html /usr/share/nginx/html
      

      The FROM instruction specifies the image to inherit from—in this case the nginx:1.18 image. 1.18 is the image tag representing the Nginx version. The nginx:latest tag references the latest Nginx release, but that could break your application in the future, which is why fixed versions are recommended.

      The COPY instruction copies the index.html file to /usr/share/nginx/html in the Docker image. This is the directory where Nginx stores static HTML content.

      Click Commit changes at the bottom of the page to create the file.

      In the next step, you’ll configure a GitLab runner to keep control of who gets to execute the deployment job.

      Step 2 — Registering a GitLab Runner

      In order to keep track of the environments that will have contact with the SSH private key, you’ll register your server as a GitLab runner.

      In your deployment pipeline you want to log in to your server using SSH. To achieve this, you’ll store the SSH private key in a GitLab CI/CD variable (Step 5). The SSH private key is a very sensitive piece of data, because it is the entry ticket to your server. Usually, the private key never leaves the system it was generated on. In the usual case, you would generate an SSH key on your host machine, then authorize it on the server (that is, copy the public key to the server) in order to log in manually and perform the deployment routine.

      Here the situation changes slightly: You want to grant an autonomous authority (GitLab CI/CD) access to your server to automate the deployment routine. Therefore the private key needs to leave the system it was generated on and be given in trust to GitLab and other involved parties. You never want your private key to enter an environment that is not either controlled or trusted by you.

      Besides GitLab, the GitLab runner is yet another system that your private key will enter. For each pipeline, GitLab uses runners to perform the heavy work, that is, execute the jobs you have specified in the CI/CD configuration. That means the deployment job will ultimately be executed on a GitLab runner, hence the private key will be copied to the runner such that it can log in to the server using SSH.

      If you use unknown GitLab Runners (for example, shared runners) to execute the deployment job, then you’d be unaware of the systems getting in contact with the private key. Even though GitLab runners clean up all data after job execution, you can avoid sending the private key to unknown systems by registering your own server as a GitLab runner. The private key will then be copied to the server controlled by you.

      Start by logging in to your server:

      In order to install the gitlab-runner service, you’ll add the official GitLab repository. Download and inspect the install script:

      • curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh > script.deb.sh
      • less script.deb.sh

      Once you are satisfied with the safety of the script, run the installer:

      It may not be obvious, but you have to enter your non-root user’s password to proceed. When you execute the previous command, the output will be like:

      Output

      [sudo] password for sammy: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 5945 100 5945 0 0 8742 0 --:--:-- --:--:-- --:--:-- 8729

      When the curl command finishes, you will receive the following message:

      Output

      The repository is setup! You can now install packages.

      Next install the gitlab-runner service:

      • sudo apt install gitlab-runner

      Verify the installation by checking the service status:

      • systemctl status gitlab-runner

      You will have active (running) in the output:

      Output

      ● gitlab-runner.service - GitLab Runner Loaded: loaded (/etc/systemd/system/gitlab-runner.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2020-06-01 09:01:49 UTC; 4s ago Main PID: 16653 (gitlab-runner) Tasks: 6 (limit: 1152) CGroup: /system.slice/gitlab-runner.service └─16653 /usr/lib/gitlab-runner/gitlab-runner run --working-directory /home/gitlab-runner --config /etc/gitla

      To register the runner, you need to get the project token and the GitLab URL:

      1. In your GitLab project, navigate to Settings > CI/CD > Runners.
      2. In the Set up a specific Runner manually section, you’ll find the registration token and the GitLab URL. Copy both to a text editor; you’ll need them for the next command. They will be referred to as https://your_gitlab.com and project_token.

      The runners section in the ci/cd settings with the copy token button

      Back to your terminal, register the runner for your project:

      • sudo gitlab-runner register -n --url https://your_gitlab.com --registration-token project_token --executor docker --description "Deployment Runner" --docker-image "docker:stable" --tag-list deployment --docker-privileged

      The command options can be interpreted as follows:

      • -n executes the register command non-interactively (we specify all parameters as command options).
      • --url is the GitLab URL you copied from the runners page in GitLab.
      • --registration-token is the token you copied from the runners page in GitLab.
      • --executor is the executor type. docker executes each CI/CD job in a Docker container (see GitLab’s documentation on executors).
      • --description is the runner’s description, which will show up in GitLab.
      • --docker-image is the default Docker image to use in CI/CD jobs, if not explicitly specified.
      • --tag-list is a list of tags assigned to the runner. Tags can be used in a pipeline configuration to select specific runners for a CI/CD job. The deployment tag will allow you to refer to this specific runner to execute the deployment job.
      • --docker-privileged executes the Docker container created for each CI/CD job in privileged mode. A privileged container has access to all devices on the host machine and has nearly the same access to the host as processes running outside containers (see Docker’s documentation about runtime privilege and Linux capabilities). The reason for running in privileged mode is so you can use Docker-in-Docker (dind) to build a Docker image in your CI/CD pipeline. It is good practice to give a container the minimum requirements it needs. For you it is a requirement to run in privileged mode in order to use Docker-in-Docker. Be aware, you registered the runner for this specific project only, where you are in control of the commands being executed in the privileged container.

      After executing the gitlab-runner register command, you will receive the following output:

      Output

      Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

      Verify the registration process by going to Settings > CI/CD > Runners in GitLab, where the registered runner will show up.

      The registered runner in the runners section of the ci/cd settings

      In the next step you’ll create a deployment user.

      Step 3 — Creating a Deployment User

      You are going to create a non-sudo user that is dedicated for the deployment task, so that its power is limited and the deployment takes place in an isolated user space. You will later configure the CI/CD pipeline to log in to the server with that user.

      On your server, create a new user:

      You’ll be guided through the user creation process. Enter a strong password and optionally any further user information you want to specify. Finally confirm the user creation with Y.

      Add the user to the Docker group:

      • sudo usermod -aG docker deployer

      This permits deployer to execute the docker command, which is required to perform the deployment.

      In the next step you’ll create an SSH key to be able to log in to the server as deployer.

      Step 4 — Setting Up an SSH Key

      You are going to create an SSH key for the deployment user. GitLab CI/CD will later use the key to log in to the server and perform the deployment routine.

      Let’s start by switching to the newly created deployer user for whom you’ll generate the SSH key:

      You’ll be prompted for the deployer password to complete the user switch.

      Next, generate a 4096-bit SSH key. It is important to answer the questions of the ssh-keygen command correctly:

      1. First question: answer it with ENTER, which stores the key in the default location (the rest of this tutorial assumes the key is stored in the default location).
      2. Second question: configures a password to protect the SSH private key (the key used for authentication). If you specify a passphrase, you’ll have to enter it each time the private key is used. In general, a passphrase adds another security layer to SSH keys, which is good practice. Somebody in possession of the private key would also require the passphrase to use the key. For the purposes of this tutorial, it is important that you have an empty passphrase, because the CI/CD pipeline will execute non-interactively and therefore does not allow to enter a passphrase.

      To summarize, run the following command and confirm both questions with ENTER to create a 4096-bit SSH key and store it in the default location with an empty passphrase:

      To authorize the SSH key for the deployer user, you need to append the public key to the authorized_keys file:

      • cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

      ~ is short for the user home in Linux. The cat program will print the contents of a file; here you use the >> operator to redirect the output of cat and append it to the authorized_keys file.

      In this step you have created an SSH key pair for the CI/CD pipeline to log in and deploy the application. Next you’ll store the private key in GitLab to make it accessible during the pipeline process.

      Step 5 — Storing the Private Key in a GitLab CI/CD Variable

      You are going to store the SSH private key in a GitLab CI/CD file variable, so that the pipeline can make use of the key to log in to the server.

      When GitLab creates a CI/CD pipeline, it will send all variables to the corresponding runner and the variables will be set as environment variables for the duration of the job. In particular, the values of file variables are stored in a file and the environment variable will contain the path to this file.

      While you’re in the variables section, you’ll also add a variable for the server IP and the server user, which will inform the pipeline about the destination server and user to log in.

      Start by showing the SSH private key:

      Copy the output to your clipboard using CTRL+C. Make sure to copy everything including the BEGIN and END line:

      ~/.ssh/id_rsa

      -----BEGIN RSA PRIVATE KEY-----
      ...
      -----END RSA PRIVATE KEY-----
      

      Now navigate to Settings > CI / CD > Variables in your GitLab project and click Add Variable. Fill out the form as follows:

      • Key: ID_RSA
      • Value: Paste your SSH private key from your clipboard with CTRL+V
      • Type: File
      • Environment Scope: All (default)
      • Protect variable: Checked
      • Mask variable: Unchecked

      Note: The variable can’t be masked because it does not meet the regular expression requirements (see GitLab’s documentation about masked variables). However, the private key will never appear in the console log, which makes masking it obsolete.

      A file containing the private key will be created on the runner for each CI/CD job and its path will be stored in the $ID_RSA environment variable.

      Create another variable with your server IP. Click Add Variable and fill out the form as follows:

      • Key: SERVER_IP
      • Value: your_server_IP
      • Type: Variable
      • Environment scope: All (default)
      • Protect variable: Checked
      • Mask variable: Checked

      Finally, create a variable with the login user. Click Add Variable and fill out the form as follows:

      • Key: SERVER_USER
      • Value: deployer
      • Type: Variable
      • Environment scope: All (default)
      • Protect variable: Checked
      • Mask variable: Checked

      You have now stored the private key in a GitLab CI/CD variable, which makes the key available during pipeline execution. In the next step, you’re moving on to configuring the CI/CD pipeline.

      Step 6 — Configuring the .gitlab-ci.yml File

      You are going to configure the GitLab CI/CD pipeline. The pipeline will build a Docker image and push it to the container registry. GitLab provides a container registry for each project. You can explore the container registry by going to Packages & Registries > Container Registry in your GitLab project (read more in GitLab’s container registry documentation.) The final step in your pipeline is to log in to your server, pull the latest Docker image, remove the old container, and start a new container.

      Now you’re going to create the .gitlab-ci.yml file that contains the pipeline configuration. In GitLab, go to the Project overview page, click the + button and select New file. Then set the File name to .gitlab-ci.yml.

      (Alternatively you can clone the repository and make all following changes to .gitlab-ci.yml on your local machine, then commit and push to the remote repository.)

      To begin add the following:

      .gitlab-ci.yml

      stages:
        - publish
        - deploy
      

      Each job is assigned to a stage. Jobs assigned to the same stage run in parallel (if there are enough runners available). Stages will be executed in the order they were specified. Here, the publish stage will go first and the deploy stage second. Successive stages only start when the previous stage finished successfully (that is, all jobs have passed). Stage names can be chosen arbitrarily.

      When you want to combine this CD configuration with your existing CI pipeline, which tests and builds the app, you may want to add the publish and deploy stages after your existing stages, such that the deployment only takes place if the tests passed.

      Following this, add this to your .gitlab-ci.yml file:

      .gitlab-ci.yml

      . . .
      variables:
        TAG_LATEST: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME:latest
        TAG_COMMIT: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME:$CI_COMMIT_SHORT_SHA
      

      The variables section defines environment variables that will be available in the context of a job’s script section. These variables will be available as usual Linux environment variables; that is, you can reference them in the script by prefixing with a dollar sign such as $TAG_LATEST. GitLab creates some predefined variables for each job that provide context specific information, such as the branch name or the commit hash the job is working on (read more about predefined variable). Here you compose two environment variables out of predefined variables. They represent:

      • CI_REGISTRY_IMAGE: Represents the URL of the container registry tied to the specific project. This URL depends on the GitLab instance. For example, registry URLs for gitlab.com projects follow the pattern: registry.gitlab.com/your_user/your_project. But since GitLab will provide this variable, you do not need to know the exact URL.
      • CI_COMMIT_REF_NAME: The branch or tag name for which project is built.
      • CI_COMMIT_SHORT_SHA: The first eight characters of the commit revision for which the project is built.

      Both of the variables are composed of predefined variables and will be used to tag the Docker image.

      TAG_LATEST will add the latest tag to the image. This is a common strategy to provide a tag that always represents the latest release. For each deployment, the latest image will be overridden in the container registry with the newly built Docker image.

      TAG_COMMIT, on the other hand, uses the first eight characters of the commit SHA being deployed as the image tag, thereby creating a unique Docker image for each commit. You will be able to trace the history of Docker images down to the granularity of Git commits. This is a common technique when doing continuous deployments, because it allows you to quickly deploy an older version of the code in case of a defective deployment.

      As you’ll explore in the coming steps, the process of rolling back a deployment to an older Git revision can be done directly in GitLab.

      $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME specifies the Docker image base name. According to GitLab’s documentation, a Docker image name has to follow this scheme:

      image name scheme

      <registry URL>/<namespace>/<project>/<image>

      $CI_REGISTRY_IMAGE represents the <registry URL>/<namespace>/<project> part and is mandatory because it is the project’s registry root. $CI_COMMIT_REF_NAME is optional but useful to host Docker images for different branches. In this tutorial you will only work with one branch, but it is good to build an extendable structure. In general, there are three levels of image repository names supported by GitLab:

      repository name levels

      registry.example.com/group/project:some-tag registry.example.com/group/project/image:latest registry.example.com/group/project/my/image:rc1

      For your TAG_COMMIT variable you used the second option, where image will be replaced with the branch name.

      Next, add the following to your .gitlab-ci.yml file:

      .gitlab-ci.yml

      . . .
      publish:
        image: docker:latest
        stage: publish
        services:
          - docker:dind
        script:
          - docker build -t $TAG_COMMIT -t $TAG_LATEST .
          - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
          - docker push $TAG_COMMIT
          - docker push $TAG_LATEST
      

      The publish section is the first job in your CI/CD configuration. Let’s break it down:

      • image is the Docker image to use for this job. The GitLab runner will create a Docker container for each job and execute the script within this container. docker:latest image ensures that the docker command will be available.
      • stage assigns the job to the publish stage.
      • services specifies Docker-in-Docker—the dind service. This is the reason why you registered the GitLab runner in privileged mode.

      The script section of the publish job specifies the shell commands to execute for this job. The working directory will be set to the repository root when these commands will be executed.

      • docker build ...: Builds the Docker image based on the Dockerfile and tags it with the latest commit tag defined in the variables section.
      • docker login ...: Logs Docker in to the project’s container registry. You use the predefined variable $CI_BUILD_TOKEN as an authentication token. GitLab will generate the token and stay valid for the job’s lifetime.
      • docker push ...: Pushes both image tags to the container registry.

      Following this, add the deploy job to your .gitlab-ci.yml:

      .gitlab-ci.yml

      . . .
      deploy:
        image: alpine:latest
        stage: deploy
        tags:
          - deployment
        script:
          - chmod og= $ID_RSA
          - apk update && apk add openssh-client
          - ssh -i $ID_RSA -o StrictHostKeyChecking=no $SERVER_USER@$SERVER_IP "docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY"
          - ssh -i $ID_RSA -o StrictHostKeyChecking=no $SERVER_USER@$SERVER_IP "docker pull $TAG_COMMIT"
          - ssh -i $ID_RSA -o StrictHostKeyChecking=no $SERVER_USER@$SERVER_IP "docker container rm -f my-app || true"
          - ssh -i $ID_RSA -o StrictHostKeyChecking=no $SERVER_USER@$SERVER_IP "docker run -d -p 80:80 --name my-app $TAG_COMMIT"
      

      Alpine is a lightweight Linux distribution and is sufficient as a Docker image here. You assign the job to the deploy stage. The deployment tag ensures that the job will be executed on runners that are tagged deployment, such as the runner you configured in Step 2.

      The script section of the deploy job starts with two configurative commands:

      • chmod og= $ID_RSA: Revokes all permissions for group and others from the private key, such that only the owner can use it. This is a requirement, otherwise SSH refuses to work with the private key.
      • apk update && apk add openssh-client: Updates Alpine’s package manager (apk) and installs the openssh-client, which provides the ssh command.

      Four consecutive ssh commands follow. The pattern for each is:

      ssh connect pattern for all deployment commands

      ssh -i $ID_RSA -o StrictHostKeyChecking=no $SERVER_USER@$SERVER_IP "command"

      In each ssh statement you are executing command on the remote server. To do so, you authenticate with your private key.

      The options are as follows:

      • -i stands for identity file and $ID_RSA is the GitLab variable containing the path to the private key file.
      • -o StrictHostKeyChecking=no makes sure to bypass the question, whether or not you trust the remote host. This question can not be answered in a non-interactive context such as the pipeline.
      • $SERVER_USER and $SERVER_IP are the GitLab variables you created in Step 5. They specify the remote host and login user for the SSH connection.
      • command will be executed on the remote host.

      The deployment ultimately takes place by executing these four commands on your server:

      1. docker login ...: Logs Docker in to the container registry.
      2. docker pull ...: Pulls the latest image from the container registry.
      3. docker container rm ...: Deletes the existing container if it exists. || true makes sure that the exit code is always successful, even if there was no container running by the name my-app. This guarantees a delete if exists routine without breaking the pipeline when the container does not exist (for example, for the first deployment).
      4. docker run ...: Starts a new container using the latest image from the registry. The container will be named my-app. Port 80 on the host will be bound to port 80 of the container (the order is -p host:container). -d starts the container in detached mode, otherwise the pipeline would be stuck waiting for the command to terminate.

      Note: It may seem odd to use SSH to run these commands on your server, considering the GitLab runner that executes the commands is the exact same server. Yet it is required, because the runner executes the commands in a Docker container, thus you would deploy inside the container instead of the server if you’d execute the commands without the use of SSH. One could argue that instead of using Docker as a runner executor, you could use the shell executor to run the commands on the host itself. But, that would create a constraint to your pipeline, namely that the runner has to be the same server as the one you want to deploy to. This is not a sustainable and extensible solution because one day you may want to migrate the application to a different server or use a different runner server. In any case it makes sense to use SSH to execute the deployment commands, may it be for technical or migration-related reasons.

      Let’s move on by adding this to the deployment job in your .gitlab-ci.yml:

      .gitlab-ci.yml

      . . .
      deploy:
      . . .
        environment:
          name: production
          url: http://your_server_IP
        only:
          - master
      

      GitLab environments allow you to control the deployments within GitLab. You can examine the environments in your GitLab project by going to Operations > Environments. If the pipeline did not finish yet, there will be no environment available, as no deployment took place so far.

      When a pipeline job defines an environment section, GitLab will create a deployment for the given environment (here production) each time the job successfully finishes. This allows you to trace all the deployments created by GitLab CI/CD. For each deployment you can see the related commit and the branch it was created for.

      There is also a button available for re-deployment that allows you to rollback to an older version of the software. The URL that was specified in the environment section will be opened when clicking the View deployment button.

      The only section defines the names of branches and tags for which the job will run. By default, GitLab will start a pipeline for each push to the repository and run all jobs (provided that the .gitlab-ci.yml file exists). The only section is one option of restricting job execution to certain branches/tags. Here you want to execute the deployment job for the master branch only. To define more complex rules on whether a job should run or not, have a look at the rules syntax.

      Note: In October 2020, GitHub has changed its naming convention for the default branch from master to main. Other providers such as GitLab and the developer community in general are starting to follow this approach. The term master branch is used in this tutorial to denote the default branch for which you may have a different name.

      Your complete .gitlab-ci.yml file will look like the following:

      .gitlab-ci.yml

      stages:
        - publish
        - deploy
      
      variables:
        TAG_LATEST: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME:latest
        TAG_COMMIT: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME:$CI_COMMIT_SHORT_SHA
      
      publish:
        image: docker:latest
        stage: publish
        services:
          - docker:dind
        script:
          - docker build -t $TAG_COMMIT -t $TAG_LATEST .
          - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
          - docker push $TAG_COMMIT
          - docker push $TAG_LATEST
      
      deploy:
        image: alpine:latest
        stage: deploy
        tags:
          - deployment
        script:
          - chmod og= $ID_RSA
          - apk update && apk add openssh-client
          - ssh -i $ID_RSA -o StrictHostKeyChecking=no $SERVER_USER@$SERVER_IP "docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY"
          - ssh -i $ID_RSA -o StrictHostKeyChecking=no $SERVER_USER@$SERVER_IP "docker pull $TAG_COMMIT"
          - ssh -i $ID_RSA -o StrictHostKeyChecking=no $SERVER_USER@$SERVER_IP "docker container rm -f my-app || true"
          - ssh -i $ID_RSA -o StrictHostKeyChecking=no $SERVER_USER@$SERVER_IP "docker run -d -p 80:80 --name my-app $TAG_COMMIT"
        environment:
          name: production
          url: http://your_server_IP
        only:
          - master
      

      Finally click Commit changes at the bottom of the page in GitLab to create the .gitlab-ci.yml file. Alternatively, when you have cloned the Git repository locally, commit and push the file to the remote.

      You’ve created a GitLab CI/CD configuration for building a Docker image and deploying it to your server. In the next step you are validating the deployment.

      Step 7 — Validating the Deployment

      Now you’ll validate the deployment in various places of GitLab as well as on your server and in a browser.

      When a .gitlab-ci.yml file is pushed to the repository, GitLab will automatically detect it and start a CI/CD pipeline. At the time you created the .gitlab-ci.yml file, GitLab started the first pipeline.

      Go to CI/CD > Pipelines in your GitLab project to see the pipeline’s status. If the jobs are still running/pending, wait until they are complete. You will see a Passed pipeline with two green checkmarks, denoting that the publish and deploy job ran successfully.

      The pipeline overview page showing a passed pipeline

      Let’s examine the pipeline. Click the passed button in the Status column to open the pipeline’s overview page. You will get an overview of general information such as:

      • Execution duration of the whole pipeline.
      • For which commit and branch the pipeline was executed.
      • Related merge requests. If there is an open merge request for the branch in charge, it would show up here.
      • All jobs executed in this pipeline as well as their status.

      Next click the deploy button to open the result page of the deploy job.

      The result page of the deploy job

      On the job result page you can see the shell output of the job’s script. This is the place to look for when debugging a failed pipeline. In the right sidebar you’ll find the deployment tag you added to this job, and that it was executed on your Deployment Runner.

      If you scroll to the top of the page, you will find the This job is deployed to production message. GitLab recognizes that a deployment took place because of the job’s environment section. Click the production link to move over to the production environment.

      The production environment in GitLab

      You will have an overview of all production deployments. There was only a single deployment so far. For each deployment there is a re-deploy button available to the very right. A re-deployment will repeat the deploy job of that particular pipeline.

      Whether a re-deployment works as intended depends on the pipeline configuration, because it will not do more than repeating the deploy job under the same circumstances. Since you have configured to deploy a Docker image using the commit SHA as a tag, a re-deployment will work for your pipeline.

      Note: Your GitLab container registry may have an expiration policy. The expiration policy regularly removes older images and tags from the container registry. As a consequence, a deployment that is older than the expiration policy would fail to re-deploy, because the Docker image for this commit will have been removed from the registry. You can manage the expiration policy in Settings > CI/CD > Container Registry tag expiration policy. The expiration interval is usually set to something high, like 90 days. But when you run into the case of trying to deploy an image that has been removed from the registry due to the expiration policy, you can solve the problem by re-running the publish job of that particular pipeline as well, which will re-create and push the image for the given commit to registry.

      Next click the View deployment button, which will open http://your_server_IP in a browser and you should see the My Personal Website headline.

      Finally we want to check the deployed container on your server. Head over to your terminal and make sure to log in again, if you have disconnected already (it works for both users, sammy and deployer):

      Now list the running containers:

      Which will list the my-app container:

      Output

      CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 5b64df4b37f8 registry.your_gitlab.com/your_gitlab_user/your_project/master:your_commit_sha "nginx -g 'daemon of…" 4 hours ago Up 4 hours 0.0.0.0:80->80/tcp my-app

      Read the How To Install and Use Docker on Ubuntu 18.04 guide to learn more about managing Docker containers.

      You have now validated the deployment. In the next step, you will go through the process of rolling back a deployment.

      Step 8 — Rolling Back a Deployment

      Next you’ll update the web page, which will create a new deployment and then re-deploy the previous deployment using GitLab environments. This covers the use case of a deployment rollback in case of a defective deployment.

      Start by making a little change in the index.html file:

      1. In GitLab, go to the Project overview and open the index.html file.
      2. Click the Edit button to open the online editor.
      3. Change the file content to the following:

      index.html

      <html>
      <body>
      <h1>My Enhanced Personal Website</h1>
      </body>
      </html>
      

      Save the changes by clicking Commit changes at the bottom of the page.

      A new pipeline will be created to deploy the changes. In GitLab, go to CI/CD > Pipelines. When the pipeline has completed, you can open http://your_server_IP in a browser for the updated web page now showing My Enhanced Personal Website instead of My Personal Website.

      When you move over to Operations > Environments > production you will see the newly created deployment. Now click the re-deploy button of the initial, older deployment:

      A list of the deployments of the production environment in GitLab with emphasize on the re-deploy button of the first deployment

      Confirm the popup by clicking the Rollback button.

      The deploy job of that older pipeline will be restarted and you will be redirected to the job’s overview page. Wait for the job to finish, then open http://your_server_IP in a browser, where you’ll see the initial headline My Personal Website showing up again.

      Let’s summarize what you have achieved throughout this tutorial.

      Conclusion

      In this tutorial, you have configured a continuous deployment pipeline with GitLab CI/CD. You created a small web project consisting of an HTML file and a Dockerfile. Then you configured the .gitlab-ci.yml pipeline configuration to:

      1. Build the Docker image.
      2. Push the Docker image to the container registry.
      3. Log in to the server, pull the latest image, stop the current container, and start a new one.

      GitLab will now deploy the web page to your server for each push to the repository.

      Furthermore you have verified a deployment in GitLab and on your server. You have also created a second deployment and rolled back to the first deployment using GitLab environments, which demonstrates how you deal with defective deployments.

      At this point you have automated the whole deployment chain. You can now share code changes more frequently with the world and/or customer. As a result, development cycles are likely to become shorter, as less time is required to gather feedback and publish the code changes.

      As a next step you could make your service accessible by a domain name and secure the communication with HTTPS for which How To Use Traefik as a Reverse Proxy for Docker Containers is a good follow up.



      Source link