One place for hosting & domains


      How To Configure Keyfile Authentication for MongoDB Replica Sets on Ubuntu 20.04


      MongoDB, also known as Mongo, is an open-source document database used in many modern web applications. It is classified as a NoSQL database because it does not rely on the relational database model. Instead, it uses JSON-like documents with dynamic schemas. This means that, unlike relational databases, MongoDB does not require a predefined schema before you add data to a database.

      When you’re working with multiple distributed MongoDB instances, as in the case of a replica set or a sharded database architecture, it’s important to ensure that the communications between them are secure. One way to do this is through keyfile authentication. This involves creating a special file that essentially functions as a shared password for each member in the cluster.

      This tutorial outlines how to update an existing replica set to use keyfile authentication. The procedure involved in this guide will also ensure that the replica set doesn’t go through any downtime, so the data within the replica set will remain available for any clients or applications that need access to it.


      To complete this tutorial, you will need:

      • Three servers, each running Ubuntu 20.04. All three of these servers should have an administrative non-root user and a firewall configured with UFW. To set this up, follow our initial server setup guide for Ubuntu 20.04.
      • MongoDB installed on each of your Ubuntu servers. Follow our tutorial on How To Install MongoDB on Ubuntu 20.04, making sure to complete each step on each of your servers.
      • All three of your MongoDB installations configured as a replica set. Follow this tutorial on How To Configure a MongoDB Replica Set on Ubuntu 20.04 to set this up.
      • SSH keys generated for each server. In addition, you should ensure that each server has the other two servers’ public keys added to its authorized_keys file. This is to ensure that each machine can communicate with one another over SSH, which will make it easier to distribute the keyfile to each of them in Step 2. To set these up, follow our guide on How To Set Up SSH Keys on Ubuntu 20.04.

      Please note that, for clarity, this guide will follow the conventions established in the prerequisite replica set tutorial and refer to the three servers as mongo0, mongo1, and mongo2. It will also assume that you’ve completed Step 1 of that guide and configured each server’s hosts file so that the following hostnames will resolve to given server’s IP address:

      Hostname Resolves to
      mongo0.replset.member mongo0
      mongo1.replset.member mongo1
      mongo2.replset.member mongo2

      There are a few instances in this guide in which you must run a command or update a file on only one of these servers. In such cases, this guide will default to using mongo0 in examples and will signify this by showing commands or file changes in a blue background, like this:

      Any commands that must be run or file changes that must be made on multiple servers will have a standard gray background, like this:

      About Keyfile Authentication

      In MongoDB, keyfile authentication relies on Salted Challenge Response Authentication Mechanism (SCRAM), the database system’s default authentication mechanism. SCRAM involves MongoDB reading and verifying credentials presented by a user against a combination of their username, password, and authentication database, all of which are known by the given MongoDB instance. This is the same mechanism used to authenticate users who supply a password when connecting to the database.

      In keyfile authentication, the keyfile acts as a shared password for each member in the cluster. A keyfile must contain between 6 and 1024 characters. Keyfiles can only contain characters from the base64 set, and note that MongoDB strips whitespace characters when reading keys. Beginning in version 4.2 of Mongo, keyfiles use YAML format, allowing you to share multiple keys in a single keyfile.

      Warning: The Community version of MongoDB comes with two authentication methods that can help keep your database secure, keyfile authentication and x.509 authentication. For production deployments that employ replication, the MongoDB documentation recommends using x.509 authentication, and it describes keyfiles as “bare-minimum forms of security” that are “best suited for testing or development environments.”

      The process of obtaining and configuring x.509 certificates comes with a number of caveats and decisions that must be made on a case-by-case basis, meaning that this procedure is beyond the scope of a DigitalOcean tutorial. If you plan on using a replica set in a production environment, we strongly encourage you to review the official MongoDB documentation on x.509 authentication.

      If you plan on using your replica set for testing or development, you can proceed with following this tutorial to add a layer of security to your cluster.

      Step 1 — Creating a User Administrator

      When you enable authentication in MongoDB, it will also enable role-based access control for the replica set. Per the MongoDB documentation:

      MongoDB uses Role-Based Access Control (RBAC) to govern access to a MongoDB system. A user is granted one or more roles that determine the user’s access to database resources and operations.

      When access control is enabled on a MongoDB instance, it means that you won’t be able to access any of the resources on the system unless you’ve authenticated as a valid MongoDB user. Even then, you must authenticate as a user with the appropriate privileges to access a given resource.

      If you don’t create a user for your MongoDB system before enabling keyfile authentication (and, consequently, access control), you will not be locked out of your replica set. You can create a MongoDB user which you can use to authenticate to the set and, if necessary, create other users through Mongo’s localhost exception. This is a special exception MongoDB makes for configurations that have enabled access control but lack users. This exception only allows you to connect to the database on the localhost and then create a user in the admin database.

      However, relying on the localhost exception to create a MongoDB user after enabling authentication means that your replica set will go through a period of downtime, since the replicas will not be able to authenticate their connection until after you create a user. This step outlines how to create a user before enabling authentication to ensure that your replica set remains available. This user will have permissions to create other users on the database, giving you the freedom to create other users with whatever permissions they need in the future. In MongoDB, a user with such permissions is known as a user administrator.

      To begin, connect to the primary member of your replica set. If you aren’t sure which of your members is the primary, you can run the rs.status() method to identify it.

      Run the following mongo command from the bash prompt of any of the Ubuntu servers hosting a MongoDB instance in your replica set. This command’s --eval option instructs the mongo operation to not open up the shell interface environment that appears when you run mongo by itself and instead run the command or method, wrapped in single quotes, that follows the --eval argument:

      • mongo --eval 'rs.status()'

      rs.status() returns a lot of information, but the relevant portion of the output is the "members" : array. In the context of MongoDB, an array is a collection of documents held between a pair of square brackets ([ and ]).

      In the "members": array you’ll find a number of documents, each of which contains information about one of the members in your replica set. Within each of these member documents, find the "stateStr" field. The member whose "stateStr" value is "PRIMARY" is the primary member of your replica set. The following example shows a situation where mongo0 is the primary:


      . . . "members" : [ { "_id" : 0, "name" : "mongo0.replset.member:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", . . . }, . . .

      Once you know which of your replica set members is the primary, SSH into the server hosting that instance. For demonstration purposes, this guide will continue to use examples in which mongo0 is the primary:

      • ssh sammy@mongo0_ip_address

      After logging into the server, connect to MongoDB by opening up the mongo shell environment:

      When creating a user in MongoDB, you must create them within a specific database which will be used as their authentication database. The combination of the user’s name and their authentication database serve as a unique identifier for that user.

      Certain administrative actions are only available to users whose authentication database is the admin database — a special privileged database included in every MongoDB installation — including the ability to create new users. Because the goal of this step is to create an user administrator that can create other users in the replica set, connect to the admin database so you can grant this user the appropriate privileges:


      switched to db admin

      MongoDB comes installed with a number of JavaScript-based shell methods you can use to manage your database. One of these, the db.createUser method, is used to create new users in the database in which the method is run.

      Initiate the db.createUser method:

      Note: Mongo won’t register the db.createUser method as complete until you enter a closing parenthesis. Until you do, the prompt will change from a greater than sign (>) to an ellipsis (...).

      This method requires you to specify a username and password for the user, as well as any roles you want the user to have. Recall that MongoDB stores its data in JSON-like documents; when you create a new user, all you’re doing is creating a document to hold the appropriate user data as individual fields.

      As with objects in JSON, documents in MongoDB begin and end with curly braces ({ and }). Enter an opening curly brace to begin the user document:

      Next, enter a user: field, with your desired username as the value in double quotes followed by a comma. The following example specifies the username UserAdminSammy, but you can enter whatever username you like:

      Next, enter a pwd field with the passwordPrompt() method as its value. When you execute the db.createUser method, the passwordPrompt() method will provide a prompt for you to enter your password. This is more secure than the alternative, which is to type out your password in cleartext as you did for your username.

      Note: The passwordPrompt() method is only compatible with MongoDB versions 4.2 and newer. If you’re using an older version of Mongo, then you will have to write out your password in cleartext, similarly to how you wrote out your username:

      Be sure to follow this field with a comma as well:

      Then enter a roles field followed by an array detailing the roles you want your administrative user to have. In MongoDB, roles define what actions the user can perform on the resources that they have access to. You can define custom roles yourself, but Mongo also comes with a number of built-in roles that grant commonly-needed permissions.

      Because you’re creating a user administrator, at a minimum you should grant them the built-in userAdminAnyDatabase role over the admin database. This will allow the user administrator to create and modify new users and roles. Because the administrative user has this role in the admin database, this will also grant it superuser access to the entire cluster:

      • roles: [ { role: "userAdminAnyDatabase", db: "admin" } ]

      Following that, enter a closing brace to signify the end of the document:

      Then enter a closing parenthesis to close and execute the db.createUser method:

      All together, here’s what your db.createUser method should look like:

      > db.createUser(
      ... {
      ... user: "UserAdminSammy",
      ... pwd: passwordPrompt(),
      ... roles: [ { role: "userAdminAnyDatabase", db: "admin" } ]
      ... }
      ... )

      If each line’s syntax is correct, the method will execute properly and you’ll be prompted to enter a password:


      Enter password:

      Enter a strong password of your choosing. Then, you’ll receive a confirmation that the user was added:


      Successfully added user: { "user" : "UserAdminSammy", "roles" : [ { "role" : "userAdminAnyDatabase", "db" : "admin" }, "readWriteAnyDatabase" ] }

      With that, you’ve added a MongoDB user profile which you can use to manage other users and roles on your system. You can test this out by creating another user, as outlined in the remainder of this step.

      Begin by authenticating as the user administrator you just created:

      • db.auth( "UserAdminSammy", passwordPrompt() )

      db.auth() will return 1 if authentication was successful:



      Note: In the future, if you want to authenticate as the user administrator when connecting to the cluster, you can do so directly from your server prompt with a command like the following:

      • mongo -u "UserAdminSammy" -p --authenticationDatabase "admin"

      In this command, the -u option tells the shell that the following argument is the username which you want to authenticate as. The -p flag tells it to prompt you to enter a password, and the --authenticationDatabase option precedes the name of the user’s authentication database. If you enter an incorrect password or the username and authentication database do not match, you won’t be able to authenticate and you’ll have to try connecting again.

      Also, be aware that in order for you to create new users in the replica set as the user administrator, you must be connected to the set’s primary member.

      The procedure for adding another user is the same as it was for the user administrator. The following example creates a new user with the clusterAdmin role, which means they will be able to perform a number of operations related to replication and sharding. Within the context of MongoDB, a user with these privileges is known as a cluster administrator.

      Having a dedicated user to perform specific functions like this is a good security practice, as it limits the number of privileged users you have on your system. After you enable keyfile authentication later in this tutorial, any client that wants to perform any of the operations allowed by the clusterAdmin role — such as any of the rs. methods, like rs.status() or rs.conf() — must first authenticate as the cluster administrator.

      That said, you can provide whatever role you’d like to this user, and likewise provide them with a different name and authentication database. However, if you want the new user to function as a cluster administrator, then you must grant them the clusterAdmin role within the admin database.

      In addition to creating a user to serve as the cluster administrator, the following method names the user ClusterAdminSammy and uses the passwordPrompt() method to prompt you to enter a password:

      • db.createUser(
      • {
      • user: "ClusterAdminSammy",
      • pwd: passwordPrompt(),
      • roles: [ { role: "clusterAdmin", db: "admin" } ]
      • }
      • )

      Again, if you’re using a version of MongoDB that precedes version 4.2, then you will have to write out your password in cleartext instead of using the passwordPrompt() method.

      If each line’s syntax is correct, the method will execute properly and you’ll be prompted to enter a password:


      Enter password:

      Enter a strong password of your choosing. Then, you’ll receive a confirmation that the user was added:


      Successfully added user: { "user" : "ClusterAdminSammy", "roles" : [ { "role" : "clusterAdmin", "db" : "admin" } ] }

      This output confirms that your user administrator is able to create new users and grant them roles. You can now close the MongoDB shell:

      Alternatively, you can close the shell by pressing CTRL + C.

      At this point, if you have any clients or applications connected to your MongoDB cluster, it would be a good time to create one or more dedicated users with the appropriate roles which they can use to authenticate to the database. Otherwise, read on to learn how to generate a keyfile, distribute it among the members of your replica set, and then configure each one to require the replica set members to authenticate with the keyfile.

      Step 2 — Creating and Distributing an Authentication Keyfile

      Before creating a keyfile, it can be helpful to create a directory on each server where you will store the keyfile in order to keep things organized. Run the following command, which creates a directory named mongo-security in the administrative Ubuntu user’s home directory, on each of your three servers:

      Then generate a keyfile on one of your servers. You can do this on any one of your servers but, for illustration purposes, this guide will generate the keyfile on mongo0.

      Navigate to the mongo-security directory you just created:

      Within that directory, create a keyfile with the following openssl command:

      • openssl rand -base64 768 > keyfile.txt

      Take note of this command’s arguments:

      • rand: instructs OpenSSL to generate pseudo-random bytes of data
      • -base64: specifies that the command should use base64 encoding to represent the pseudo-random data as printable text. This is important because, as mentioned previously, MongoDB keyfiles can only contain characters in the base64 set
      • 768: the number of bytes the command should generate. In base64 encoding, three binary bytes of data are represented as four characters. Because MongoDB keyfiles can have a maximum of 1024 characters, 768 is the maximum number of bytes you can generate for a valid keyfile

      Following this command’s 768 argument is a greater-than sign (>). This redirects the command’s output into a new file named keyfile.txt which will serve as your keyfile. Feel free to name the keyfile something other than keyfile.txt if you’d like, but be sure to change the filename whenever it appears in later commands.

      Next, modify the keyfile’s permissions so that only the owner has read access:

      Following this, distribute the keyfile to the other two servers hosting the MongoDB instances in your replica set. Assuming you followed the prerequisite guide on How To Set Up SSH Keys, you can do so with the scp command:

      • scp keyfile.txt sammy@mongo1.replset.member:/home/sammy/mongo-security
      • scp keyfile.txt sammy@mongo2.replset.member:/home/sammy/mongo-security

      Notice that each of these commands copies the keyfile directly to the ~/mongo-security/ directories you created previously on mongo1 and mongo2. Be sure to change sammy to the name of the administrative Ubuntu user profile you created on each server.

      Next, change the file’s owner to the mongodb user profile. This is a special user that was created when you installed MongoDB, and it’s used to run the mongod service. This user must have access to the keyfile in order for MongoDB to use it for authentication.

      Run the following command on each of your servers to change the keyfile’s owner to the mongodb user account:

      • sudo chown mongodb:mongodb ~/mongo-security/keyfile.txt

      After changing the keyfiles’ owner on each server, you’re ready to reconfigure each of your MongoDB instances to enforce keyfile authentication.

      Step 3 — Enabling Keyfile Authentication

      Now that you’ve generated a keyfile and distributed it to each of the servers in your replica set, you can update the MongoDB configuration file on each server to enforce keyfile authentication.

      In order to avoid any downtime while configuring the members of your replica set to require authentication, this step involves reconfiguring the secondary members of the set first. Then, you’ll direct your primary member to step down and become a secondary member. This will cause the secondary members to hold an election to select a new primary, keeping your cluster available to whatever clients or applications need access to it. You’ll then reconfigure the former primary node to enable authentication.

      On each of your servers hosting a secondary member of your replica set, open up MongoDB’s configuration file with your preferred text editor:

      • sudo nano /etc/mongod.conf

      Within the file, find the security section. It will look like this by default:


      . . .
      . . .

      Uncomment this line by removing the pound sign (#). Then, on the next line, add a keyFile: directive followed by the full path to the keyfile you created in the previous step:


      . . .
        keyFile: /home/sammy/mongo-security/keyfile.txt
      . . .

      Note that there are two spaces at the beginning of this new line. These are necessary for the configuration file to be read correctly. When you enter this line in your own configuration files, make sure that the path you provide reflects the actual path of the keyfile on each server.

      Below the keyFile directive, add a transitionToAuth directive with a value of true. When set to true, this configuration option allows the MongoDB instance to accept both authenticated and non-authenticated connections. This is useful when reconfiguring a replica set to enforce authentication, as it will ensure that your data remains available as you restart each member of the set:


      . . .
        keyFile: /home/sammy/mongo-security/keyfile.txt
        transitionToAuth: true
      . . .

      Again, make sure that you include two blank spaces before the transitionToAuth directive.

      After making those changes, save and close the file. If you used nano to edit it, you can do so by pressing CTRL + X, Y, and then ENTER.

      Then restart the mongod service on both of the secondary instances’ servers to immediately put these changes into effect:

      • sudo systemctl restart mongod

      With that, you’ve configured keyfile authentication for the secondary members of your replica set. At this point, both authenticated and non-authenticated users can access these members without restriction.

      Next, you’ll repeat this procedure on the primary member. Before doing so, though, you must step down the member so it’s no longer the primary. To do this, open up the MongoDB shell on the server hosting the primary member. For illustration purposes, this guide will again assume this is mongo0:

      From the prompt, run the rs.stepDown() method. This will instruct the primary to become a secondary member, and will cause the current secondary members to hold an election to determine which will serve as the new primary:

      If the method returns "ok" : 1 in the output, it means the primary member successfully stepped down to become a secondary:


      { "ok" : 1, "$clusterTime" : { "clusterTime" : Timestamp(1614795467, 1), "signature" : { "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="), "keyId" : NumberLong(0) } }, "operationTime" : Timestamp(1614795467, 1) }

      After stepping down the primary, you can close the Mongo shell:

      Next, open up the MongoDB configuration file on this server:

      • sudo nano /etc/mongod.conf

      Find the security section and uncomment the security header by removing the pound sign. Then add the same keyFile and transitionToAuth directives you added to the other MongoDB instances. After making these changes, the security section will look like this:


      . . .
        keyFile: /home/sammy/mongo-security/keyfile.txt
        transitionToAuth: true
      . . .

      Again, make sure that the file path after the keyFile directive reflects the keyfile’s actual location on this server.

      When finished, save and close the file. Then restart the mongod process:

      • sudo systemctl restart mongod

      Following that, all of your MongoDB instances are able to accept both authenticated and non-authenticated connections. In the final step of this guide, you’ll configure your instances to require users to authenticate before performing privileged actions.

      Step 4 — Restarting Each Member Without transitionToAuth to Enforce Authentication

      At this point, each of your MongoDB instances are configured with the transitionToAuth set to true. This means that even though you’ve enabled each server to use the keyfile you created to authenticate connections internally, they’re still able to accept non-authenticated connections.

      To change this and require each member to enforce authentication, reopen the mongod.conf file on each server:

      • sudo nano /etc/mongod.conf

      Find the security section and disable the transitionToAuth directive. You can do this by commenting the line out by prepending it with a pound sign:


      . . .
        keyFile: /home/sammy/mongo-security/keyfile.txt
        #transitionToAuth: true
      . . .

      After disabling the transitionToAuth directive in each instance’s configuration file, save and close each file.

      Then, restart the mongod service on each server:

      • sudo systemctl restart mongod

      Following that, each of the MongoDB instances in your replica set will require you to authenticate to perform privileged actions.

      To test this, try running a MongoDB method that works when invoked by an authenticated user that has the appropriate privileges. Try running the following command from any of your Ubuntu servers’ prompts:

      • mongo --eval 'rs.status()'

      Even though you ran this method successfully in Step 1, the rs.status() method can now only be run by a user that has been granted the clusterAdmin or clusterManager roles since you’ve enabled keyfile authentication. Regardless of whether you run this command on a server hosting the primary member or one of the secondary members, it will not work because you have not authenticated:


      . . . MongoDB server version: 4.4.4 { "operationTime" : Timestamp(1616184183, 1), "ok" : 0, "errmsg" : "command replSetGetStatus requires authentication", "code" : 13, "codeName" : "Unauthorized", "$clusterTime" : { "clusterTime" : Timestamp(1616184183, 1), "signature" : { "hash" : BinData(0,"huJUmB/lrrxpx9YfnONM4mayJwo="), "keyId" : NumberLong("6941116945081040899") } } }

      Recall that, after enabling access control, all of the cluster administration methods (including rs. methods like rs.status()) will only work when invoked by an authenticated user that has been granted the appropriate cluster management roles. If you’ve created a cluster administrator — as outlined in Step 1 — and authenticate as that user, then this method will work as expected:

      • mongo -u "ClusterAdminSammy" -p --authenticationDatabase "admin" --eval 'rs.status()'

      After entering the user’s password when prompted, you will see the rs.status() method’s output:


      . . . MongoDB server version: 4.4.4 { "set" : "shard2", "date" : ISODate("2021-03-19T20:21:45.528Z"), "myState" : 2, "term" : NumberLong(4), "syncSourceHost" : "mongo1.replset.member:27017", "syncSourceId" : 2, "heartbeatIntervalMillis" : NumberLong(2000), "majorityVoteCount" : 2, . . .

      This confirms that the replica set is enforcing authentication, and that you’re able to authenticate successfully.


      By completing this tutorial, you created a keyfile with OpenSSL and then configured a MongoDB replica set to require its members to use it for internal authentication. You also created a user administrator which will allow you to manage users and roles in the future. Throughout all of this, your replica set will not have gone through any downtime and your data will have remained available to your clients and applications.

      If you’d like to learn more about MongoDB, we encourage you to check out our entire library of MongoDB content.

      Source link

      How To Set Up Multi-Factor Authentication for SSH on CentOS 8

      Not using CentOS 8?

      Choose a different version or distribution.

      The author selected the COVID-19 Relief Fund to receive a donation as part of the Write for DOnations program.


      SSH uses passwords for authentication by default, and most SSH hardening instructions recommend using an SSH key instead. However, an SSH key is still only a single factor, though a much more secure factor. The channel is the terminal on your computer sending the data via an encrypted tunnel to the remote machine. But like a hacker can guess a password, they can steal an SSH key, and then in either case, with that single piece of data, an attacker can gain access to your remote systems.

      In this tutorial, we’ll set up multi-factor authentication to combat that. Multi-factor authentication (MFA) or Two-factor authentication (2FA) requires more than one factor to authenticate or log in. This means a bad actor would have to compromise multiple things, like your computer and your phone, to get in. There are several types of factors used in authentication:

      1. Something you know, like a password or security question
      2. Something you have, like an authenticator app or security token
      3. Something you are, like your fingerprint or voice

      One common factor is an OATH-TOTP app, like Google Authenticator. OATH-TOTP (Open Authentication Time-Based One-Time Password) is an open protocol that generates a one-time use password, commonly a six-digit number recycled every 30 seconds.

      This article will go over how to enable SSH authentication using an OATH-TOTP app in addition to an SSH key. Logging into your server via SSH will require two factors across two channels, thereby making it more secure than a password or SSH key alone. Also, we’ll go over some additional use cases for MFA and some helpful tips and tricks.

      And if you are seeking further guidance on securing SSH connections, check out these tutorials on Hardening OpenSSH and Hardening OpenSSH Client.


      To follow this tutorial, you will need:

      • One CentOS 8 server with a sudo non-root user and SSH key, which you can set up by following this Initial Server Setup tutorial.
      • A smartphone or tablet with an OATH-TOTP app installed, like Google Authenticator (iOS, Android).
      • Alternatively, you can also use a Linux command line app called ‘oathtool’ to generate an OATH-TOTP code. It’s available in various distribution repos
      • If you really want secure your SSH connection there are several good steps outlined in this SSH Essentials article, such as whitelisting users, disabling root login, and changing which port SSH uses.

      Step 1 — Installing Google’s PAM

      In this step, we’ll install and configure Google’s PAM.

      PAM, which stands for Pluggable Authentication Module, is an authentication infrastructure used on Linux systems to authenticate a user. Because Google made an OATH-TOTP app, they also made a PAM that generates TOTPs and is fully compatible with any OATH-TOTP app, like Google Authenticator or Authy.

      First, add the EPEL (Extra Packages for Enterprise Linux) repo:

      If your repositories have the EPEL install package, you will see the following output:


      ===== Name Matched: epel ===== epel-release.noarch : Extra Packages for Enterprise Linux repository configuration

      Now install the epel-release package to enable the EPEL repository:

      • sudo yum install epel-release

      However, if you don’t have the package epel-release, then you can install the repository information manually:

      • sudo yum install

      Next, install the PAM. You might be prompted to accept the EPEL key if this is the first time using the repo. Once accepted, you won’t be prompted for again to accept the key:

      • sudo yum install google-authenticator qrencode-libs

      With the PAM installed, we’ll use a helper app that comes with the PAM to generate a TOTP key for the user you want to add a second factor to. This key is generated on a user-by-user basis, not system-wide. This means every user that wants to use a TOTP auth app will need to log in and run the helper app to get their own key; you can’t just run it once to enable it for everyone (but there are some tips at the end of this tutorial to set up or require MFA for many users).

      Run these two commands to initialize the app:

      • google-authenticator -s ~/.ssh/google_authenticator

      Normally, all you need to do is run the google-authenticator command with no arguments, but SELinux doesn’t allow the ssh daemon to write to files outside of the .ssh directory in your home folder. This prevents authentication.

      SELinux is a powerful tool that protects your system from potential attacks, and it’s worth running in Enforcing mode. As such, turning off SELinux is not considered a best practice. Instead, we’ll move the default location of the google_authenticator file into your ~/.ssh directory.

      After you run the command, you’ll be asked a few questions. The first one asks if authentication tokens should be time-based:


      Do you want authentication tokens to be time-based (y/n) y

      This PAM allows for time-based or sequential-based tokens. Using sequential-based tokens mean the code starts at a certain point and then increments the code after every use. Using time-based tokens mean the code changes after a certain time frame. We’ll stick with time-based because that is what apps like Google Authenticator anticipate, so answer y for yes.

      After answering this question, a lot of output will scroll past, including a large QR code. Use your authenticator app on your phone to scan the QR code or manually type in the secret key. If the QR code is too big to scan, you can use the URL above the QR code to get a smaller version. Once it’s added, you’ll see a six-digit code that changes every 30 seconds in your app.

      Note: Make sure you record the secret key, verification code, and the emergency scratch codes in a safe place, like a password manager. The emergency scratch codes are the only way to regain access if you, for example, lose access to your TOTP app.

      The remaining questions inform the PAM on how to function. We’ll go through them one by one:


      Do you want me to update your "~/.google_authenticator" file (y/n) y

      This writes the key and options to the google_authenticator file. If you say no, the program quits and nothing is written, which means the authenticator won’t work:


      Do you want to disallow multiple uses of the same authentication token? This restricts you to one login about every 30s, but it increases your chances to notice or even prevent man-in-the-middle attacks (y/n) y

      By answering yes here, you are preventing a replay attack by making each code expire immediately after use. This prevents an attacker from capturing a code you just used and logging in with it:


      By default, a new token is generated every 30 seconds by the mobile app. In order to compensate for possible time-skew between the client and the server, we allow an extra token before and after the current time. This allows for a time skew of up to 30 seconds between authentication server and client. If you experience problems with poor time synchronization, you can increase the window from its default size of 3 permitted codes (one previous code, the current code, the next code) to 17 permitted codes (the 8 previous codes, the current code, and the 8 next codes). This will permit for a time skew of up to 4 minutes between client and server. Do you want to do so? (y/n) n

      Answering yes here allows up to 17 valid codes in a moving four minute window. By answering no, you limit it to 3 valid codes in a 1:30 minute rolling window. Unless you find issues with the 1:30 minute window, answering no is the more secure choice. If you answer no and later realize you need more time, this setting can be adjusted in the .google_authenticator file stored at the root of your home directory:


      If the computer that you are logging into isn't hardened against brute-force login attempts, you can enable rate-limiting for the authentication module. By default, this limits attackers to no more than 3 login attempts every 30s. Do you want to enable rate-limiting (y/n) y

      Rate limiting means a remote attacker can only attempt a certain number of guesses before being forced to wait some time before being able to try again. If you haven’t previously configured rate limiting directly into SSH, doing so now is a great hardening technique.

      Once you finish this setup, if you want to back up your secret key, you can copy the ~/.ssh/google-authenticator file to a trusted location. From there, you can deploy it on additional systems or redeploy it after a clean install. Be aware that by using the same key on multiple machines you are introducing the possibility of a replay attack if an attacker is able to sniff the token for one server then use it, within the valid window, against a different server. You must evaluate the likelihood of that risk versus having every system use a different token and managing those different tokens.

      Note: If you are using an encrypted home folder, which is beyond the scope of this article, you may need to store the ~./google-authenticator file in a directory outside of your home folder. The project README has details on how to do that.

      Since we stored the config file in a non-standard location, we need to restore the SELinux context based on its new location.

      Use the following command to do so:

      With those two changes done we now have the Google Authenticator PAM installed and configured, the next step is to configure SSH to use your TOTP key. We’ll need to tell SSH about the PAM and then configure SSH to use it.

      Step 2 — Configuring OpenSSH to Use MFA/2FA

      Because we’ll be making SSH changes over SSH, it’s important to never close your initial SSH connection. Instead, open a second SSH session to do testing. This is to avoid locking yourself out of your server if there was a mistake in your SSH configuration. Once everything works, then you can safely close any sessions. Another safety precaution is to create a backup of the system files you will be editing so in case something goes wrong you can simply revert to the vanilla file and start over again with a clean configuration.

      To begin, back up the sshd configuration file and then edit it. Here, we’re using nano, which isn’t installed on CentOS by default. You can install it with sudo yum install nano, or use your favorite alternative text editor.

      Backup the file and then open it:

      • sudo cp /etc/pam.d/sshd /etc/pam.d/sshd.bak
      • sudo nano /etc/pam.d/sshd

      Add the following line to the end of the file:


      auth       required secret=/home/${USER}/.ssh/google_authenticator nullok
      auth       required

      Since we had to put the google_authenticator config file in a non-standard location, we have to provide PAM with the path to the config file. The secret option tells PAM where the non-default location of the config file is stored.

      The nullok word at the end of the line tells the PAM that this authentication method is optional. This allows users without a OATH-TOTP token to still log in just using their SSH key. Once all users have an OATH-TOTP token, you can remove nullok from this line to make MFA mandatory.

      The second line with is required to allow authentication if I user doesn’t use an MFA token to login. When logging in each method needs a SUCCESS to allow authentication. If a user doesn’t use the MFA auth tool by utilizing the nullok option that returns an IGNORE for the interactive keyboard authentication. So if that line is ignored then the next line triggers which returns SUCCESS and allows authentication to proceed.

      Save and close the file.

      Next, we’ll configure SSH to support this kind of authentication.

      Back up the the SSH configuration file then open it for editing:

      • sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak
      • sudo nano /etc/ssh/sshd_config

      Look for lines beginning ChallengeResponseAuthentication. Comment out the no line and uncomment the yes line.

      You file will look like this:


      . . .
      # Change to no to disable s/key passwords
      ChallengeResponseAuthentication yes
      #ChallengeResponseAuthentication no
      . . .

      Save and close the file and then restart SSH to reload the configuration files. Restarting the sshd service won’t close our current open connections, so you won’t risk locking yourself out with this command:

      • sudo systemctl restart sshd.service

      To test that everything’s working so far, open ANOTHER terminal window and try logging in over SSH. It is very important that you keep your current SSH session open and test with an additional session or you will lock yourself out at some point and will need to use the web console to get yourself back in.

      Note: If you’ve previously created an SSH key and are using it, you’ll notice you didn’t have to type in your user’s password or the MFA verification code. This is because an SSH key overrides all other authentication options by default. Otherwise, you should have gotten a password and verification code prompt.

      Next, to enable an SSH key as one factor and the verification code as a second, we need to tell SSH which factors to use and prevent the SSH key from overriding all other types.

      Step 3 — Making SSH Aware of MFA

      Reopen the sshd configuration file:

      • sudo nano /etc/ssh/sshd_config

      Add the following line at the bottom of the file. This tells SSH which authentication methods are required. This line tells SSH we need a SSH key and either a password or a verification code (or all three):


      . . .
      AuthenticationMethods publickey,password publickey,keyboard-interactive

      Save and close the file.

      Next, open the PAM sshd configuration file again:

      • sudo nano /etc/pam.d/sshd

      Find the line auth substack password-auth towards the top of the file. Comment it out by adding a # character as the first character on the line. This tells PAM not to prompt for a password:


      . . .
      #auth       substack     password-auth
      . . .

      Save and close the file, then restart SSH:

      • sudo systemctl restart sshd.service

      Now try logging into the server again with a different terminal session/window. Unlike last time, SSH should ask for your verification code. Upon entering it, you’ll be logged in. Even though you don’t see any indication that your SSH key was used, your login attempt used two factors. If you want to verify, you can add -v (for verbose) after the SSH command:

      Example SSH Output

      . . . debug1: Authentications that can continue: publickey debug1: Next authentication method: publickey debug1: Offering RSA public key: /Users/sammy/.ssh/id_rsa debug1: Server accepts key: pkalg ssh-rsa blen 279 Authenticated with partial success. debug1: Authentications that can continue: keyboard-interactive debug1: Next authentication method: keyboard-interactive Verification code:

      Towards the end of the output, you’ll see where SSH uses your SSH key and then asks for the verification code. You can now log in over SSH with a SSH key and a one-time password. If you want to enforce all three authentication types, you can follow the next step.

      You have now successfully added a second factor when logging in remotely to your server over SSH. If this is what you wanted — to use your SSH key and a TOTP token to enable MFA for SSH (for most people, this is the optimal configuration) — then you’re done.

      What follows are some tips and tricks for recovery, automated usage, and more.

      Step 4 — Adding a Third Factor (Optional)

      In Step 3, we listed the approved types of authentication in the sshd_config file:

      1. publickey (SSH key)
      2. password publickey (password)
      3. keyboard-interactive (verification code)

      Although we listed three different factors, with the options we’ve chosen so far, they only allow for an SSH key and the verification code. If you’d like to have all three factors (SSH key, password, and verification code), one quick change will enable all three.

      Open the PAM sshd configuration file:

      • sudo nano /etc/pam.d/sshd

      Locate the line you commented out previously, #auth substack password-auth, and uncomment the line by removing the # character. Save and close the file. Now once again, restart SSH:

      • sudo systemctl restart sshd.service

      By enabling the option auth substack password-auth, PAM will now prompt for a password in addition the checking for an SSH key and asking for a verification code, which we had working previously. Now we can use something we know (password) and two different types of things we have (SSH key and verification code) over two different channels (your computer for the SSH key and your phone for the TOTP token).

      Step 5 — Recovering Access to Google MFA (optional)

      As with any system that you harden and secure, you become responsible for managing that security. In this case, that means not losing your SSH key or your TOTP secret key and making sure you have access to your TOTP app. However, sometimes things happen, and you can lose control of the keys or apps you need to get in.

      Losing Access to the Secret Key

      If you lose your TOTP secret key, recovery can be broken up into a couple of steps. The first is getting back in without knowing the verification code and the second is finding the secret key or regenerating it for normal MFA login. This often can happen if you get a new phone and don’t transfer over your secrets to a new authenticator app.

      To get in after losing the TOTP secret key on a DigitalOcean Droplet, you can simply use the virtual console from your dashboard to log in using your username and password. This works because we only protected your user account with MFA for ssh connections. Non-ssh connections, such as a console login, doesn’t use the Google Authenticator PAM module.

      If you’re on a non-Droplet system then you have two options for regaining access:

      1. Console (local/non-ssh) access to the system (typically physical or via something like iDrac)
      2. Have a different user that doesn’t have MFA enabled

      The second option is the less secure option, since the point in using MFA is to harden all ssh connections, but it’s one fail safe if you lose access to your MFA authenticator app.

      Once you’re logged in, there are two ways to get the TOTP secret:

      1. Recover the existing key
      2. Generate a new key

      In each user’s home directory, the secret key and Google Authenticator settings are saved in the file ~/.ssh/google-authenticator. The very first line of this file is a secret key. A quick way to get the key is to execute the following command, which displays the first line of the google-authenticator file (i.e. the secret key). Then, take that secret key and manually type it into a TOTP app:

      • head -n 1 /home/sammy/.ssh/google_authenticator

      Once you’ve recovered your existing key, you can either manually type it into your authenticator app and then you should be back in business or you can fill in the relevant details in the URL below and have Google generate a QR code for you to scan. You’ll need to add your username, hostname, the secret key from the google-authenticator file, and then any name of your choosing for 'entry-name-in-auth-app’ to easily identify this key versus a different TOTP token:|0&cht=qr&chl=otpauth://totp/username@hostname%3Fsecret%3D16-char-secret%26issuer%3Dentry-name-in-auth-app

      If there is a reason not to use the existing key (for example, being unable to easily share the secret key with the impacted user securely), you can remove the ~/.ssh/google-authenticator file outright. This will allow the user to log in again using only a single factor, assuming you haven’t enforced MFA by removing the nullok option in the ’/etc/pam.d/sshd’ file. They can then run google-authenticator to generate a new key.

      Losing Access to the TOTP App

      If you need to log in to your server but don’t have access to your TOTP app to get your verification code, you can still log in using the recovery codes that were displayed when you first created your secret key. Note that these recovery codes are one-time use. Once one is used to log in, it cannot be used as a verification code again. Hopefully you saved them in a place that is accessible if you don’t have your TOTP app and yet still secure.

      Step 6 — Changing Authentication Settings (optional)

      If you want to change your MFA settings after the initial configuration, instead of generating a new configuration with the updated settings, you can just edit the ~/.ssh/google-authenticator file. This file is laid out in the following manner:

      .google-authenticator layout

      <secret key>
      <recovery codes>

      Options that are set in this file have a line in the options section; if you answered “no” to a particular option during the initial setup, the corresponding line is excluded from the file. In other words, the file only contains enabled options. If an option isn’t present it is by default disabled.

      Here are the changes you can make to this file:

      • To enable sequential codes instead of time based codes, change the line " TOTP_AUTH to " HOTP_COUNTER 1.
      • To allow multiple uses of a single code, remove the line " DISALLOW_REUSE.
      • To extend the code expiration window to 4 minutes, add the line " WINDOW_SIZE 17.
      • To disable multiple failed logins (rate limiting), remove the line " RATE_LIMIT 3 30.
      • To change the threshold of rate limiting, find the line " RATE_LIMIT 3 30 and adjust the numbers. The 3 in the original indicates the number of attempts over a period of time, and the 30 indicates the period of time in seconds.
      • To disable the use of recovery codes, remove the five 8 digit codes at bottom of the file.

      Step 7 — Avoiding MFA for Some Accounts (optional)

      There may be a situation in which a single user or a few service accounts (i.e. accounts used by applications, not humans) need SSH access without MFA enabled. For example, some applications that use SSH, like some FTP clients, may not support MFA. If an application doesn’t have a way to request the verification code, the request may get stuck until the SSH connection times out.

      As long as a couple of options in /etc/pam.d/sshd are set correctly, you can control which factors are used on a user-by-user basis.

      To allow MFA for some accounts and SSH key only for others, make sure the following settings in /etc/pam.d/sshd are active:


      auth       required
      #auth       substack     password-auth
      . . .
      # Used with polkit to reauthorize users in remote sessions
      -session   optional prepare
      auth       required nullok

      Here, auth substack password-auth is commented out because passwords need to be disabled. MFA cannot be forced if some accounts are meant to have MFA disabled, so leave the nullok option on the final line.

      After setting this configuration, simply run google-authenticator as any users that need MFA, and don’t run it for users where only SSH keys will be used.

      Step 8 — Automating Setup with Configuration Management (optional)

      Many system administrators use configuration management tools, like Puppet, Chef, or Ansible, to manage their systems. If you want to use a system like this to install set up a secret key when a new user’s account is created, there is a method to do that.

      google-authenticator supports command line switches to set all the options in a single, non-interactive command. To see all the options, you can type google-authenticator --help. Below is the command that would set everything up as outlined in Step 1:

      • google-authenticator -t -d -f -r 3 -R 30 -w 3 -s ~/.ssh/google_authenticator

      The options referenced above are as follows:

      • -t => Time based counter
      • -d => Disallow token reuse
      • -f => Force writing the settings to file without prompting the user
      • -r => How many attempts to enter the correct code
      • -R => How long in seconds a user can attempt to enter the correct code
      • -w => How many codes can are valid at a time (this references the 1:30 min - 4 min window of valid codes)
      • -s => Path to where the authentication file should be stored

      These answers all the questions we answered manually, saves it to a file, and then outputs the secret key, QR code, and recovery codes. (If you add the flag -q, then there won’t be any output.) If you do use this command in an automated fashion, make sure to capture the secret key and/or recovery codes and make them available to the user. Also remember to have your automation reset the SELinux context of the ’.ssh’ directory (restorecon -Rv ~/.ssh/).

      Step 9 — Forcing MFA for All Users (optional)

      If you want to force MFA for all users even on the first login, or if you would prefer not to rely on your users to generate their own keys, there’s an easy way to handle this. You can simply use the same google-authenticator file for each user, as there’s no user-specific data stored in the file.

      To do this, after the configuration file is initially created, a privileged user needs to copy the file to the .ssh directory of every home directory and change its permissions to the appropriate user. You can also copy the file to /etc/skel/ so it’s automatically copied over to a new user’s home directory upon creation.

      Warning: This can be a security risk because everyone is sharing the same second factor. This means that if it’s leaked, it’s as if every user had only one factor. Take this into consideration if you want to use this approach.

      Another method to force the creation of a user’s secret key is to use a bash script that:

      1. Creates a TOTP token,
      2. Prompts them to download the Google Authenticator app and scan the QR code that will be displayed, and
      3. Runs the google-authenticator application for them after checking if the google-authenticator file already exists.

      To make sure the script runs when a user logs in, you can name it .bash_login and place it at the root of their home directory.


      In this tutorial, you added two factors (an SSH key + MFA token) across two channels (your computer + your phone) to your server. You’ve made it very difficult for an outside agent to brute force their way into your machine via SSH and greatly increased the security of your machine.

      And remember, if you are seeking further guidance on securing SSH connections, check out these tutorials on Hardening OpenSSH and Hardening OpenSSH Client.

      Source link