Secrets Archives - XpresServers Hosting

Use HashiCorp Vault to Manage Secrets

Updated Tuesday, June 4, 2019 by Linode Contributed by Linode

HashiCorp Vault is a secrets management tool that helps to provide secure, automated access to sensitive data. Vault meets these use cases by coupling authentication methods (such as application tokens) to secret engines (such as simple key/value pairs) using policies to control how access is granted. In this guide, you will install, configure, and access Vault in an example deployment to illustrate Vault’s features and API.

This guide will use the latest version of Vault, which is 1.1.0 at the time of this writing.

Why Use Vault?

A service such as Vault requires operational effort to run securely and effectively. Given the added complexity of using Vault as part of an application, in what way does it add value?

Consider a simple application that must use an API token or other secret value. How should this sensitive credential be given to the application at runtime?

Committing the secret alongside the rest of the application code in a version control system such as git is a poor security practice for a number of reasons, including that the sensitive value is recorded in plaintext and not protected in any way.
Recording a secret in a file that is passed to an application requires that the file be securely populated in the first place and strictly access-controlled.
Static credentials are challenging to rotate or restrict access to if an application is compromised.

Vault solves these and other problems in a number of ways, including:

Services and applications that run without operator interaction can authenticate to Vault using values that can be rotated, revoked, and permission-controlled.
Some secrets engines can generate temporary, dynamically-generated secrets to ensure that credentials expire after a period of time.
Policies for users and machine accounts can be strictly controlled for specific types of access to particular paths.

Concepts

Before continuing, you should familiarize yourself with important Vault terms and concepts that will be used later in this guide.

A token is the the underlying mechanism that underpins access to Vault resources. Whether a user authenticates to Vault using a GitHub token or an application-driven service authenticates using an AppRole RoleID and SecretID, all forms of authentication are eventually normalized to a token. Tokens are typically short-lived (that is, expire after a period or time-to-live, or ttl) and have one or more policies attached to them.
A Vault policy dictates certain actions that may be performed upon a Vault path. Capabilities such as the ability to read a secret, write secrets, and delete them are all examples of actions that are defined in a policy for a particular path.
A path in Vault is similar in form to a Unix filesystem path (like /etc) or a URL (such as /blog/title). Users and machine accounts interact with Vault over particular paths in order to retrieve secrets, change settings, or otherwise interact with a running Vault service. All Vault access is performed over a REST interface, so these paths eventually take the form of an HTTP URL. While some paths interact with the Vault service itself to manage resources such as policies or settings, many paths serve as an endpoint to either authenticate to Vault or interact with a secret engine.
A secret engine is a backend used in Vault to provide secrets to Vault users. The simplest example of a secret engine is the key/value backend, which simply returns plain text values that may be stored at particular paths (these secrets remain encrypted on the backend). Other examples of secret backends include the PKI backend, which can generate and manage TLS certificates, and the TOTP backend, which can generate temporary one-time passwords for web sites that require multi-factor authentication (including the Linode Manager).

Installation

This guide will setup Vault in a simple, local filesystem-only configuration. The steps listed here apply equally to any distribution.

These installation steps will:

Procure a TLS certificate to ensure that all communications between Vault and clients are encrypted.
Configure Vault for local filesystem storage.
Install the vault binary and set up the operating system to operate Vault as a service.

Note

The configuration outlined in this guide is suitable for small deployments. In situations that call for highly-available or fault-tolerant services, consider running more than one Vault instance with a highly-available storage backend such as Consul.

Before you Begin

Familiarize yourself with Linode’s Getting Started guide and complete the steps for deploying and setting up a Linode running a recent Linux distribution (such as Ubuntu 18.04 or CentOS 7), including setting the hostname and timezone.

Note

Setting the full hostname correctly in /etc/hosts is important in this guide in order to terminate TLS on Vault correctly. Your Linode’s fully qualified domain name and short hostname should be present in the /etc/hosts file before continuing.
This guide uses sudo wherever possible. Complete the sections of our Securing Your Server guide to create a standard user account, harden SSH access, and remove unnecessary network services.
Follow our UFW Guide in order to install and configure a firewall on your Ubuntu or Debian-based system, or our FirewallD Guide for rpm or CentOS-based systems. Consider reviewing Vault’s Production Hardening recommendations if this will be used in a production environment.

Note

When configuring a firewall, keep in mind that Vault listens on port 8200 by default and Let’s Encrypt utilizes ports 80 (HTTP) and 443 (HTTPS).
Ensure your system is up to date. On Debian-based systems, use:
```
sudo apt update && sudo apt upgrade
```
While on rpm-based systems, such as CentOS, use:
```
sudo yum update
```

Acquire a TLS Certificate

Follow the steps in our Secure HTTP Traffic with Certbot guide to acquire a TLS certificate.
Add a system group in order to grant limited read access to the TLS files created by Certbot.
```
sudo groupadd tls
```
Change the group ownership of certificate files in the Let’s Encrypt directory to tls.
```
sudo chgrp -R tls /etc/letsencrypt/{archive,live}
```

Grant members of the tls group read access to the necessary directories and files.

sudo chmod g+rx /etc/letsencrypt/{archive,live}
sudo find /etc/letsencrypt/archive -name 'privkey*' -exec chmod g+r {} ';'

Download Vault files

Download the release binary for Vault.
```
wget https://releases.hashicorp.com/vault/1.1.0/vault_1.1.0_linux_amd64.zip
```
Note

If you receive an error that indicates wget is missing from your system, install the wget package and try again.
Download the checksum file, which will verify that the zip file is not corrupt.
```
wget https://releases.hashicorp.com/vault/1.1.0/vault_1.1.0_SHA256SUMS
```
Download the checksum signature file, which verifies that the checksum file has not been tampered with.
```
wget https://releases.hashicorp.com/vault/1.1.0/vault_1.1.0_SHA256SUMS.sig
```

Verify the Downloads

Import the HashiCorp Security GPG key (listed on the HashiCorp Security page under Secure Communications):
```
gpg --recv-keys 51852D87348FFC4C
```
The output should show that the key was imported:
```
  
gpg: /home/user/.gnupg/trustdb.gpg: trustdb created
gpg: key 51852D87348FFC4C: public key "HashiCorp Security " imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1
```
Note

If an error occurs with the error message keyserver receive failed: Syntax error in URI, simply try rerunning the gpg command again.

Note

If you receive errors that indicate the dirmngr software is missing or inaccessible, install dirmngr using your package manager and run the GPG command again.

Verify the checksum file’s GPG signature:

gpg --verify vault*.sig vault*SHA256SUMS

The output should contain the Good signature from "HashiCorp Security <security@hashicorp.com>" confirmation message:

  
gpg: Signature made Mon 18 Mar 2019 01:44:51 PM MDT
gpg:                using RSA key 91A6E7F85D05C65630BEF18951852D87348FFC4C
gpg: Good signature from "HashiCorp Security <security@hashicorp.com>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 91A6 E7F8 5D05 C656 30BE  F189 5185 2D87 348F FC4C

Verify that the fingerprint output matches the fingerprint listed in the Secure Communications section of the HashiCorp Security page.
Verify the .zip archive’s checksum:
```
sha256sum -c vault*SHA256SUMS 2>&1 | grep OK
```
The output should show the file’s name as given in the vault*SHA256SUMS file:
```
  
vault_1.1.0_linux_amd64.zip: OK
```

Install the Vault Executable

Extract the Vault executable to the local directory.
```
unzip vault_*_linux_amd64.zip
```
Note

If you receive an error that indicates unzip is missing from your system, install the unzip package and try again.
Move the vault executable into a system-wide location.
```
sudo mv vault /usr/local/bin
```

Reset the ownership and permissions on the executable.

sudo chown root:root /usr/local/bin/vault
sudo chmod 755 /usr/local/bin/vault

Set executable capabilities on the vault binary. This will grant Vault privileges to lock memory, which is a best practice for running Vault securely (see the Vault documentation for additional information).
```
sudo setcap cap_ipc_lock=+ep /usr/local/bin/vault
```
Verify that vault is now available in the local shell.
```
vault --version
```
The output of this command should return the following.
```
  
Vault v1.1.0 ('36aa8c8dd1936e10ebd7a4c1d412ae0e6f7900bd')
```

System Vault Configuration

Create a system user that vault will run as when the service is started.
```
sudo useradd --system -d /etc/vault.d -s /bin/nologin vault
```
Add the vault user to the previously created tls group, which will grant the user the ability to read Let’s Encrypt certificates.
```
sudo gpasswd -a vault tls
```

Create the data directory and configuration directory for vault with limited permissions.

sudo install -o vault -g vault -m 750 -d /var/lib/vault
sudo install -o vault -g vault -m 750 -d /etc/vault.d

Create a systemd service file that will control how to run vault persistently as a system daemon.

/etc/systemd/system/vault.service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32



[Unit]
Description="a tool for managing secrets"
Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/vault.d/vault.hcl

[Service]
User=vault
Group=vault
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
Capabilities=CAP_IPC_LOCK+ep
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/local/bin/vault server -config=/etc/vault.d/vault.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
StartLimitIntervalSec=60
StartLimitBurst=3
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

These systemd service options define a number of important settings to ensure that Vault runs securely and reliably. Review the Vault documentation for a complete explanation of what these options achieve.

Configuration

Configure Vault

Create a configuration file for Vault with the following contents, replacing example.com with the domain used in your Let’s Encrypt certificates.

/etc/vault.d/vault.hcl

1
2
3
4
5
6
7
8
9



listener "tcp" {
  address = "0.0.0.0:8200"
  tls_cert_file = "/etc/letsencrypt/live/example.com/fullchain.pem"
  tls_key_file = "/etc/letsencrypt/live/example.com/privkey.pem"
}

storage "file" {
  path = "/var/lib/vault"
}

This configuration will use the Let’s Encrypt certificates created in the previous steps to terminate TLS for the Vault service. This ensures that secrets will never be transmitted in plaintext. The actual storage for Vault will be on the local filesystem at /var/lib/vault.

Run The Vault Service

Vault is now ready to run. Start the service using systemctl.
```
sudo systemctl start vault
```
If desired, enable the service as well so that Vault starts at system boot time.
```
sudo systemctl enable vault
```
Confirm that Vault is operational by using the vault executable to check for the service’s status. Set the VAULT_ADDR environment variable to https://example.com:8200, replacing example.com with your own domain:
```
export VAULT_ADDR=https://example.com:8200
```

vault commands should now be sent to your local Vault instance. To confirm this, run the vault status command:

vault status

The command should return output similar to the following:

  
Key                Value
---                -----
Seal Type          shamir
Initialized        false
Sealed             true
Total Shares       0
Threshold          0
Unseal Progress    0/0
Unseal Nonce       n/a
Version            n/a
HA Enabled         false

The remainder of this tutorial assumes that the environment variable VAULT_ADDR is set to this value to ensure that requests are sent to the correct Vault host.

Initializing Vault

At this stage, Vault is installed and running, but not yet initialized. The following steps will initialize the Vault backend, which sets unseal keys and returns the initial root token. Initialization occurs only one time for a Vault deployment.

There are two configurable options to choose when performing the initialization step. The first value is the number of key shares, which controls the total number of unseal keys that Vault will generate. The second value is the key threshold, which controls how many of these unseal key shares are required before Vault will successfully unseal itself. Unsealing is required whenever Vault is restarted or otherwise brought online after being in a previously offline state.

To illustrate this concept, consider a secure server in a data center. Because the Vault database is only decrypted in-memory, stealing or bringing the server offline for any reason will leave the only copy of Vault’s database on the filesystem in encrypted form, or “sealed”.

When starting the server again, a key share of 3 and key threshold of 2 means that 3 keys exist, but at least 2 must be provided at startup for Vault to derive its decryption key and load its database into memory for access once again.

The key share count ensure that multiple keys can exist at different locations for a degree of fault tolerance and backup purposes. The key threshold count ensures that compromising one unseal key alone is not sufficient to decrypt Vault data.

Choose a value for the number of key shares and key threshold. Your situation may vary, but as an example, consider a team of three people in charge of operating Vault. A key share of 3 ensures that each member holds one unseal key. A key threshold of 2 means that no single operator can lose their key and compromise the system or steal the Vault database without coordinating with another operator.

Using these chosen values, execute the initialization command. Be prepared to save the output that is returned from the following command, as it is only viewable once.

vault operator init -key-shares=3 -key-threshold=2

This command will return output similar to the following:

  
Unseal Key 1: BaR6GUWRY8hIeNyuzAn7FTa82DiIldgvEZhOKhVsl0X5
Unseal Key 2: jzh7lji1NX9TsNVGycUudSIy/X4lczJgsCpRfm3m8Q03
Unseal Key 3: JfdH8LqEyc4B+xLMBX6/LT9o8G/6isC2ZFfz+iNMIW/0

Initial Root Token: s.YijNa8lqSDeho1tJBtY02983

Vault initialized with 3 key shares and a key threshold of 2. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 2 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 2 key to
reconstruct the master key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

In a production scenario, these unseal keys should be stored in separate locations. For example, store one in a password manager such as LastPass, encrypted one with gpg, and store another offline on a USB key. Doing so ensures that compromising one storage location is not sufficient to recover the number of unseal keys required to decrypt the Vault database.
The Initial Root Token is equivalent to the “root” or superuser account for the Vault API. Record and protect this token in a similar fashion. Like the root account on a Unix system, this token should be used to create less-privileged accounts to use for day-to-day interactions with Vault and the root token should be used infrequently due to its widespread privileges.

Unseal Vault

After initialization, Vault will be sealed. The following unseal steps must be performed any time the vault service is brought down and then brought up again, such as when performing systemctl restart vault or restarting the host machine.

With VAULT_ADDR set appropriately, execute the unseal command.
```
vault operator unseal
```
A prompt will appear:
```
  
Unseal Key (will be hidden):
```

Paste or enter one unseal key and press Enter. The command will finish with output similar to the following:

  
Unseal Key (will be hidden):
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       3
Threshold          2
Unseal Progress    1/2
Unseal Nonce       0124ce2a-6229-fac1-0e3f-da3e97e00583
Version            1.1.0
HA Enabled         false

Notice that the output indicates that the one out of two required unseal keys have been provided.

Perform the unseal command again.
```
vault operator unseal
```
Enter a different unseal key when the prompt appears.
```
  
Unseal Key (will be hidden):
```

The resulting output should indicate that Vault is now unsealed (notice the Sealed false line).

  
Unseal Key (will be hidden):
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    3
Threshold       2
Version         1.1.0
Cluster Name    vault-cluster-a397153e
Cluster ID      a065557e-3ee8-9d26-4d90-b90c8d69fa5d
HA Enabled      false

Vault is now operational.

Using Vault

Token Authentication

When interacting with Vault over its REST API, Vault identifies and authenticates most requests by the presence of a token. While the initial root token can be used for now, the Policies section of this guide explains how to provision additional tokens.

Set the VAULT_TOKEN environment variable to the value of the previously-obtained root token. This token is the authentication mechanism that the vault command will rely on for future interaction with Vault. The actual root token will be different in your environment.
```
export VAULT_TOKEN=s.YijNa8lqSDeho1tJBtY02983
```
Use the token lookup subcommand to confirm that the token is valid and has the expected permissions.
```
vault token lookup
```
The output of this command should include the following:
```
  
policies            [root]
```

The KV Secret Backend

Vault backends are the core mechanism Vault uses to permit users to read and write secret values. The simplest backend to illustrate this functionality is the KV backend. This backend lets clients write key/value pairs (such as mysecret=apikey) that can be read later.

Enable the secret backend by using the enable Vault subcommand.
```
vault secrets enable -version=2 kv
```

Write an example value to the KV backend using the kv put Vault subcommand.

vault kv put kv/myservice api_token=secretvalue

This command should return output similar to the following:

  
Key              Value
---              -----
created_time     2019-03-31T04:35:38.631167678Z
deletion_time    n/a
destroyed        false
version          1

Read this value from the kv/myservice path.

vault kv get kv/myservice

This command should return output similar to the following:

  
====== Metadata ======
Key              Value
---              -----
created_time     2019-03-31T04:35:38.631167678Z
deletion_time    n/a
destroyed        false
version          1

====== Data ======
Key          Value
---          -----
api_token    secretvalue

Many utilities and script are better suited to process json output. Use the -format=json flag to do a read once more, with the results return in JSON form.

vault kv get -format=json kv/myservice

{
  "request_id": "2734ea81-6f39-c017-4c73-2719b2018b65",
  "lease_id": "",
  "lease_duration": 0,
  "renewable": false,
  "data": {
    "data": {
      "api_token": "secretvalue"
    },
    "metadata": {
      "created_time": "2019-03-31T04:35:38.631167678Z",
      "deletion_time": "",
      "destroyed": false,
      "version": 1
    }
  },
  "warnings": null
}

Policies

Up until this point, we have performed API calls to Vault with the root token. Production best practices dictate that this token should rarely be used and most operations should be performed with lesser-privileged tokens associated with controlled policies.

Policies are defined by specifying a particular path and the set of capabilities that are permitted by a user upon the path. In our previous commands, the path has been kv/myservice, so we can create a policy to only read this secret and perform no other operations, including reading or listing secrets. When no policy exists for a particular path, Vault denies operations by default.

In the case of the KV backend, Vault distinguishes operations upon the stored data, which are the actual stored values, and metadata, which includes information such as version history. In this example, we will create a policy to control access to the key/value data alone.

Create the following Vault policy file.
policy.hcl
1 2 3

path "kv/data/myservice" { capabilities = ["read"] }
This simple policy will permit any token associated with it to read the secret stored at the KV secret backend path kv/myservice.
Load this policy into Vault using the policy write subcommand. The following command names the aforementioned policy read-myservice.
```
vault policy write read-myservice policy.hcl
```

To illustrate the use of this policy, create a new token with this new policy associated with it.

vault token create -policy=read-myservice

This command should return output similar to the following.

  
Key                  Value
---                  -----
token                s.YdpJWRRaEIgdOW4y72sSVygy
token_accessor       07akQfzg0TDjj3YoZSGMPkHA
token_duration       768h
token_renewable      true
token_policies       ["default" "read-myservice"]
identity_policies    []
policies             ["default" "read-myservice"]

Open another terminal window or tab and login to the same host that Vault is running on. Set the VAULT_ADDR to ensure that new vault commands point at the local instance of Vault, replacing example.com with your domain.
```
export VAULT_ADDR=https://example.com:8200
```
Set the VAULT_TOKEN environment variable to the new token just created by the token create command. Remember that your actual token will be different than the one in this example.
```
export VAULT_TOKEN=s.YdpJWRRaEIgdOW4y72sSVygy
```

Now attempt to read our secret in Vault at the kv/myservice path.

vault kv get kv/myservice

Vault should return the key/value data.

  
====== Metadata ======
Key              Value
---              -----
created_time     2019-03-31T04:35:38.631167678Z
deletion_time    n/a
destroyed        false
version          1

====== Data ======
Key          Value
---          -----
api_token    secretvalue

To illustrate forbidden operations, attempt to list all secrets in the KV backend.

vault kv list kv/

Vault should deny this request.

  
Error listing kv/metadata: Error making API request.

URL: GET https://example.com:8200/v1/kv/metadata?list=true
Code: 403. Errors:

* 1 error occurred:
        * permission denied

In contrast, attempt to perform the same operation in the previous terminal window that has been configured with the root token.
```
vault kv list kv/
```
```
  
Keys
----
myservice
```
The root token should have sufficient rights to return a list of all secret keys under the kv/ path.

Authentication Methods

In practice, when services that require secret values are deployed, a token should not be distributed as part of the deployment or configuration management. Rather, services should authenticate themselves to Vault in order to acquire a token that has a limited lifetime. This ensures that credentials eventually expire and cannot be reused if they are ever leaked or disclosed.

Vault supports many types of authentication methods. For example, the Kubernetes authentication method can retrieve a token for individual pods. As a simple illustrative example, the following steps will demonstrate how to use the AppRole method.

The AppRole authentication method works by requiring that clients provide two pieces of information: the AppRole RoleID and SecretID. The recommendation approach to using this method is to store these two pieces of information in separate locations, as one alone is not sufficient to authenticate against Vault, but together, they permit a client to retrieve a valid Vault token. For example, in a production service, a RoleID might be present in a service’s configuration file, while the SecretID could be provided as an environment variable.

Enable the AppRole authentication method using the auth subcommand. Remember to perform these steps in the terminal window with the root token stored in the VAULT_TOKEN environment variable, otherwise Vault commands will fail.
```
vault auth enable approle
```
Create a named role. This will define a role that can be used to “log in” to Vault and retrieve a token with a policy associated with it. The following command creates a named role named my-application which creates tokens valid for 10 minutes which will have the read-myservice policy associated with them.
```
vault write auth/approle/role/my-application 
    token_ttl=10m 
    policies=read-myservice
```
Retrieve the RoleID of the named role, which uniquely identifies the AppRole. Note this value for later use.
```
vault read auth/approle/role/my-application/role-id
```
```
  
Key        Value
---        -----
role_id    147cd412-d1c2-4d2c-c57e-d660da0b1fa8
```
In this example case, RoleID is 147cd412-d1c2-4d2c-c57e-d660da0b1fa8. Note that your value will be different.

Finally, read the secret-id of the named role, and save this value for later use as well.

vault write -f auth/approle/role/my-application/secret-id

  
Key                   Value
---                   -----
secret_id             2225c0c3-9b9f-9a9c-a0a5-10bf06df7b25
secret_id_accessor    30cbef6a-8834-94fe-6cf3-cf2e4598dd6a

In this example output, the SecretID is 2225c0c3-9b9f-9a9c-a0a5-10bf06df7b25.

Use these values to generate a limited-use token by performing a write operation against the AppRole API. Replace the RoleID and SecretID values here with your own.

vault write auth/approle/login 
    role_id=147cd412-d1c2-4d2c-c57e-d660da0b1fa8 
    secret_id=2225c0c3-9b9f-9a9c-a0a5-10bf06df7b25

The resulting output should include a new token, which in this example case is s.coRl4UR6YL1sqw1jXhJbuZfq

  
Key                     Value
---                     -----
token                   s.3uu4vwFO8D1mG5S76IG04mck
token_accessor          fi3aW4W9kZNB3FAC20HRXeoT
token_duration          10m
token_renewable         true
token_policies          ["default" "read-myservice"]
identity_policies       []
policies                ["default" "read-myservice"]
token_meta_role_name    my-application

Open one more terminal tab or window and log in to your remote host running Vault.
Once again, set the VAULT_ADDR environment variable to the correct value to communicate with your local Vault instance.
```
export VAULT_ADDR=https://example.com:8200
```
Set the VAULT_TOKEN environment variable to this newly created token. From the previous example output, this would be the following (note that your token will be different).
```
export VAULT_TOKEN=s.3uu4vwFO8D1mG5S76IG04mck
```
Read the KV path that this token should be able to access.
```
vault kv get kv/myservice
```
The example should should be read and accessible.
If you read this value using this Vault token after more than 10 minutes have elapsed, the token will have expired and any read operations using the token should be denied. Performing another vault write auth/approle/login operation (detailed in step 5) can generate new tokens to use.

More Information

You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

Find answers, ask questions, and help others.

This guide is published under a CC BY-ND 4.0 license.

Source link

Secrets Management with Terraform

Updated Wednesday, December 12, 2018 by Linode Contributed by Linode

Terraform is an Infrastructure as Code (IaC) tool that allows you to write declarative code to manage your infrastructure. In order to implement IaC with Terraform it is necessary to supply secrets, such as server passwords and API tokens, within your code. This guide will discuss methods for securing those secrets within Terraform.

Keeping Secrets Out of .tf Files

In Terraform, .tf files contain the declarative code used to create, manage, and destroy infrastructure. This code is often committed to a version control system like Git, using a platform like GitHub, and shared within a team. Because it is easy for this information to become public-facing, it is important that you make sure your committed code is free of secrets.

Input Variables

Terraform configurations in .tf files can accept values from input variables. These variables are included in your configuration using Terraform’s interpolation syntax.

For example, you might have a linode-infrastructure.tf file with a provider block that requires an API access token. The token variable definition is declared inside your .tf file and is then interpolated inside the provider declaration with the "${var.token}" syntax:

linode-infrastructure.tf

1
2
3
4
5
6
7



variable "token" {
  description = "Your API access token"
}

provider "linode" {
    token = "${var.token}"
}

Variable definitions are written in .tf files. In this example, it’s the same file as your provider configuration, but the definition could have been in a separate .tf file too.

Note
Your variable definitions can have default values assigned to them. Here’s an example that encodes Linode’s Newark data center as the default value for a region variable:
variables.tf
1
2
3
4
variable "region" {
  description = "The region to deploy Linode instances in"
  default = "us-east"
}
You could later use this variable when declaring your Linode instances.

Assigning Variable Values in a File

The values assigned to your variables (aside from default values) are not included in the variable definitions in your .tf files. Instead, the values are stored in separate files with the .tfvars extension. When Terraform runs a command like plan or apply, it automatically looks through the working directory for a file named terraform.tfvars, or for files with the .auto.tfvars extension.

Here’s an example terraform.tfvars which supplies a value for the token variable from the previous example:

terraform.tfvars

1

token = 'your-token-value'

You would then add the terraform.tfvars file to your .gitignore file and keep it out of version control. This strategy allows you to safely commit the linode-infrastructure.tf file.

For ease of use with large terraform.tfvars files, it might be beneficial to include an example terraform.tfvars.example in your Git repository with all of the variable names recorded (but none of the values entered). Team members could then copy this example into their local repository’s terraform.tfvars and enter the appropriate values.

Note
Variable value files with names that don’t match terraform.tfvars or *.auto.tfvars can be specified with the -var-file option:
terraform apply -var-file=myvars.tfvars
Supplying multiple .tfvars files is another way to further separate secret variables and non-secret variables; e.g.:
terraform apply 
-var-file=non-secret-variables.tfvars 
-var-file=secret-variables.tfvars

Assigning Values in Environment Variables

Terraform allows you to keep input variable values in environment variables. These variables have the prefix TF_VAR_ and are supplied at the command line. Using the above example of an API access token, you could export the variable and use it like so:

export TF_VAR_token=your-token-value
terraform apply

You could also include the variable on the same line when running terraform plan or terraform apply:

TF_VAR_token=your-token-value terraform apply

Caution

This method commits the environment variable to your shell’s history, so take care when using this method.

Assigning Values in Command-Line Flags

Variable values can be set with the -var option:

terraform apply -var 'token=your-token-value'

Caution

This method commits the command-line variable to your shell’s history, so take care when using this method.

Supply Variables at Prompt

If Terraform does not find a default value for a defined variable; or a value from a .tfvars file, environment variable, or CLI flag; it will prompt you for a value before running an action:

  
$ terraform plan
var.token
  Your API access token

  Enter a value:

This method is a bit easier to use than supplying environment variables, and has the added benefit of displaying the description you set up when defining your variable.

How to Manage Your State File

While it is relatively easy to keep secrets out of .tf files using any of the above methods, there is another file you need to be aware of when managing secrets, and that is the terraform.tfstate file.

This state file contains a JSON object that holds your managed infrastructure’s current state. This state is a snapshot of the various attributes of your infrastructure at the time it was last modified. It is generated on terraform apply and is a necessary part of the Terraform process, as it maps the declarative code of your .tf files to your real world infrastructure.

As of the writing of this guide, sensitive information used to generate your Terraform state can be stored as plain text in the terraform.tfstate file. For example, if you are working with the Linode provider and have supplied a root password for your Linode instance, that root password will be stored as plain text in the state file. Avoid checking your terraform.tfstate file into your version control repository. Instead, the following are some strategies for storing and sharing your state files.

Remote Backends

Terraform backends allow the user to securely store their state in a remote location, such as a key/value store like Consul, or an S3 compatible bucket storage like Minio. This allows the Terraform state to be read from the remote store, and because the state only ever exists locally in memory, there is no worry about storing secrets in plain text.

Some backends, like Consul, also allow for state locking. If one user is applying a state, another user will be unable to make any changes.

Using a Terraform backend is the preferred way to share a Terraform state file.

Encrypting Secrets

Third-party tools exist that allow you to encrypt your secrets. If you encrypt the secrets in your terraform.tfstate (or your .tfvars files), you can check them into version control securely:

git-crypt allows you to encrypt files when they are committed to a Git repository. git-crypt also decrypts files when they are checked out.

Note

You must initialize git-crypt in a repository before committing your state file or variable value files, or they will not be eligible for encryption.
Terrahelp allows you to encrypt and decrypt a whole state file, or just the variables you have include in your terraform.tfvars file.

Use a Dummy Password

It is possible to supply a dummy password to Terraform and later change that password manually to a more secure one. For instance, if you were to create a Linode instance with a dummy root password, you could later change that password from the command line or in the Linode Manager.

Note

Any attempt to change the password in a .tf file will result in the creation of new resources on terraform apply.

Privatize Version Control

If you are unwilling or unable to use the above options to help manage your state file, and if you are using a platform like GitHub or GitLab to share your state files, then at minimum the repository should be private.

More Information

Find answers, ask questions, and help others.

This guide is published under a CC BY-ND 4.0 license.

Source link

Secrets Management with Salt

Updated Tuesday, November 6, 2018 by Linode Contributed by Linode

Salt is a powerful configuration management tool which helps you manage your server deployments with configuration state files. These files are easily shared with others on your team and can be checked in to version control systems like Git.

A common problem when working with Salt’s state files is the need access to sensitive data, like API keys and database passwords, within those files. Directly embedding that information as plain-text inside your state files can represent a security vulnerability, especially if you were to check those files into version control. This guide will explore some common methods for securing your secrets within Salt.

Salt Pillar

A primary method for storing secrets in Salt is to keep them in Salt’s Pillar feature. Salt Pillar is designed to maintain secrets and other variable information in a single location (generally, on the Salt master) and then deliver that information to specific minions. If you separate your secrets out from your states and into pillar files, you can ignore those files in your version control system.

Note

In addition to storing secrets, Salt Pillar can also maintain non-sensitive data; for example, the versions of the packages you want to install on your minions. So, you may still want to track some pillar files in version control.

To handle this distinction, you could create a special directory at /srv/pillar/secrets and add set your version control system to ignore that directory (when using Git, list this directory in your .gitignore file). Keep all sensitive data inside pillar files within this directory, and maintain non-sensitive data in pillar files in /srv/pillar or another subfolder.

Anatomy of Pillar Data Files

Pillar data is kept in .sls files which are written in the same YAML syntax as states. These are generally stored within /srv/pillar on the Salt master, but this location can be configured via the pillar_roots option in your master’s configuration.

For example, let’s say your minion runs an application which accesses the Linode API. This example pillar file records your API token in a variable called linode_api_token:

/srv/pillar/app_secrets.sls

1

linode_api_token: YOUR_API_TOKEN

As with state files, a top file (separate from your states’ top file) maps pillar data to minions. This example top file maps your app_secrets pillar data to your app server:

/srv/pillar/top.sls

1
2
3



base:
  'appserver':
    - app_secrets

Note

You may want to create a pillar.example file (like those provided by Salt formulas) that lists all the known variable keys for your pillar but does not contain the actual secrets. If you check this file into your version control, other users that clone your states’ repository can duplicate this example pillar file and more quickly set up their own deployments.

Accessing Pillar Data inside Salt States

To inject pillar data into your states, use Salt’s Jinja template syntax. While Salt uses the YAML syntax for state and pillar files, the files are first interpreted as Jinja templates (by default).

This example state embeds the API token in a file on your Linode; the data is accessed through the pillar dictionary:

/srv/salt/setup_app.sls

1
2
3
4



api_token:
  file.managed:
    - name: /var/your_app/api_token
    - contents: {{ pillar['linode_api_token'] }}

Caution

There are times when pillar data could show up in the output that Salt generates, like when file.managed displays diffs of a modified file. To avoid displaying these diffs, you can set file.managed’s show_diff flag to false.

Passing Pillar Data at the Command Line

You can also supply pillar values as a dictionary through the command line, and those values will override any values set in your pillar files. This example command would apply the A_DIFFERENT_API_TOKEN value instead of the original YOUR_API_TOKEN from the previous example:

salt '*' state.apply pillar='{"linode_api_token": "A_DIFFERENT_API_TOKEN"}'

Environment Variables

Another way to keep sensitive values out of version control is to use environment variables. The method for passing environment variables to your states is similar to how pillar data can be passed via the command line. The environment variable prefixes your salt command, as in this example:

LINODE_API_TOKEN="YOUR_API_TOKEN" salt 'appserver' state.apply setup_app

The environment variable is referenced by a Salt state file through the salt['environ.get']('ENVIRONMENT_VARIABLE_NAME') syntax. The previous setup_app example state can be adapted to use an environment variable as follows:

/srv/salt/setup_app.sls

1
2
3
4



api_token:
  file.managed:
    - name: /var/your_app/api_token
    - contents: {{ salt['environ.get']('LINODE_API_TOKEN') }}

As with the previous pillar example, you’ll want to keep file.managed’s diffs from appearing on screen when dealing with sensitive information by setting show_diff: false. For more information, see Using Environment Variables in SLS Modules.

GPG Encryption

You can use Salt’s GPG renderer to decrypt GPG ciphers that are located in your pillar files. This decryption step happens before your pillar data is passed to your minions. As a result, any value in a pillar file can be encrypted. Because the values are encrypted, you can store your pillar files in version control securely.

This approach requires that the GPG secret key is stored on your Salt master. It also makes sense to include the public key in version control so that your team members can use it to encrypt new data for your pillar files.

SDB

Salt comes with a database interface called SDB that was initially created to store non-minion-specific data, such as passwords. It was designed to connect to a package like Salt’s keyring module, but other options are available, such as Consul and Vault.

These databases are set up using a configuration profile in /srv/salt/master.d. To access data, you supply an sdb:// url, such as password: sdb://mysecrets/mypassword. For more information on SDB, reference the Salt SDB documentation.

Note

More Information

Find answers, ask questions, and help others.

This guide is published under a CC BY-ND 4.0 license.

Source link

Shared

Cloud VPS

Dedicated

Secrets

Use HashiCorp Vault to Manage Secrets

Why Use Vault?

Concepts

Installation

Before you Begin

Acquire a TLS Certificate

Download Vault files

Verify the Downloads

Install the Vault Executable

System Vault Configuration

Configuration

Configure Vault

Run The Vault Service

Initializing Vault

Unseal Vault

Using Vault

Token Authentication

The KV Secret Backend

Policies

Authentication Methods

More Information

Secrets Management with Terraform

Keeping Secrets Out of .tf Files

Input Variables

Assigning Variable Values in a File

Assigning Values in Environment Variables

Assigning Values in Command-Line Flags

Supply Variables at Prompt

How to Manage Your State File

Remote Backends

Encrypting Secrets

Use a Dummy Password

Privatize Version Control

More Information

Secrets Management with Salt

Salt Pillar

Anatomy of Pillar Data Files

Accessing Pillar Data inside Salt States

Passing Pillar Data at the Command Line

Environment Variables

GPG Encryption

SDB

More Information