One place for hosting & domains

      Terraform

      How To Use Terraform With Your Team


      The author selected the Free and Open Source Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      When multiple people are working on the same Terraform project from different locations simultaneously, it is important to handle the infrastructure code and project state correctly to avoid overwriting errors. The solution is to store the state remotely instead of locally. A remote system is available to all members of your team, and it is possible for them to lock the state while they’re working.

      One such remote backend is pg, which stores the state in a PostgreSQL database. During the course of this tutorial, you’ll use it with a DigitalOcean Managed Database to ensure data availability.

      Terraform also supports the official, managed cloud offering by HashiCorp called Terraform Cloud—a proprietary app that syncs your team’s work in one place and offers a user interface for configuration and management.

      In this tutorial, you’ll create an organization in Terraform Cloud to which you’ll connect your project. You’ll then use your organization to set up workspaces and resources. You will store your state in the managed cloud so it is always available. You’ll also set up the pg backend with an accompanying managed PostgreSQL database.

      Prerequisites

      • A DigitalOcean Personal Access Token, which you can create via the DigitalOcean Control Panel. You can find instructions to create this in How to Generate a Personal Access Token.
      • Terraform installed on your local machine and a project set up with the DigitalOcean provider. Complete Step 1 of the How To Use Terraform with DigitalOcean tutorial.
      • If you would like to use a pg backend, you will need a Managed PostgreSQL database cluster created and accessible. For more information, visit the Quickstart guide. You can use a separate database for this tutorial.
      • If you would like to use HashiCorp’s managed cloud, you will need an account with Terraform Cloud. You can create one on their sign-up page.

      Note: We have specifically tested this tutorial using Terraform 0.13.

      Storing State in Terraform Cloud

      In this step, you’ll create a project that deploys a Droplet, but instead of storing the state locally, you’ll use Terraform Cloud as the backend with the remote provider. This entails creating the organization and workspace in Terraform Cloud, writing the infrastructure code, and planning it.

      Creating an Organization

      Terraform Cloud allows you to have multiple organizations, which house your workspaces and modules. Paid-plan organizations can have multiple teams with access-level control features, while the free plan you’ll use provides only one team per organization. You can invite team members to join the organization.

      Start off by heading over to Terraform Cloud and logging in. If you haven’t yet created an organization, it will prompt you to do so.

      Terraform Cloud - Create a new organization

      Enter an organization name of your choosing and remember that it must be unique among all names in Terraform Cloud. You’ll receive an error if the name already exists. The email address should already be filled in with the address of your account. Once you’re finished, click the Create organization button to continue.

      It will then ask you to select the type of workspace.

      Terraform Cloud - Choosing a workspace type

      Since you’ll interface with Terraform Cloud using the command line, click the CLI-driven workflow option. Then, input a name for your workspace.

      Terraform Cloud - Setting workspace name

      Type in a workspace name of your choosing (we’ll call it sammy), then click Create workspace to finalize the organization creation process. It will then direct you to a workspace settings page.

      Terraform Cloud - Workspace settings

      You’ve now created your workspace, which is a part of your organization. Since you just created it, your workspace contains no infrastructure code. In the central part of the interface, Terraform Cloud gives you starting instructions for connecting to this workspace.

      Before connecting to it, you’ll need to configure the version of Terraform that the cloud will use to execute your commands. To set it, click the Settings dropdown in the upper-right corner and select General from the list. When the page opens, navigate to the Terraform Version dropdown and select 0.13.1 (for this tutorial).

      Terraform Cloud - Setting Terraform Version

      Then, click the Save settings button to save the changes.

      To connect your project to your organization and workspace, you’ll first need to log in using the command line. Before you run the command, navigate to the tokens page to create a new access token for your server, which will provide access to your account. You’ll receive a prompt to create an API token.

      Terraform Cloud - Create API token

      The default description is fine, so click Create API token to create it.

      Terraform Cloud - Created API token

      Click the token value, or the icon after it, to copy the API token. You’ll use this token to connect your project to your Terraform Cloud account.

      In the command line, run the following command to log in:

      You’ll receive the following output:

      Output

      Terraform will request an API token for app.terraform.io using your browser. If login is successful, Terraform will store the token in plain text in the following file for use by subsequent commands: /home/sammy/.terraform.d/credentials.tfrc.json Do you want to proceed? Only 'yes' will be accepted to confirm. ...

      Terraform is warning you that the token will be stored locally. Enter yes when it prompts you:

      Output

      --------------------------------------------------------------------------------- Open the following URL to access the tokens page for app.terraform.io: https://app.terraform.io/app/settings/tokens?source=terraform-login --------------------------------------------------------------------------------- Generate a token using your browser, and copy-paste it into this prompt. Terraform will store the token in plain text in the following file for use by subsequent commands: /home/sammy/.terraform.d/credentials.tfrc.json Token for app.terraform.io: Enter a value:

      Paste in the token you’ve copied and confirm with ENTER. Terraform will show a success message:

      Output

      Retrieved token for user your_username --------------------------------------------------------------------------------- Success! Terraform has obtained and saved an API token. The new API token will be used for any future Terraform command that must make authenticated requests to app.terraform.io.

      You’ve configured your local Terraform installation to access your Terraform Cloud account. You’ll now create a project that deploys a Droplet and configure it to use Terraform Cloud for storing its state.

      Setting Up the Project

      First, create a directory named terraform-team-remote where you’ll store the project:

      • mkdir ~/terraform-team-remote

      Navigate to it:

      • cd ~/terraform-team-remote

      To set up your project, you’ll need to:

      • define and configure the remote provider, which interfaces with Terraform Cloud.
      • require the digitalocean provider to be able to deploy DigitalOcean resources.
      • define and initialize variables that you’ll use.

      You’ll store the provider and module requirements specifications in a file named provider.tf. Create and open it for editing by running:

      Add the following lines:

      ~/terraform-team-remote/provider.tf

      terraform {
        required_version = "0.13.1"
      
        required_providers {
          digitalocean = {
            source = "digitalocean/digitalocean"
            version = ">1.22.2"
          }
        }
      
        backend "remote" {
          hostname = "app.terraform.io"
          organization = "your_organization_name"
      
          workspaces {
            name = "your_workspace_name"
          }
        }
      }
      
      variable "do_token" {}
      
      provider "digitalocean" {
        token = var.do_token
      }
      

      Here, you first specify your Terraform version. Then, you specify the digitalocean provider as required and set the backend to remote.

      Its hostname is set to app.terraform.io, which is the address of Terraform Cloud. For the organization and workspaces.name, replace the highlighted values with the names you specified.

      Next, you define a variable called do_token, which you pass to the digitalocean provider created after it. You’ve now configured your project to connect to your organization, so save and close the file.

      Initialize your project with the following command:

      The output will be similar to this:

      Output

      Initializing the backend... Successfully configured the backend "remote"! Terraform will automatically use this backend unless the backend configuration changes. Initializing provider plugins... - Finding digitalocean/digitalocean versions matching "> 1.22.2"... - Installing digitalocean/digitalocean v2.3.0... - Installed digitalocean/digitalocean v2.3.0 (signed by a HashiCorp partner, key ID F82037E524B9C0E8) Partner and community providers are signed by their developers. If you'd like to know more about provider signing, you can read about it here: https://www.terraform.io/docs/plugins/signing.html Terraform has been successfully initialized! ...

      Next, define the Droplet in a file called droplets.tf. Create and open it for editing by running:

      Add the following lines:

      ~/terraform-team-remote/droplets.tf

      resource "digitalocean_droplet" "web" {
        image  = "ubuntu-18-04-x64"
        name   = "web-1"
        region = "fra1"
        size   = "s-1vcpu-1gb"
      }
      

      This code will deploy a Droplet called web-1 in the fra1 region, running Ubuntu 18.04 on 1GB RAM and one CPU core. That is all you need to define, so save and close the file.

      What’s left to define are the variable values. The remote provider does not support passing in values to variables through the command line, so you’ll have to pass them in using variable files or set them in Terraform Cloud. Terraform reads variable values from files with a filename ending in .auto.tfvars. Create and open a file called vars.auto.tfvars for editing, in which you’ll define the do_token variable:

      Add the following line, replacing your_do_token with your DigitalOcean API token:

      vars.auto.tfvars

      do_token = "your_do_token"
      

      When you’re done, save and close the file. Terraform will automatically read this file when planning actions.

      Your project is now complete and set up to use Terraform Cloud as its backend. You’ll now plan and apply the Droplet and review how that reflects in the Cloud app.

      Applying the Configuration

      Since you haven’t yet planned or applied your project, the workspace in Terraform Cloud is currently empty. You can try applying the project by running the following command to update it:

      You’ll notice that the output is different from when you use local as your backend:

      Output

      Running apply in the remote backend. Output will stream here. Pressing Ctrl-C will cancel the remote apply if it's still pending. If the apply started it will stop streaming the logs, but will not stop the apply running remotely. Preparing the remote apply... To view this run in a browser, visit: https://app.terraform.io/app/sammy-shark/sammy/runs/run-QnAh2HDwx6zWbNV1 Waiting for the plan to start... Terraform v0.13.1 Configuring remote state backend... Initializing Terraform configuration... Refreshing Terraform state in-memory prior to plan... The refreshed state will be used to calculate this plan, but will not be persisted to local or remote state storage. ------------------------------------------------------------------------ An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # digitalocean_droplet.web will be created + resource "digitalocean_droplet" "web" { + backups = false + created_at = (known after apply) + disk = (known after apply) + id = (known after apply) + image = "ubuntu-18-04-x64" + ipv4_address = (known after apply) + ipv4_address_private = (known after apply) + ipv6 = false + ipv6_address = (known after apply) + locked = (known after apply) + memory = (known after apply) + monitoring = false + name = "web-1" + price_hourly = (known after apply) + price_monthly = (known after apply) + private_networking = (known after apply) + region = "fra1" + resize_disk = true + size = "s-1vcpu-1gb" + status = (known after apply) + urn = (known after apply) + vcpus = (known after apply) + volume_ids = (known after apply) + vpc_uuid = (known after apply) } Plan: 1 to add, 0 to change, 0 to destroy. ...

      When using the remote backend, Terraform is not planning or applying configuration from the local machine. Instead, it delegates those tasks to the cloud, and only streams the output to the console in real time.

      Enter yes when prompted. Terraform will soon finish applying the configuration, and you can navigate to the workspace on the Terraform Cloud website to find that it has applied a new action.

      Terraform Cloud - New Run Applied

      You can now destroy the deployed resources by running the following:

      In this section, you’ve connected your project to Terraform Cloud. You’ll now use another backend, pg, which stores the state in a PostgreSQL database.

      Storing State in a Managed PostgreSQL Database

      In this section, you’ll set up a project that deploys a Droplet, much like the the previous step. This time, however, you’ll store the state in a DigitalOcean Managed PostgreSQL database using the pg provider. This provider supports state locking, so the state won’t ever be overwritten by two or more changes happening at the same time.

      Start by creating a directory named terraform-team-pg in which you’ll store the project:

      • mkdir ~/terraform-team-pg

      Navigate to it:

      Like the previous section, you’ll first define the provider and then pass in the connection string for the database and the digitalocean module. Create and open provider.tf for editing:

      Add the following lines:

      ~/terraform-team-pg/provider.tf

      terraform {
        required_providers {
          digitalocean = {
            source = "digitalocean/digitalocean"
            version = ">1.22.2"
          }
        }
      
        backend "pg" {
          conn_str = "your_db_connection_string"
        }
      }
      
      variable "do_token" {}
      
      provider "digitalocean" {
        token = var.do_token
      }
      

      Here you require the digitalocean provider and define the pg backend, which accepts a connection string. Then, you define the do_token variable and pass it to the instance of the digitalocean provider.

      Remember to replace your_db_connection_string with the connection string for your managed database from your DigitalOcean Control Panel. Then save and close the file.

      Warning: To continue, in the Settings of your database, make sure you have the IP address of the machine from which you’re running Terraform on an allowlist.

      Initialize the project by running:

      The output will be similar to the following:

      Output

      Initializing the backend... Successfully configured the backend "pg"! Terraform will automatically use this backend unless the backend configuration changes. Error: No existing workspaces. Use the "terraform workspace" command to create and select a new workspace. If the backend already contains existing workspaces, you may need to update the backend configuration.

      Terraform successfully initialized the backend; meaning it connected to the database. However, it complains about not having a workspace, since it does not create one during initialization. To resolve this, create a default workspace and switch to it by running:

      • terraform workspace new default

      The output will be the following:

      Output

      Created and switched to workspace "default"! You're now on a new, empty workspace. Workspaces isolate their state, so if you run "terraform plan" Terraform will not see any existing state for this configuration.

      To finish the initialization process, run terraform init again:

      You’ll receive output showing it has successfully completed:

      Output

      Initializing the backend... Initializing provider plugins... - Finding digitalocean/digitalocean versions matching "> 1.22.2"... - Installing digitalocean/digitalocean v2.3.0... - Installed digitalocean/digitalocean v2.3.0 (signed by a HashiCorp partner, key ID F82037E524B9C0E8) Partner and community providers are signed by their developers. If you'd like to know more about provider signing, you can read about it here: https://www.terraform.io/docs/plugins/signing.html Terraform has been successfully initialized!

      Since the Droplet definition is the same as in the previous project, you can copy it over by running:

      • cp ../terraform-team-remote/droplets.tf .

      You’ll need your DigitalOcean token in an environment variable. Create one, replacing your_do_token with your token:

      • export DO_PAT="your_do_token"

      To check that the connection to the database is working, try planning the configuration:

      • terraform plan -var "do_token=${DO_PAT}"

      The output will be similar to the following:

      Output

      Refreshing Terraform state in-memory prior to plan... The refreshed state will be used to calculate this plan, but will not be persisted to local or remote state storage. ------------------------------------------------------------------------ An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # digitalocean_droplet.web will be created + resource "digitalocean_droplet" "web" { + backups = false + created_at = (known after apply) + disk = (known after apply) + id = (known after apply) + image = "ubuntu-18-04-x64" + ipv4_address = (known after apply) + ipv4_address_private = (known after apply) + ipv6 = false + ipv6_address = (known after apply) + locked = (known after apply) + memory = (known after apply) + monitoring = false + name = "web-1" + price_hourly = (known after apply) + price_monthly = (known after apply) + private_networking = (known after apply) + region = "fra1" + resize_disk = true + size = "s-1vcpu-1gb" + status = (known after apply) + urn = (known after apply) + vcpus = (known after apply) + volume_ids = (known after apply) + vpc_uuid = (known after apply) } Plan: 1 to add, 0 to change, 0 to destroy. ...

      Terraform reported no errors and planned out the actions as usual. It successfully connected to your PostgreSQL database and stored its state. Multiple people can now work on this simultaneously with the project remaining synchronized.

      Conclusion

      In this tutorial, you’ve used two different backends: Terraform Cloud, which is HashiCorp’s managed cloud offering for Terraform; and pg, which allows you to store the project’s state in a PostgreSQL database. You used a managed PostgreSQL database from DigitalOcean, which you can provision and use with Terraform within minutes.

      For more information about the features of Terraform Cloud, visit the official docs.

      To learn more about using Terraform, check out our series on How To Manage Infrastructure with Terraform.



      Source link

      How To Troubleshoot Terraform


      The author selected the Free and Open Source Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      Things do not always go according to plan: deployments can fail, existing resources may break unexpectedly, and you and your team could be forced to fix the issue as soon as possible. Understanding the methods to approach debugging your Terraform project is crucial when you need to make a swift response.

      Similarly to developing with other programming languages and frameworks, setting log levels in Terraform to gain insight into its internal workflows with the necessary verbosity is a feature that can help you when troubleshooting. By logging the internal actions, you can uncover implicitly hidden errors, such as variables defaulting to an unsuitable data type. Also common with frameworks is the ability to import and use third-party modules (or libraries) to reduce code duplication between projects.

      In this tutorial, you’ll verify that variables always have sensible values and you’ll specify exactly which versions of providers and modules you need to prevent conflicts. You’ll also enable various levels of debug mode verbosity, which can help you diagnose an underlying issue in Terraform itself.

      Prerequisites

      • A DigitalOcean Personal Access Token, which you can create via the DigitalOcean Control Panel. You can find instructions to do this in How to Generate a Personal Access Token.
      • Terraform installed on your local machine and a project set up with the DigitalOcean provider. Complete Step 1 and Step 2 of the How To Use Terraform with DigitalOcean tutorial, and be sure to name the project folder terraform-troubleshooting, instead of loadbalance. During Step 2, do not include the pvt_key variable and the SSH key resource.

      Note: We have specifically tested this tutorial using Terraform 0.13.

      Setting Version Constraints

      Although the ability to make use of third-party modules and providers can minimize code duplication and effort when writing your Terraform code, it is also possible for developers of third-party modules to release new versions that can potentially bring breaking changes to your specific code. To prevent this, Terraform allows you to specify version boundaries to ensure that only the versions you want are installed and used. Specifying versions means that you will require the versions you’ve tested in your code, but it also leaves the possibility for a future update.

      Version constraints are specified as strings and passed in to the version parameter when you define module or provider requirements. As part of the Prerequisites, you’ve already requested the digitalocean provider in provider.tf. Run the following command to review what version requirements it specifies:

      You’ll find the provider code as follows:

      terraform-troubleshooting/provider.tf

      terraform {
        required_providers {
          digitalocean = {
            source = "digitalocean/digitalocean"
            version = "1.22.2"
          }
        }
      }
      ...
      

      In this case, you have requested the digitalocean provider, and explicitly set the version to 1.22.2. When your project only requires a specific version, this is the most effortless way to accomplish that.

      Version constraints can be more complex than just specifying one version. They can contain one or more groups of conditions, separated by a comma (,). The groups each define an acceptable range of versions and may include operators, such as:

      • >, <, >=, <=: for comparisons, such as >=1.0, which would require the version to be equal to or greater than 1.0.
      • !=: for excluding a specific version—!= 1.0 would deny version 1.0 from being used and requested.
      • ~>: for matching the specified version up to the right-most version part, which is allowed to increment (~>1.5.10 will match 1.5.10 and 1.5.11, but won’t match 1.5.9).

      Here are two examples of version constraints with multiple groups:

      • >=1.0, <2.0: allows all versions from the 1.0 series onward, up to 2.0.
      • >1.0, != 1.5: allows versions greater than, but not equal to 1.0, with the exception of 1.5, which it also excludes.

      For a potential available version to be selected, it must pass every specified constraint and remain compatible with other modules and providers, as well as the version of Terraform that you’re using. If Terraform deems no combination acceptable, it won’t be able to perform any tasks because the dependencies remain unresolved. When Terraform identifies acceptable versions satisfying the constraints, it uses the latest one available.

      In this section, you’ve learned about locking the range of module and resource versions that you can install in your project by specifying version constraints. This is useful when you want stability by using only tested and approved versions of third-party code. In the next section, you’ll configure Terraform to show more verbose logs, which are necessary for bug reports and further debugging in case of crashes.

      Enabling Debug Mode

      There could be a bug or malformed input within your workflow, which may result in your resources not provisioning as intended. In such rare cases, it’s important to know how to access detailed logs describing what Terraform is doing. They may aid in pinpointing the cause of the error, tell you if it’s user-made, or prompt you to report the issue to Terraform developers if it’s an internal bug.

      Terraform exposes the TF_LOG environment variable for setting the level of logging verbosity, of which there are five:

      • TRACE: the most elaborate verbosity, shows every step taken by Terraform and produces enormous outputs with internal logs.
      • DEBUG: describes what happens internally in a more concise way compared to TRACE.
      • ERROR: shows errors that prevent Terraform from continuing.
      • WARN: logs warnings, which may indicate misconfiguration or mistakes, but are not critical to execution.
      • INFO: shows general, high-level messages about the execution process.

      To specify a desired log level, you’ll have to set the environment variable to the appropriate value:

      If TF_LOG is defined, but the value is not one of the five listed verbosity levels, Terraform will default to TRACE.

      You’ll now define a Droplet resource and try deploying it with different log levels. You’ll store the Droplet definition in a file named droplets.tf, so create and open it for editing:

      Add the following lines:

      terraform-troubleshooting/droplets.tf

      resource "digitalocean_droplet" "test-droplet" {
        image  = "ubuntu-18-04-x64"
        name   = "test-droplet"
        region = "fra1"
        size   = "s-1vcpu-1gb"
      }
      

      This Droplet will run Ubuntu 18.04 with one CPU core and 1GB RAM in the fra1 region; you’ll call it test-droplet. That is all you need to define, so save and close the file.

      Before deploying the Droplet, set the log level to DEBUG by running:

      Then, plan the Droplet provisioning:

      • terraform plan -var "do_token=${DO_PAT}"

      The output will be very long, and you can inspect it more closely to find that each line starts with the level of verbosity (importance) in double brackets. You’ll see that most of the lines start with [DEBUG].

      [WARN] and [INFO] are also present; that’s because TF_LOG sets the lowest log level. This means that you’d have to set TF_LOG to TRACE to show TRACE and all other log levels at the same time.

      If an internal error occurred, Terraform will show the stack trace and exit, stopping execution. From there, you’ll be able to locate where in the source code the error occurred, and if it’s a bug, report it to Terraform developers. Otherwise, if it’s an error in your code, Terraform will point it out to you, so you can fix it in your project.

      Here is how the log output would be when the DigitalOcean backend can’t verify your API token. It throws a user error because of incorrect input:

      Output

      ... digitalocean_droplet.test-droplet: Creating... 2021/01/20 06:54:35 [ERROR] eval: *terraform.EvalApplyPost, err: Error creating droplet: POST https://api.digitalocean.com/v2/droplets: 401 Unable to authenticate you 2021/01/20 06:54:35 [ERROR] eval: *terraform.EvalSequence, err: Error creating droplet: POST https://api.digitalocean.com/v2/droplets: 401 Unable to authenticate you Error: Error creating droplet: POST https://api.digitalocean.com/v2/droplets: 401 Unable to authenticate you on droplets.tf line 1, in resource "digitalocean_droplet" "test-droplet": 1: resource "digitalocean_droplet" "test-droplet" { ...

      You’ve now learned to enable more verbose logging modes. They are very useful for diagnosing crashes and unexpected Terraform behavior. In the next section, you’ll review verifying variables and preventing edge cases.

      Validating Variables

      In this section, you’ll ensure that variables always have sensible and appropriate values according to their type and validation parameters.

      In HCL (HashiCorp Configuration Language), when defining a variable you do not necessarily need to specify anything except its name. You would declare an example variable called test_ip like this:

      variable "test_ip" { }
      

      You can then use this value through the code; passing its value in when you run Terraform.

      While that will work, this definition has two shortcomings: first, you can not pass a value in at runtime; and second, it can be of any type (bool, string and so on), which may not be suitable for its purpose. To remedy this, you should always specify its default value and type:

      variable "test_ip" {
        type    = string
        default = "8.8.8.8"
      }
      

      By setting a default value, you ensure that the code referencing the variable remains operational in the event that a more specific value was not provided. When you specify a type, Terraform can validate the new value the variable should be set to, and show an error if it’s non-conforming to the type. An instance of this behavior would be trying to fit a string into a number.

      A new feature of Terraform 0.13 is that you can provide a validation routine for variables that can give an error message if the validation fails. Examples of validation would be checking the length of the new value if it’s a string, or looking for at least one match with a RegEx expression in case of structured data.

      To add input validation to your variable, define a validation block:

      variable "test_ip" {
        type    = string
        default = "8.8.8.8"
      
        validation {
          condition     = can(regex("d{1,3}.d{1,3}.d{1,3}.d{1,3}", var.test_ip))
          error_message = "The provided value is not a valid IP address."
        }
      }
      

      Under validation, you can specify two parameters within the curly braces:

      • A condition that accepts a bool it will calculate, which will signify if the validation passes.
      • An error_message that specifies the error message in case the validation does not pass.

      In this example, you compute the condition by searching for a regex match in the variable value. You pass that to the can function. The can function returns true if the function that’s passed in as a parameter ran without errors, so it’s useful for checking when a function completed successfully or returned results.

      The regex function we’re using here accepts a Regular Expression (RegEx), applies it to a given string, and returns the matched substrings. The RegEx matches four pairs of three digit numbers, separated with dots in-between. You can learn more about RegEx by visiting the Introduction to Regular Expressions tutorial.

      You now know how to specify a default value for a variable, how to set its type, and how to enable input validation using RegEx expressions.

      Conclusion

      In this tutorial, you’ve troubleshooted Terraform by enabling debug mode and setting the log verbosity to appropriate levels. You’ve also learned about some of the advanced features of variables, such as declaring validation procedures and setting good defaults. Leaving out default values is a common pitfall that may cause strange issues further along in your project’s development.

      For the stability of your project, locking third-party module and provider versions is recommended, since it leaves you with a stable system and an upgrade path when it becomes necessary.

      Verification of input values for variables is not confined to matching with regex. For more built-in functions that you can make use of, visit the official docs.

      To learn more about Terraform, check out our How To Manage Infrastructure with Terraform series.



      Source link

      How To Deploy Multiple Environments in Your Terraform Project Without Duplicating Code


      The author selected the Free and Open Source Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      Terraform offers advanced features that become increasingly useful as your project grows in size and complexity. It’s possible to alleviate the cost of maintaining complex infrastructure definitions for multiple environments by structuring your code to minimize repetitions and introducing tool-assisted workflows for easier testing and deployment.

      Terraform associates a state with a backend, which determines where and how state is stored and retrieved. Every state has only one backend and is tied to an infrastructure configuration. Certain backends, such as local or s3, may contain multiple states. In that case, the pairing of state and infrastructure to the backend is describing a workspace. Workspaces allow you to deploy multiple distinct instances of the same infrastructure configuration without storing them in separate backends.

      In this tutorial, you’ll first deploy multiple infrastructure instances using different workspaces. You’ll then deploy a stateful resource, which, in this tutorial, will be a DigitalOcean Volume. Finally, you’ll reference pre-made modules from the Terraform Registry, which you can use to supplement your own.

      Prerequisites

      • A DigitalOcean Personal Access Token, which you can create via the DigitalOcean Control Panel. You can find instructions for this in the How to Generate a Personal Access Token tutorial.
      • Terraform installed on your local machine and a project set up with the DO provider. Complete Step 1 and Step 2 of the How To Use Terraform with DigitalOcean tutorial, and be sure to name the project folder terraform-advanced, instead of loadbalance. During Step 2, do not include the pvt_key variable and the SSH key resource.

      Note: We have specifically tested this tutorial using Terraform 0.13.

      Deploying Multiple Infrastructure Instances Using Workspaces

      Multiple workspaces are useful when you want to deploy or test a modified version of your main infrastructure without creating a separate project and setting up authentication keys again. Once you have developed and tested a feature using the separate state, you can incorporate the new code into the main workspace and possibly delete the additional state. When you init a Terraform project, regardless of backend, Terraform creates a workspace called default. It is always present and you can never delete it.

      However, multiple workspaces are not a suitable solution for creating multiple environments, such as for staging and production. Therefore workspaces, which only track the state, do not store the code or its modifications.

      Since workspaces do not track the actual code, you should manage the code separation between multiple workspaces at the version control (VCS) level by matching them to their infrastructure variants. How you can achieve this is dependent on the VCS tool itself; for example, in Git branches would be a fitting abstraction. To make it easier to manage the code for multiple environments, you can break them up into reusable modules, so that you avoid repeating similar code for each environment.

      Deploying Resources in Workspaces

      You’ll now create a project that deploys a Droplet, which you’ll apply from multiple workspaces.

      You’ll store the Droplet definition in a file called droplets.tf.

      Assuming you’re in the terraform-advanced directory, create and open it for editing by running:

      Add the following lines:

      droplets.tf

      resource "digitalocean_droplet" "web" {
        image  = "ubuntu-18-04-x64"
        name   = "web-${terraform.workspace}"
        region = "fra1"
        size   = "s-1vcpu-1gb"
      }
      

      This definition will create a Droplet running Ubuntu 18.04 with one CPU core and 1 GB RAM in the fra1 region. Its name will contain the name of the current workspace it is deployed from. When you’re done, save and close the file.

      Apply the project for Terraform to run its actions with:

      • terraform apply -var "do_token=${DO_PAT}"

      Your output will be similar to the following:

      Output

      An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # digitalocean_droplet.web will be created + resource "digitalocean_droplet" "web" { + backups = false + created_at = (known after apply) + disk = (known after apply) + id = (known after apply) + image = "ubuntu-18-04-x64" + ipv4_address = (known after apply) + ipv4_address_private = (known after apply) + ipv6 = false + ipv6_address = (known after apply) + ipv6_address_private = (known after apply) + locked = (known after apply) + memory = (known after apply) + monitoring = false + name = "web-default" + price_hourly = (known after apply) + price_monthly = (known after apply) + private_networking = (known after apply) + region = "fra1" + resize_disk = true + size = "s-1vcpu-1gb" + status = (known after apply) + urn = (known after apply) + vcpus = (known after apply) + volume_ids = (known after apply) + vpc_uuid = (known after apply) } Plan: 1 to add, 0 to change, 0 to destroy. ...

      Enter yes when prompted to deploy the Droplet in the default workspace.

      The name of the Droplet will be web-default, because the workspace you start with is called default. You can list the workspaces to confirm that it’s the only one available:

      You’ll receive the following output:

      Output

      * default

      The asterisk (*) means that you currently have that workspace selected.

      Create and switch to a new workspace called testing, which you’ll use to deploy a different Droplet, by running workspace new:

      • terraform workspace new testing

      You’ll have output similar to:

      Output

      Created and switched to workspace "testing"! You're now on a new, empty workspace. Workspaces isolate their state, so if you run "terraform plan" Terraform will not see any existing state for this configuration.

      You plan the deployment of the Droplet again by running:

      • terraform plan -var "do_token=${DO_PAT}"

      The output will be similar to the previous run:

      Output

      An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # digitalocean_droplet.web will be created + resource "digitalocean_droplet" "web" { + backups = false + created_at = (known after apply) + disk = (known after apply) + id = (known after apply) + image = "ubuntu-18-04-x64" + ipv4_address = (known after apply) + ipv4_address_private = (known after apply) + ipv6 = false + ipv6_address = (known after apply) + ipv6_address_private = (known after apply) + locked = (known after apply) + memory = (known after apply) + monitoring = false + name = "web-testing" + price_hourly = (known after apply) + price_monthly = (known after apply) + private_networking = (known after apply) + region = "fra1" + resize_disk = true + size = "s-1vcpu-1gb" + status = (known after apply) + urn = (known after apply) + vcpus = (known after apply) + volume_ids = (known after apply) + vpc_uuid = (known after apply) } Plan: 1 to add, 0 to change, 0 to destroy. ...

      Notice that Terraform plans to deploy a Droplet called web-testing, which it has named differently from web-default. This is because the default and testing workspaces have separate states and have no knowledge of each other’s resources—even though they stem from the same code.

      To confirm that you’re in the testing workspace, output the current one you’re in with workspace show:

      The output will be the name of the current workspace:

      Output

      testing

      To delete a workspace, you first need to destroy all its deployed resources. Then, if it’s active, you need to switch to another one using workspace select. Since the testing workspace here is empty, you can switch to default right away:

      • terraform workspace select default

      You’ll receive output of Terraform confirming the switch:

      Output

      Switched to workspace "default".

      You can then delete it by running workspace delete:

      • terraform workspace delete testing

      Terraform will then perform the deletion:

      Output

      Deleted workspace "testing"!

      You can destroy the Droplet you’ve deployed in the default workspace by running:

      • terraform destroy -var "do_token=${DO_PAT}"

      Enter yes when prompted to finish the process.

      In this section, you’ve worked in multiple Terraform workspaces. In the next section, you’ll deploy a stateful resource.

      Deploying Stateful Resources

      Stateless resources do not store data, so you can create and replace them quickly, because they are not unique. Stateful resources, on the other hand, contain data that is unique or not simply re-creatable; therefore, they require persistent data storage.

      Since you may end up destroying such resources, or multiple resources require their data, it’s best to store it in a separate entity, such as DigitalOcean Volumes.

      Volumes are objects that you can attach to Droplets (servers), but are separate from them, and provide additional storage space. In this step, you’ll define the Volume and connect it to a Droplet in droplets.tf.

      Open it for editing:

      Add the following lines:

      droplets.tf

      resource "digitalocean_droplet" "web" {
        image  = "ubuntu-18-04-x64"
        name   = "web-${terraform.workspace}"
        region = "fra1"
        size   = "s-1vcpu-1gb"
      }
      
      resource "digitalocean_volume" "volume" {
        region                  = "fra1"
        name                    = "new-volume"
        size                    = 10
        initial_filesystem_type = "ext4"
        description             = "New Volume for Droplet"
      }
      
      resource "digitalocean_volume_attachment" "volume_attachment" {
        droplet_id = digitalocean_droplet.web.id
        volume_id  = digitalocean_volume.volume.id
      }
      

      Here you define two new resources, the Volume itself and a Volume attachment. The Volume will be 10GB, formatted as ext4, called new-volume, and located in the same region as the Droplet. To connect the Volume to the Droplet, since they are separate entities, you define a Volume attachment object. volume_attachment takes the Droplet and Volume IDs and instructs the DigitalOcean cloud to make the Volume available to the Droplet as a disk device.

      When you’re done, save and close the file.

      Plan this configuration by running:

      • terraform plan -var "do_token=${DO_PAT}"

      The actions that Terraform will plan will be the following:

      Output

      An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # digitalocean_droplet.web will be created + resource "digitalocean_droplet" "web" { + backups = false + created_at = (known after apply) + disk = (known after apply) + id = (known after apply) + image = "ubuntu-18-04-x64" + ipv4_address = (known after apply) + ipv4_address_private = (known after apply) + ipv6 = false + ipv6_address = (known after apply) + ipv6_address_private = (known after apply) + locked = (known after apply) + memory = (known after apply) + monitoring = false + name = "web-default" + price_hourly = (known after apply) + price_monthly = (known after apply) + private_networking = (known after apply) + region = "fra1" + resize_disk = true + size = "s-1vcpu-1gb" + status = (known after apply) + urn = (known after apply) + vcpus = (known after apply) + volume_ids = (known after apply) + vpc_uuid = (known after apply) } # digitalocean_volume.volume will be created + resource "digitalocean_volume" "volume" { + description = "New Volume for Droplet" + droplet_ids = (known after apply) + filesystem_label = (known after apply) + filesystem_type = (known after apply) + id = (known after apply) + initial_filesystem_type = "ext4" + name = "new-volume" + region = "fra1" + size = 10 + urn = (known after apply) } # digitalocean_volume_attachment.volume_attachment will be created + resource "digitalocean_volume_attachment" "volume_attachment" { + droplet_id = (known after apply) + id = (known after apply) + volume_id = (known after apply) } Plan: 3 to add, 0 to change, 0 to destroy. ...

      The output details that Terraform would create a Droplet, a Volume, and a Volume attachment, which connects the Volume to the Droplet.

      You’ve now defined and connected a Volume (a stateful resource) to a Droplet. In the next section, you’ll review public, pre-made Terraform modules that you can incorporate in your project.

      Referencing Pre-made Modules

      Aside from creating your own custom modules for your projects, you can also use pre-made modules and providers from other developers, which are publicly available at Terraform Registry.

      In the modules section you can search the database of available modules and sort by provider in order to find the module with the functionality you need. Once you’ve found one, you can read its description, which lists the inputs and outputs the module provides, as well as its external module and provider dependencies.

      Terraform Registry - SSH key Module

      You’ll now add the DigitalOcean SSH key module to your project. You’ll store the code separate from existing definitions in a file called ssh-key.tf. Create and open it for editing by running:

      Add the following lines:

      ssh-key.tf

      module "ssh-key" {
        source         = "clouddrove/ssh-key/digitalocean"
        key_path       = "~/.ssh/id_rsa.pub"
        key_name       = "new-ssh-key"
        enable_ssh_key = true
      }
      

      This code defines an instance of the clouddrove/droplet/digitalocean module from the registry and sets some of the parameters it offers. It should add a public SSH key to your account by reading it from the ~/.ssh/id_rsa.pub.

      When you’re done, save and close the file.

      Before you plan this code, you must download the referenced module by running:

      You’ll receive output similar to the following:

      Output

      Initializing modules... Downloading clouddrove/ssh-key/digitalocean 0.13.0 for ssh-key... - ssh-key in .terraform/modules/ssh-key Initializing the backend... Initializing provider plugins... - Using previously-installed digitalocean/digitalocean v1.22.2 Terraform has been successfully initialized! ...

      You can now plan the code for the changes:

      • terraform plan -var "do_token=${DO_PAT}"

      You’ll receive output similar to this:

      Output

      Refreshing Terraform state in-memory prior to plan... The refreshed state will be used to calculate this plan, but will not be persisted to local or remote state storage. ------------------------------------------------------------------------ An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: ... # module.ssh-key.digitalocean_ssh_key.default[0] will be created + resource "digitalocean_ssh_key" "default" { + fingerprint = (known after apply) + id = (known after apply) + name = "devops" + public_key = "ssh-rsa ... demo@clouddrove" } Plan: 4 to add, 0 to change, 0 to destroy. ...

      The output shows that you would create the SSH key resource, which means that you downloaded and invoked the module from your code.

      Conclusion

      Bigger projects can make use of some advanced features Terraform offers to help reduce complexity and make maintenance easier. Workspaces allow you to test new additions to your code without touching the stable main deployments. You can also couple workspaces with a version control system to track code changes. Using pre-made modules can also shorten development time, but may incur additional expenses or time in the future if the module becomes obsolete.

      For further resources on using Terraform, check out our How To Manage Infrastructure With Terraform series.



      Source link