One place for hosting & domains

      DigitalOcean

      How To Transfer DigitalOcean Spaces Between Regions Using Rclone


      Introduction

      DigitalOcean Spaces is an object storage service designed to make it easy and cost effective to store and serve large amounts of data.

      In this guide, we will cover how to migrate data between Spaces regions, by using Rclone to transfer data between two Spaces. We will demonstrate how to install Rclone, the configuration settings needed to access multiple regions, and the commands that you can use to synchronize your files between regions and verify their integrity.

      Creating API Keys and Finding Spaces Endpoint Information

      Before we begin installing and configuring Rclone to copy our objects between Spaces, we will need some information about our DigitalOcean Spaces account. We will need a Spaces API key, and we will need to know the regions and names of our source and destination Spaces.

      Generating a DigitalOcean Spaces API Key

      To create a DigitalOcean Spaces API key, follow the “Creating an Access Key” section of our How To Create a DigitalOcean Space API Key documentation.

      Save the access key ID and the secret key. We will use them later to configure Rclone to access our account.

      Finding the Spaces S3-Compatible Endpoint

      Next, we need to find the endpoint for each Space. You can view the Space’s endpoint within the DigitalOcean Control Panel by selecting the Space and viewing the Settings tab:

      DigitalOcean Spaces endpoint

      The endpoint will always be the region you created the Space in, followed by .digitaloceanspaces.com. Make note of the endpoint for both of your Spaces. Will we use this information when creating our rclone configuration.

      Installing Rclone

      You’re now ready to install Rclone. You can do this on your local machine, or – if you are bandwidth limited – you may want to install Rclone on a Droplet located in the source or destination Spaces region.

      Visit the Downloads section of the project’s website to find binaries of the utility compiled for different platforms. Download the zipped binary that matches your computer’s operating system to get started.

      Once you have the Rclone zip file downloaded to your computer, follow the section below that matches your platform.

      Linux

      Before we can extract the archive, we will need to ensure that the unzip utility is available.

      If you are running Ubuntu or Debian, you can update the local package index and install unzip by typing:

      • sudo apt update
      • sudo apt install unzip

      If you are running CentOS or Fedora, you can install unzip by typing:

      With unzip installed, navigate to the directory where you downloaded the rclone zip file:

      Next, unzip the archive and move into the newly created directory:

      • unzip rclone*
      • cd rclone-v*

      From here, we can copy the binary to the /usr/local/bin directory so that it is available system-wide:

      • sudo cp rclone /usr/local/bin

      Next, we add the manual page to our system, so that we can easily get help on the command syntax and available options. Make sure that the local manual directory is available and then copy the rclone.1 file:

      • sudo mkdir -p /usr/local/share/man/man1
      • sudo cp rclone.1 /usr/local/share/man/man1

      Then update the man database to add the new manual page to the system:

      Finally, we can create the Rclone configuration directory and open up a configuration file:

      • mkdir -p ~/.config/rclone
      • nano ~/.config/rclone/rclone.conf

      This will open your text editor with a new blank file. Skip ahead to the section on Configuring Rclone to continue.

      macOS

      If you are running macOS, begin by navigating in the terminal to the directory where you downloaded the rclone zip file:

      Next, unzip the file and move into the newly created directory:

      • unzip -a rclone*
      • cd rclone-v*

      Next, make sure the /usr/local/bin directory is available and then copy the rclone binary to it:

      • sudo mkdir -p /usr/local/bin
      • sudo cp rclone /usr/local/bin

      Finally, we can create the configuration directory and open up a configuration file:

      • mkdir -p ~/.config/rclone
      • nano ~/.config/rclone/rclone.conf

      This will open up your text editor with a new blank file. Skip ahead to the section on Configuring Rclone to continue.

      Windows

      If you are running Windows, begin by navigating to the Downloads directory in the Windows File Explorer. Select the rclone zip file and right-click. In the context menu that appears, click Extract All…:

      Windows extract rclone zip file

      Follow the prompts to extract the files from the zip archive.

      The rclone.exe utility must be run from the command line. Open a new Command Prompt (the cmd.exe program) window by clicking the Windows button in the lower-left corner, typing cmd, and selecting Command Prompt.

      Inside, navigate to the rclone path you extracted by typing:

      • cd "%HOMEPATH%Downloadsrclone*rclone*"

      List the directory contents to verify that you are in the correct location:

      Output

      10/23/2017 01:02 PM <DIR> . 10/23/2017 01:02 PM <DIR> .. 10/23/2017 01:02 PM 17 git-log.txt 10/23/2017 01:02 PM 296,086 rclone.1 10/23/2017 01:02 PM 16,840,192 rclone.exe 10/23/2017 01:02 PM 315,539 README.html 10/23/2017 01:02 PM 261,497 README.txt 5 File(s) 17,713,331 bytes 2 Dir(s) 183,296,266,240 bytes free

      You will need to be in this directory whenever you want to use the rclone.exe command.

      Note: On macOS and Linux, we run the tool by typing rclone, but on Windows, the command is called rclone.exe. Throughout the rest of this guide, we will be providing commands as rclone, so be sure to substitute rclone.exe each time when running on Windows.

      Next, we can create the configuration directory and open up a configuration file to define our S3 and Spaces credentials:

      • mkdir "%HOMEPATH%.configrclone"
      • notepad "%HOMEPATH%.configrclonerclone.conf"

      This will open up your text editor with a new blank file. Continue ahead to learn how to define your Spaces regions in the configuration file.

      Configuring Rclone

      We will configure our two DigitalOcean Spaces regions as Rclone “remotes” in the Rclone configuration file. Paste the following section in the configuration file to define the first region:

      ~/.config/rclone/rclone.conf

      [spaces-sfo2]
      type = s3
      env_auth = false
      access_key_id = your_spaces_access_key
      secret_access_key = your_spaces_secret_key
      endpoint = sfo2.digitaloceanspaces.com
      acl = private
      

      Here, we define a new rclone “remote” named spaces-sfo2. Change the region name to match the Spaces region you are configuring.

      We set the type to s3 so that rclone knows the appropriate way to interact with and manage the remote storage resource. We will define the Spaces access credentials in this configuration file, so we can set env_auth to false.

      Next, we set the access_key_id and secret_access_key variables to our Spaces access key and secret key, respectively. Be sure to change the values to the credentials associated with your account.

      We set the endpoint to the Spaces endpoint we looked up earlier.

      Finally, we set the acl to private to protect our assets until we want to share them.

      Next, make a duplicate of the configuration block you just created, then update the name and endpoint region to reflect your second region:

      ~/.config/rclone/rclone.conf

      . . .
      
      [spaces-nyc3]
      type = s3
      env_auth = false
      access_key_id = your_spaces_access_key
      secret_access_key = your_spaces_secret_key
      endpoint = nyc3.digitaloceanspaces.com
      acl = private
      

      The rest of the configuration should remain the same as for the first region. Save and close the file when you are finished.

      On macOS and Linux, be sure to lock down the permissions of the configuration file since our credentials are inside:

      • chmod 600 ~/.config/rclone/rclone.conf

      On Windows, permissions are denied to non-administrative users unless explicitly granted, so we shouldn’t need to adjust access manually.

      Next, we’ll use rclone to explore our Spaces and sync data between them.

      Copying Objects from S3 to Spaces

      Now that our configuration is complete, we are ready to transfer our files.

      Begin by checking the rclone configured remotes:

      Output

      spaces-nyc3: spaces-sfo2:

      Both of the regions we defined are displayed.

      We can view the available Spaces by asking rclone to list the “directories” associated with the remotes (make sure to add the colon to the end of the remote name):

      Output

      -1 2019-09-23 13:07:54 -1 source-space

      The above output indicates that one Space, called source-space was found in the sfo2 region.

      You can repeat the procedure to view the other region:

      Output

      -1 2019-09-23 13:08:28 -1 destination-space

      To view the contents of a Space, you can use the tree command. Pass in the remote name, followed by a colon and the name of the “directory” you wish to list (the Space name):

      • rclone tree spaces-sfo2:source-space

      Output

      / ├── Photos │ ├── 2019.01.24-23.10.27.png │ ├── 2019.01.24-23.11.39.png │ ├── 2019.01.24-23.18.00.png │ ├── 2019.01.24-23.18.18.png │ ├── 2019.01.24-23.18.30.png │ ├── 2019.01.24-23.19.32.png │ ├── 2019.01.24-23.23.06.png │ ├── 2019.01.24-23.23.53.png │ ├── 2019.01.24-23.25.14.png │ ├── 2019.01.24-23.26.22.png │ ├── 2019.01.25-12.43.35.png │ ├── 2019.03.13-14.35.34.png │ └── 2019.03.13-14.40.52.png └── Photos.zip 1 directories, 14 files

      When you are ready, you can copy the files between Spaces by typing:

      • rclone sync spaces-sfo2:source-space spaces-nyc3:destination-space

      Assuming everything went well, rclone will begin copying objects between the two Spaces.

      Note: If you hadn’t previously created the destination Space in the specified region, rclone will attempt to create one for you with the given name. This will fail if the name provided is already being used by another account or if the name doesn’t meet the naming requirements for DigitalOcean Spaces (lowercase letters, numbers, and dashes only).

      When the transfer is complete, you can check that the objects have transferred by viewing them with the tree subcommand:

      • rclone tree spaces-nyc3:destination-space

      Output

      / ├── Photos │ ├── 2019.01.24-23.10.27.png │ ├── 2019.01.24-23.11.39.png │ ├── 2019.01.24-23.18.00.png │ ├── 2019.01.24-23.18.18.png │ ├── 2019.01.24-23.18.30.png │ ├── 2019.01.24-23.19.32.png │ ├── 2019.01.24-23.23.06.png │ ├── 2019.01.24-23.23.53.png │ ├── 2019.01.24-23.25.14.png │ ├── 2019.01.24-23.26.22.png │ ├── 2019.01.25-12.43.35.png │ ├── 2019.03.13-14.35.34.png │ └── 2019.03.13-14.40.52.png └── Photos.zip 1 directories, 14 files

      For more robust verification, use the check subcommand to compare the objects in both regions:

      • rclone check spaces-sfo2:source-space spaces-nyc3:destination-space

      Output

      2019/09/23 14:29:11 NOTICE: S3 bucket destination-space: 0 differences found 2019/09/23 14:29:11 NOTICE: S3 bucket destination-space: 14 matching files

      This will compare the hash values of each object in both remotes. You may receive a message indicating that some hashes could not be compared. In that case, you can rerun the command with the --size-only flag (which just compares based on file size) or the --download flag (which downloads each object from both remotes to compare locally) to verify the transfer integrity.

      Conclusion

      In this guide, we’ve covered how to transfer objects between two DigitalOcean Spaces regions. We gathered API credentials and endpoint information from the Spaces service, installed and configured the rclone utility on our local computer, and then copied all objects from a source Space to a destination Space.

      The rclone client can be used for many other object storage management tasks including uploading or downloading files, mounting Spaces on the local filesystem, and creating or deleting additional Spaces. Check out the man page to learn more about the functionality the tool provides.



      Source link

      How To Autoscale Your Workloads on DigitalOcean Kubernetes


      Introduction

      When working with an application built on Kubernetes, developers will often need to schedule additional pods to handle times of peak traffic or increased load processing. By default, scheduling these additional pods is a manual step; the developer must change the number of desired replicas in the deployment object to account for the increased traffic, then change it back when the additional pods are no longer needed. This dependency on manual intervention can be less than ideal in many scenarios. For example, your workload could hit peak hours in the middle of the night when no one is awake to scale the pods, or your website could get an unexpected increase in traffic when a manual response would not be quick enough to deal with the load. In these situations, the most efficient and least error prone approach is to automate your clusters scaling with the Horizontal Pod Autoscaler (HPA).

      By using information from the Metrics Server, the HPA will detect increased resource usage and respond by scaling your workload for you. This is especially useful with microservice architectures, and will give your Kubernetes cluster the ability to scale your deployment based on metrics such as CPU utilization. When combined with DigitalOcean Kubernetes (DOKS), a managed Kubernetes offering that provides developers with a platform for deploying containerized applications, using HPA can create an automated infrastructure that quickly adjusts to changes in traffic and load.

      Note: When considering whether to use autoscaling for your workload, keep in mind that autoscaling works best for stateless applications, especially ones that are capable of having multiple instances of the application running and accepting traffic in parallel. This parallelism is important because the main objective of autoscaling is to dynamically distribute an application’s workload across multiple instances in your Kubernetes cluster to ensure your application has the resources necessary to service the traffic in a timely and stable manner without overloading any single instance.

      An example of a workload that does not exhibit this parrallelism is database autoscaling. Setting up autoscaling for a database would be vastly more complex, as you would need to account for race conditions, issues with data integrity, data synchronization, and constant additions and removals of database cluster members. For reasons like these, we do not recommend using this tutorial’s autoscaling strategy for databases.

      In this tutorial, you will set up a sample Nginx deployment on DOKS that can autoscale horizontally to account for increased CPU load. You will accomplish this by deploying Metrics Server into your cluster to gather pod metrics for HPA to use when determining when to scale.

      Prerequisites

      Before you begin this guide you’ll need the following:

      • A DigitalOcean Kubernetes cluster with your connection configured as the kubectl default. Instructions on how to configure kubectl are shown under the Connect to your Cluster step when you create your cluster. To create a Kubernetes cluster on DigitalOcean, see Kubernetes Quickstart.

      • The Helm package manager installed on your local machine, and Tiller installed on your cluster. To do this, complete Steps 1 and 2 of the How To Install Software on Kubernetes Clusters with the Helm Package Manager tutorial.

      Step 1 — Creating a Test Deployment

      In order to show the effect of the HPA, you will first deploy an application that you will use to autoscale. This tutorial uses a standard Nginx Docker image as a deployment because it is fully capable of operating in parallel, is widely used within Kubernetes with such tools as the Nginx Ingress Controller, and is lightweight to set up. This Nginx deployment will serve a static Welcome to Nginx! page that comes standard in the base image. If you already have a deployment you would like to scale, feel free to use that deployment and skip this step.

      Create the sample deployment using the Nginx base image by issuing the following command. You can replace the name web if you would like to give your deployment a different name:

      • kubectl create deployment web --image=nginx:latest

      The --image=nginx:latest flag will create the deployment from the latest version of the Nginx base image.

      After a few seconds, your web pod will spin up. To see this pod, run the following command, which will show you the pods running in the current namespace:

      This will give you output similar to the following:

      Output

      NAME READY STATUS RESTARTS AGE web-84d7787df5-btf9h 1/1 Running 0 11s

      Take note that there is only one pod originally deployed. Once autoscaling triggers, more pods will spin up automatically.

      You now have a basic deployment up and running within the cluster. This is the deployment you are going to configure for autoscaling. Your next step is to configure this deployment to define its resource requests and limits.

      Step 2 — Setting CPU Limits and Requests on Your Deployment

      In this step, you are going to set requests and limits on CPU usage for your deployment. Limits in Kubernetes are set on the deployment to describe the maximum amount of the resource (either CPU or Memory) that the pod can use. Requests are set on the deployment to describe how much of that resource is needed on a node in order for that node to be considered as a valid node for scheduling. For example, if your webserver had a memory request set at 1GB, only nodes with at least 1GB of free memory would be considered for scheduling. For autoscaling, it is necessary to set these limits and requests because the HPA will need to have this information when making scaling and scheduling decisions.

      To set the requests and limits, you will need to make changes to the deployment you just created. This tutorial will use the following kubectl edit command to modify the API object configuration stored in the cluster. The kubectl edit command will open the editor defined by your KUBE_EDITOR or EDITOR environment variables, or fall back to vi for Linux or notepad for Windows by default.

      Edit your deployment:

      • kubectl edit deployment web

      You will see the configuration for the deployment. You can now set resource limits and requests specified for your deployment’s CPU usage. These limits set the baseline for how much of each resource a pod of this deployment can use individually. Setting this will give the HPA a frame of reference to know whether a pod is being overworked. For example, if you expect your pod to have an upper limit of 100 millicores of CPU and the pod is currently using 95 millicores, HPA will know that it is at 95% capacity. Without providing that limit of 100 milicores, the HPA can’t decipher the pod’s full capacity.

      We can set the limits and requests in the resources section:

      Deployment Configuration File

      . . .
        template:
          metadata:
            creationTimestamp: null
            labels:
              app: web
          spec:
            containers:
            - image: nginx:latest
              imagePullPolicy: Always
              name: nginx
              resources: {}
              terminationMessagePath: /dev/termination-log
              terminationMessagePolicy: File
            dnsPolicy: ClusterFirst
            restartPolicy: Always
            schedulerName: default-scheduler
            securityContext: {}
            terminationGracePeriodSeconds: 30
      status:
        availableReplicas: 1
      . . .
      

      For this tutorial, you will be setting requests for CPU to 100m and memory to 250Mi. These values are meant for demonstration purposes; every workload is different, so these values may not make sense for other workloads. As a general rule, these values should be set to the maximum that a pod of this workload should be expected to use. Monitoring the application and gathering resource usage data on how it performs during low and peak times is recommended to help determine these values. These values can also be tweaked and changed at any time, so you can always come back and optimize your deployment later.

      Go ahead and insert the following highlighted lines under the resources section of your Nginx container:

      Deployment Configuration File

      . . .
        template:
          metadata:
            creationTimestamp: null
            labels:
              app: web
          spec:
            containers:
            - image: nginx:latest
              imagePullPolicy: Always
              name: nginx
              resources:
                limits:
                  cpu: 300m
                requests:
                  cpu: 100m
                  memory: 250Mi
              terminationMessagePath: /dev/termination-log
              terminationMessagePolicy: File
            dnsPolicy: ClusterFirst
            restartPolicy: Always
            schedulerName: default-scheduler
            securityContext: {}
            terminationGracePeriodSeconds: 30
      status:
        availableReplicas: 1
      . . .
      

      Once you’ve inserted these lines, save and quit the file. If there is an issue with the syntax, kubectl will reopen the file for you with an error posted for more information.

      Now that you have your limits and requests set, you need to ensure that your metrics are being gathered so that the HPA can monitor and correctly adhere to these limits. In order to do this, you will set up a service to gather the CPU metrics. For this tutorial, you will use the Metrics Server project for collecting these metrics, which you will install with a Helm chart.

      Step 3 — Installing Metrics Server

      Next, you will install the Kubernetes Metric Server. This is the server that scrapes pod metrics, which will gather the metrics that the HPA will use to decide if autoscaling is necessary.

      To install the Metrics Server using Helm, run the following command:

      • helm install stable/metrics-server --name metrics-server

      This will install the latest stable version of Metrics Server. The --name flag names this release metrics-server.

      Once you wait for this pod to initialize, try to use the kubectl top pod command to display your pod’s metrics:

      This command is meant to give a pod-level view of resource usage in your cluster, but because of the way that DOKS handles DNS, this command will return an error at this point:

      Output

      Error: Metrics not available for pod Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods.metrics.k8s.io)

      This error occurs because DOKS nodes do not create a DNS record for themselves, and since Metrics Server contacts nodes through their hostnames, the hostnames do not resolve properly. To fix this problem, change how the Metrics Server communicates with nodes by adding runtime flags to the Metrics Server container using the following command:

      • kubectl edit deployment metrics-server

      You will be adding a flag under the command section.

      metrics-server Configuration File

      . . .
        template:
          metadata:
            creationTimestamp: null
            labels:
              app: metrics-server
              release: metrics-server
          spec:
            affinity: {}
            containers:
            - command:
              - /metrics-server
              - --cert-dir=/tmp
              - --logtostderr
              - --secure-port=8443
              image: gcr.io/google_containers/metrics-server-amd64:v0.3.4
              imagePullPolicy: IfNotPresent
              livenessProbe:
                failureThreshold: 3
                httpGet:
                  path: /healthz
      . . .
      

      The flag you are adding is --kubelet-preferred-address-types=InternalIP. This flag tells the metrics server to contact nodes using their internalIP as opposed to their hostname. You can use this flag as a workaround to communicate with the nodes via internal IP addresses.

      Also, add the --metric-resolution flag to change the default rate at which the Metrics Server scrapes metrics. For this tutorial, we will set Metrics Server to make data points every 60s, but if you would like more metrics data, you could ask for the Metrics Server to scrape metrics every 10s or 20s. This will give you more data points of resource usage per period of time. Feel free to fine-tune this resolution to meet your needs.

      Add the following highlighted lines to the file:

      metrics-server Configuration File

      . . .
        template:
          metadata:
            creationTimestamp: null
            labels:
              app: metrics-server
              release: metrics-server
          spec:
            affinity: {}
            containers:
            - command:
              - /metrics-server
              - --cert-dir=/tmp
              - --logtostderr
              - --secure-port=8443
              - --metric-resolution=60s
              - --kubelet-preferred-address-types=InternalIP
              image: gcr.io/google_containers/metrics-server-amd64:v0.3.4
              imagePullPolicy: IfNotPresent
              livenessProbe:
                failureThreshold: 3
                httpGet:
                  path: /healthz
      . . .
      

      After the flag is added, save and exit your editor.

      To verify your Metrics Server is running, use kubectl top pod after a few minutes. As before, this command will give us resource usage on a pod level. This time, a working Metrics Server will allow you to see metrics on each pod:

      This will give the following output, with your Metrics Server pod running:

      Output

      NAME CPU(cores) MEMORY(bytes) metrics-server-db745fcd5-v8gv6 3m 12Mi web-555db5bf6b-f7btr 0m 2Mi

      You now have a functional Metrics Server and are able to view and monitor resource usage of pods within your cluster. Next, you are going to configure the HPA to monitor this data and react to periods of high CPU usage.

      Step 4 — Creating and Validating the Horizontal Pod Autoscaler

      Lastly, it’s time to create the Horizontal Pod Autoscaler (HPA) for your deployment. The HPA is the actual Kubernetes object that routinely checks the CPU usage data collected from your Metrics Server and scales your deployment based on the thresholds you set in Step 2.

      Create the HPA using the kubectl autoscale command:

      • kubectl autoscale deployment web --max=4 --cpu-percent=80

      This command creates the HPA for your web deployment. It also uses the --max flag to set the max replicas that web can be scaled to, which in this case you set as 4.

      The --cpu-percent flag tells the HPA at what percent usage of the limit you set in Step 2 you want to trigger the autoscale to occur. This also uses the requests to help schedule the scaled up pods to a node that can accomodate the initial resource allocation. In this example, if the limit you set on your deployment in Step 1 was 100 millicores (100m), this command would trigger an autoscale once the pod hit 80m in average CPU usage. This would allow the deployment to autoscale prior to maxing out its CPU resources.

      Now that your deployment can automatically scale, it’s time to put this to the test.

      To validate, you are going to generate a load that will put your cluster over your threshold and then watch the autoscaler take over. To start, open up a second terminal to watch the currently scheduled pods and refresh the list of pods every 2 seconds. To accomplish this, use the watch command in this second terminal:

      The watch command issues the command given as its arguments continuously, displaying the output in your terminal. The duration between repetitions can be further configured with the -n flag. For the purposes of this tutorial, the default two seconds setting will suffice.

      The terminal will now display the output of kubectl top pods initially and then every 2 seconds it will refresh the output that that command generates, which will look similar to this:

      Output

      Every 2.0s: kubectl top pods NAME CPU(cores) MEMORY(bytes) metrics-server-6fd5457684-7kqtz 3m 15Mi web-7476bb659d-q5bjv 0m 2Mi

      Take note of the number of pods currently deployed for web.

      Switch back to your original terminal. You will now open a terminal inside your current web pod using kubectl exec and create an artificial load. You can accomplish this by going into the pod and installing the stress CLI tool.

      Enter your pod using kubectl exec, replacing the highlighted pod name with the name of your web pod:

      • kubectl exec -it web-f765fd676-s9729 /bin/bash

      This command is very similar in concept to using ssh to log in to another machine. /bin/bash establishes a bash shell in your pod.

      Next, from the bash shell inside your pod, update the repository metadata and install the stress package.

      • apt update; apt-get install -y stress

      Note: For CentOS-based containers, this would be:

      Next, generate some CPU load on your pod using the stress command and let it run:

      Now, go back to your watch command in the second terminal. Wait a few minutes for the Metrics Server to gather CPU data that is above the HPA’s defined threshold. Note that metrics by default are gathered at whichever rate you set --metric-resolution equal to when configuring the metrics server. It may take a minute or so for the usage metrics to update.

      After about two minutes, you will see additional web pods spin up:

      Output

      Every 2.0s: kubectl top pods NAME CPU(cores) MEMORY(bytes) metrics-server-db745fcd5-v8gv6 6m 16Mi web-555db5bf6b-ck98q 0m 2Mi web-555db5bf6b-f7btr 494m 21Mi web-555db5bf6b-h5cbx 0m 1Mi web-555db5bf6b-pvh9f 0m 2Mi

      You can now see that the HPA scheduled new pods based off the CPU load gathered by Metrics Server. When you are satisfied with this validation, use CTRL+C to stop the stress command in your first terminal, then exit your pod’s bash shell.

      Conclusion

      In this article you created a deployment that will autoscale based on CPU load. You added CPU resource limits and requests to your deployment, installed and configured Metrics Server in your cluster through the use of Helm, and created an HPA to make scaling decisions.

      This was a demonstration deployment of both Metrics Server and HPA. Now you can tweak the configuration to fit your particular use cases. Be sure to poke around the Kubernetes HPA docs for help and info on requests and limitations. Also, check out the Metrics Server project see all the tunable settings that may apply to your use case.

      If you would like to do more with Kubernetes, head over to our Kubernetes Community page or explore our Managed Kubernetes service.



      Source link

      How to Manage DigitalOcean and Kubernetes Infrastructure with Pulumi


      The author selected the Diversity in Tech Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      Pulumi is a tool for creating, deploying, and managing infrastructure using code written in general purpose programming languages. It supports automating all of DigitalOcean’s managed services—such as Droplets, managed databases, DNS records, and Kubernetes clusters—in addition to application configuration. Deployments are performed from an easy-to-use command-line interface that also integrates with a wide variety of popular CI/CD systems.

      Pulumi supports multiple languages but in this tutorial you will use TypeScript, a statically typed version of JavaScript that uses the Node.js runtime. This means you will get IDE support and compile-time checking that will help to ensure you’ve configured the right resources, used correct slugs, etc., while still being able to access any NPM modules for utility tasks.

      In this tutorial, you will provision a DigitalOcean Kubernetes cluster, a load balanced Kubernetes application, and a DigitalOcean DNS domain that makes your application available at a stable domain name of your choosing. This can all be provisioned in 60 lines of infrastructure-as-code and a single pulumi up command-line gesture. After this tutorial, you’ll be ready to productively build powerful cloud architectures using Pulumi infrastructure-as-code that leverages the full surface area of DigitalOcean and Kubernetes.

      Prerequisites

      To follow this tutorial, you will need:

      • A DigitalOcean Account to deploy resources to. If you do not already have one, register here.
      • A DigitalOcean API Token to perform automated deployments. Generate a personal access token here and keep it handy as you’ll use it in Step 2.
      • Because you’ll be creating and using a Kubernetes cluster, you’ll need to install kubectl. Don’t worry about configuring it further — you’ll do that later.
      • You will write your infrastructure-as-code in TypeScript, so you will need Node.js 8 or later. Download it here or install it using your system’s package manager.
      • You’ll use Pulumi to deploy infrastructure, so you’ll need to install the open source Pulumi SDK.
      • To perform the optional Step 5, you will need a domain name configured to use DigitalOcean nameservers. This guide explains how to do this for your registrar of choice.

      Step 1 — Scaffolding a New Project

      The first step is to create a directory that will store your Pulumi project. This directory will contain the source code for your infrastructure definitions, in addition to metadata files describing the project and its NPM dependencies.

      First, create the directory:

      Next, move in to the newly created directory:

      From now on, run commands from your newly created do-k8s directory.

      Next, create a new Pulumi project. There are different ways to accomplish this, but the easiest way is to use the pulumi new command with the typescript project template. This command will first prompt you to log in to Pulumi so that your project and deployment state are saved, and will then create a simple TypeScript project in the current directory:

      Here you have passed the -y option to the new command which tells it to accept default project options. For example, the project name is taken from the current directory’s name, and so will be do-k8s. If you’d like to use different options for your project name, simply elide the -y.

      After running the command, list the contents of the directory with ls:

      The following files will now be present:

      Output

      Pulumi.yaml index.ts node_modules package-lock.json package.json tsconfig.json

      The primary file you’ll be editing is index.ts. Although this tutorial only uses this single file, you can organize your project any way you see fit using Node.js modules. This tutorial also describes one step at a time, leveraging the fact that Pulumi can detect and incrementally deploy only what has changed. If you prefer, you can just populate the entire program, and deploy it all in one go using pulumi up.

      Now that you’ve scaffolded your new project, you are ready to add the dependencies needed to follow the tutorial.

      Step 2 — Adding Dependencies

      The next step is to install and add dependencies on the DigitalOcean and Kubernetes packages. First, install them using NPM:

      This will download the NPM packages, Pulumi plugins, and save them as dependencies.

      Next, open the index.ts file with your favorite editor. This tutorial will use nano:

      Replace the contents of your index.ts with the following:

      index.ts

      import * as digitalocean from "@pulumi/digitalocean";
      import * as kubernetes from "@pulumi/kubernetes";
      

      This makes the full contents of these packages available to your program. If you type "digitalocean." using an IDE that understands TypeScript and Node.js, you should see a list of DigitalOcean resources supported by this package, for instance.

      Save and close the file after adding the content.

      Note: We will be using a subset of what’s available in those packages. For complete documentation of resources, properties, and associated APIs, please refer to the relevant API documentation for the @pulumi/digitalocean and @pulumi/kubernetes packages.

      Next, you will configure your DigitalOcean token so that Pulumi can provision resources in your account:

      • pulumi config set digitalocean:token YOUR_TOKEN_HERE --secret

      Notice the --secret flag, which uses Pulumi’s encryption service to encrypt your token, ensuring that it is stored in cyphertext. If you prefer, you can use the DIGITALOCEAN_TOKEN environment variable instead, but you’ll need to remember to set it every time you update your program, whereas using configuration automatically stores and uses it for your project.

      In this step you added the necessary dependencies and configured your API token with Pulumi so that you can provision your Kubernetes cluster.

      Step 3 — Provisioning a Kubernetes Cluster

      Now you’re ready to create a DigitalOcean Kubernetes cluster. Get started by reopening the index.ts file:

      Add these lines at the end of your index.ts file:

      index.ts

      …
      const cluster = new digitalocean.KubernetesCluster("do-cluster", {
          region: digitalocean.Regions.SFO2,
          version: "latest",
          nodePool: {
              name: "default",
              size: digitalocean.DropletSlugs.DropletS2VPCU2GB,
              nodeCount: 3,
          },
      });
      
      export const kubeconfig = cluster.kubeConfigs[0].rawConfig;
      

      This new code allocates an instance of digitalocean.KubernetesCluster and sets a number of properties on it. This includes using the sfo2 region slug, the latest supported version of Kubernetes, the s-2vcpu-2gb Droplet size slug, and states your desired count of three Droplet instances. Feel free to change any of these, but be aware that DigitalOcean Kubernetes is only available in certain regions at the time of this writing. You can refer to the product documentation for updated information about region availability.

      For a complete list of properties you can configure on your cluster, please refer to the KubernetesCluster API documentation.

      The final line in that code snippet exports the resulting Kubernetes cluster’s kubeconfig file so that it’s easy to use. Exported variables are printed to the console and also accessible to tools. You will use this momentarily to access our cluster from standard tools like kubectl.

      Now you’re ready to deploy your cluster. To do so, run pulumi up:

      This command takes the program, generates a plan for creating the infrastructure described, and carries out a series of steps to deploy those changes. This works for the initial creation of infrastructure in addition to being able to diff and update your infrastructure when subsequent updates are made. In this case, the output will look something like this:

      Output

      Previewing update (dev): Type Name Plan + pulumi:pulumi:Stack do-k8s-dev create + └─ digitalocean:index:KubernetesCluster do-cluster create Resources: + 2 to create Do you want to perform this update? yes > no details

      This says that proceeding with the update will create a single Kubernetes cluster named do-cluster. The yes/no/details prompt allows us to confirm that this is the desired outcome before any changes are actually made. If you select details, a full list of resources and their properties will be shown. Choose yes to begin the deployment:

      Output

      Updating (dev): Type Name Status + pulumi:pulumi:Stack do-k8s-dev created + └─ digitalocean:index:KubernetesCluster do-cluster created Outputs: kubeconfig: "…" Resources: + 2 created Duration: 6m5s Permalink: https://app.pulumi.com/…/do-k8s/dev/updates/1

      It takes a few minutes to create the cluster, but then it will be up and running, and the full kubeconfig will be printed out to the console. Save the kubeconfig to a file:

      • pulumi stack output kubeconfig > kubeconfig.yml

      And then use it with kubectl to perform any Kubernetes command:

      • KUBECONFIG=./kubeconfig.yml kubectl get nodes

      You will receive output similar to the following:

      Output

      NAME STATUS ROLES AGE VERSION default-o4sj Ready <none> 4m5s v1.14.2 default-o4so Ready <none> 4m3s v1.14.2 default-o4sx Ready <none> 3m37s v1.14.2

      At this point you’ve set up infrastructure-as-code and have a repeatable way to bring up and configure new DigitalOcean Kubernetes clusters. In the next step, you will build on top of this to define the Kubernetes infrastructure in code and learn how to deploy and manage them similarly.

      Step 4 — Deploying an Application to Your Cluster

      Next, you will describe a Kubernetes application’s configuration using infrastructure-as-code. This will consist of three parts:

      1. A Provider object, which tells Pulumi to deploy Kubernetes resources to the DigitalOcean cluster, rather than the default of whatever kubectl is configured to use.
      2. A Kubernetes Deployment, which is the standard Kubernetes way of deploying a Docker container image that is replicated across any number of Pods.
      3. A Kubernetes Service, which is the standard way to tell Kubernetes to load balance access across a target set of Pods (in this case, the Deployment above).

      This is a fairly standard reference architecture for getting up and running with a load balanced service in Kubernetes.

      To deploy all three of these, open your index.ts file again:

      Once the file is open, append this code to the end of the file:

      index.ts

      …
      const provider = new kubernetes.Provider("do-k8s", { kubeconfig })
      
      const appLabels = { "app": "app-nginx" };
      const app = new kubernetes.apps.v1.Deployment("do-app-dep", {
          spec: {
              selector: { matchLabels: appLabels },
              replicas: 5,
              template: {
                  metadata: { labels: appLabels },
                  spec: {
                      containers: [{
                          name: "nginx",
                          image: "nginx",
                      }],
                  },
              },
          },
      }, { provider });
      const appService = new kubernetes.core.v1.Service("do-app-svc", {
          spec: {
              type: "LoadBalancer",
              selector: app.spec.template.metadata.labels,
              ports: [{ port: 80 }],
          },
      }, { provider });
      
      export const ingressIp = appService.status.loadBalancer.ingress[0].ip;
      

      This code is similar to standard Kubernetes configuration, and the behavior of objects and their properties is equivalent, except that it’s written in TypeScript alongside your other infrastructure declarations.

      Save and close the file after making the changes.

      Just like before, run pulumi up to preview and then deploy the changes:

      After selecting yes to proceed, the CLI will print out detailed status updates, including diagnostics around Pod availability, IP address allocation, and more. This will help you understand why your deployment might be taking time to complete or getting stuck.

      The full output will look something like this:

      Output

      Updating (dev): Type Name Status pulumi:pulumi:Stack do-k8s-dev + ├─ pulumi:providers:kubernetes do-k8s created + ├─ kubernetes:apps:Deployment do-app-dep created + └─ kubernetes:core:Service do-app-svc created Outputs: + ingressIp : "157.230.199.202" Resources: + 3 created 2 unchanged Duration: 2m52s Permalink: https://app.pulumi.com/…/do-k8s/dev/updates/2

      After this completes, notice that the desired number of Pods are running:

      • KUBECONFIG=./kubeconfig.yml kubectl get pods

      Output

      NAME READY STATUS RESTARTS AGE do-app-dep-vyf8k78z-758486ff68-5z8hk 1/1 Running 0 1m do-app-dep-vyf8k78z-758486ff68-8982s 1/1 Running 0 1m do-app-dep-vyf8k78z-758486ff68-94k7b 1/1 Running 0 1m do-app-dep-vyf8k78z-758486ff68-cqm4c 1/1 Running 0 1m do-app-dep-vyf8k78z-758486ff68-lx2d7 1/1 Running 0 1m

      Similar to how the program exports the cluster’s kubeconfig file, this program also exports the Kubernetes service’s resulting load balancer’s IP address. Use this to curl the endpoint and see that it is up and running:

      • curl $(pulumi stack output ingressIp)

      Output

      <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> <p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/> Commercial support is available at <a href="http://nginx.com/">nginx.com</a>.</p> <p><em>Thank you for using nginx.</em></p> </body> </html>

      From here, you can easily edit and redeploy your application infrastructure. For example, try changing the replicas: 5 line to say replicas: 7, and then rerun pulumi up:

      Notice that it just shows what has changed, and that selecting details displays the precise diff:

      Output

      Previewing update (dev): Type Name Plan Info pulumi:pulumi:Stack do-k8s-dev ~ └─ kubernetes:apps:Deployment do-app-dep update [diff: ~spec] Resources: ~ 1 to update 4 unchanged Do you want to perform this update? details pulumi:pulumi:Stack: (same) [urn=urn:pulumi:dev::do-k8s::pulumi:pulumi:Stack::do-k8s-dev] ~ kubernetes:apps/v1:Deployment: (update) [id=default/do-app-dep-vyf8k78z] [urn=urn:pulumi:dev::do-k8s::kubernetes:apps/v1:Deployment::do-app-dep] [provider=urn:pulumi:dev::do-k8s::pulumi:providers:kubernetes::do-k8s::80f36105-337f-451f-a191-5835823df9be] ~ spec: { ~ replicas: 5 => 7 }

      Now you have both a fully functioning Kubernetes cluster and a working application. With your application up and running, you may want to configure a custom domain to use with your application. The next step will guide you through configuring DNS with Pulumi.

      Step 5 — Creating a DNS Domain (Optional)

      Although the Kubernetes cluster and application are up and running, the application’s address is dependent upon the whims of automatic IP address assignment by your cluster. As you adjust and redeploy things, this address might change. In this step, you will see how to assign a custom DNS name to the load balancer IP address so that it’s stable even as you subsequently change your infrastructure.

      Note: To complete this step, ensure you have a domain using DigitalOcean’s DNS nameservers, ns1.digitalocean.com, ns2.digitalocean.com, and ns3.digitalocean.com. Instructions to configure this are available in the Prerequisites section.

      To configure DNS, open the index.ts file and append the following code to the end of the file:

      index.ts

      …
      const domain = new digitalocean.Domain("do-domain", {
          name: "your_domain",
          ipAddress: ingressIp,
      });
      

      This code creates a new DNS entry with an A record that refers to your Kubernetes service’s IP address. Replace your_domain in this snippet with your chosen domain name.

      It is common to want additional sub-domains, like www, to point at the web application. This is easy to accomplish using a DigitalOcean DNS record. To make this example more interesting, also add a CNAME record that points www.your_domain.com to your_domain.com:

      index.ts

      …
      const cnameRecord = new digitalocean.DnsRecord("do-domain-cname", {
          domain: domain.name,
          type: "CNAME",
          name: "www",
          value: "@",
      });
      

      Save and close the file after making these changes.

      Finally, run pulumi up to deploy the DNS changes to point at your existing application and cluster:

      Output

      Updating (dev): Type Name Status pulumi:pulumi:Stack do-k8s-dev + ├─ digitalocean:index:Domain do-domain created + └─ digitalocean:index:DnsRecord do-domain-cname created Resources: + 2 created 5 unchanged Duration: 6s Permalink: https://app.pulumi.com/…/do-k8s/dev/updates/3

      After the DNS changes have propagated, you will be able to access your content at your custom domain:

      You will receive output similar to the following:

      Output

      <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> <p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/> Commercial support is available at <a href="http://nginx.com/">nginx.com</a>.</p> <p><em>Thank you for using nginx.</em></p> </body> </html>

      With that, you have successfully set up a new DigitalOcean Kubernetes cluster, deployed a load balanced Kubernetes application to it, and given that application’s load balancer a stable domain name using DigitalOcean DNS, all in 60 lines of code and a pulumi up command.

      The next step will guide you through removing the resources if you no longer need them.

      Step 6 — Removing the Resources (Optional)

      Before concluding the tutorial, you may want to destroy all of the resources created above. This will ensure you don’t get charged for resources that aren’t being used. If you prefer to keep your application up and running, feel free to skip this step.

      Run the following command to destroy the resources. Be careful using this, as it cannot be undone!

      Just as with the up command, destroy displays a preview and prompt before taking action:

      Output

      Previewing destroy (dev): Type Name Plan - pulumi:pulumi:Stack do-k8s-dev delete - ├─ digitalocean:index:DnsRecord do-domain-cname delete - ├─ digitalocean:index:Domain do-domain delete - ├─ kubernetes:core:Service do-app-svc delete - ├─ kubernetes:apps:Deployment do-app-dep delete - ├─ pulumi:providers:kubernetes do-k8s delete - └─ digitalocean:index:KubernetesCluster do-cluster delete Resources: - 7 to delete Do you want to perform this destroy? yes > no details

      Assuming this is what you want, select yes and watch the deletions occur:

      Output

      Destroying (dev): Type Name Status - pulumi:pulumi:Stack do-k8s-dev deleted - ├─ digitalocean:index:DnsRecord do-domain-cname deleted - ├─ digitalocean:index:Domain do-domain deleted - ├─ kubernetes:core:Service do-app-svc deleted - ├─ kubernetes:apps:Deployment do-app-dep deleted - ├─ pulumi:providers:kubernetes do-k8s deleted - └─ digitalocean:index:KubernetesCluster do-cluster deleted Resources: - 7 deleted Duration: 7s Permalink: https://app.pulumi.com/…/do-k8s/dev/updates/4

      At this point, nothing remains: the DNS entries are gone and the Kubernetes cluster—along with the application running inside of it—are gone. The permalink is still available, so you can still go back and see the full history of updates for this stack. This could help you recover if the destruction was a mistake, since the service keeps full state history for all resources.

      If you’d like to destroy your project in its entirety, remove the stack:

      You will receive output asking you to confirm the deletion by typing in the stack’s name:

      Output

      This will permanently remove the 'dev' stack! Please confirm that this is what you'd like to do by typing ("dev"):

      Unlike the destroy command, which deletes the cloud infrastructure resources, the removal of a stack erases completely the full history of your stack from Pulumi’s purview.

      Conclusion

      In this tutorial, you’ve deployed DigitalOcean infrastructure resources—a Kubernetes cluster and a DNS domain with A and CNAME records—in addition to the Kubernetes application configuration that uses this cluster. You have done so using infrastructure-as-code written in a familiar programming language, TypeScript, that works with existing editors, tools, and libraries, and leverages existing communities and packages. You’ve done it all using a single command line workflow for doing deployments that span your application and infrastructure.

      From here, there are a number of next steps you might take:

      The entire sample from this tutorial is available on GitHub. For extensive details about how to use Pulumi infrastructure-as-code in your own projects today, check out the Pulumi Documentation, Tutorials, or Getting Started guides. Pulumi is open source and free to use.



      Source link