One place for hosting & domains


      How To Install and Secure Grafana on Ubuntu 18.04

      The author selected Dev Color to receive a donation as part of the Write for DOnations program.


      Grafana is an open-source data visualization and monitoring tool that integrates with complex data from sources like Prometheus, InfluxDB, Graphite, and ElasticSearch. Grafana lets you create alerts, notifications, and ad-hoc filters for your data while also making collaboration with your teammates easier through built-in sharing features.

      In this tutorial, you will install Grafana and secure it with an SSL certificate and an Nginx reverse proxy. Once you have set up Grafana, you’ll have the option to configure user authentication through GitHub, allowing you to better organize your team permissions.


      To follow this tutorial, you will need:

      Step 1 — Installing Grafana

      In this first step, you will install Grafana onto your Ubuntu 18.04 server. You can install Grafana either by downloading it directly from its official website or by going through an APT repository. Because an APT repository makes it easier to install and manage Grafana’s updates, you’ll use that method in this tutorial.

      Although Grafana is available in the official Ubuntu 18.04 packages repository, the version of Grafana there may not be the latest, so use Grafana’s official repository.

      Download the Grafana GPG key with wget, then pipe the output to apt-key. This will add the key to your APT installation’s list of trusted keys, which will allow you to download and verify the GPG-signed Grafana package.

      • wget -q -O - | sudo apt-key add -

      In this command, the option -q turns off the status update message for wget, and -O outputs the file that you downloaded to the terminal. These two options ensure that only the contents of the downloaded file are pipelined to apt-key.

      Next, add the Grafana repository to your APT sources:

      • sudo add-apt-repository "deb stable main"

      Refresh your APT cache to update your package lists:

      Next, make sure Grafana will be installed from the Grafana repository:

      The output of the previous command tells you the version of Grafana that you are about to install, and where you will retrieve the package from. Verify that the installation candidate at the top of the list will come from the official Grafana repository at

      Output of apt-cache policy grafana

      grafana: Installed: (none) Candidate: 6.3.3 Version table: 6.3.3 500 500 stable/main amd64 Packages …

      You can now proceed with the installation:

      Once Grafana is installed, use systemctl to start the Grafana server:

      • sudo systemctl start grafana-server

      Next, verify that Grafana is running by checking the service’s status:

      • sudo systemctl status grafana-server

      You will receive output similar to this:

      Output of grafana-server status

      ● grafana-server.service - Grafana instance Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; disabled; vendor preset: enabled) Active: active (running) since Tue 2019-08-13 08:22:30 UTC; 11s ago Docs: Main PID: 13630 (grafana-server) Tasks: 7 (limit: 1152) …

      This output contains information about Grafana’s process, including its status, Main Process Identifier (PID), and more. active (running) shows that the process is running correctly.

      Lastly, enable the service to automatically start Grafana on boot:

      • sudo systemctl enable grafana-server

      You will receive the following output:

      Output of systemctl enable grafana-server

      Synchronizing state of grafana-server.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable grafana-server Created symlink /etc/systemd/system/ → /usr/lib/systemd/system/grafana-server.service.

      This confirms that systemd has created the necessary symbolic links to autostart Grafana.

      Grafana is now installed and ready for use. Next, you wil secure your connection to Grafana with a reverse proxy and SSL certificate.

      Step 2 — Setting Up the Reverse Proxy

      Using an SSL certificate will ensure that your data is secure by encrypting the connection to and from Grafana. But, to make use of this connection, you’ll first need to reconfigure Nginx as a reverse proxy for Grafana.

      Open the Nginx configuration file you created when you set up the Nginx server block with Let’s Encrypt in the Prerequisites. You can use any text editor, but for this tutorial we’ll use nano:

      • sudo nano /etc/nginx/sites-available/your_domain

      Locate the following block:


          location / {
              try_files $uri $uri/ =404;

      Because you already configured Nginx to communicate over SSL and because all web traffic to your server already passes through Nginx, you just need to tell Nginx to forward all requests to Grafana, which runs on port 3000 by default.

      Delete the existing try_files line in this location block and replace it with the following proxy_pass option.


          location / {
              proxy_pass http://localhost:3000;

      This will map the proxy to the appropriate port. Once you’re done, save and close the file by pressing CTRL+X, followed by Y and then ENTER if you’re using nano.

      Now, test the new settings to make sure everything is configured correctly:

      You will receive the following output:


      nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful

      Finally, activate the changes by reloading Nginx:

      • sudo systemctl reload nginx

      You can now access the default Grafana login screen by pointing your web browser to https://your_domain. If you’re unable to reach Grafana, verify that your firewall is set to allow traffic on port 443 and then re-trace the previous instructions.

      With the connection to Grafana encrypted, you can now implement additional security measures, starting with changing Grafana’s default administrative credentials.

      Step 3 — Updating Credentials

      Because every Grafana installation uses the same administrative credentials by default, it is best practice to change your login information as soon as possible. In this step, you’ll update the credentials to improve security.

      Start by navigating to https://your_domain from your web browser. This will bring up the default login screen where you’ll see the Grafana logo, a form asking you to enter an email or username and password, a Log in button, and a Forgot your password? link.

      Grafana Login

      Enter admin into both the User and Password fields and then click on the Log in button.

      On the next screen, you’ll be asked to make your account more secure by changing the default password:

      Change Password

      Enter the password you’d like to start using into the New password and Confirm new password fields.

      From here, you can click Save to save the new information or press Skip to skip this step. If you skip, you will be prompted to change the password next time you login.

      In order to increase the security of your Grafana setup, click Save. You’ll return to the Home Dashboard page:

      Home Dashboard

      You’ve now secured your account by changing the default credentials. Next, you will make changes to your Grafana configuration so that nobody can create a new Grafana account without your permission.

      Step 4 — Disabling Grafana Registrations and Anonymous Access

      Grafana provides options that allow visitors to create user accounts for themselves and preview dashboards without registering. When Grafana isn’t accessible via the internet or when it’s working with publicly available data like service statuses, you may want to allow these features. However, when using Grafana online to work with sensitive data, anonymous access could be a security problem. To fix this problem, make some changes to your Grafana configuration.

      Start by opening Grafana’s main configuration file for editing:

      • sudo nano /etc/grafana/grafana.ini

      Locate the following allow_sign_up directive under the [users] heading:


      # disable user signup / registration
      ;allow_sign_up = true

      Enabling this directive with true adds a Sign Up button to the login screen, allowing users to register themselves and access Grafana.

      Disabling this directive with false removes the Sign Up button and strengthens Grafana’s security and privacy.

      Uncomment this directive by removing the ; at the beginning of the line and then setting the option to false:


      # disable user signup / registration
      allow_sign_up = false

      Next, locate the following enabled directive under the [auth.anonymous] heading:


      # enable anonymous access
      ;enabled = false

      Setting enabled to true gives non-registered users access to your dashboards; setting this option to false limits dashboard access to registered users only.

      Uncomment this directive by removing the ; at the beginning of the line and then setting the option to false.


      enabled = false

      Save the file and exit your text editor.

      To activate the changes, restart Grafana:

      • sudo systemctl restart grafana-server

      Verify that everything is working by checking Grafana’s service status:

      • sudo systemctl status grafana-server

      Like before, the output will report that Grafana is active (running).

      Now, point your web browser to https://your_domain. To return to the Sign Up screen, bring your cursor to your avatar in the lower left of the screen and click on the Sign out option that appears.

      Once you have signed out, verify that there is no Sign Up button and that you can’t sign in without entering login credentials.

      At this point, Grafana is fully configured and ready for use. Next, you can simplify the login process for your organization by authenticating through GitHub.

      (Optional) Step 5 — Setting Up a GitHub OAuth App

      For an alternative approach to signing in, you can configure Grafana to authenticate through GitHub, which provides login access to all members of authorized GitHub organizations. This can be particularly useful when you want to allow multiple developers to collaborate and access metrics without having to create Grafana-specific credentials.

      Start by logging into a GitHub account associated with your organization and then navigate to your GitHub profile page at

      Click on your organization’s name under Organization settings in the navigation menu on the left-hand side of the screen.

      GitHub Organization settings

      On the next screen, you’ll see your Organization profile where you can change settings like your Organization display name, organization Email, and organization URL.

      Because Grafana uses OAuth — an open standard for granting remote third parties access to local resources — to authenticate users through GitHub, you’ll need to create a new OAuth application within GitHub.

      Click the OAuth Apps link under Developer settings on the lower left-hand side of the screen.

      If you don’t already have any OAuth applications associated with your organization on GitHub, you’ll be told there are No Organization Owned Applications. Otherwise, you’ll see a list of the OAuth applications already connected to your account.

      Click the Register an application button to continue.

      On the next screen, fill in the following details about your Grafana installation:

      • Application name - This helps you distinguish your different OAuth applications from one another.
      • Homepage URL - This tells GitHub where to find Grafana. Type https://your_domain into this field, replacing your_domain with your domain.
      • Application Description - This provides a description of your OAuth application’s purpose.
      • Application callback URL - This is the address where users will be sent once successfully authenticated. For Grafana, this field must be set to https://your_domain/login/github.

      Keep in mind that Grafana users logging in through GitHub will see the values you entered in the first three preceding fields, so be sure to enter something meaningful and appropriate.

      When completed, the form will look something like:

      GitHub Register OAuth Application

      Click the green, Register application button.

      You will now be redirected to a page containing the Client ID and Client Secret associated with your new OAuth application. Make note of both values, because you will need to add them to Grafana’s main configuration file to complete the setup.

      Warning: Make sure to keep your Client ID and Client Secret in a secure and non-public location, because they could be used as the basis of an attack.

      With your GitHub OAuth application created, you’re now ready to reconfigure Grafana to use GitHub for authentication.

      (Optional) Step 6 — Configuring Grafana as a GitHub OAuth App

      To complete GitHub authentication for your Grafana setup, you will now make some changes to your Grafana configuration files.

      To begin, open the main Grafana configuration file.

      • sudo nano /etc/grafana/grafana.ini

      Locate the [auth.github] heading, and uncomment this section by removing the ; at the beginning of every line except ;team_ids=, which will not be changed in this tutorial.

      Next, configure Grafana to use GitHub with your OAuth application’s client_id and client_secret values.

      • Set enabled and allow_sign_up to true. This will enable GitHub Authentication and permit members of the allowed organization to create accounts themselves. Note that this setting is different than the allow_sign_up property under [users] that you changed in Step 4.
      • Set client_id and client_secret to the values you got while creating your GitHub OAuth application.
      • Set allowed_organizations to the name of your organization to ensure that only members of your organization can sign up and log into Grafana.

      The complete configuration will look like:


      enabled = true
      allow_sign_up = true
      client_id = your_client_id_from_github
      client_secret = your_client_secret_from_github
      scopes = user:email,read:org
      auth_url =
      token_url =
      api_url =
      ;team_ids =
      allowed_organizations = your_organization_name

      You’ve now told Grafana everything it needs to know about GitHub. To complete the setup, you’ll need to enable redirects behind a reverse proxy. This is done by setting a root_url value under the [server] heading.


      root_url = https://your_domain

      Save your configuration and close the file.

      Then, restart Grafana to activate the changes:

      • sudo systemctl restart grafana-server

      Lastly, verify that the service is up and running.

      • sudo systemctl status grafana-server

      The output will indicate that the service is active (running).

      Now, test your new authentication system by navigating to https://your_domain. If you are already logged into Grafana, hover your mouse over the avatar log in the lower left-hand corner of the screen, and click on Sign out in the secondary menu that appears next to your name.

      On the login page, you’ll see a new section under the original Log in button that includes a Sign in with GitHub button with the GitHub logo.

      Grafana Login page with GitHub

      Click on the Sign in with GitHub button to be redirected to GitHub, where you’ll sign into your GitHub account and confirm your intention to Authorize Grafana.

      Click the green, Authorize your_github_organization button.

      Note: Make sure your GitHub account is a member of your approved organization and your Grafana email address matches your GitHub email address. If you try to authenticate with a GitHub account that isn’t a member of your approved organization, you’ll get a Login Failed message telling you, User not a member of one of the required organizations.

      You will now be logged in with your existing Grafana account. If a Grafana account doesn’t already exist for the user you logged in as, Grafana will create a new user account with Viewer permissions, ensuring that new users can only use existing dashboards.

      To change the default permissions for new users, open the main Grafana configuration file for editing.

      • sudo nano /etc/grafana/grafana.ini

      Locate the auto_assign_org_role directive under the [users] heading, and uncomment the setting by removing the ; at the beginning of the line.

      Set the directive to one of the following values:

      • Viewer — can only use existing dashboards
      • Editor — can change use, modify, and add dashboards
      • Admin — has permission to do everything

      This tutorial will set the auto-assign to Viewer:


      auto_assign_org_role = Viewer

      Once you’ve saved your changes, close the file and restart Grafana:

      • sudo systemctl restart grafana-server

      Check the service’s status:

      • sudo systemctl status grafana-server

      Like before, the status will read active (running).

      At this point, you have fully configured Grafana to allow members of your GitHub organization to register and use your Grafana installation.


      In this tutorial you installed, configured, and secured Grafana, and you also learned how to permit members of your organization to authenticate through GitHub.

      To extend your current Grafana installation, see the list of official and community-built dashboards. To learn more about using Grafana in general, see the official Grafana documentation, or check out our other monitoring tutorials.

      Source link

      How to Set Up a Prometheus, Grafana and Alertmanager Monitoring Stack on DigitalOcean Kubernetes


      Along with tracing and logging, monitoring and alerting are essential components of a Kubernetes observability stack. Setting up monitoring for your DigitalOcean Kubernetes cluster allows you to track your resource usage and analyze and debug application errors.

      A monitoring system usually consists of a time-series database that houses metric data and a visualization layer. In addition, an alerting layer creates and manages alerts, handing them off to integrations and external services as necessary. Finally, one or more components generate or expose the metric data that will be stored, visualized, and processed for alerts by the stack.

      One popular monitoring solution is the open-source Prometheus, Grafana, and Alertmanager stack, deployed alongside kube-state-metrics and node_exporter to expose cluster-level Kubernetes object metrics as well as machine-level metrics like CPU and memory usage.

      Rolling out this monitoring stack on a Kubernetes cluster requires configuring individual components, manifests, Prometheus metrics, and Grafana dashboards, which can take some time. The DigitalOcean Kubernetes Cluster Monitoring Quickstart, released by the DigitalOcean Community Developer Education team, contains fully defined manifests for a Prometheus-Grafana-Alertmanager cluster monitoring stack, as well as a set of preconfigured alerts and Grafana dashboards. It can help you get up and running quickly, and forms a solid foundation from which to build your observability stack.

      In this tutorial, we’ll deploy this preconfigured stack on DigitalOcean Kubernetes, access the Prometheus, Grafana, and Alertmanager interfaces, and describe how to customize it.


      Before you begin, you’ll need a DigitalOcean Kubernetes cluster available to you, and the following tools installed in your local development environment:

      • The kubectl command-line interface installed on your local machine and configured to connect to your cluster. You can read more about installing and configuring kubectl in its official documentation.
      • The git version control system installed on your local machine. To learn how to install git on Ubuntu 18.04, consult How To Install Git on Ubuntu 18.04.
      • The Coreutils base64 tool installed on your local machine. If you’re using a Linux machine, this will most likely already be installed. If you’re using OS X, you can use openssl base64, which comes installed by default.

      Note: The Cluster Monitoring Quickstart has only been tested on DigitalOcean Kubernetes clusters. To use the Quickstart with other Kubernetes clusters, some modification to the manifest files may be necessary.

      Step 1 — Cloning the GitHub Repository and Configuring Environment Variables

      To start, clone the DigitalOcean Kubernetes Cluster Monitoring GitHub repository onto your local machine using git:

      • git clone

      Then, navigate into the repo:

      You should see the following directory structure:


      LICENSE changes.txt manifest

      The manifest directory contains Kubernetes manifests for all of the monitoring stack components, including Service Accounts, Deployments, StatefulSets, ConfigMaps, etc. To learn more about these manifest files and how to configure them, skip ahead to Configuring the Monitoring Stack.

      If you just want to get things up and running, begin by setting the APP_INSTANCE_NAME and NAMESPACE environment variables, which will be used to configure a unique name for the stack's components and configure the Namespace into which the stack will be deployed:

      • export APP_INSTANCE_NAME=sammy-cluster-monitoring
      • export NAMESPACE=default

      In this tutorial, we set APP_INSTANCE_NAME to sammy-cluster-monitoring, which will prepend all of the monitoring stack Kubernetes object names. You should substitute in a unique descriptive prefix for your monitoring stack. We also set the Namespace to default. If you’d like to deploy the monitoring stack to a Namespace other than default, ensure that you first create it in your cluster:

      • kubectl create namespace "$NAMESPACE"

      You should see the following output:


      namespace/sammy created

      In this case, the NAMESPACE environment variable was set to sammy. Throughout the rest of the tutorial we'll assume that NAMESPACE has been set to default.

      Now, use the base64 command to base64-encode a secure Grafana password. Be sure to substitute a password of your choosing for your_grafana_password:

      • export GRAFANA_GENERATED_PASSWORD="$(echo -n 'your_grafana_password' | base64)"

      If you're using macOS, you can substitute the openssl base64 command which comes installed by default.

      At this point, you've grabbed the stack's Kubernetes manifests and configured the required environment variables, so you're now ready to substitute the configured variables into the Kubernetes manifest files and create the stack in your Kubernetes cluster.

      Step 2 — Creating the Monitoring Stack

      The DigitalOcean Kubernetes Monitoring Quickstart repo contains manifests for the following monitoring, scraping, and visualization components:

      • Prometheus is a time series database and monitoring tool that works by polling metrics endpoints and scraping and processing the data exposed by these endpoints. It allows you to query this data using PromQL, a time series data query language. Prometheus will be deployed into the cluster as a StatefulSet with 2 replicas that uses Persistent Volumes with DigitalOcean Block Storage. In addition, a preconfigured set of Prometheus Alerts, Rules, and Jobs will be stored as a ConfigMap. To learn more about these, skip ahead to the Prometheus section of Configuring the Monitoring Stack.
      • Alertmanager, usually deployed alongside Prometheus, forms the alerting layer of the stack, handling alerts generated by Prometheus and deduplicating, grouping, and routing them to integrations like email or PagerDuty. Alertmanager will be installed as a StatefulSet with 2 replicas. To learn more about Alertmanager, consult Alerting from the Prometheus docs.
      • Grafana is a data visualization and analytics tool that allows you to build dashboards and graphs for your metrics data. Grafana will be installed as a StatefulSet with one replica. In addition, a preconfigured set of Dashboards generated by kubernetes-mixin will be stored as a ConfigMap.
      • kube-state-metrics is an add-on agent that listens to the Kubernetes API server and generates metrics about the state of Kubernetes objects like Deployments and Pods. These metrics are served as plaintext on HTTP endpoints and consumed by Prometheus. kube-state-metrics will be installed as an auto-scalable Deployment with one replica.
      • node-exporter, a Prometheus exporter that runs on cluster nodes and provides OS and hardware metrics like CPU and memory usage to Prometheus. These metrics are also served as plaintext on HTTP endpoints and consumed by Prometheus. node-exporter will be installed as a DaemonSet.

      By default, along with scraping metrics generated by node-exporter, kube-state-metrics, and the other components listed above, Prometheus will be configured to scrape metrics from the following components:

      • kube-apiserver, the Kubernetes API server.
      • kubelet, the primary node agent that interacts with kube-apiserver to manage Pods and containers on a node.
      • cAdvisor, a node agent that discovers running containers and collects their CPU, memory, filesystem, and network usage metrics.

      To learn more about configuring these components and Prometheus scraping jobs, skip ahead to Configuring the Monitoring Stack. We'll now substitute the environment variables defined in the previous step into the repo's manifest files, and concatenate the individual manifests into a single master file.

      Begin by using awk and envsubst to fill in the APP_INSTANCE_NAME, NAMESPACE, and GRAFANA_GENERATED_PASSWORD variables in the repo's manifest files. After substituting in the variable values, the files will be combined and saved into a master manifest file called sammy-cluster-monitoring_manifest.yaml.

      • awk 'FNR==1 {print "---"}{print}' manifest/*
      • > "${APP_INSTANCE_NAME}_manifest.yaml"

      You should consider storing this file in version control so that you can track changes to the monitoring stack and roll back to previous versions. If you do this, be sure to scrub the admin-password variable from the file so that you don't check your Grafana password into version control.

      Now that you've generated the master manifest file, use kubectl apply -f to apply the manifest and create the stack in the Namespace you configured:

      • kubectl apply -f "${APP_INSTANCE_NAME}_manifest.yaml" --namespace "${NAMESPACE}"

      You should see output similar to the following:


      serviceaccount/alertmanager created configmap/sammy-cluster-monitoring-alertmanager-config created service/sammy-cluster-monitoring-alertmanager-operated created service/sammy-cluster-monitoring-alertmanager created . . . created configmap/sammy-cluster-monitoring-prometheus-config created service/sammy-cluster-monitoring-prometheus created statefulset.apps/sammy-cluster-monitoring-prometheus created

      You can track the stack’s deployment progress using kubectl get all. Once all of the stack components are RUNNING, you can access the preconfigured Grafana dashboards through the Grafana web interface.

      Step 3 — Accessing Grafana and Exploring Metrics Data

      The Grafana Service manifest exposes Grafana as a ClusterIP Service, which means that it's only accessible via a cluster-internal IP address. To access Grafana outside of your Kubernetes cluster, you can either use kubectl patch to update the Service in-place to a public-facing type like NodePort or LoadBalancer, or kubectl port-forward to forward a local port to a Grafana Pod port. In this tutorial we'll forward ports, so you can skip ahead to Forwarding a Local Port to Access the Grafana Service. The following section on exposing Grafana externally is included for reference purposes.

      Exposing the Grafana Service using a Load Balancer (optional)

      If you'd like to create a DigitalOcean Load Balancer for Grafana with an external public IP, use kubectl patch to update the existing Grafana Service in-place to the LoadBalancer Service type:

      • kubectl patch svc "$APP_INSTANCE_NAME-grafana"
      • --namespace "$NAMESPACE"
      • -p '{"spec": {"type": "LoadBalancer"}}'

      The kubectl patch command allows you to update Kubernetes objects in-place to make changes without having to re-deploy the objects. You can also modify the master manifest file directly, adding a type: LoadBalancer parameter to the Grafana Service spec. To learn more about kubectl patch and Kubernetes Service types, you can consult the Update API Objects in Place Using kubectl patch and Services resources in the official Kubernetes docs.

      After running the above command, you should see the following:


      service/sammy-cluster-monitoring-grafana patched

      It may take several minutes to create the Load Balancer and assign it a public IP. You can track its progress using the following command with the -w flag to watch for changes:

      • kubectl get service "$APP_INSTANCE_NAME-grafana" -w

      Once the DigitalOcean Load Balancer has been created and assigned an external IP address, you can fetch its external IP using the following commands:

      • SERVICE_IP=$(kubectl get svc $APP_INSTANCE_NAME-grafana
      • --namespace $NAMESPACE
      • --output jsonpath='{.status.loadBalancer.ingress[0].ip}')
      • echo "http://${SERVICE_IP}/"

      You can now access the Grafana UI by navigating to http://SERVICE_IP/.

      Forwarding a Local Port to Access the Grafana Service

      If you don't want to expose the Grafana Service externally, you can also forward local port 3000 into the cluster directly to a Grafana Pod using kubectl port-forward.

      • kubectl port-forward --namespace ${NAMESPACE} ${APP_INSTANCE_NAME}-grafana-0 3000

      You should see the following output:


      Forwarding from -> 3000 Forwarding from [::1]:3000 -> 3000

      This will forward local port 3000 to containerPort 3000 of the Grafana Pod sammy-cluster-monitoring-grafana-0. To learn more about forwarding ports into a Kubernetes cluster, consult Use Port Forwarding to Access Applications in a Cluster.

      Visit http://localhost:3000 in your web browser. You should see the following Grafana login page:

      Grafana Login Page

      To log in, use the default username admin (if you haven't modified the admin-user parameter), and the password you configured in Step 1.

      You'll be brought to the following Home Dashboard:

      Grafana Home Page

      In the left-hand navigation bar, select the Dashboards button, then click on Manage:

      Grafana Dashboard Tab

      You'll be brought to the following dashboard management interface, which lists the dashboards configured in the dashboards-configmap.yaml manifest:

      Grafana Dashboard List

      These dashboards are generated by kubernetes-mixin, an open-source project that allows you to create a standardized set of cluster monitoring Grafana dashboards and Prometheus alerts. To learn more, consult the kubernetes-mixin GitHub repo.

      Click in to the Kubernetes / Nodes dashboard, which visualizes CPU, memory, disk, and network usage for a given node:

      Grafana Nodes Dashboard

      Describing how to use these dashboards is outside of this tutorial’s scope, but you can consult the following resources to learn more:

      In the next step, we'll follow a similar process to connect to and explore the Prometheus monitoring system.

      Step 4 — Accessing Prometheus and Alertmanager

      To connect to the Prometheus Pods, we can use kubectl port-forward to forward a local port. If you’re done exploring Grafana, you can close the port-forward tunnel by hitting CTRL-C. Alternatively, you can open a new shell and create a new port-forward connection.

      Begin by listing running Pods in the default namespace:

      • kubectl get pod -n default

      You should see the following Pods:


      sammy-cluster-monitoring-alertmanager-0 1/1 Running 0 17m sammy-cluster-monitoring-alertmanager-1 1/1 Running 0 15m sammy-cluster-monitoring-grafana-0 1/1 Running 0 16m sammy-cluster-monitoring-kube-state-metrics-d68bb884-gmgxt 2/2 Running 0 16m sammy-cluster-monitoring-node-exporter-7hvb7 1/1 Running 0 16m sammy-cluster-monitoring-node-exporter-c2rvj 1/1 Running 0 16m sammy-cluster-monitoring-node-exporter-w8j74 1/1 Running 0 16m sammy-cluster-monitoring-prometheus-0 1/1 Running 0 16m sammy-cluster-monitoring-prometheus-1 1/1 Running 0 16m

      We are going to forward local port 9090 to port 9090 of the sammy-cluster-monitoring-prometheus-0 Pod:

      • kubectl port-forward --namespace ${NAMESPACE} sammy-cluster-monitoring-prometheus-0 9090

      You should see the following output:


      Forwarding from -> 9090 Forwarding from [::1]:9090 -> 9090

      This indicates that local port 9090 is being forwarded successfully to the Prometheus Pod.

      Visit http://localhost:9090 in your web browser. You should see the following Prometheus Graph page:

      Prometheus Graph Page

      From here you can use PromQL, the Prometheus query language, to select and aggregate time series metrics stored in its database. To learn more about PromQL, consult Querying Prometheus from the official Prometheus docs.

      In the Expression field, type kubelet_node_name and hit Execute. You should see a list of time series with the metric kubelet_node_name that reports the Nodes in your Kubernetes cluster. You can see which node generated the metric and which job scraped the metric in the metric labels:

      Prometheus Query Results

      Finally, in the top navigation bar, click on Status and then Targets to see the list of targets Prometheus has been configured to scrape. You should see a list of targets corresponding to the list of monitoring endpoints described at the beginning of Step 2.

      To learn more about Prometheus and how to query your cluster metrics, consult the official Prometheus docs.

      To connect to Alertmanager, which manages Alerts generated by Prometheus, we'll follow a similar process to what we used to connect to Prometheus. . In general, you can explore Alertmanager Alerts by clicking into Alerts in the Prometheus top navigation bar.

      To connect to the Alertmanager Pods, we will once again use kubectl port-forward to forward a local port. If you’re done exploring Prometheus, you can close the port-forward tunnel by hitting CTRL-Cor open a new shell to create a new connection. .

      We are going to forward local port 9093 to port 9093 of the sammy-cluster-monitoring-alertmanager-0 Pod:

      • kubectl port-forward --namespace ${NAMESPACE} sammy-cluster-monitoring-alertmanager-0 9093

      You should see the following output:


      Forwarding from -> 9093 Forwarding from [::1]:9093 -> 9093

      This indicates that local port 9093 is being forwarded successfully to an Alertmanager Pod.

      Visit http://localhost:9093 in your web browser. You should see the following Alertmanager Alerts page:

      Alertmanager Alerts Page

      From here, you can explore firing alerts and optionally silencing them. To learn more about Alertmanager, consult the official Alertmanager documentation.

      In the next step, you'll learn how to optionally configure and scale some of the monitoring stack components.

      Step 6 — Configuring the Monitoring Stack (optional)

      The manifests included in the DigitalOcean Kubernetes Cluster Monitoring Quickstart repository can be modified to use different container images, different numbers of Pod replicas, different ports, and customized configuration files.

      In this step, we'll provide a high-level overview of each manifest’s purpose, and then demonstrate how to scale Prometheus up to 3 replicas by modifying the master manifest file.

      To begin, navigate into the manifests subdirectory in the repo, and list the directory’s contents:


      alertmanager-0serviceaccount.yaml alertmanager-configmap.yaml alertmanager-operated-service.yaml alertmanager-service.yaml . . . node-exporter-ds.yaml prometheus-0serviceaccount.yaml prometheus-configmap.yaml prometheus-service.yaml prometheus-statefulset.yaml

      Here you'll find manifests for the different monitoring stack components. To learn more about specific parameters in the manifests, click into the links and consult the comments included throughout the YAML files:






      • prometheus-0serviceaccount.yaml: The Prometheus Service Account, ClusterRole and ClusterRoleBinding.
      • prometheus-configmap.yaml: A ConfigMap that contains three configuration files:

        • alerts.yaml: Contains a preconfigured set of alerts generated by kubernetes-mixin (which was also used to generate the Grafana dashboards). To learn more about configuring alerting rules, consult Alerting Rules from the Prometheus docs.
        • prometheus.yaml: Prometheus's main configuration file. Prometheus has been preconfigured to scrape all the components listed at the beginning of Step 2. Configuring Prometheus goes beyond the scope of this article, but to learn more, you can consult Configuration from the official Prometheus docs.
        • rules.yaml: A set of Prometheus recording rules that enable Prometheus to compute frequently needed or computationally expensive expressions, and save their results as a new set of time series. These are also generated by kubernetes-mixin, and configuring them goes beyond the scope of this article. To learn more, you can consult Recording Rules from the official Prometheus documentation.
      • prometheus-service.yaml: The Service that exposes the Prometheus StatefulSet.

      • prometheus-statefulset.yaml: The Prometheus StatefulSet, configured with 2 replicas. This parameter can be scaled depending on your needs.

      Example: Scaling Prometheus

      To demonstrate how to modify the monitoring stack, we'll scale the number of Prometheus replicas from 2 to 3.

      Open the sammy-cluster-monitoring_manifest.yaml master manifest file using your editor of choice:

      • nano sammy-cluster-monitoring_manifest.yaml

      Scroll down to the Prometheus StatefulSet section of the manifest:


      . . . apiVersion: apps/v1beta2 kind: StatefulSet metadata: name: sammy-cluster-monitoring-prometheus labels: &Labels k8s-app: prometheus sammy-cluster-monitoring prometheus spec: serviceName: "sammy-cluster-monitoring-prometheus" replicas: 2 podManagementPolicy: "Parallel" updateStrategy: type: "RollingUpdate" selector: matchLabels: *Labels template: metadata: labels: *Labels spec: . . .

      Change the number of replicas from 2 to 3:


      . . . apiVersion: apps/v1beta2 kind: StatefulSet metadata: name: sammy-cluster-monitoring-prometheus labels: &Labels k8s-app: prometheus sammy-cluster-monitoring prometheus spec: serviceName: "sammy-cluster-monitoring-prometheus" replicas: 3 podManagementPolicy: "Parallel" updateStrategy: type: "RollingUpdate" selector: matchLabels: *Labels template: metadata: labels: *Labels spec: . . .

      When you're done, save and close the file.

      Apply the changes using kubectl apply -f:

      • kubectl apply -f sammy-cluster-monitoring_manifest.yaml --namespace default

      You can track progress using kubectl get pods. Using this same technique, you can update many of the Kubernetes parameters and much of the configuration for this observability stack.


      In this tutorial, you installed a Prometheus, Grafana, and Alertmanager monitoring stack into your DigitalOcean Kubernetes cluster with a standard set of dashboards, Prometheus rules, and alerts.

      You may also choose to deploy this monitoring stack using the Helm Kubernetes package manager. To learn more, consult How to Set Up DigitalOcean Kubernetes Cluster Monitoring with Helm and Prometheus. One additional way to get this stack up and running is to use the DigitalOcean Marketplace Kubernetes Monitoring Stack solution, currently in beta.

      The DigitalOcean Kubernetes Cluster Monitoring Quickstart repository is heavily based on and modified from Google Cloud Platform’s click-to-deploy Prometheus solution. A full manifest of modifications and changes from the original repository can be found in the Quickstart repo’s changes.txt file.

      Source link