One place for hosting & domains

      Deploy

      How to Deploy Prometheus Operator and Grafana on Linode Kubernetes Engine


      Updated by Linode Written by Ben Bigger

      In this guide, you will deploy the Prometheus Operator to your Linode Kubernetes Engine (LKE) cluster using Helm, either as:

      The Prometheus Operator Monitoring Stack

      When administrating any system, effective monitoring tools can empower users to perform quick and effective issue diagnosis and resolution. This need for monitoring solutions has led to the development of several prominent open source tools designed to solve problems associated with monitoring diverse systems.

      Since its release in 2016, Prometheus has become a leading monitoring tool for containerized environments including Kubernetes. Alertmanager is often used with Prometheus to send and manage alerts with tools such as Slack. Grafana, an open source visualization tool with a robust web interface, is commonly deployed along with Prometheus to provide centralized visualization of system metrics.

      The community-supported Prometheus Operator Helm Chart provides a complete monitoring stack including each of these tools along with Node Exporter and kube-state-metrics, and is designed to provide robust Kubernetes monitoring in its default configuration.

      While there are several options for deploying the Prometheus Operator, using Helm, a Kubernetes “package manager,” to deploy the community-supported the Prometheus Operator enables you to:

      • Control the components of your monitoring stack with a single configuration file.
      • Easily manage and upgrade your deployments.
      • Utilize out-of-the-box Grafana interfaces built for Kubernetes monitoring.

      Before You Begin

      Note

      1. Deploy an LKE Cluster. This guide was written using an example node pool with three 2 GB Linodes. Depending on the workloads you will be deploying on your cluster, you may consider using Linodes with more available resources.

      2. Install Helm 3 to your local environment.

      3. Install kubectl to your local environment and connect to your cluster.

      4. Create the monitoring namespace on your LKE cluster:

        kubectl create namespace monitoring
        
      5. Create a directory named lke-monitor to store all of your Helm values and Kubernetes manifest files and move into the new directory:

        mkdir ~/lke-monitor && cd ~/lke-monitor
        
      6. Add the Google stable Helm charts repository to your Helm repos:

        helm repo add stable https://kubernetes-charts.storage.googleapis.com/
        
      7. Update your Helm repositories:

        helm repo update
        
      8. (Optional) For public access with HTTPS and basic auth configured for your web interfaces of your monitoring tools:

        • Purchase a domain name from a reliable domain registrar and configure your registrar to use Linode’s nameservers with your domain. Using Linode’s DNS Manager, create a new Domain for the one that you have purchased.

        • Ensure that htpasswd is installed to your local environment. For many systems, this tool has already been installed. Debian and Ubuntu users will have to install the apache2-utils package with the following command:

          sudo apt install apache2-utils
          

      Prometheus Operator Minimal Deployment

      In this section, you will complete a minimal deployment of the Prometheus Operator for individual/local access with kubectl Port-Forward. If you require your monitoring interfaces to be publicly accessible over the internet, you can skip to the following section on completing a Prometheus Operator Deployment with HTTPS and Basic Auth.

      Deploy Prometheus Operator

      In this section, you will create a Helm chart values file and use it to deploy Prometheus Operator to your LKE cluster.

      1. Using the text editor of your choice, create a file named values.yaml in the ~/lke-monitor directory and save it with the configurations below. Since the control plane is Linode-managed, as part of this step we will also disable metrics collection for the control plane component:

        Caution

        The below configuration will establish persistent data storage with three separate 10GiB Block Storage Volumes for Prometheus, Alertmanager, and Grafana. Because the Prometheus Operator deploys as StatefulSets, these Volumes and their associated Persistent Volume resources must be deleted manually if you later decide to tear down this Helm release.
        ~/lke-monitor/values.yaml
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        16
        17
        18
        19
        20
        21
        22
        23
        24
        25
        26
        27
        28
        29
        30
        31
        32
        33
        34
        35
        36
        37
        
        # Prometheus Operator Helm Chart values for Linode Kubernetes Engine minimal deployment
        prometheus:
          prometheusSpec:
            storageSpec:
              volumeClaimTemplate:
                spec:
                  storageClassName: linode-block-storage-retain
                  resources:
                    requests:
                      storage: 10Gi
        
        alertmanager:
          alertmanagerSpec:
            storage:
              volumeClaimTemplate:
                spec:
                  storageClassName: linode-block-storage-retain
                  resources:
                    requests:
                      storage: 10Gi
        
        grafana:
          persistence:
            enabled: true
            storageClassName: linode-block-storage-retain
            size: 10Gi
        
        # Disable metrics for Linode-managed Kubernetes control plane elements
        kubeEtcd:
          enabled: false
        
        kubeControllerManager:
          enabled: false
        
        kubeScheduler:
          enabled: false
            
      2. Export an environment variable to store your Grafana admin password:

        Note

        Replace prom-operator in the below command with a secure password and save the password for later reference.

        export GRAFANA_ADMINPASSWORD="prom-operator"
        
      3. Using Helm, deploy a Prometheus Operator release labeled lke-monitor in the monitoring namespace on your LKE cluster with the settings established in your values.yaml file:

        helm install 
        lke-monitor stable/prometheus-operator 
        -f ~/lke-monitor/values.yaml 
        --namespace monitoring 
        --set grafana.adminPassword=$GRAFANA_ADMINPASSWORD 
        --set prometheusOperator.createCustomResource=false
        

        Note

        You can safely ignore messages similar to manifest_sorter.go:192: info: skipping unknown hook: "crd-install" as discussed in this Github issues thread.

        Alternatively, you can add --set prometheusOperator.createCustomResource=false to the above command to prevent the message from appearing.

      4. Verify that the Prometheus Operator has been deployed to your LKE cluster and its components are running and ready by checking the pods in the monitoring namespace:

        kubectl -n monitoring get pods
        

        You should see a similar output to the following:

          
        NAME                                                     READY   STATUS    RESTARTS   AGE
        alertmanager-lke-monitor-prometheus-ope-alertmanager-0   2/2     Running   0          45s
        lke-monitor-grafana-84cbb54f98-7gqtk                     2/2     Running   0          54s
        lke-monitor-kube-state-metrics-68c56d976f-n587d          1/1     Running   0          54s
        lke-monitor-prometheus-node-exporter-6xt8m               1/1     Running   0          53s
        lke-monitor-prometheus-node-exporter-dkc27               1/1     Running   0          53s
        lke-monitor-prometheus-node-exporter-pkc65               1/1     Running   0          53s
        lke-monitor-prometheus-ope-operator-f87bc9f7c-w56sw      2/2     Running   0          54s
        prometheus-lke-monitor-prometheus-ope-prometheus-0       3/3     Running   1          35s
            
        

      Access Monitoring Interfaces with Port-Forward

      1. List the services running in the monitoring namespace and review their respective ports:

        kubectl -n monitoring get svc
        

        You should see an output similar to the following:

          
        NAME                                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                     AGE
        alertmanager-operated                     ClusterIP   None            <none>        9093/TCP,9094/TCP,9094/UDP  115s
        lke-monitor-grafana                       ClusterIP   10.128.140.155  <none>        80/TCP                      2m3s
        lke-monitor-kube-state-metrics            ClusterIP   10.128.165.34   <none>        8080/TCP                    2m3s
        lke-monitor-prometheus-node-exporter      ClusterIP   10.128.192.213  <none>        9100/TCP                    2m3s
        lke-monitor-prometheus-ope-alertmanager   ClusterIP   10.128.153.6    <none>        9093/TCP                    2m3s
        lke-monitor-prometheus-ope-operator       ClusterIP   10.128.198.160  <none>        8080/TCP,443/TCP            2m3s
        lke-monitor-prometheus-ope-prometheus     ClusterIP   10.128.121.47   <none>        9090/TCP                    2m3s
        prometheus-operated                       ClusterIP   None            <none>        9090/TCP                    105s
            
        

        From the above output, the resource services you will access have the corresponding ports:

        Resource Service Name Port
        Prometheus lke‑monitor‑prometheus‑ope‑prometheus 9090
        Alertmanager lke‑monitor‑prometheus‑ope‑alertmanager 9093
        Grafana lke‑monitor‑grafana 80
      2. Use kubectl port-forward to open a connection to a service, then access the service’s interface by entering the corresponding address in your web browser:

        Note

        Press control+C on your keyboard to terminate a port-forward process after entering any of the following commands.

        • To provide access to the Prometheus interface at the address 127.0.0.1:9090 in your web browser, enter:

          kubectl -n monitoring 
          port-forward 
          svc/lke-monitor-prometheus-ope-prometheus 
          9090
          
        • To provide access to the Alertmanager interface at the address 127.0.0.1:9093 in your web browser, enter:

          kubectl -n monitoring 
          port-forward 
          svc/lke-monitor-prometheus-ope-alertmanager  
          9093
          
        • To provide access to the Grafana interface at the address 127.0.0.1:8081 in your web browser, enter:

          kubectl -n monitoring 
          port-forward 
          svc/lke-monitor-grafana  
          8081:80
          

          Log in with the username admin and the password you exported as $GRAFANA_ADMINPASSWORD. The Grafana dashboards are accessible at Dashboards > Manage from the left navigation bar.

      Prometheus Operator Deployment with HTTPS and Basic Auth

      Note

      Before you start on this section, ensure that you have completed all of the steps in Before you Begin.

      This section will show you how to install and configure the necessary components for secure, path-based, public access to the Prometheus, Alertmanager, and Grafana interfaces using the domain you have configured for use with Linode.

      An Ingress is used to provide external routes, via HTTP or HTTPS, to your cluster’s services. An Ingress Controller, like the NGINX Ingress Controller, fulfills the requirements presented by the Ingress using a load balancer.

      To enable HTTPS on your monitoring interfaces, you will create a Transport Layer Security (TLS) certificate from the Let’s Encrypt certificate authority (CA) using the ACME protocol. This will be facilitated by cert-manager, the native Kubernetes certificate management controller.

      While the Grafana interface is natively password-protected, the Prometheus and Alertmanager interfaces must be secured by other means. This guide covers basic authentication configurations to secure the Prometheus and Alertmanager interfaces.

      If you are completing this section of the guide after completing a Prometheus Operator Minimal Deployment, you can use Helm to upgrade your release and maintain the persistent data storage for your monitoring stack.

      Install the NGINX Ingress Controller

      In this section, you will install the NGINX Ingress Controller using Helm, which will create a NodeBalancer to handle your cluster’s traffic.

      1. Install the Google stable NGINX Ingress Controller Helm chart:

        helm install nginx-ingress stable/nginx-ingress
        
      2. Access your NodeBalancer’s assigned external IP address.

        kubectl -n default get svc -o wide nginx-ingress-controller
        

        The command will return a similar output to the following:

          
        NAME                       TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                      AGE   SELECTOR
        nginx-ingress-controller   LoadBalancer   10.128.41.200   192.0.2.0      80:30889/TCP,443:32300/TCP   59s   app.kubernetes.io/component=controller,app=nginx-ingress,release=nginx-ingress
            
        
      3. Copy the IP address of the EXTERNAL IP field and navigate to Linode’s DNS Manager and create an A record using this external IP address and a hostname value corresponding to the subdomain you plan to use with your domain.

      Now that your NGINX Ingress Controller has been deployed and your domain’s A record has been updated, you are ready to enable HTTPS on your monitoring interfaces.

      Install cert-manager

      Note

      Before performing the commands in this section, ensure that your DNS has had time to propagate across the internet. You can query the status of your DNS by using the following command, substituting example.com for your domain (including a subdomain if you have configured one).

      dig +short example.com
      

      If successful, the output should return the IP address of your NodeBalancer.

      1. Install cert-manager’s CRDs.

        kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.15.2/cert-manager.crds.yaml
        
      2. Add the Helm repository which contains the cert-manager Helm chart.

        helm repo add jetstack https://charts.jetstack.io
        
      3. Update your Helm repositories.

        helm repo update
        
      4. Install the cert-manager Helm chart. These basic configurations should be sufficient for many use cases, however, additional cert-manager configurable parameters can be found in cert-manager’s official documentation.

        helm install 
        cert-manager jetstack/cert-manager 
        --namespace cert-manager 
        --version v0.15.2
        
      5. Verify that the corresponding cert-manager pods are running and ready.

        kubectl -n cert-manager get pods
        

        You should see a similar output:

          
        NAME                                       READY   STATUS    RESTARTS   AGE
        cert-manager-749df5b4f8-mc9nj              1/1     Running   0          19s
        cert-manager-cainjector-67b7c65dff-4fkrw   1/1     Running   0          19s
        cert-manager-webhook-7d5d8f856b-4nw9z      1/1     Running   0          19s
            
        

      Create a ClusterIssuer Resource

      Now that cert-manager is installed and running on your cluster, you will need to create a ClusterIssuer resource which defines which CA can create signed certificates when a certificate request is received. A ClusterIssuer is not a namespaced resource, so it can be used by more than one namespace.

      1. Using the text editor of your choice, create a file named acme-issuer-prod.yaml with the example configurations, replacing the value of email with your own email address for the ACME challenge:

        ~/lke-monitor/acme-issuer-prod.yaml
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        
        apiVersion: cert-manager.io/v1alpha2
        kind: ClusterIssuer
        metadata:
          name: letsencrypt-prod
        spec:
          acme:
            email: [email protected]
            server: https://acme-v02.api.letsencrypt.org/directory
            privateKeySecretRef:
              name: letsencrypt-secret-prod
            solvers:
            - http01:
                ingress:
                  class: nginx
            
        • This manifest file creates a ClusterIssuer resource that will register an account on an ACME server. The value of spec.acme.server designates Let’s Encrypt’s production ACME server, which should be trusted by most browsers.

          Note

          Let’s Encrypt provides a staging ACME server that can be used to test issuing trusted certificates, while not worrying about hitting Let’s Encrypt’s production rate limits. The staging URL is https://acme-staging-v02.api.letsencrypt.org/directory.
        • The value of privateKeySecretRef.name provides the name of a secret containing the private key for this user’s ACME server account (this is tied to the email address you provide in the manifest file). The ACME server will use this key to identify you.

        • To ensure that you own the domain for which you will create a certificate, the ACME server will issue a challenge to a client. cert-manager provides two options for solving challenges, http01 and DNS01. In this example, the http01 challenge solver will be used and it is configured in the solvers array. cert-manager will spin up challenge solver Pods to solve the issued challenges and use Ingress resources to route the challenge to the appropriate Pod.

      2. Create the ClusterIssuer resource:

        kubectl apply -f ~/lke-monitor/acme-issuer-prod.yaml
        

      Create a Certificate Resource

      After you have a ClusterIssuer resource, you can create a Certificate resource. This will describe your x509 public key certificate and will be used to automatically generate a CertificateRequest which will be sent to your ClusterIssuer.

      1. Using the text editor of your choice, create a file named certificate-prod.yaml with the example configurations:

        Note

        Replace the value of spec.dnsNames with the domain, including subdomains, that you will use to host your monitoring interfaces.

        ~/lke-monitor/certificate-prod.yaml
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        
        apiVersion: cert-manager.io/v1alpha2
        kind: Certificate
        metadata:
          name: prometheus-operator-prod
          namespace: monitoring
        spec:
          secretName: letsencrypt-secret-prod
          duration: 2160h # 90d
          renewBefore: 360h # 15d
          issuerRef:
            name: letsencrypt-prod
            kind: ClusterIssuer
          dnsNames:
          - example.com
            

        Note

        The configurations in this example create a Certificate in the monitoring namespace that is valid for 90 days and renews 15 days before expiry.

      2. Create the Certificate resource:

        kubectl apply -f ~/lke-monitor/certificate-prod.yaml
        
      3. Verify that the Certificate has been successfully issued:

        kubectl -n monitoring get certs
        

        When your certificate is ready, you should see a similar output:

          
        NAME          READY   SECRET                    AGE
        lke-monitor   True    letsencrypt-secret-prod   33s
            
        

      Next, you will create the necessary resources for basic authentication of the Prometheus and Alertmanager interfaces.

      Configure Basic Auth Credentials

      In this section, you will use htpasswd to generate credentials for basic authentication and create a Kubernetes Secret, which will then be applied to your Ingress configuration to secure access to your monitoring interfaces.

      1. Create a basic authentication password file for the user admin:

        htpasswd -c ~/lke-monitor/auth admin
        

        Follow the prompts to create a secure password, then store your password securely for future reference.

      2. Create a Kubernetes Secret for the monitoring namespace using the file you created above:

        kubectl -n monitoring create secret generic basic-auth --from-file=auth
        
      3. Verify that the basic-auth secret has been created on your LKE cluster:

        kubectl -n monitoring get secret basic-auth
        

        You should see a similar output to the following:

          
        NAME         TYPE     DATA   AGE
        basic-auth   Opaque   1      81s
            
        

      All the necessary components are now in place to be able to enable HTTPS on your monitoring interfaces. In the next section, you will complete the steps needed to deploy Prometheus Operator.

      Deploy or Upgrade Prometheus Operator

      In this section, you will create a Helm chart values file and use it to deploy Prometheus Operator to your LKE cluster.

      1. Using the text editor of your choice, create a file named values-https-basic-auth.yaml in the ~/lke-monitor directory and save it with the configurations below. Since the control plane is Linode-managed, as part of this step we will also disable metrics collection for the control plane component:

        Note

        Caution

        The below configuration will establish persistent data storage with three separate 10GiB Block Storage Volumes for Prometheus, Alertmanager, and Grafana. Because the Prometheus Operator deploys as StatefulSets, these Volumes and their associated Persistent Volume resources must be deleted manually if you later decide to tear down this Helm release.
        ~/lke-monitor/values-https-basic-auth.yaml
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        16
        17
        18
        19
        20
        21
        22
        23
        24
        25
        26
        27
        28
        29
        30
        31
        32
        33
        34
        35
        36
        37
        38
        39
        40
        41
        42
        43
        44
        45
        46
        47
        48
        49
        50
        51
        52
        53
        54
        55
        56
        57
        58
        59
        60
        61
        62
        63
        64
        65
        66
        67
        68
        69
        70
        71
        72
        73
        74
        75
        76
        77
        78
        79
        80
        81
        82
        83
        84
        85
        86
        87
        88
        89
        90
        91
        92
        93
        94
        95
        
        # Helm chart values for Prometheus Operator with HTTPS and basic auth
        prometheus:
          ingress:
            enabled: true
            annotations:
              kubernetes.io/ingress.class: nginx
              nginx.ingress.kubernetes.io/rewrite-target: /$2
              cert-manager.io/cluster-issuer: letsencrypt-prod
              nginx.ingress.kubernetes.io/auth-type: basic
              nginx.ingress.kubernetes.io/auth-secret: basic-auth
              nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required'
            hosts:
            - example.com
            paths:
            - /prometheus(/|$)(.*)
            tls:
            - secretName: lke-monitor-tls
              hosts:
              - example.com
          prometheusSpec:
            routePrefix: /
            externalUrl: https://example.com/prometheus
            storageSpec:
              volumeClaimTemplate:
                spec:
                  storageClassName: linode-block-storage-retain
                  resources:
                    requests:
                      storage: 10Gi
        
        alertmanager:
          ingress:
            enabled: true
            annotations:
              kubernetes.io/ingress.class: nginx
              nginx.ingress.kubernetes.io/rewrite-target: /$2
              cert-manager.io/cluster-issuer: letsencrypt-prod
              nginx.ingress.kubernetes.io/auth-type: basic
              nginx.ingress.kubernetes.io/auth-secret: basic-auth
              nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required'
            hosts:
            - example.com
            paths:
            - /alertmanager(/|$)(.*)
            tls:
            - secretName: lke-monitor-tls
              hosts:
              - example.com
          alertmanagerSpec:
            routePrefix: /
            externalUrl: https://example.com/alertmanager
            storage:
              volumeClaimTemplate:
                spec:
                  storageClassName: linode-block-storage-retain
                  resources:
                    requests:
                      storage: 10Gi
        
        grafana:
          persistence:
            enabled: true
            storageClassName: linode-block-storage-retain
            size: 10Gi
          ingress:
            enabled: true
            annotations:
              kubernetes.io/ingress.class: nginx
              nginx.ingress.kubernetes.io/rewrite-target: /$2
              nginx.ingress.kubernetes.io/auth-type: basic
              nginx.ingress.kubernetes.io/auth-secret: basic-auth
              nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required'
            hosts:
            - example.com
            path: /grafana(/|$)(.*)
            tls:
            - secretName: lke-monitor-tls
              hosts:
              - example.com
          grafana.ini:
            server:
              domain: example.com
              root_url: "%(protocol)s://%(domain)s/grafana/"
              enable_gzip: "true"
        
        # Disable control plane metrics
        kubeEtcd:
          enabled: false
        
        kubeControllerManager:
          enabled: false
        
        kubeScheduler:
          enabled: false
            
      2. Export an environment variable to store your Grafana admin password:

        Note

        Replace prom-operator in the below command with a secure password and save the password for later reference.

        export GRAFANA_ADMINPASSWORD="prom-operator"
        
      3. Using Helm, deploy a Prometheus Operator release labeled lke-monitor in the monitoring namespace on your LKE cluster with the settings established in your values-https-basic-auth.yaml file:

        Note

        If you have already deployed a Prometheus Operator release, you can upgrade it by replacing helm install with helm upgrade in the below command.

        helm install 
        lke-monitor stable/prometheus-operator 
        -f ~/lke-monitor/values-https-basic-auth.yaml 
        --namespace monitoring 
        --set grafana.adminPassword=$GRAFANA_ADMINPASSWORD
        

        Once completed, you will see output similar to the following:

          
        NAME: lke-monitor
        LAST DEPLOYED: Mon Jul 27 17:03:46 2020
        NAMESPACE: monitoring
        STATUS: deployed
        REVISION: 1
        NOTES:
        The Prometheus Operator has been installed. Check its status by running:
          kubectl --namespace monitoring get pods -l "release=lke-monitor"
        
        Visit https://github.com/coreos/prometheus-operator for instructions on how
        to create & configure Alertmanager and Prometheus instances using the Operator.
        
        
      4. Verify that the Prometheus Operator has been deployed to your LKE cluster and its components are running and ready by checking the pods in the monitoring namespace:

        kubectl -n monitoring get pods
        

        You should see a similar output to the following, confirming that you are ready to access your monitoring interfaces using your domain:

          
        NAME                                                     READY   STATUS    RESTARTS   AGE
        alertmanager-lke-monitor-prometheus-ope-alertmanager-0   2/2     Running   0          45s
        lke-monitor-grafana-84cbb54f98-7gqtk                     2/2     Running   0          54s
        lke-monitor-kube-state-metrics-68c56d976f-n587d          1/1     Running   0          54s
        lke-monitor-prometheus-node-exporter-6xt8m               1/1     Running   0          53s
        lke-monitor-prometheus-node-exporter-dkc27               1/1     Running   0          53s
        lke-monitor-prometheus-node-exporter-pkc65               1/1     Running   0          53s
        lke-monitor-prometheus-ope-operator-f87bc9f7c-w56sw      2/2     Running   0          54s
        prometheus-lke-monitor-prometheus-ope-prometheus-0       3/3     Running   1          35s
            
        

      Access Monitoring Interfaces from your Domain

      Your monitoring interfaces are now publicly accessible with HTTPS and basic auth from the domain you have configured for use with this guide at the following paths:

      Resource Domain and path
      Prometheus example.com/prometheus
      Alertmanager example.com/alertmanager
      Grafana example.com/grafana

      When accessing an interface for the first time, log in as admin with the password you configured for basic auth credentials.

      When accessing the Grafana interface, you will then log in again as admin with the password you exported as $GRAFANA_ADMINPASSWORD on your local environment. The Grafana dashboards are accessible at Dashboards > Manage from the left navigation bar.

      More Information

      You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

      This guide is published under a CC BY-ND 4.0 license.



      Source link

      How to Deploy Jenkins X in Linode Kubernetes Engine


      Updated by Linode Contributed by Daniele Polencic

      Marquee image for How to Deploy Jenkins X in Linode Kubernetes Engine

      What is Jenkins X?

      Jenkins X is CI/CD tool that automates the management of development environments and the promotion of new versions of applications between environments. This guide illustrates how to install Jenkins X on Linode Kubernetes Engine.

      Before You Begin

      1. You will need a personal GitHub account. If you don’t have one, you can register here.
      2. You need a GitHub account for your bot. This bot automates actions like triaging issues, merging Pull Requests, etc. You can register your bot account here.
      3. You will need an account on DockerHub. If you don’t have one, you can register here.

      Create an LKE Cluster

      Follow the instructions in Deploying and Managing a Cluster with Linode Kubernetes Engine Tutorial to create and connect to an LKE cluster.

      We recommend using three 16GB Linodes to start with.

      Creating a Kubernetes cluster on Linode

      You can verify that the installation is successful with:

      kubectl get nodes
      

      The output should similar to:

        
      NAME                        STATUS   ROLES    AGE   VERSION
      lke7189-9006-5f05145fc9a3   Ready       8h    v1.17.3
      lke7189-9006-5f051460a1e2   Ready       8h    v1.17.3
      lke7189-9006-5f0514617a87   Ready       8h    v1.17.3
      
      

      Install jx

      To install Jenkins X, you need the jx command-line tool. You can find the instruction on how to install jx on the official documentation.

      You can verify that the installation is successful with:

      jx version
      

      The output should be similar to:

        
      Version        2.1.80
      Commit         b2bd447
      Build date     2020-06-23T15:34:17Z
      Go version     1.13.8
      Git tree state clean
      
      

      Bootstrap Jenkins X

      Note

      If you used the export command with the KUBECONFIG variable so that you can use kubectl to connect to your cluster, you will need to perform the following steps in the same terminal terminal window where you executed that command in order for jx to also be able to connect to your cluster.

      1. Create a new folder, navigate to the new folder, and execute the jx boot command to start a new project:

        mkdir jenkins-x
        cd jenkins-x
        jx boot
        

        The command prompts the following message:

          
        Creating boot config with defaults, as not in an existing boot directory with a git repository.
        No Jenkins X pipeline file jenkins-x.yml or no jx boot requirements file jx-requirements.yml found. You are not running this command from inside a Jenkins X Boot git clone
        To continue we will clone https://github.com/jenkins-x/jenkins-x-boot-config.git @ master to jenkins-x-boot-config
        ? Do you want to clone the Jenkins X Boot Git repository? [? for help] (Y/n)
        
        

        Yes is the default answer, so you can press enter and continue to the next step.

      2. In the next step, the CLI clones the Jenkins X configuration repo and validates your configuration.

        The command-line tool expects the cluster name to be in the format <project id>_<zone>_<cluster name> — which is the default if you want to use Google Kubernetes Engine. Since Linode Kubernetes Engine doesn’t follow the same naming pattern, the validation fails:

          
        Cloning https://github.com/jenkins-x/jenkins-x-boot-config.git @ master to jenkins-x-boot-config
        Attempting to resolve version for boot config https://github.com/jenkins-x/jenkins-x-boot-config.git from https://github.com/jenkins-x/jenkins-x-versions.git
        Booting Jenkins X
        
        STEP: validate-git command: /bin/sh -c jx step git validate in dir: /tmp/jenkinsx/jenkins-x-boot-config/env
        
        STEP: verify-preinstall command: /bin/sh -c jx step verify preinstall --provider-values-dir="kubeProviders" in dir: /tmp/jenkinsx/jenkins-x-boot-config
        
        error: : unable to parse lke7389 as <project id>_<zone>_<cluster name>
        error: failed to interpret pipeline file /tmp/jenkinsx/jenkins-x-boot-config/jenkins-x.yml: failed to run '/bin/sh -c jx step verify preinstall --provider-values-dir="kubeProviders"' command in directory '/tmp/jenkinsx/jenkins-x-boot-config', output: ''
        
        

        But don’t worry, this error can be fixed. If you inspect the current folder, you should notice that jx created a jenkins-x-boot-config folder. In that folder, there’s a jx-requirements.yml file with the details of your cluster.

        Open that with your favourite editor and identify the line that starts with: provider: gke. You should change that to provider: kubernetes.

        jx-requirements.yml
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        
        autoUpdate:
          enabled: false
          schedule: ""
        bootConfigURL: https://github.com/jenkins-x/jenkins-x-boot-config.git
        cluster:
          gitKind: github
          gitName: github
          gitServer: https://github.com
          namespace: jx
          provider: gke # <- you should change this line to "kubernetes"!
      3. It’s time to run the jx boot command again, but this time from within the jenkins-x-boot-config folder:

        cd jenkins-x-boot-config
        jx boot
        

        This time, the command stops at a different step, which has you confirm you’d like to run jx on a service that is not GKE or EKS:

          
        Attempting to resolve version for boot config https://github.com/jenkins-x/jenkins-x-boot-config from https://github.com/jenkins-x/jenkins-x-versions.git
        Booting Jenkins X
        
        STEP: validate-git command: /bin/sh -c jx step git validate in dir: /tmp/jenkinsx/jenkins-x-boot-config/env
        
        STEP: verify-preinstall command: /bin/sh -c jx step verify preinstall --provider-values-dir="kubeProviders" in dir: /tmp/jenkinsx/jenkins-x-boot-config
        
        jx boot has only been validated on GKE and EKS, we'd love feedback and contributions for other Kubernetes providers
        ? Continue execution anyway? (Y/n)
        
        

        Yes is the default answer, so press enter and continue executing.

      4. The next question is the cluster name:

          
        ? Cluster name
        
        

        Enter jenkins-x-lke.

      5. Next, make yourself the Git Owner.

          
        ? Git Owner name for environment repositories
        
        

        Type in your GitHub username, and press enter.

      6. In the next step, name the users you wish to associate as an approver. This should include your username and the names of any other members of your team which you wish to grant this ability, and the input should be a comma separated list.

          
        Environment repos will be private, if you want to create public environment repos, please set environmentGitPublic to true in jx-requirements.yml
        ? Comma-separated git provider usernames of approvers for development environment repository
        
        
      7. Pay attention to the next question, which describes that the webhooks will be called over HTTP, not HTTPS:

          
        Locking version stream https://github.com/jenkins-x/jenkins-x-versions.git to release v1.0.529. Jenkins X will use this release rather than master to resolve all versions from now on.
        writing the following to the OWNERS file for the development environment repository:
        approvers:
        - <your_GitHub_username>
        reviewers:
        - <your_GitHub_username>
        WARNING: TLS is not enabled so your webhooks will be called using HTTP. This means your webhook secret will be sent to your cluster in the clear. See https://jenkins-x.io/docs/getting-started/setup/boot/#ingress for more information
        ? Do you wish to continue? [? for help] (y/N)
        
        

        Notice how No is the default answer. However, you should answer Yes and move on to the next question.

      8. In the next steps, jx configures several components in your cluster. It might take a while before you’re requested to interact with the terminal. When you do, the next questions are about the Jenkins X admin:

          
        ? Jenkins X Admin Username (admin)
        ? Jenkins X Admin Password
        
        

        You should choose a name and a password for the admin user.

      9. Next, you enter in the info about your Jenkins X Bot.

        The bot will help you triage issues, Pull Requests and execute chores. In this part, you should enter the details for your bot (the GitHub account that you created in advance):

          
        ? Pipeline bot Git username
        ? Pipeline bot Git email address
        ? Pipeline bot Git token
        
        

        You can create a token for your bot by visiting this URL.

        Note

        Please note that you should create the token with the Bot’s account and not yours.

        Creating an API token on GitHub

      10. You can use the same token for the next question (and just press enter):

          
        ? HMAC token, used to validate incoming webhooks. Press enter to use the generated token
        
        
      11. The next question is crucial, as it concerns how Jenkins X will interact with the docker registry:

          
        ? Do you want to configure non default Docker Registry? (y/N)
        
        

        Linode does not offer a container registry at the moment, so you will use Docker Hub as your container registry. You should answer Yes which is NOT the default option. Answer the following questions accordingly:

          
        ? Do you want to configure non default Docker Registry? Yes
        ? Docker Registry Url <accept-the-default-value>
        ? Docker Registry username <enter-your-dockerhub-username>
        ? Docker Registry password <enter-your-dockerhub-password>
        ? Docker Registry email <enter-your-dockerhub-email>
        
        

      The installation should complete with the following output:

        
      Verifying the git config
      Verifying username <bot-name> at git server github at https://github.com
      Found 1 organisation in git server https://github.com: learnk8s
      Validated pipeline user <bot-name> on git server https://github.com
      Git tokens seem to be setup correctly
      Installation is currently looking: GOOD
      Using namespace 'jx' from context named 'lke7411-ctx' on server 'https://c50d6328-b182-4fe9-9746-7f6e2b2e1b4d.ap-south-1.linodelke.net:443'.
      
      

      Jenkins X is installed!

      If you inspect your GitHub profile, you might have noticed that there are three new private repositories:

      • environment-jenkins-x-lke-dev
      • environment-jenkins-x-lke-staging
      • environment-jenkins-x-lke-prod

      The repositories hold the configuration for each environment.

      Your First Project

      In this part, you will set up a CI/CD pipeline for a Java (Spring Boot) application. The same steps are valid for any other language or framework.

      1. You should fork the Jenkins X Demo repository from LearnK8s.

      2. Next, clone the repository locally and change your current directory, being sure to add your GitHub username to the cloned repository’s address:

        git clone [email protected]:your-github-username/jenkins-x-demo
        cd jenkins-x-demo
        
      3. You can import the project in Jenkins X with the following command:

        jx import
        

        You will see a similar output to this:

          
        PipelineActivity for <your-gh-username>-jenkins-x-demo-master-1
        upserted PipelineResource meta-<your-gh-username>-jenkins-x-dlncr for the git repository https://github.com/<your-gh-username>/jenkins-x-demo.git
        upserted Task meta-<your-gh-username>-jenkins-x-dlncr-meta-pipeline-1
        upserted Pipeline meta-<your-gh-username>-jenkins-x-dlncr-1
        created PipelineRun meta-<your-gh-username>-jenkins-x-dlncr-1
        created PipelineStructure meta-<your-gh-username>-jenkins-x-dlncr-1
        
        Watch pipeline activity via:    jx get activity -f jenkins-x-demo -w
        Browse the pipeline log via:    jx get build logs <your-gh-username>/jenkins-x-demo/master
        You can list the pipelines via: jx get pipelines
        When the pipeline is complete:  jx get applications
        
        For more help on available commands see: https://jenkins-x.io/developing/browsing/
        
        
      4. You can now run the following command to follow the pipeline output:

        jx get build logs <yourgithubuser>/jenkins-x-demo/master
        

        The pipeline will take around three or four minutes to complete, as it has to:

        • Compile the project.
        • Create tags in the repository.
        • Build and push the container image to Docker Hub.
        • Package the Helm chart for the application.
      5. In the last step of the pipeline, Jenkins X raises a Pull Request to the repository that holds the configuration of your staging environment (created by default during installation).

        At the end of the pipeline you will see the following output:

          
        Created Pull Request: https://github.com//environment-jenkins-x-lke-staging/pull/1
        
        

        The Pull Request is an automatic promotion to the staging environment. If you merge the Pull Request, the application is automatically promoted to the staging environment.

        You can merge the Pull Request to see the pipeline applying the changes in the staging environment.

        If the promotion is successful, you can inspect the app with:

        jx get applications
        
          
        APPLICATION      STAGING PODS URL
        jenkins-x-demo   0.0.10       http://jenkins-x-staging.178.79.175.247.nip.io
        
        

      It might take some time for your application to start. However, when it’s started, you can visit the app at the URL provided and see it running.

      More Information

      You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

      This guide is published under a CC BY-ND 4.0 license.



      Source link

      How to deploy Kubeflow on Linode Kubernetes Engine


      Updated by Linode Contributed by Daniele Polencic

      Marquee image for How to deploy Kubeflow on Linode Kubernetes Engine

      What is Kubeflow?

      Kubeflow is an excellent alternative for training and evaluating machine learning models in public and private clouds. Kubeflow is designed to make your machine learning experiments portable and scalable.

      Start by creating Jupyter notebooks in the cloud. Once you’re confident in your model, you can scale your model to run on thousands of machines.

      Kubeflow optimises your model and breaks it down into smaller tasks that can be processed in parallel. Then, it distributes the tasks to several computers and waits until the results are ready.

      Are you ready to train your model at scale?

      Before You Begin

      1. You can run this tutorial locally using minikf. However, you should have at least 12GB of RAM, 2CPUs and 50GB of disk space available.

      2. The kfctl command line interface (CLI) used to deploy Kubeflow currently only works on Linux and Mac. If you’re on Windows, you can use WSL2 or a Docker container to work around this limitation.

      3. Most Kubeflow pipelines require Kubernetes Volumes that can be attached to several nodes at once (ReadWriteMany). Currently, the only mode supported by the Linode Block Storage CSI driver is ReadWriteOnce, meaning that it can only be connected to one Kubernetes node at a time.

      Caution

      This guide’s example instructions create several billable resources on your Linode account. If you do not want to keep using the example cluster created with this guide, be sure to delete it when you have finished. If you remove the resources afterward, you will only be billed for the hour(s) that the resources were present on your account. For more information see our How Linode Billing Works guide. For a full list of plan prices, visit our Pricing page.

      Create an LKE Cluster

      Follow the instructions in Deploying and Managing a Cluster with Linode Kubernetes Engine Tutorial to create and connect to an LKE cluster.

      The official Kubeflow documentation recommends provisioning a cluster with at least 4 CPU cores, 12GB of memory and 50GB of space available. We recommend running three 16GB Linodes — that should give you enough resources to scale your models.

      Creating a Kubernetes cluster on Linode

      You can verify that the installation is successful with:

      kubectl get nodes
      

      The output should be similar to:

        
      NAME                        STATUS   ROLES    AGE   VERSION
      lke7189-9006-5f05145fc9a3   Ready       8h    v1.17.3
      lke7189-9006-5f051460a1e2   Ready       8h    v1.17.3
      lke7189-9006-5f0514617a87   Ready       8h    v1.17.3
      
      

      Install Kubeflow

      To install Kubeflow, you need three parts:

      1. A Kubernetes cluster, which you already provisioned in the previous step.
      2. kfctl — a command-line tool that configures and deploys Kubeflow.
      3. A KfDef file — a recipe of components that should be included in your Kubeflow installation.

      Download and Install kfctl

      You can download and install kfctl from the official repository:

      1. Download the latest release of kfctl v1.0.2 from the Kubeflow releases page.

      2. Unpack the tar ball with:

        tar -xvf kfctl_v1.0.2_.tar.gz
        
      3. Add the location of kfctl binary to the path environment variable. If you don’t add the location of the binary to the path variable, you must use the full path to the kfctl binary each time you run it.

        export PATH=$PATH:<path to where kfctl was unpacked>
        

      Note, at the time of writing this guide, there is no Windows release available. However, you can use WSL2 or a Docker container to work around this limitation.

      Verify that the binary is installed correctly with:

      kfctl version
      

      The KfDef file

      The last piece of the puzzle is the KfDef file. Think of the KfDef file as a list of components that should be installed with Kubeflow. As an example, Kubeflow can be configured to use the Spark operator, Katib (Hyperparameter Tuning), Seldon serving, etc.

      1. You can download the following KfDef that includes a lighter version of Kubeflow as well as Dex — an identity provider useful for managing user access.

        wget https://raw.githubusercontent.com/learnk8s/kubeflow-pipelines-demo/master/kfctl_istio_dex.v1.0.2-pipelines.yaml
        

        If you open the file, you can see the various components that are about to be installed.

        Pay attention to line 116 and 118. Those lines contain the default username and password that you will use to log in into Kubeflow, [email protected] and 12341234. It’s a good idea to change these to non-default values.

      2. Next, generate the configuration with:

        kfctl build -f kfctl_istio_dex.v1.0.2-pipelines.yaml
        

        You should notice that a new folder named kustomize was created. If you inspect the folder, you should find a collection of components and configurations.

      3. Apply the configuration with:

        kfctl apply -f kfctl_istio_dex.v1.0.2-pipelines.yaml
        

        The command reads the KfDef definition and the kustomize folder, and submits all resources to the cluster.

        The process could easily take 15 to 20 minutes, depending on your cluster specs. You can monitor the progress of the installation from another terminal with:

        kubectl get pods --all-namespaces
        

        As soon as kfctl completes the installation, it might take a little longer for all Pods to be in a Running state.

      Accessing Kubeflow

      Once the installation is completed, you can decide how you will use Kubeflow. You have two options:

      1. You can temporarily create a tunnel to your Kubeflow cluster. The cluster is private, and only you can access it.

      2. You can expose the cluster to the internet with a NodeBalancer.

      Option 1: Creating a Tunnel to Kubeflow

      If you prefer creating a tunnel, execute the following command:

      kubectl port-forward --namespace istio-system service/istio-ingressgateway 8080:80
      

      Visit http://localhost:8080 in your browser. Skip to the Logging In section.

      Option 2: Expose Kubeflow with a NodeBalancer

      If you prefer a more permanent solution, you can expose the login page with a NodeBalancer. Execute the following command:

      kubectl patch service --namespace istio-system istio-ingressgateway -p '{"spec": {"type": "LoadBalancer"}}'
      

      The command exposes the Istio Ingress Gateway to external traffic.

      You can obtain the IP address for the load balancer with:

      kubectl get service --namespace istio-system istio-ingressgateway
      

      The output should look like:

        
      NAME                   TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)                                                                                                                                      AGE
      istio-ingressgateway   LoadBalancer   10.128.29.15   139.999.26.160   15020:31944/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:32622/TCP,15030:31175/TCP,15031:31917/TCP,15032:31997/TCP,15443:30575/TCP   8m2s
      
      

      Note the value of the EXTERNAL-IP IP address, and open that address in your browser.

      Logging In

      Dex — Identity provider

      If you haven’t changed the credentials as previously suggested, the default username and password are:

      Once logged in, you should see Kubeflow’s dashboard.

      Kubeflow's dashboard

      You have no successfully installed Kubeflow.

      Kubeflow has two frequently used features: pipelines and Jupyter notebooks. In the next section, you will create a Jupyter notebook.

      Jupyter notebooks

      A Jupyter notebook is a convenient environment to explore and create machine learning model. The notebook runs in Kubeflow, so you can create environments that are as big as your Linodes. For example, if you selected 16GB Linode nodes, you might be able to create notebook servers with up to 14GB of memory and 3 CPUs.

      Creating a notebook is straightforward, as demonstrated in this animation:

      Notebook server in Kubeflow

      Please note, you can’t create notebook servers that have specs higher than a single Linode. The notebook server is not distributed and runs as a single Pod in the Kubernetes cluster. If you wish to run your model at scale, you might want to use Kubeflow’s Pipelines, which are demonstrated in the next section.

      Kubeflow pipelines

      Kubeflow pipelines are similar to CI/CD pipelines. You define sequential (or parallel) tasks and connect them.

      In CI/CD pipelines, the tasks are usually a sequence of build, test and deploy. In Kubeflow, pipelines you have similar stages: train, predict and serve.

      There are a few examples in the “Pipelines” section that you can test. As an example, you can select the “Data passing in Python components” pipeline.

      Pipelines in Kubeflow

      The pipeline has three independent tasks that are meant to demonstrate how to:

      • Generate data (i.e. writing to a file).
      • Consume data (i.e. reading from a file).
      • Transform data (i.e. reading, processing and writing to a file).

      You can explore the Python code using a Jupyter notebook.

      The same code is used to generate a pipeline in Kubeflow.

      Pipelines are written using Python and a Kubeflow domain-specific language (DSL). While you can leverage your existing Python models, there are a small amount of changes necessary to teach Kubeflow how to break your model into smaller parts for distributed processing.

      The best resource to continue learning Kubeflow pipelines is the official documentation.

      More Information

      You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

      This guide is published under a CC BY-ND 4.0 license.



      Source link