One place for hosting & domains

      Deployments

      Monitoramento para Deployments Distribuídos e de Microsserviços


      Introdução

      O monitoramento de sistemas e da infraestrutura é uma responsabilidade central de equipes de operações de todos os tamanhos. A indústria desenvolveu coletivamente muitas estratégias e ferramentas para ajudar a monitorar servidores, coletar dados importantes e responder a incidentes e condições em alteração em ambientes variados. No entanto, à medida que as metodologias de software e os projetos de infraestrutura evoluem, o monitoramento deve se adaptar para atender a novos desafios e fornecer insights em um território relativamente desconhecido.

      Até agora, nesta série, discutimos o que são métricas, monitoramento e alertas, e as qualidades de bons sistemas de monitoramento. Conversamos sobre coletar métricas de sua infraestrutura e aplicações e os sinais importantes para monitorar toda a sua infraestrutura. Em nosso último guia, cobrimos como colocar em prática métricas e alertas entendendo os componentes individuais e as qualidades do bom projeto de alertas.

      Neste guia, vamos dar uma olhada em como o monitoramento e a coleta de métricas são alterados para arquiteturas e microsserviços altamente distribuídos. A crescente popularidade da computação em nuvem, dos clusters de big data e das camadas de orquestração de instâncias forçou os profissionais de operações a repensar como projetar o monitoramento em escala e a enfrentar problemas específicos com uma melhor instrumentação. Vamos falar sobre o que diferencia os novos modelos de deployment e quais estratégias podem ser usadas para atender a essas novas demandas.

      Quais Desafios Criam as Arquiteturas Altamente Distribuídas?

      Para modelar e espelhar os sistemas monitorados, a infraestrutura de monitoramento sempre foi um pouco distribuída. No entanto, muitas práticas modernas de desenvolvimento — incluindo projetos em torno de microsserviços, containers e instâncias de computação intercambiáveis e efêmeras — alteraram drasticamente o cenário de monitoramento. Em muitos casos, os principais recursos desses avanços são os fatores que tornam o monitoramento mais difícil. Vamos analisar algumas das maneiras pelas quais elas diferem dos ambientes tradicionais e como isso afeta o monitoramento.

      O Trabalho é Dissociado dos Recursos Subjacentes

      Algumas das mudanças mais fundamentais na forma como muitos sistemas se comportam são devido a uma explosão em novas camadas de abstração em torno das quais o software pode ser projetado. A tecnologia de containers mudou o relacionamento entre o software deployado e o sistema operacional subjacente. Os aplicações com deploy em containers têm relacionamentos diferentes com o mundo externo, com outros programas e com o sistema operacional do host, do que com as aplicações cujo deploy foi feito por meios convencionais. As abstrações de kernel e rede podem levar a diferentes entendimentos do ambiente operacional, dependendo de qual camada você verificar.

      Esse nível de abstração é incrivelmente útil de várias maneiras, criando estratégias de deployment consistentes, facilitando a migração do trabalho entre hosts e permitindo que os desenvolvedores controlem de perto os ambientes de runtime de suas aplicações. No entanto, esses novos recursos surgem às custas do aumento da complexidade e de um relacionamento mais distante com os recursos que suportam cada processo.

      Aumento na Comunicação Baseada em Rede

      Uma semelhança entre paradigmas mais recentes é uma dependência crescente da comunicação de rede interna para coordenar e realizar tarefas. O que antes era o domínio de uma única aplicação, agora pode ser distribuído entre muitos componentes que precisam coordenar e compartilhar informações. Isso tem algumas repercussões em termos de infraestrutura de comunicação e monitoramento.

      Primeiro, como esses modelos são construídos na comunicação entre serviços pequenos e discretos, a saúde da rede se torna mais importante do que nunca. Em arquiteturas tradicionais, mais monolíticas, tarefas de coordenação, compartilhamento de informações e organização de resultados foram amplamente realizadas em aplicações com lógica de programação regular ou através de uma quantidade comparativamente pequena de comunicação externa. Em contraste, o fluxo lógico de aplicações altamente distribuídas usa a rede para sincronizar, verificar a integridade dos pares e passar informações. A saúde e o desempenho da rede impactam diretamente mais funcionalidades do que anteriormente, o que significa que é necessário um monitoramento mais intensivo para garantir a operação correta.

      Embora a rede tenha se tornado mais crítica do que nunca, a capacidade de monitorá-la é cada vez mais desafiadora devido ao número estendido de participantes e linhas de comunicação individuais. Em vez de rastrear interações entre algumas aplicações, a comunicação correta entre dezenas, centenas ou milhares de pontos diferentes torna-se necessária para garantir a mesma funcionalidade. Além das considerações de complexidade, o aumento do volume de tráfego também sobrecarrega os recursos de rede disponíveis, aumentando ainda mais a necessidade de um monitoramento confiável.

      Funcionalidade e Responsabilidade Particionada para um Nível Maior

      Acima, mencionamos de passagem a tendência das arquiteturas modernas de dividir o trabalho e a funcionalidade entre muitos componentes menores e discretos. Esses projetos podem ter um impacto direto no cenário de monitoramento porque tornam a clareza e a compreensão especialmente valiosas, mas cada vez mais evasivas.

      Ferramentas e instrumentação mais robustas são necessárias para garantir um bom funcionamento. No entanto, como a responsabilidade de concluir qualquer tarefa é fragmentada e dividida entre diferentes workers (possivelmente em muitos hosts físicos diferentes), entender onde a responsabilidade reside em questões de desempenho ou erros pode ser difícil. Solicitações e unidades de trabalho que tocam dezenas de componentes, muitos dos quais são selecionados de um pool de possíveis candidatos, podem tornar impraticável a visualização do caminho da solicitação ou a análise da causa raiz usando mecanismos tradicionais.

      Unidades de Vida Curta e Efêmera

      Uma batalha adicional na adaptação do monitoramento convencional é monitorar sensivelmente as unidades de vida curta ou efêmeras. Independentemente de as unidades de interesse serem instâncias de computação em nuvem, instâncias de container ou outras abstrações, esses componentes geralmente violam algumas das suposições feitas pelo software de monitoramento convencional.

      Por exemplo, para distinguir entre um node problemático e uma instância intencionalmente destruída para reduzir a escala, o sistema de monitoramento deve ter um entendimento mais íntimo de sua camada de provisionamento e gerenciamento do que era necessário anteriormente. Para muitos sistemas modernos, esses eventos ocorrem com muito mais frequência, portanto, ajustar manualmente o domínio de monitoramento a cada vez não é prático. O ambiente de deployment muda mais rapidamente com esses projetos, portanto, a camada de monitoramento deve adotar novas estratégias para permanecer valiosa.

      Uma questão que muitos sistemas devem enfrentar é o que fazer com os dados das instâncias destruídas. Embora as work units possam ser aprovisionadas e desprovisionadas rapidamente para acomodar demandas variáveis, é necessário tomar uma decisão sobre o que fazer com os dados relacionados às instâncias antigas. Os dados não perdem necessariamente seu valor imediatamente porque o worker subjacente não está mais disponível. Quando centenas ou milhares de nodes podem entrar e sair todos os dias, pode ser difícil saber como melhorar a construção de uma narrativa sobre a integridade operacional geral de seu sistema a partir dos dados fragmentados de instâncias de vida curta.

      Quais Alterações são Necessárias para Escalar seu Monitoramento?

      Agora que identificamos alguns dos desafios únicos das arquiteturas e microsserviços distribuídos, podemos falar sobre como os sistemas de monitoramento podem funcionar dentro dessas realidades. Algumas das soluções envolvem reavaliar e isolar o que é mais valioso sobre os diferentes tipos de métricas, enquanto outras envolvem novas ferramentas ou novas formas de entender o ambiente em que elas habitam.

      Granularidade e Amostragem

      O aumento no volume total de tráfego causado pelo elevado número de serviços é um dos problemas mais simples de se pensar. Além do aumento nos números de transferência causados por novas arquiteturas, a própria atividade de monitoramento pode começar a atolar a rede e roubar recursos do host. Para lidar melhor com o aumento de volume, você pode expandir sua infraestrutura de monitoramento ou reduzir a resolução dos dados com os quais trabalha. Vale à pena olhar ambas as abordagens, mas vamos nos concentrar na segunda, pois representa uma solução mais extensível e amplamente útil.

      Alterar suas taxas de amostragem de dados pode minimizar a quantidade de dados que seu sistema precisa coletar dos hosts. A amostragem é uma parte normal da coleção de métricas que representa com que frequência você solicita novos valores para uma métrica. Aumentar o intervalo de amostragem reduzirá a quantidade de dados que você precisa manipular, mas também reduzirá a resolução — o nível de detalhes — de seus dados. Embora você deva ter cuidado e compreender sua resolução mínima útil, ajustar as taxas de coleta de dados pode ter um impacto profundo em quantos clientes de monitoramento seu sistema pode atender adequadamente.

      Para diminuir a perda de informações resultante de resoluções mais baixas, uma opção é continuar a coletar dados em hosts na mesma frequência, mas compilá-los em números mais digeríveis para transferência pela rede. Computadores individuais podem agregar e calcular valores médios de métricas e enviar resumos para o sistema de monitoramento. Isso pode ajudar a reduzir o tráfego da rede, mantendo a precisão, já que um grande número de pontos de dados ainda é levado em consideração. Observe que isso ajuda a reduzir a influência da coleta de dados na rede, mas não ajuda, por si só, com a pressão envolvida na coleta desses números no host.

      Tome Decisões com Base em Dados Agregados de Várias Unidades

      Como mencionado acima, um dos principais diferenciais entre sistemas tradicionais e arquiteturas modernas é a quebra de quais componentes participam no processamento de solicitações. Em sistemas distribuídos e microsserviços, é muito mais provável que uma unidade de trabalho ou worker seja dado a um grupo de workers por meio de algum tipo de camada de agendamento ou arbitragem. Isso tem implicações em muitos dos processos automatizados que você pode construir em torno do monitoramento.

      Em ambientes que usam grupos de workers intercambiáveis, as políticas de verificação de integridade e de alerta podem ter relações complexas com a infraestrutura que eles monitoram. As verificações de integridade em workers individuais podem ser úteis para desativar e reciclar unidades defeituosas automaticamente. No entanto, se você tiver a automação em funcionamento, em escala, não importa muito se um único servidor web falhar em um grande pool ou grupo. O sistema irá se auto-corrigir para garantir que apenas as unidades íntegras estejam no pool ativo recebendo solicitações.

      Embora as verificações de integridade do host possam detectar unidades defeituosas, a verificação da integridade do pool em si é mais apropriada para alertas. A capacidade do pool de satisfazer a carga de trabalho atual tem maior importância na experiência do usuário do que os recursos de qualquer worker individual. Os alertas com base no número de membros íntegros, na latência do agregado do pool ou na taxa de erros do pool podem notificar os operadores sobre problemas mais difíceis de serem mitigados automaticamente e mais propensos a causar impacto nos usuários.

      Integração com a Camada de Provisionamento

      Em geral, a camada de monitoramento em sistemas distribuídos precisa ter um entendimento mais completo do ambiente de deploy e dos mecanismos de provisionamento. O gerenciamento automatizado do ciclo de vida se torna extremamente valioso devido ao número de unidades individuais envolvidas nessas arquiteturas. Independentemente de as unidades serem containers puros, containers em uma estrutura de orquestração ou nodes de computação em um ambiente de nuvem, existe uma camada de gerenciamento que expõe informações de integridade e aceita comandos para dimensionar e responder a eventos.

      O número de peças em jogo aumenta a probabilidade estatística de falha. Com todos os outros fatores sendo iguais, isso exigiria mais intervenção humana para responder e mitigar esses problemas. Como o sistema de monitoramento é responsável por identificar falhas e degradação do serviço, se ele puder conectar-se às interfaces de controle da plataforma, isso pode aliviar uma grande classe desses problemas. Uma resposta imediata e automática desencadeada pelo software de monitoramento pode ajudar a manter a integridade operacional do seu sistema.

      Essa relação estreita entre o sistema de monitoramento e a plataforma de deploy não é necessariamente obrigatória ou comum em outras arquiteturas. Mas os sistemas distribuídos automatizados visam ser auto-reguláveis, com a capacidade de dimensionar e ajustar com base em regras pré-configuradas e status observado. O sistema de monitoramento, neste caso, assume um papel central no controle do ambiente e na decisão sobre quando agir.

      Outro motivo pelo qual o sistema de monitoramento deve ter conhecimento da camada de provisionamento é lidar com os efeitos colaterais de instâncias efêmeras. Em ambientes onde há rotatividade frequente nas instâncias de trabalho, o sistema de monitoramento depende de informações de um canal paralelo para entender quando as ações foram intencionais ou não. Por exemplo, sistemas que podem ler eventos de API de um provisionador podem reagir de maneira diferente quando um servidor é destruído intencionalmente por um operador do que quando um servidor repentinamente não responde sem nenhum evento associado. A capacidade de diferenciar esses eventos pode ajudar seu monitoramento a permanecer útil, preciso e confiável, mesmo que a infraestrutura subjacente possa mudar com frequência.

      Rastreio Distribuído

      Um dos aspectos mais desafiadores de cargas de trabalho altamente distribuídas é entender a interação entre os diferentes componentes e isolar a responsabilidade ao tentar a análise da causa raiz. Como uma única solicitação pode afetar dúzias de pequenos programas para gerar uma resposta, pode ser difícil interpretar onde os gargalos ou alterações de desempenho se originam. Para fornecer melhores informações sobre como cada componente contribui para a latência e sobrecarga de processamento, surgiu uma técnica chamada rastreamento distribuído.

      O rastreamento distribuído é uma abordagem dos sistemas de instrumentação que funciona adicionando código a cada componente para iluminar o processamento da solicitação à medida que ela percorre seus serviços. Cada solicitação recebe um identificador exclusivo na borda de sua infraestrutura que é transmitido conforme a tarefa atravessa sua infraestrutura. Cada serviço usa essa ID para relatar erros e os registros de data e hora de quando viu a solicitação pela primeira vez e quando ela foi entregue para a próxima etapa. Ao agregar os relatórios dos componentes usando o ID da solicitação, um caminho detalhado com dados de tempo precisos pode ser rastreado através de sua infraestrutura.

      Esse método pode ser usado para entender quanto tempo é gasto em cada parte de um processo e identificar claramente qualquer aumento sério na latência. Essa instrumentação extra é uma maneira de adaptar a coleta de métricas a um grande número de componentes de processamento. Quando mapeado visualmente com o tempo no eixo x, a exibição resultante mostra o relacionamento entre diferentes estágios, por quanto tempo cada processo foi executado e o relacionamento de dependência entre os eventos que devem ser executados em paralelo. Isso pode ser incrivelmente útil para entender como melhorar seus sistemas e como o tempo está sendo gasto.

      Melhorando a Capacidade de Resposta Operacional para Sistemas Distribuídos

      Discutimos como as arquiteturas distribuídas podem tornar a análise da causa raiz e a clareza operacional difíceis de se obter. Em muitos casos, mudar a forma como os humanos respondem e investigam questões é parte da resposta a essas ambiguidades. Configurar as ferramentas para expor as informações de uma maneira que permita analisar a situação metodicamente pode ajudar a classificar as várias camadas de dados disponíveis. Nesta seção, discutiremos maneiras de se preparar para o sucesso ao solucionar problemas em ambientes grandes e distribuídos.

      Definindo Alertas para os Quatro Sinais de Ouro em Todas as Camadas

      O primeiro passo para garantir que você possa responder a problemas em seus sistemas é saber quando eles estão ocorrendo. Em nosso guia Coletando Métricas de sua Infraestrutura e Aplicações, apresentamos os quatro sinais de ouro – indicadores de monitoramento identificados pela equipe de SRE do Google como os mais vitais para rastrear. Os quatro sinais são:

      • latência
      • tráfego
      • taxa de erro
      • saturação

      Esses ainda são os melhores locais para começar quando estiver instrumentando seus sistemas, mas o número de camadas que devem ser observadas geralmente aumenta para sistemas altamente distribuídos. A infraestrutura subjacente, o plano de orquestração e a camada de trabalho precisam de um monitoramento robusto com alertas detalhados definidos para identificar alterações importantes.

      Obtendo uma Visão Completa

      Depois que seus sistemas identificarem uma anomalia e notificarem sua equipe, esta precisa começar a coletar dados. Antes de continuar a partir desta etapa, eles devem ter uma compreensão de quais componentes foram afetados, quando o incidente começou e qual condição de alerta específica foi acionada.

      A maneira mais útil de começar a entender o escopo de um incidente é começar em um nível alto. Comece a investigar verificando dashboards e visualizações que coletam e generalizam informações de seus sistemas. Isso pode ajudá-lo a identificar rapidamente os fatores correlacionados e a entender o impacto imediato que o usuário enfrenta. Durante esse processo, você deve conseguir sobrepor informações de diferentes componentes e hosts.

      O objetivo deste estágio é começar a criar um inventário mental ou físico de itens para verificar com mais detalhes e começar a priorizar sua investigação. Se você puder identificar uma cadeia de problemas relacionados que percorrem diferentes camadas, a camada mais baixa deve ter precedência: as correções para as camadas fundamentais geralmente resolvem os sintomas em níveis mais altos. A lista de sistemas afetados pode servir como uma lista de verificação informal de locais para validar as correções posteriormente quando a mitigação é implementada.

      Detalhando Problemas Específicos

      Quando você perceber que tem uma visão razoável do incidente, faça uma pesquisa detalhada sobre os componentes e sistemas da sua lista em ordem de prioridade. As métricas detalhadas sobre unidades individuais ajudarão você a rastrear a rota da falha até o recurso responsável mais baixo. Ao examinar painéis de controle e entradas de log mais refinados, consulte a lista de componentes afetados para tentar entender melhor como os efeitos colaterais estão sendo propagados pelo sistema. Com microsserviços, o número de componentes interdependentes significa que os problemas se espalham para outros serviços com mais frequência.

      Este estágio é focado em isolar o serviço, componente ou sistema responsável pelo incidente inicial e identificar qual problema específico está ocorrendo. Isso pode ser um código recém-implantado, uma infraestrutura física com defeito, um erro ou bug na camada de orquestração ou uma alteração na carga de trabalho que o sistema não pôde manipular normalmente. Diagnosticar o que está acontecendo e porquê permite descobrir como mitigar o problema e recuperar a saúde operacional. Entender até que ponto a resolução deste problema pode corrigir problemas relatados em outros sistemas pode ajudá-lo a continuar priorizando as tarefas de mitigação.

      Mitigando e Resolvendo os Problemas

      Depois que os detalhes forem identificados, você poderá resolver ou mitigar o problema. Em muitos casos, pode haver uma maneira óbvia e rápida de restaurar o serviço fornecendo mais recursos, revertendo ou redirecionando o tráfego para uma implementação alternativa. Nestes cenários, a resolução será dividida em três fases:

      • Execução de ações para contornar o problema e restaurar o serviço imediato
      • Resolução do problema subjacente para recuperar a funcionalidade total e a integridade operacional
      • Avaliação completa do motivo da falha e implementação de correções de longo prazo para evitar recorrência

      Em muitos sistemas distribuídos, a redundância e os componentes altamente disponíveis garantirão que o serviço seja restaurado rapidamente, embora seja necessário mais trabalho em segundo plano para restaurar a redundância ou tirar o sistema de um estado degradado. Você deve usar a lista de componentes impactados compilados anteriormente como uma base de medição para determinar se a mitigação inicial resolve problemas de serviço em cascata. À medida que a sofisticação dos sistemas de monitoramento evolui, ele também pode automatizar alguns desses processos de recuperação mais completos enviando comandos para a camada de provisionamento para lançar novas instâncias de unidades com falha ou para eliminar unidades que não se comportam corretamente.

      Dada a automação possível nas duas primeiras fases, o trabalho mais importante para a equipe de operações geralmente é entender as causas-raiz de um evento. O conhecimento obtido a partir desse processo pode ser usado para desenvolver novos gatilhos e políticas para ajudar a prever ocorrências futuras e automatizar ainda mais as reações do sistema. O software de monitoramento geralmente obtém novos recursos em resposta a cada incidente para proteger contra os cenários de falha recém-descobertos. Para sistemas distribuídos, rastreamentos distribuídos, entradas de log, visualizações de séries temporais e eventos como deploys recentes podem ajudá-lo a reconstruir a sequência de eventos e identificar onde o software e os processos humanos podem ser aprimorados.

      Devido à complexidade específica inerente aos grandes sistemas distribuídos, é importante tratar o processo de resolução de qualquer evento significativo como uma oportunidade para aprender e ajustar seus sistemas. O número de componentes separados e os caminhos de comunicação envolvidos forçam uma grande dependência da automação e das ferramentas para ajudar a gerenciar a complexidade. A codificação de novas lições nos mecanismos de resposta e conjuntos de regras desses componentes (bem como nas políticas operacionais que sua equipe segue) é a melhor maneira de seu sistema de monitoramento manter a pegada de gerenciamento de sua equipe sob controle.

      Conclusão

      Neste guia, falamos sobre alguns dos desafios específicos que as arquiteturas distribuídas e os projetos de microsserviço podem introduzir para o software de monitoramento e visibilidade. As maneiras modernas de se construir sistemas quebram algumas suposições dos métodos tradicionais, exigindo abordagens diferentes para lidar com os novos ambientes de configuração. Exploramos os ajustes que você precisará considerar ao passar de sistemas monolíticos para aqueles que dependem cada vez mais de workers efêmeros, baseados em nuvem ou em containers e alto volume de coordenação de rede. Posteriormente, discutimos algumas maneiras pelas quais a arquitetura do sistema pode afetar a maneira como você responde a incidentes e a resolução.



      Source link

      How To Automate Deployments to DigitalOcean Kubernetes with CircleCI


      The author selected the Tech Education Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      Having an automated deployment process is a requirement for a scalable and resilient application, and GitOps, or Git-based DevOps, has rapidly become a popular method of organizing CI/CD with a Git repository as a “single source of truth.” Tools like CircleCI integrate with your GitHub repository, allowing you to test and deploy your code automatically every time you make a change to your repository. When this kind of CI/CD is combined with the flexibility of Kubernetes infrastructure, you can build an application that scales easily with changing demand.

      In this article you will use CircleCI to deploy a sample application to a DigitalOcean Kubernetes (DOKS) cluster. After reading this tutorial, you’ll be able to apply these same techniques to deploy other CI/CD tools that are buildable as Docker images.

      Prerequisites

      To follow this tutorial, you’ll need to have:

      For this tutorial, you will use Kubernetes version 1.13.5 and kubectl version 1.10.7.

      Step 1 — Creating Your DigitalOcean Kubernetes Cluster

      Note: You can skip this section if you already have a running DigitalOcean Kubernetes cluster.

      In this first step, you will create the DigitalOcean Kubernetes (DOKS) cluster from which you will deploy your sample application. The kubectl commands executed from your local machine will change or retrieve information directly from the Kubernetes cluster.

      Go to the Kubernetes page on your DigitalOcean account.

      Click Create a Kubernetes cluster, or click the green Create button at the top right of the page and select Clusters from the dropdown menu.

      [Creating a Kubernetes Cluster on DigitalOcean](assets.digitalocean.com/articles/cart64920/CreateDOKS.gif)

      The next page is where you are going to specify the details of your cluster. On Select a Kubernetes version pick version 1.13.5-do.0. If this one is not available, choose a higher one.

      For Choose a datacenter region, choose the region closest to you. This tutorial will use San Francisco – 2.

      You then have the option to build your Node pool(s). On Kubernetes, a node is a worker machine, which contains the services necessary to run pods. On DigitalOcean, each node is a Droplet. Your node pool will consist of a single Standard node. Select the 2GB/1vCPU configuration and change to 1 Node on the number of nodes.

      You can add extra tags if you want; this can be useful if you plan to use DigitalOcean API or just to better organize your node pools.

      On Choose a name, for this tutorial, use kubernetes-deployment-tutorial. This will make it easier to follow throughout while reading the next sections. Finally, click the green Create Cluster button to create your cluster.

      After cluster creation, there will be a button on the UI to download a configuration file called Download Config File. This is the file you will be using to authenticate the kubectl commands you are going to run against your cluster. Download it to your kubectl machine.

      The default way to use that file is to always pass the --kubeconfig flag and the path to it on all commands you run with kubectl. For example, if you downloaded the config file to Desktop, you would run the kubectl get pods command like this:

      • kubectl --kubeconfig ~/Desktop/kubernetes-deployment-tutorial-kubeconfig.yaml get pods

      This would yield the following output:

      Output

      No resources found.

      This means you accessed your cluster. The No resources found. message is correct, since you don’t have any pods on your cluster.

      If you are not maintaining any other Kubernetes clusters you can copy the kubeconfig file to a folder on your home directory called .kube. Create that directory in case it does not exist:

      Then copy the config file into the newly created .kube directory and rename it config:

      • cp current_kubernetes-deployment-tutorial-kubeconfig.yaml_file_path ~/.kube/config

      The config file should now have the path ~/.kube/config. This is the file that kubectl reads by default when running any command, so there is no need to pass --kubeconfig anymore. Run the following:

      You will receive the following output:

      Output

      No resources found.

      Now access the cluster with the following:

      You will receive the list of nodes on your cluster. The output will be similar to this:

      Output

      NAME STATUS ROLES AGE VERSION kubernetes-deployment-tutorial-1-7pto Ready <none> 1h v1.13.5

      In this tutorial you are going to use the default namespace for all kubectl commands and manifest files, which are files that define the workload and operating parameters of work in Kubernetes. Namespaces are like virtual clusters inside your single physical cluster. You can change to any other namespace you want; just make sure to always pass it using the --namespace flag to kubectl, and/or specifying it on the Kubernetes manifests metadata field. They are a great way to organize the deployments of your team and their running environments; read more about them in the official Kubernetes overview on Namespaces.

      By finishing this step you are now able to run kubectl against your cluster. In the next step, you will create the local Git repository you are going to use to house your sample application.

      Step 2 — Creating the Local Git Repository

      You are now going to structure your sample deployment in a local Git repository. You will also create some Kubernetes manifests that will be global to all deployments you are going to do on your cluster.

      Note: This tutorial has been tested on Ubuntu 18.04, and the individual commands are styled to match this OS. However, most of the commands here can be applied to other Linux distributions with little to no change needed, and commands like kubectl are platform-agnostic.

      First, create a new Git repository locally that you will push to GitHub later on. Create an empty folder called do-sample-app in your home directory and cd into it:

      • mkdir ~/do-sample-app
      • cd ~/do-sample-app

      Now create a new Git repository in this folder with the following command:

      Inside this repository, create an empty folder called kube:

      • mkdir ~/do-sample-app/kube/

      This will be the location where you are going to store the Kubernetes resources manifests related to the sample application that you will deploy to your cluster.

      Now, create another folder called kube-general, but this time outside of the Git repository you just created. Make it inside your home directory:

      This folder is outside of your Git repository because it will be used to store manifests that are not specific to a single deployment on your cluster, but common to multiple ones. This will allow you to reuse these general manifests for different deployments.

      With your folders created and the Git repository of your sample application in place, it's time to arrange the authentication and authorization of your DOKS cluster.

      Step 3 — Creating a Service Account

      It's generally not recommended to use the default admin user to authenticate from other Services into your Kubernetes cluster. If your keys on the external provider got compromised, your whole cluster would become compromised.

      Instead you are going to use a single Service Account with a specific Role, which is all part of the RBAC Kubernetes authorization model.

      This authorization model is based on Roles and Resources. You start by creating a Service Account, which is basically a user on your cluster, then you create a Role, in which you specify what resources it has access to on your cluster. Finally, you create a Role Binding, which is used to make the connection between the Role and the Service Account previously created, granting to the Service Account access to all resources the Role has access to.

      The first Kubernetes resource you are going to create is the Service Account for your CI/CD user, which this tutorial will name cicd.

      Create the file cicd-service-account.yml inside the ~/kube-general folder, and open it with your favorite text editor:

      • nano ~/kube-general/cicd-service-account.yml

      Write the following content on it:

      ~/kube-general/cicd-service-account.yml

      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: cicd
        namespace: default
      

      This is a YAML file; all Kubernetes resources are represented using one. In this case you are saying this resource is from Kubernetes API version v1 (internally kubectl creates resources by calling Kubernetes HTTP APIs), and it is a ServiceAccount.

      The metadata field is used to add more information about this resource. In this case, you are giving this ServiceAccount the name cicd, and creating it on the default namespace.

      You can now create this Service Account on your cluster by running kubectl apply, like the following:

      • kubectl apply -f ~/kube-general/

      You will recieve output similar to the following:

      Output

      serviceaccount/cicd created

      To make sure your Service Account is working, try to log in to your cluster using it. To do that you first need to obtain their respective access token and store it in an environment variable. Every Service Account has an access token which Kubernetes stores as a Secret.

      You can retrieve this secret using the following command:

      • TOKEN=$(kubectl get secret $(kubectl get secret | grep cicd-token | awk '{print $1}') -o jsonpath='{.data.token}' | base64 --decode)

      Some explanation on what this command is doing:

      $(kubectl get secret | grep cicd-token | awk '{print $1}')
      

      This is used to retrieve the name of the secret related to our cicd Service Account. kubectl get secret returns the list of secrets on the default namespace, then you use grep to search for the lines related to your cicd Service Account. Then you return the name, since it is the first thing on the single line returned from the grep.

      kubectl get secret preceding-command -o jsonpath='{.data.token}' | base64 --decode
      

      This will retrieve only the secret for your Service Account token. You then access the token field using jsonpath, and pass the result to base64 --decode. This is necessary because the token is stored as a Base64 string. The token itself is a JSON Web Token.

      You can now try to retrieve your pods with the cicd Service Account. Run the following command, replacing server-from-kubeconfig-file with the server URL that can be found after server: in ~kube/config. This command will give a specific error that you will learn about later in this tutorial:

      • kubectl --insecure-skip-tls-verify --kubeconfig="/dev/null" --server=server-from-kubeconfig-file --token=$TOKEN get pods

      --insecure-skip-tls-verify skips the step of verifying the certificate of the server, since you are just testing and do not need to verify this. --kubeconfig="/dev/null" is to make sure kubectl does not read your config file and credentials but instead uses the token provided.

      The output should be similar to this:

      Output

      Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:default:cicd" cannot list resource "pods" in API group "" in the namespace "default"

      This is an error, but it shows us that the token worked. The error you received is about your Service Account not having the neccessary authorization to list the resource secrets, but you were able to access the server itself. If your token had not worked, the error would have been the following one:

      Output

      error: You must be logged in to the server (Unauthorized)

      Now that the authentication was a success, the next step is to fix the authorization error for the Service Account. You will do this by creating a role with the necessary permissions and binding it to your Service Account.

      Step 4 — Creating the Role and the Role Binding

      Kubernetes has two ways to define roles: using a Role or a ClusterRole resource. The difference between the former and the latter is that the first one applies to a single namespace, while the other is valid for the whole cluster.

      As you are using a single namespace on this tutorial, you will use a Role.

      Create the file ~/kube-general/cicd-role.yml and open it with your favorite text editor:

      • nano ~/kube-general/cicd-role.yml

      The basic idea is to grant access to do everything related to most Kubernetes resources in the default namespace. Your Role would look like this:

      ~/kube-general/cicd-role.yml

      kind: Role
      apiVersion: rbac.authorization.k8s.io/v1
      metadata:
        name: cicd
        namespace: default
      rules:
        - apiGroups: ["", "apps", "batch", "extensions"]
          resources: ["deployments", "services", "replicasets", "pods", "jobs", "cronjobs"]
          verbs: ["*"]
      

      This YAML has some similarities with the one you created previously, but here you are saying this resource is a Role, and it's from the Kubernetes API rbac.authorization.k8s.io/v1. You are naming your role cicd, and creating it on the same namespace you created your ServiceAccount, the default one.

      Then you have the rules field, which is a list of resources this role has access to. In Kubernetes resources are defined based on the API group they belong to, the resource kind itself, and what actions you can do on then, which is represented by a verb. Those verbs are similar to the HTTP ones.

      In our case you are saying that your Role is allowed to do everything, *, on the following resources: deployments, services, replicasets, pods, jobs, and cronjobs. This also applies to those resources belonging to the following API groups: "" (empty string), apps, batch, and extensions. The empty string means the root API group. If you use apiVersion: v1 when creating a resource it means this resource is part of this API group.

      A Role by itself does nothing; you must also create a RoleBinding, which binds a Role to something, in this case, a ServiceAccount.

      Create the file ~/kube-general/cicd-role-binding.yml and open it:

      • nano ~/kube-general/cicd-role-binding.yml

      Add the following lines to the file:

      ~/kube-general/cicd-role-binding.yml

      kind: RoleBinding
      apiVersion: rbac.authorization.k8s.io/v1
      metadata:
        name: cicd
        namespace: default
      subjects:
        - kind: ServiceAccount
          name: cicd
          namespace: default
      roleRef:
        kind: Role
        name: cicd
        apiGroup: rbac.authorization.k8s.io
      

      Your RoleBinding has some specific fields that have not yet been covered in this tutorial. roleRef is the Role you want to bind to something; in this case it is the cicd role you created earlier. subjects is the list of resources you are binding your role to; in this case it's a single ServiceAccount called cicd.

      Note: If you had used a ClusterRole, you would have to create a ClusterRoleBinding instead of a RoleBinding. The file would be almost the same. The only difference would be that it would have no namespace field inside the metadata.

      With those files created you will be able to use kubectl apply again. Create those new resources on your Kubernetes cluster by running the following command:

      • kubectl apply -f ~/kube-general/

      You will receive output similar to the following:

      Output

      rolebinding.rbac.authorization.k8s.io/cicd created role.rbac.authorization.k8s.io/cicd created serviceaccount/cicd created

      Now, try the command you ran previously:

      • kubectl --insecure-skip-tls-verify --kubeconfig="/dev/null" --server=server-from-kubeconfig-file --token=$TOKEN get pods

      Since you have no pods, this will yield the following output:

      Output

      No resources found.

      In this step, you gave the Service Account you are going to use on CircleCI the necessary authorization to do meaningful actions on your cluster like listing, creating, and updating resources. Now it's time to create your sample application.

      Step 5 — Creating Your Sample Application

      Note: All commands and files created from now on will start from the folder ~/do-sample-app you created earlier. This is becase you are now creating files specific to the sample application that you are going to deploy to your cluster.

      The Kubernetes Deployment you are going to create will use the Nginx image as a base, and your application will be a simple static HTML page. This is a great start because it allows you to test if your deployment works by serving a simple HTML directly from Nginx. As you will see later on, you can redirect all traffic coming to a local address:port to your deployment on your cluster to test if it's working.

      Inside the repository you set up earlier, create a new Dockerfile file and open it with your text editor of choice:

      • nano ~/do-sample-app/Dockerfile

      Write the following on it:

      ~/do-sample-app/Dockerfile

      FROM nginx:1.14
      
      COPY index.html /usr/share/nginx/html/index.html
      

      This will tell Docker to build the application container from an nginx image.

      Now create a new index.html file and open it:

      • nano ~/do-sample-app/index.html

      Write the following HTML content:

      ~/do-sample-app/index.html

      <!DOCTYPE html>
      <title>DigitalOcean</title>
      <body>
        Kubernetes Sample Application
      </body>
      

      This HTML will display a simple message that will let you know if your application is working.

      You can test if the image is correct by building and then running it.

      First, build the image with the following command, replacing dockerhub-username with your own Docker Hub username. You must specify your username here so when you push it later on to Docker Hub it will just work:

      • docker build ~/do-sample-app/ -t dockerhub-username/do-kubernetes-sample-app

      Now run the image. Use the following command, which starts your image and forwards any local traffic on port 8080 to the port 80 inside the image, the port Nginx listens to by default:

      • docker run --rm -it -p 8080:80 dockerhub-username/do-kubernetes-sample-app

      The command prompt will stop being interactive while the command is running. Instead you will see the Nginx access logs. If you open localhost:8080 on any browser it should show an HTML page with the content of ~/do-sample-app/index.html. In case you don't have a browser available, you can open a new terminal window and use the following curl command to fetch the HTML from the webpage:

      You will receive the following output:

      Output

      <!DOCTYPE html> <title>DigitalOcean</title> <body> Kubernetes Sample Application </body>

      Stop the container (CTRL + C on the terminal where it's running), and submit this image to your Docker Hub account. To do this, first log in to Docker Hub:

      Fill in the required information about your Docker Hub account, then push the image with the following command (don't forget to replace the dockerhub-username with your own):

      • docker push dockerhub-username/do-kubernetes-sample-app

      You have now pushed your sample application image to your Docker Hub account. In the next step, you will create a Deployment on your DOKS cluster from this image.

      Step 6 — Creating the Kubernetes Deployment and Service

      With your Docker image created and working, you will now create a manifest telling Kubernetes how to create a Deployment from it on your cluster.

      Create the YAML deployment file ~/do-sample-app/kube/do-sample-deployment.yml and open it with your text editor:

      • nano ~/do-sample-app/kube/do-sample-deployment.yml

      Write the following content on the file, making sure to replace dockerhub-username with your Docker Hub username:

      ~/do-sample-app/kube/do-sample-deployment.yml

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: do-kubernetes-sample-app
        namespace: default
        labels:
          app: do-kubernetes-sample-app
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: do-kubernetes-sample-app
        template:
          metadata:
            labels:
              app: do-kubernetes-sample-app
          spec:
            containers:
              - name: do-kubernetes-sample-app
                image: dockerhub-username/do-kubernetes-sample-app:latest
                ports:
                  - containerPort: 80
                    name: http
      

      Kubernetes deployments are from the API group apps, so the apiVersion of your manifest is set to apps/v1. On metadata you added a new field you have not used previously, called metadata.labels. This is useful to organize your deployments. The field spec represents the behavior specification of your deployment. A deployment is responsible for managing one or more pods; in this case it's going to have a single replica by the spec.replicas field. That is, it's going to create and manage a single pod.

      To manage pods, your deployment must know which pods it's responsible for. The spec.selector field is the one that gives it that information. In this case the deployment will be responsible for all pods with tags app=do-kubernetes-sample-app. The spec.template field contains the details of the Pod this deployment will create. Inside the template you also have a spec.template.metadata field. The labels inside this field must match the ones used on spec.selector. spec.template.spec is the specification of the pod itself. In this case it contains a single container, called do-kubernetes-sample-app. The image of that container is the image you built previously and pushed to Docker Hub.

      This YAML file also tells Kubernetes that this container exposes the port 80, and gives this port the name http.

      To access the port exposed by your Deployment, create a Service. Make a file named ~/do-sample-app/kube/do-sample-service.yml and open it with your favorite editor:

      • nano ~/do-sample-app/kube/do-sample-service.yml

      Next, add the following lines to the file:

      ~/do-sample-app/kube/do-sample-service.yml

      apiVersion: v1
      kind: Service
      metadata:
        name: do-kubernetes-sample-app
        namespace: default
        labels:
          app: do-kubernetes-sample-app
      spec:
        type: ClusterIP
        ports:
          - port: 80
            targetPort: http
            name: http
        selector:
          app: do-kubernetes-sample-app
      

      This file gives your Service the same labels used on your deployment. This is not required, but it helps to organize your applications on Kubernetes.

      The service resource also has a spec field. The spec.type field is responsible for the behavior of the service. In this case it's a ClusterIP, which means the service is exposed on a cluster-internal IP, and is only reachable from within your cluster. This is the default spec.type for services. spec.selector is the label selector criteria that should be used when picking the pods to be exposed by this service. Since your pod has the tag app: do-kubernetes-sample-app, you used it here. spec.ports are the ports exposed by the pod's containers that you want to expose from this service. Your pod has a single container which exposes port 80, named http, so you are using it here as targetPort. The service exposes that port on port 80 too, with the same name, but you could have used a different port/name combination than the one from the container.

      With your Service and Deployment manifest files created, you can now create those resources on your Kubernetes cluster using kubectl:

      • kubectl apply -f ~/do-sample-app/kube/

      You will receive the following output:

      Output

      deployment.apps/do-kubernetes-sample-app created service/do-kubernetes-sample-app created

      Test if this is working by forwarding one port on your machine to the port that the service is exposing inside your Kubernetes cluster. You can do that using kubectl port-forward:

      • kubectl port-forward $(kubectl get pod --selector="app=do-kubernetes-sample-app" --output jsonpath='{.items[0].metadata.name}') 8080:80

      The subshell command $(kubectl get pod --selector="app=do-kubernetes-sample-app" --output jsonpath='{.items[0].metadata.name}') retrieves the name of the pod matching the tag you used. Otherwise you could have retrieved it from the list of pods by using kubectl get pods.

      After you run port-forward, the shell will stop being interactive, and will instead output the requests redirected to your cluster:

      Output

      Forwarding from 127.0.0.1:8080 -> 80 Forwarding from [::1]:8080 -> 80

      Opening localhost:8080 on any browser should render the same page you saw when you ran the container locally, but it's now coming from your Kubernetes cluster! As before, you can also use curl in a new terminal window to check if it's working:

      You will receive the following output:

      Output

      <!DOCTYPE html> <title>DigitalOcean</title> <body> Kubernetes Sample Application </body>

      Next, it's time to push all the files you created to your GitHub repository. To do this you must first create a repository on GitHub called digital-ocean-kubernetes-deploy.

      In order to keep this repository simple for demonstration purposes, do not initialize the new repository with a README, license, or .gitignore file when asked on the GitHub UI. You can add these files later on.

      With the repository created, point your local repository to the one on GitHub. To do this, press CTRL + C to stop kubectl port-forward and get the command line back, then run the following commands to add a new remote called origin:

      • cd ~/do-sample-app/
      • git remote add origin https://github.com/your-github-account-username/digital-ocean-kubernetes-deploy.git

      There should be no output from the preceding command.

      Next, commit all the files you created up to now to the GitHub repository. First, add the files:

      Next, commit the files to your repository, with a commit message in quotation marks:

      • git commit -m "initial commit"

      This will yield output similar to the following:

      Output

      [master (root-commit) db321ad] initial commit 4 files changed, 47 insertions(+) create mode 100644 Dockerfile create mode 100644 index.html create mode 100644 kube/do-sample-deployment.yml create mode 100644 kube/do-sample-service.yml

      Finally, push the files to GitHub:

      • git push -u origin master

      You will be prompted for your username and password. Once you have entered this, you will see output like this:

      Output

      Counting objects: 7, done. Delta compression using up to 8 threads. Compressing objects: 100% (7/7), done. Writing objects: 100% (7/7), 907 bytes | 0 bytes/s, done. Total 7 (delta 0), reused 0 (delta 0) To github.com:your-github-account-username/digital-ocean-kubernetes-deploy.git * [new branch] master -> master Branch master set up to track remote branch master from origin.

      If you go to your GitHub repository page you will now see all the files there. With your project up on GitHub, you can now set up CircleCI as your CI/CD tool.

      Step 7 — Configuring CircleCI

      For this tutorial, you will use CircleCI to automate deployments of your application whenever the code is updated, so you will need to log in to CircleCI using your GitHub account and set up your repository.

      First, go to their homepage https://circleci.com, and press Sign Up.

      circleci-home-page

      You are using GitHub, so click the green Sign Up with GitHub button.

      CircleCI will redirect to an authorization page on GitHub. CircleCI needs some permissions on your account to be able to start building your projects. This allows CircleCI to obtain your email, deploy keys and permission to create hooks on your repositories, and add SSH keys to your account. If you need more information on what CircleCI is going to do with your data, check their documentation about GitHub integration.

      circleci-github-authorization

      After authorizing CircleCI you will be redirected to their dashboard.

      circleci-project-dashboard

      Next, set up your GitHub repository in CircleCI. Click on Set Up New Projects from the CircleCI Dashboard, or as a shortcut, open the following link changing the highlighted text with your own GitHub username: https://circleci.com/setup-project/gh/your-github-username/digital-ocean-kubernetes-deploy.

      After that press Start Building. Do not create a config file in your repository just yet, and don't worry if the first build fails.

      circleci-start-building

      Next, specify some environment variables in the CircleCI settings. You can find the settings of the project by clicking on the small button with a cog icon on the top right section of the page then selecting Environment Variables, or you can go directly to the environment variables page by using the following URL (remember to fill in your username): https://circleci.com/gh/your-github-username/digital-ocean-kubernetes-deploy/edit#env-vars. Press Add Variable to create new environment variables.

      First, add two environment variables called DOCKERHUB_USERNAME and DOCKERHUB_PASS which will be needed later on to push the image to Docker Hub. Set the values to your Docker Hub username and password, respectively.

      Then add three more: KUBERNETES_TOKEN, KUBERNETES_SERVER, and KUBERNETES_CLUSTER_CERTIFICATE.

      The value of KUBERNETES_TOKEN will be the value of the local environment variable you used earlier to authenticate on your Kubernetes cluster using your Service Account user. If you have closed the terminal, you can always run the following command to retrieve it again:

      • kubectl get secret $(kubectl get secret | grep cicd-token | awk '{print $1}') -o jsonpath='{.data.token}' | base64 --decode

      KUBERNETES_SERVER will be the string you passed as the --server flag to kubectl when you logged in with your cicd Service Account. You can find this after server: in the ~/.kube/config file, or in the file kubernetes-deployment-tutorial-kubeconfig.yaml downloaded from the DigitalOcean dashboard when you made the initial setup of your Kubernetes cluster.

      KUBERNETES_CLUSTER_CERTIFICATE should also be available on your ~/.kube/config file. It's the certificate-authority-data field on the clusters item related to your cluster. It should be a long string; make sure to copy all of it.

      Those environment variables must be defined here because most of them contain sensitive information, and it is not secure to place them directly on the CircleCI YAML config file.

      With CircleCI listening for changes on your repository, and the environment variables configured, it's time to create the configuration file.

      Make a directory called .circleci inside your sample application repository:

      • mkdir ~/do-sample-app/.circleci/

      Inside this directory, create a file named config.yml and open it with your favorite editor:

      • nano ~/do-sample-app/.circleci/config.yml

      Add the following content to the file, making sure to replace dockerhub-username with your Docker Hub username:

      ~/do-sample-app/.circleci/config.yml

      version: 2.1
      jobs:
        build:
          docker:
            - image: circleci/buildpack-deps:stretch
          environment:
            IMAGE_NAME: dockerhub-username/do-kubernetes-sample-app
          working_directory: ~/app
          steps:
            - checkout
            - setup_remote_docker
            - run:
                name: Build Docker image
                command: |
                  docker build -t $IMAGE_NAME:latest .
            - run:
                name: Push Docker Image
                command: |
                  echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin
                  docker push $IMAGE_NAME:latest
      workflows:
        version: 2
        build-master:
          jobs:
            - build:
                filters:
                  branches:
                    only: master
      

      This sets up a Workflow with a single job, called build, that runs for every commit to the master branch. This job is using the image circleci/buildpack-deps:stretch to run its steps, which is an image from CircleCI based on the official buildpack-deps Docker image, but with some extra tools installed, like Docker binaries themselves.

      The workflow has four steps:

      • checkout retrieves the code from GitHub.
      • setup_remote_docker sets up a remote, isolated environment for each build. This is required before you use any docker command inside a job step. This is necessary because as the steps are running inside a docker image, setup_remote_docker allocates another machine to run the commands there.
      • The first run step builds the image, as you did previously locally. For that you are using the environment variable you declared in environment:, IMAGE_NAME (remember to change the highlighted section with your own information).
      • The last run step pushes the image to Dockerhub, using the environment variables you configured on the project settings to authenticate.

      Commit the new file to your repository and push the changes upstream:

      • cd ~/do-sample-app/
      • git add .circleci/
      • git commit -m "add CircleCI config"
      • git push

      This will trigger a new build on CircleCI. The CircleCI workflow is going to correctly build and push your image to Docker Hub.

      CircleCI build page with success build info

      Now that you have created and tested your CircleCI workflow, you can set your DOKS cluster to retrieve the up-to-date image from Docker Hub and deploy it automatically when changes are made.

      Step 8 — Updating the Deployment on the Kubernetes Cluster

      Now that your application image is being built and sent to Docker Hub every time you push changes to the master branch on GitHub, it's time to update your deployment on your Kubernetes cluster so that it retrieves the new image and uses it as a base for deployment.

      To do that, first fix one issue with your deployment: it's currently depending on an image with the latest tag. This tag does not tell us which version of the image you are using. You cannot easily lock your deployment to that tag because it's overwritten everytime you push a new image to Docker Hub, and by using it like that you lose one of the best things about having containerized applications: Reproducibility.

      You can read more about that on this article about why depending on Docker latest tag is a anti-pattern.

      To correct this, you first must make some changes to your Push Docker Image build step in the ~/do-sample-app/.circleci/config.yml file. Open up the file:

      • nano ~/do-sample-app/.circleci/config.yml

      Then add the highlighted lines to your Push Docker Image step:

      ~/do-sample-app/.circleci/config.yml:16-22

      ...
            - run:
                name: Push Docker Image
                command: |
                  echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin
                  docker tag $IMAGE_NAME:latest $IMAGE_NAME:$CIRCLE_SHA1
                  docker push $IMAGE_NAME:latest
                  docker push $IMAGE_NAME:$CIRCLE_SHA1
      ...
      

      Save and exit the file.

      CircleCI has some special environment variables set by default. One of them is CIRCLE_SHA1, which contains the hash of the commit it's building. The changes you made to ~/do-sample-app/.circleci/config.yml will use this environment variable to tag your image with the commit it was built from, always tagging the most recent build with the latest tag. That way, you always have specific images available, without overwriting them when you push something new to your repository.

      Next, change your deployment manifest file to point to that file. This would be simple if inside ~/do-sample-app/kube/do-sample-deployment.yml you could set your image as dockerhub-username/do-kubernetes-sample-app:$COMMIT_SHA1, but kubectl doesn't do variable substitution inside the manifests when you use kubectl apply. To account for this, you can use envsubst. envsubst is a cli tool, part of the GNU gettext project. It allows you to pass some text to it, and if it finds any variable inside the text that has a matching environment variable, it's replaced by the respective value. The resulting text is then returned as their output.

      To use this, you will create a simple bash script which will be responsible for your deployment. Make a new folder called scripts inside ~/do-sample-app/:

      • mkdir ~/do-sample-app/scripts/

      Inside that folder create a new bash script called ci-deploy.sh and open it with your favorite text editor:

      • nano ~/do-sample-app/scripts/ci-deploy.sh

      Inside it write the following bash script:

      ~/do-sample-app/scripts/ci-deploy.sh

      #! /bin/bash
      # exit script when any command ran here returns with non-zero exit code
      set -e
      
      COMMIT_SHA1=$CIRCLE_SHA1
      
      # We must export it so it's available for envsubst
      export COMMIT_SHA1=$COMMIT_SHA1
      
      # since the only way for envsubst to work on files is using input/output redirection,
      #  it's not possible to do in-place substitution, so we need to save the output to another file
      #  and overwrite the original with that one.
      envsubst <./kube/do-sample-deployment.yml >./kube/do-sample-deployment.yml.out
      mv ./kube/do-sample-deployment.yml.out ./kube/do-sample-deployment.yml
      
      echo "$KUBERNETES_CLUSTER_CERTIFICATE" | base64 --decode > cert.crt
      
      ./kubectl 
        --kubeconfig=/dev/null 
        --server=$KUBERNETES_SERVER 
        --certificate-authority=cert.crt 
        --token=$KUBERNETES_TOKEN 
        apply -f ./kube/
      

      Let's go through this script, using the comments in the file. First, there is the following:

      set -e
      

      This line makes sure any failed command stops the execution of the bash script. That way if one command fails, the next ones are not executed.

      COMMIT_SHA1=$CIRCLE_SHA1
      export COMMIT_SHA1=$COMMIT_SHA1
      

      These lines export the CircleCI $CIRCLE_SHA1 environment variable with a new name. If you had just declared the variable without exporting it using export, it would not be visible for the envsubst command.

      envsubst <./kube/do-sample-deployment.yml >./kube/do-sample-deployment.yml.out
      mv ./kube/do-sample-deployment.yml.out ./kube/do-sample-deployment.yml
      

      envsubst cannot do in-place substitution. That is, it cannot read the content of a file, replace the variables with their respective values, and write the output back to the same file. Therefore, you will redirect the output to another file and then overwrite the original file with the new one.

      echo "$KUBERNETES_CLUSTER_CERTIFICATE" | base64 --decode > cert.crt
      

      The environment variable $KUBERNETES_CLUSTER_CERTIFICATE you created earlier on CircleCI's project settings is in reality a Base64 encoded string. To use it with kubectl you must decode its contents and save it to a file. In this case you are saving it to a file named cert.crt inside the current working directory.

      ./kubectl 
        --kubeconfig=/dev/null 
        --server=$KUBERNETES_SERVER 
        --certificate-authority=cert.crt 
        --token=$KUBERNETES_TOKEN 
        apply -f ./kube/
      

      Finally, you are running kubectl. The command has similar arguments to the one you ran when you were testing your Service Account. You are calling apply -f ./kube/, since on CircleCI the current working directory is the root folder of your project. ./kube/ here is your ~/do-sample-app/kube folder.

      Save the file and make sure it's executable:

      • chmod +x ~/do-sample-app/scripts/ci-deploy.sh

      Now, edit ~/do-sample-app/kube/do-sample-deployment.yml:

      • nano ~/do-sample-app/kube/do-sample-deployment.yml

      Change the tag of the container image value to look like the following one:

      ~/do-sample-app/kube/do-sample-deployment.yml

            # ...
            containers:
              - name: do-kubernetes-sample-app
                image: dockerhub-username/do-kubernetes-sample-app:$COMMIT_SHA1
                ports:
                  - containerPort: 80
                    name: http
      

      Save and close the file. You must now add some new steps to your CI configuration file to update the deployment on Kubernetes.

      Open ~/do-sample-app/.circleci/config.yml on your favorite text editor:

      • nano ~/do-sample-app/.circleci/config.yml

      Write the following new steps, right below the Push Docker Image one you had before:

      ~/do-sample-app/.circleci/config.yml

      ...
            - run:
                name: Install envsubst
                command: |
                  sudo apt-get update && sudo apt-get -y install gettext-base
            - run:
                name: Install kubectl
                command: |
                  curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
                  chmod u+x ./kubectl
            - run:
                name: Deploy Code
                command: ./scripts/ci-deploy.sh
      

      The first two steps are installing some dependencies, first envsubst, and then kubectl. The Deploy Code step is responsible for running our deploy script.

      To make sure the changes are really going to be reflected on your Kubernetes deployment, edit your index.html. Change the HTML to something else, like:

      ~/do-sample-app/index.html

      <!DOCTYPE html>
      <title>DigitalOcean</title>
      <body>
        Automatic Deployment is Working!
      </body>
      

      Once you have saved the above change, commit all the modified files to the repository, and push the changes upstream:

      • cd ~/do-sample-app/
      • git add --all
      • git commit -m "add deploy script and add new steps to circleci config"
      • git push

      You will see the new build running on CircleCI, and successfully deploying the changes to your Kubernetes cluster.

      Wait for the build to finish, then run the same command you ran previously:

      • kubectl port-forward $(kubectl get pod --selector="app=do-kubernetes-sample-app" --output jsonpath='{.items[0].metadata.name}') 8080:80

      Make sure everything is working by opening your browser on the URL localhost:8080 or by making a curl request to it. It should show the updated HTML:

      You will receive the following output:

      Output

      <!DOCTYPE html> <title>DigitalOcean</title> <body> Automatic Deployment is Working! </body>

      Congratulations, you have set up automated deployment with CircleCI!

      Conclusion

      This was a basic tutorial on how to do deployments to DigitalOcean Kubernetes using CircleCI. From here, you can improve your pipeline in many ways. The first thing you can do is create a single build job for multiple deployments, each one deploying to different Kubernetes clusters or different namespaces. This can be extremely useful when you have different Git branches for development/staging/production environments, ensuring that the deployments are always separated.

      You could also build your own image to be used on CircleCI, instead of using buildpack-deps. This image could be based on it, but could already have kubectl and envsubst dependencies installed.

      If you would like to learn more about CI/CD on Kubernetes, check out the tutorials for our CI/CD on Kubernetes Webinar Series, or for more information about apps on Kubernetes, see Modernizing Applications for Kubernetes.



      Source link

      How To Set Up Multi-Node Deployments With Rancher 2.1, Kubernetes, and Docker Machine on Ubuntu 18.04


      The author selected Code Org to receive a donation as part of the Write for DOnations program.

      Introduction

      Rancher is a popular open-source container management platform. Released in early 2018, Rancher 2.X works on Kubernetes and has incorporated new tools such as multi-cluster management and built-in CI pipelines. In addition to the enhanced security, scalability, and straightforward deployment tools already in Kubernetes, Rancher offers a graphical user interface that makes managing containers easier. Through Rancher’s GUI, users can manage secrets, securely handle roles and permissions, scale nodes and pods, and set up load balancers and volumes without needing a command line tool or complex YAML files.

      In this tutorial, you will deploy a multi-node Rancher 2.1 server using Docker Machine on Ubuntu 18.04. By the end, you’ll be able to provision new DigitalOcean Droplets and container pods via the Rancher UI to quickly scale up or down your hosting environment.

      Prerequisites

      Before you start this tutorial, you’ll need a DigitalOcean account, in addition to the following:

      • A DigitalOcean Personal Access Token, which you can create following the instructions in this tutorial. This token will allow Rancher to have API access to your DigitalOcean account.

      • A fully registered domain name with an A record that points to the IP address of the Droplet you create in Step 1. You can learn how to point domains to DigitalOcean Droplets by reading through DigitalOcean’s Domains and DNS documentation. Throughout this tutorial, substitute your domain for example.com.

      Step 1 — Creating a Droplet With Docker Installed

      To start and configure Rancher, you’ll need to create a new Droplet with Docker installed. To accomplish this, you can use DigitalOcean’s Docker image.

      First, log in to your DigitalOcean account and choose Create Droplet. Then, under the Choose an Image section, select the Marketplace tab. Select Docker 18.06.1~ce~3 on 18.04.

      Choose the Docker 18.06 image from the One-click Apps menu

      Next, select a Droplet no smaller than 2GB and choose a datacenter region for your Droplet.

      Finally, add your SSH keys, provide a host name for your Droplet, and press the Create button.

      It will take a few minutes for the server to provision and for Docker to download. Once the Droplet deploys successfully, you’re ready to start Rancher in a new Docker container.

      Step 2 — Starting and Configuring Rancher

      The Droplet you created in Step 1 will run Rancher in a Docker container. In this step, you will start the Rancher container and ensure it has a Let’s Encrypt SSL certificate so that you can securely access the Rancher admin panel. Let’s Encrypt is an automated, open-source certificate authority that allows developers to provision ninety-day SSL certificates for free.

      Log in to your new Droplet:

      To make sure Docker is running, enter:

      Check that the listed Docker version is what you expect. You can start Rancher with a Let's Encrypt certificate already installed by running the following command:

      • docker run -d --restart=unless-stopped -p 80:80 -p 443:443 -v /host/rancher:/var/lib/rancher rancher/rancher --acme-domain example.com

      The --acme-domain option installs an SSL certificate from Let's Encrypt to ensure your Rancher admin is served over HTTPS. This script also instructs the Droplet to fetch the rancher/rancher Docker image and start a Rancher instance in a container that will restart automatically if it ever goes down accidentally. To ease recovery in the event of data loss, the script mounts a volume on the host machine (at /host/rancher) that contains the Rancher data.

      To see all the running containers, enter:

      You'll see output similar to the following (with a unique container ID and name):

      Output

      CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 7b2afed0a599 rancher/rancher "entrypoint.sh" 12 seconds ago Up 11 seconds 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp wizardly_fermat

      If the container is not running, you can execute the docker run command again.

      Before you can access the Rancher admin panel, you'll need to set your admin password and Rancher server URL. The Rancher admin interface will give you access to all of your running nodes, pods, and secrets, so it is important that you use a strong password for it.

      Go to the domain name that points to your new Droplet in your web browser. The first time you visit this address, Rancher will let you set a password:

      Set your Rancher password using the prompt

      When asked for your Rancher server URL, use the domain name pointed at your Droplet.

      You have now completed your Rancher server setup, and you will see the Rancher admin home screen:

      The Rancher admin home screen

      You're ready to continue to the Rancher cluster setup.

      Step 3 — Configuring a Cluster With a Single Node

      To use Rancher, you'll need to create a cluster with at least one node. A cluster is a group of one or more nodes. This guide will give you more information about the Kubernetes Architecture. In this tutorial, nodes correspond to Droplets that Rancher will manage. Pods represent a group of running Docker containers within the Droplet. Each node can run many pods. Using the Rancher UI, you can set up clusters and nodes in an underlying Kubernetes environment.

      By the end of this step, you will have set up a cluster with a single node ready to run your first pod.

      In Rancher, click Add Cluster, and select DigitalOcean as the infrastructure provider.

      Select DigitalOcean from the listed infrastructure providers

      Enter a Cluster Name and scroll down to the Node Pools section. Enter a Name Prefix, leave the Count at 1 for now, and check etcd, Control Plane, and Worker.

      • etcd is Kubernetes' key value storage system for keeping your entire environment's state. In order to maintain high availability, you should run three or five etcd nodes so that if one goes down your environment will still be manageable.
      • Control Plane checks through all of the Kubernetes Objects — such as pods — in your environment and keeps them up to date with the configuration you provide in the Rancher admin interface.
      • Workers run the actual workloads and monitoring agents that ensure your containers stay running and networked. Worker nodes are where your pods will run the software you deploy.

      Create a Node Pool with a single Node

      Before creating the cluster, click Add Node Template to configure the specific options for your new node.

      Enter your DigitalOcean Personal Access Token in the Access Token input box and click Next: Configure Droplet.

      Next, select the same Region and Droplet Size as Step 1. For Image, be sure to select Ubuntu 16.04.5 x64 as there's currently a compatibility issue with Rancher and Ubuntu 18.04. Hit Create to save the template.

      Finally, click Create at the Add Cluster page to kick off the provisioning process. It will take a few minutes for Rancher to complete this step, but you will see a new Droplet in your DigitalOcean Droplets dashboard when it's done.

      In this step, you've created a new cluster and node onto which you will deploy a workload in the next section.

      Step 4 — Deploying a Web Application Workload

      Once the new cluster and node are ready, you can deploy your first pod in a workload. A Kubernetes Pod is the smallest unit of work available to Kubernetes and by extension Rancher. Workloads describe a single group of pods that you deploy together. For example, you may run multiple pods of your webserver in a single workload to ensure that if one pod slows down with a particular request, other instances can handle incoming requests. In this section, you're going to deploy a Nginx Hello World image to a single pod.

      Hover over Global in the header and select Default. This will bring you to the Default project dashboard. You'll focus on deploying a single project in this tutorial, but from this dashboard you can also create multiple projects to achieve isolated container hosting environments.

      To start configuring your first pod, click Deploy.

      Enter a Name, and put nginxdemos/hello in the Docker Image field. Next, map port 80 in the container to port 30000 on the host nodes. This will ensure that the pods you deploy are available on each node at port 30000. You can leave Protocol set to TCP, and the next dropdown as NodePort.

      Note: While this method of running the pod on every node's port is easier to get started, Rancher also includes Ingress to provide load balancing and SSL termination for production use.

      The input form for deploying a Workload

      To launch the pod, scroll to the bottom and click Launch.

      Rancher will take you back to the default project home page, and within a few seconds your pod will be ready. Click the link 30000/tcp just below the name of the workload and Rancher will open a new tab with information about the running container's environment.

      Server address, Server name, and other output from the running NGINX container

      The Server address and port you see on this page are those of the internal Docker network, and not the public IP address you see in your browser. This means that Rancher is working and routing traffic from http://first_node_ip:30000/ to the workload as expected.

      At this point, you've successfully deployed your first workload of one pod to a single Rancher node. Next, you'll see how to scale your Rancher environment.

      Step 5 — Scaling Nodes and Pods

      Rancher gives you two ways to scale your hosting resources: increasing the number of pods in your workload or increasing the number of nodes in your cluster.

      Adding pods to your workload will give your application more running processes. This will allow it to handle more traffic and enable zero-downtime deployments, but each node can handle only a finite number of pods. Once all your nodes have hit their pod limit, you will have to increase the number of nodes if you want to continue scaling up.

      Another consideration is that while increasing pods is typically free, you will have to pay for each node you add to your environment. In this step, you will scale up both nodes and pods, and add another node to your Rancher cluster.

      Note: This part of the tutorial will provision a new DigitalOcean Droplet automatically via the API, so be aware that you will incur extra charges while the second node is running.

      Navigate to the cluster home page of your Rancher installation by selecting Cluster: your-cluster-name from the top navigation bar. Next click Nodes from the top navigation bar.

      Use the top navbar dropdown to select your Cluster

      This page shows that you currently have one running node in the cluster. To add more nodes, click Edit Cluster, and scroll to the Node Pools section at the bottom of the page. Click Add Node Pool, enter a prefix, and check the Worker box. Click Save to update the cluster.

      Add a Node Pool as a Worker only

      Within 2–5 minutes, Rancher will provision a second droplet and indicate the node as Active in the cluster's dashboard. This second node is only a worker, which means it will not run the Rancher etcd or Control Plane containers. This allows the Worker more capacity for running workloads.

      Note: Having an uneven number of etcd nodes will ensure that they can always reach a quorum (or consensus). If you only have one etcd node, you run the risk of your cluster being unreachable if that one node goes down. In a production environment it is a better practice to run three or five etcd nodes.

      When the second node is ready, you will be able to see the workload you deployed in the previous step on this node by navigating to http://second_node_ip:30000/ in your browser.

      Scaling up nodes gives you more Droplets to distribute your workloads on, but you may also want to run more instances of each pod within a workload. To add more pods, return to the Default project page, press the arrow to the left of your hello-world workload, and click + twice to add two more pods.

      Running three Hello World Pods in a Workload

      Rancher will automatically deploy more pods and distribute the running containers to each node depending on where there is availability.

      You can now scale your nodes and pods to suit your application's requirements.

      Conclusion

      You've now set up multi-node deployments using Rancher 2.1 on Ubuntu 18.04, and have scaled up to two running nodes and multiple pods within a workload. You can use this strategy to host and scale any kind of Docker container that you need to run in your application and use Rancher's dashboard and alerts to help you maximize the performance of your workloads and nodes within each cluster.



      Source link