One place for hosting & domains

      Ubuntu

      How To Install Elasticsearch, Logstash, and Kibana (Elastic Stack) on Ubuntu 20.04

      Introduction

      The Elastic Stack — formerly known as the ELK Stack — is a collection of open-source software produced by Elastic which allows you to search, analyze, and visualize logs generated from any source in any format, a practice known as centralized logging. Centralized logging can be useful when attempting to identify problems with your servers or applications as it allows you to search through all of your logs in a single place. It’s also useful because it allows you to identify issues that span multiple servers by correlating their logs during a specific time frame.

      The Elastic Stack has four main components:

      • Elasticsearch: a distributed RESTful search engine which stores all of the collected data.
      • Logstash: the data processing component of the Elastic Stack which sends incoming data to Elasticsearch.
      • Kibana: a web interface for searching and visualizing logs.
      • Beats: lightweight, single-purpose data shippers that can send data from hundreds or thousands of machines to either Logstash or Elasticsearch.

      In this tutorial, you will install the Elastic Stack on an Ubuntu 20.04 server. You will learn how to install all of the components of the Elastic Stack — including Filebeat, a Beat used for forwarding and centralizing logs and files — and configure them to gather and visualize system logs. Additionally, because Kibana is normally only available on the localhost, we will use Nginx to proxy it so it will be accessible over a web browser. We will install all of these components on a single server, which we will refer to as our Elastic Stack server.

      Note: When installing the Elastic Stack, you must use the same version across the entire stack. In this tutorial we will install the latest versions of the entire stack which are, at the time of this writing, Elasticsearch 7.7.1, Kibana 7.7.1, Logstash 7.7.1, and Filebeat 7.7.1.

      To complete this tutorial, you will need the following:

      Additionally, because the Elastic Stack is used to access valuable information about your server that you would not want unauthorized users to access, it’s important that you keep your server secure by installing a TLS/SSL certificate. This is optional but strongly encouraged.

      However, because you will ultimately make changes to your Nginx server block over the course of this guide, it would likely make more sense for you to complete the Let’s Encrypt on Ubuntu 20.04 guide at the end of this tutorial’s second step. With that in mind, if you plan to configure Let’s Encrypt on your server, you will need the following in place before doing so:

      • A fully qualified domain name (FQDN). This tutorial will use your_domain throughout. You can purchase a domain name on Namecheap, get one for free on Freenom, or use the domain registrar of your choice.

      • Both of the following DNS records set up for your server. You can follow this introduction to DigitalOcean DNS for details on how to add them.

        • An A record with your_domain pointing to your server’s public IP address.
        • An A record with www.your_domain pointing to your server’s public IP address.

      The Elasticsearch components are not available in Ubuntu’s default package repositories. They can, however, be installed with APT after adding Elastic’s package source list.

      All of the packages are signed with the Elasticsearch signing key in order to protect your system from package spoofing. Packages which have been authenticated using the key will be considered trusted by your package manager. In this step, you will import the Elasticsearch public GPG key and add the Elastic package source list in order to install Elasticsearch.

      To begin, use cURL, the command line tool for transferring data with URLs, to import the Elasticsearch public GPG key into APT. Note that we are using the arguments -fsSL to silence all progress and possible errors (except for a server failure) and to allow cURL to make a request on a new location if redirected. Pipe the output of the cURL command into the apt-key program, which adds the public GPG key to APT.

      1. curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

      Next, add the Elastic source list to the sources.list.d directory, where APT will search for new sources:

      1. echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

      Next, update your package lists so APT will read the new Elastic source:

      1. sudo apt update

      Then install Elasticsearch with this command:

      1. sudo apt install elasticsearch

      Elasticsearch is now installed and ready to be configured. Use your preferred text editor to edit Elasticsearch’s main configuration file, elasticsearch.yml. Here, we’ll use nano:

      1. sudo nano /etc/elasticsearch/elasticsearch.yml

      Note: Elasticsearch’s configuration file is in YAML format, which means that we need to maintain the indentation format. Be sure that you do not add any extra spaces as you edit this file.

      The elasticsearch.yml file provides configuration options for your cluster, node, paths, memory, network, discovery, and gateway. Most of these options are preconfigured in the file but you can change them according to your needs. For the purposes of our demonstration of a single-server configuration, we will only adjust the settings for the network host.

      Elasticsearch listens for traffic from everywhere on port 9200. You will want to restrict outside access to your Elasticsearch instance to prevent outsiders from reading your data or shutting down your Elasticsearch cluster through its [REST API] (https://en.wikipedia.org/wiki/Representational_state_transfer). To restrict access and therefore increase security, find the line that specifies network.host, uncomment it, and replace its value with localhost like this:

      /etc/elasticsearch/elasticsearch.yml

      . . .
      # ---------------------------------- Network -----------------------------------
      #
      # Set the bind address to a specific IP (IPv4 or IPv6):
      #
      network.host: localhost
      . . .
      

      We have specified localhost so that Elasticsearch listens on all interfaces and bound IPs. If you want it to listen only on a specific interface, you can specify its IP in place of localhost. Save and close elasticsearch.yml. If you’re using nano, you can do so by pressing CTRL+X, followed by Y and then ENTER .

      These are the minimum settings you can start with in order to use Elasticsearch. Now you can start Elasticsearch for the first time.

      Start the Elasticsearch service with systemctl. Give Elasticsearch a few moments to start up. Otherwise, you may get errors about not being able to connect.

      1. sudo systemctl start elasticsearch

      Next, run the following command to enable Elasticsearch to start up every time your server boots:

      1. sudo systemctl enable elasticsearch

      You can test whether your Elasticsearch service is running by sending an HTTP request:

      1. curl -X GET "localhost:9200"

      You will see a response showing some basic information about your local node, similar to this:

      Output

      { "name" : "Elasticsearch", "cluster_name" : "elasticsearch", "cluster_uuid" : "qqhFHPigQ9e2lk-a7AvLNQ", "version" : { "number" : "7.7.1", "build_flavor" : "default", "build_type" : "deb", "build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f", "build_date" : "2020-03-26T06:34:37.794943Z", "build_snapshot" : false, "lucene_version" : "8.5.1", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_version" : "6.0.0-beta1" }, "tagline" : "You Know, for Search" }

      Now that Elasticsearch is up and running, let’s install Kibana, the next component of the Elastic Stack.

      According to the official documentation, you should install Kibana only after installing Elasticsearch. Installing in this order ensures that the components each product depends on are correctly in place.

      Because you’ve already added the Elastic package source in the previous step, you can just install the remaining components of the Elastic Stack using apt:

      1. sudo apt install kibana

      Then enable and start the Kibana service:

      1. sudo systemctl enable kibana
      2. sudo systemctl start kibana

      Because Kibana is configured to only listen on localhost, we must set up a reverse proxy to allow external access to it. We will use Nginx for this purpose, which should already be installed on your server.

      First, use the openssl command to create an administrative Kibana user which you’ll use to access the Kibana web interface. As an example we will name this account kibanaadmin, but to ensure greater security we recommend that you choose a non-standard name for your user that would be difficult to guess.

      The following command will create the administrative Kibana user and password, and store them in the htpasswd.users file. You will configure Nginx to require this username and password and read this file momentarily:

      1. echo "kibanaadmin:`openssl passwd -apr1`" | sudo tee -a /etc/nginx/htpasswd.users

      Enter and confirm a password at the prompt. Remember or take note of this login, as you will need it to access the Kibana web interface.

      Next, we will create an Nginx server block file. As an example, we will refer to this file as your_domain, although you may find it helpful to give yours a more descriptive name. For instance, if you have a FQDN and DNS records set up for this server, you could name this file after your FQDN.

      Using nano or your preferred text editor, create the Nginx server block file:

      1. sudo nano /etc/nginx/sites-available/your_domain

      Add the following code block into the file, being sure to update your_domain to match your server’s FQDN or public IP address. This code configures Nginx to direct your server’s HTTP traffic to the Kibana application, which is listening on localhost:5601. Additionally, it configures Nginx to read the htpasswd.users file and require basic authentication.

      Note that if you followed the prerequisite Nginx tutorial through to the end, you may have already created this file and populated it with some content. In that case, delete all the existing content in the file before adding the following:

      /etc/nginx/sites-available/your_domain

      server {
          listen 80;
      
          server_name your_domain;
      
          auth_basic "Restricted Access";
          auth_basic_user_file /etc/nginx/htpasswd.users;
      
          location / {
              proxy_pass http://localhost:5601;
              proxy_http_version 1.1;
              proxy_set_header Upgrade $http_upgrade;
              proxy_set_header Connection 'upgrade';
              proxy_set_header Host $host;
              proxy_cache_bypass $http_upgrade;
          }
      }
      

      When you’re finished, save and close the file.

      Next, enable the new configuration by creating a symbolic link to the sites-enabled directory. If you already created a server block file with the same name in the Nginx prerequisite, you do not need to run this command:

      1. sudo ln -s /etc/nginx/sites-available/your_domain /etc/nginx/sites-enabled/your_domain

      Then check the configuration for syntax errors:

      1. sudo nginx -t

      If any errors are reported in your output, go back and double check that the content you placed in your configuration file was added correctly. Once you see syntax is ok in the output, go ahead and restart the Nginx service:

      1. sudo systemctl reload nginx

      If you followed the initial server setup guide, you should have a UFW firewall enabled. To allow connections to Nginx, we can adjust the rules by typing:

      1. sudo ufw allow 'Nginx Full'

      Note: If you followed the prerequisite Nginx tutorial, you may have created a UFW rule allowing the Nginx HTTP profile through the firewall. Because the Nginx Full profile allows both HTTP and HTTPS traffic through the firewall, you can safely delete the rule you created in the prerequisite tutorial. Do so with the following command:

      1. sudo ufw delete allow 'Nginx HTTP'

      Kibana is now accessible via your FQDN or the public IP address of your Elastic Stack server. You can check the Kibana server’s status page by navigating to the following address and entering your login credentials when prompted:

      http://your_domain/status
      

      This status page displays information about the server’s resource usage and lists the installed plugins.

      |Kibana status page

      Note: As mentioned in the Prerequisites section, it is recommended that you enable SSL/TLS on your server. You can follow the Let’s Encrypt guide now to obtain a free SSL certificate for Nginx on Ubuntu 20.04. After obtaining your SSL/TLS certificates, you can come back and complete this tutorial.

      Now that the Kibana dashboard is configured, let’s install the next component: Logstash.

      Although it’s possible for Beats to send data directly to the Elasticsearch database, it is common to use Logstash to process the data. This will allow you more flexibility to collect data from different sources, transform it into a common format, and export it to another database.

      Install Logstash with this command:

      1. sudo apt install logstash

      After installing Logstash, you can move on to configuring it. Logstash’s configuration files reside in the /etc/logstash/conf.d directory. For more information on the configuration syntax, you can check out the configuration reference that Elastic provides. As you configure the file, it’s helpful to think of Logstash as a pipeline which takes in data at one end, processes it in one way or another, and sends it out to its destination (in this case, the destination being Elasticsearch). A Logstash pipeline has two required elements, input and output, and one optional element, filter. The input plugins consume data from a source, the filter plugins process the data, and the output plugins write the data to a destination.

      Logstash pipeline

      Create a configuration file called 02-beats-input.conf where you will set up your Filebeat input:

      1. sudo nano /etc/logstash/conf.d/02-beats-input.conf

      Insert the following input configuration. This specifies a beats input that will listen on TCP port 5044.

      /etc/logstash/conf.d/02-beats-input.conf

      input {
        beats {
          port => 5044
        }
      }
      

      Save and close the file.

      Next, create a configuration file called 30-elasticsearch-output.conf:

      1. sudo nano /etc/logstash/conf.d/30-elasticsearch-output.conf

      Insert the following output configuration. Essentially, this output configures Logstash to store the Beats data in Elasticsearch, which is running at localhost:9200, in an index named after the Beat used. The Beat used in this tutorial is Filebeat:

      /etc/logstash/conf.d/30-elasticsearch-output.conf

      output {
        if [@metadata][pipeline] {
      	elasticsearch {
        	hosts => ["localhost:9200"]
        	manage_template => false
        	index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
        	pipeline => "%{[@metadata][pipeline]}"
      	}
        } else {
      	elasticsearch {
        	hosts => ["localhost:9200"]
        	manage_template => false
        	index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      	}
        }
      }
      
      
      

      Save and close the file.

      Test your Logstash configuration with this command:

      1. sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t

      If there are no syntax errors, your output will display Config Validation Result: OK. Exiting Logstash after a few seconds. If you don’t see this in your output, check for any errors noted in your output and update your configuration to correct them. Note that you’ll receive warnings from OpenJDK, but they should not cause any problems and can be ignored.

      If your configuration test is successful, start and enable Logstash to put the configuration changes into effect:

      1. sudo systemctl start logstash
      2. sudo systemctl enable logstash

      Now that Logstash is running correctly and is fully configured, let’s install Filebeat.

      The Elastic Stack uses several lightweight data shippers called Beats to collect data from various sources and transport them to Logstash or Elasticsearch. Here are the Beats that are currently available from Elastic:

      • Filebeat: collects and ships log files.
      • Metricbeat: collects metrics from your systems and services.
      • Packetbeat: collects and analyzes network data.
      • Winlogbeat: collects Windows event logs.
      • Auditbeat: collects Linux audit framework data and monitors file integrity.
      • Heartbeat: monitors services for their availability with active probing.

      In this tutorial we will use Filebeat to forward local logs to our Elastic Stack.

      Install Filebeat using apt:

      1. sudo apt install filebeat

      Next, configure Filebeat to connect to Logstash. Here, we will modify the example configuration file that comes with Filebeat.

      Open the Filebeat configuration file:

      1. sudo nano /etc/filebeat/filebeat.yml

      Note: As with Elasticsearch, Filebeat’s configuration file is in YAML format. This means that proper indentation is crucial, so be sure to use the same number of spaces that are indicated in these instructions.

      Filebeat supports numerous outputs, but you’ll usually only send events directly to Elasticsearch or to Logstash for additional processing. In this tutorial, we’ll use Logstash to perform additional processing on the data collected by Filebeat. Filebeat will not need to send any data directly to Elasticsearch, so let’s disable that output. To do so, find the output.elasticsearch section and comment out the following lines by preceding them with a #:

      /etc/filebeat/filebeat.yml

      ...
      #output.elasticsearch:
        # Array of hosts to connect to.
        #hosts: ["localhost:9200"]
      ...
      

      Then, configure the output.logstash section. Uncomment the lines output.logstash: and hosts: ["localhost:5044"] by removing the #. This will configure Filebeat to connect to Logstash on your Elastic Stack server at port 5044, the port for which we specified a Logstash input earlier:

      /etc/filebeat/filebeat.yml

      output.logstash:
        # The Logstash hosts
        hosts: ["localhost:5044"]
      

      Save and close the file.

      The functionality of Filebeat can be extended with Filebeat modules. In this tutorial we will use the system module, which collects and parses logs created by the system logging service of common Linux distributions.

      Let’s enable it:

      1. sudo filebeat modules enable system

      You can see a list of enabled and disabled modules by running:

      1. sudo filebeat modules list

      You will see a list similar to the following:

      Output

      Enabled: system Disabled: apache2 auditd elasticsearch icinga iis kafka kibana logstash mongodb mysql nginx osquery postgresql redis traefik

      By default, Filebeat is configured to use default paths for the syslog and authorization logs. In the case of this tutorial, you do not need to change anything in the configuration. You can see the parameters of the module in the /etc/filebeat/modules.d/system.yml configuration file.

      Next, we need to set up the Filebeat ingest pipelines, which parse the log data before sending it through logstash to elasticsearch. To load the ingest pipeline for the system module, enter the following command:

      1. sudo filebeat setup --pipelines --modules system

      Next, load the index template into Elasticsearch. An Elasticsearch index is a collection of documents that have similar characteristics. Indexes are identified with a name, which is used to refer to the index when performing various operations within it. The index template will be automatically applied when a new index is created.

      To load the template, use the following command:

      1. sudo filebeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

      Output

      Index setup finished.

      Filebeat comes packaged with sample Kibana dashboards that allow you to visualize Filebeat data in Kibana. Before you can use the dashboards, you need to create the index pattern and load the dashboards into Kibana.

      As the dashboards load, Filebeat connects to Elasticsearch to check version information. To load dashboards when Logstash is enabled, you need to disable the Logstash output and enable Elasticsearch output:

      1. sudo filebeat setup -E output.logstash.enabled=false -E output.elasticsearch.hosts=['localhost:9200'] -E setup.kibana.host=localhost:5601

      You should receive output similar to this:

      Output

      Overwriting ILM policy is disabled. Set `setup.ilm.overwrite:true` for enabling. Index setup finished. Loading dashboards (Kibana must be running and reachable) Loaded dashboards Setting up ML using setup --machine-learning is going to be removed in 8.0.0. Please use the ML app instead. See more: https://www.elastic.co/guide/en/elastic-stack-overview/current/xpack-ml.html Loaded machine learning job configurations Loaded Ingest pipelines

      Now you can start and enable Filebeat:

      1. sudo systemctl start filebeat
      2. sudo systemctl enable filebeat

      If you’ve set up your Elastic Stack correctly, Filebeat will begin shipping your syslog and authorization logs to Logstash, which will then load that data into Elasticsearch.

      To verify that Elasticsearch is indeed receiving this data, query the Filebeat index with this command:

      1. curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'

      You should receive output similar to this:

      Output

      ... { { "took" : 4, "timed_out" : false, "_shards" : { "total" : 2, "successful" : 2, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 4040, "relation" : "eq" }, "max_score" : 1.0, "hits" : [ { "_index" : "filebeat-7.7.1-2020.06.04", "_type" : "_doc", "_id" : "FiZLgXIB75I8Lxc9ewIH", "_score" : 1.0, "_source" : { "cloud" : { "provider" : "digitalocean", "instance" : { "id" : "194878454" }, "region" : "nyc1" }, "@timestamp" : "2020-06-04T21:45:03.995Z", "agent" : { "version" : "7.7.1", "type" : "filebeat", "ephemeral_id" : "cbcefb9a-8d15-4ce4-bad4-962a80371ec0", "hostname" : "june-ubuntu-20-04-elasticstack", "id" : "fbd5956f-12ab-4227-9782-f8f1a19b7f32" }, ...

      If your output shows 0 total hits, Elasticsearch is not loading any logs under the index you searched for, and you will need to review your setup for errors. If you received the expected output, continue to the next step, in which we will see how to navigate through some of Kibana’s dashboards.

      Let’s return to the Kibana web interface that we installed earlier.

      In a web browser, go to the FQDN or public IP address of your Elastic Stack server. After entering the login credentials you defined in Step 2, you will see the Kibana homepage:

      Kibana Homepage

      Click the Discover link in the left-hand navigation bar (you may have to click the the Expand icon at the very bottom left to see the navigation menu items). On the Discover page, select the predefined filebeat-* index pattern to see Filebeat data. By default, this will show you all of the log data over the last 15 minutes. You will see a histogram with log events, and some log messages below:

      Discover page

      Here, you can search and browse through your logs and also customize your dashboard. At this point, though, there won’t be much in there because you are only gathering syslogs from your Elastic Stack server.

      Use the left-hand panel to navigate to the Dashboard page and search for the Filebeat System dashboards. Once there, you can select the sample dashboards that come with Filebeat’s system module.

      For example, you can view detailed stats based on your syslog messages:

      Syslog Dashboard

      You can also view which users have used the sudo command and when:

      Sudo Dashboard

      Kibana has many other features, such as graphing and filtering, so feel free to explore.

      In this tutorial, you’ve learned how to install and configure the Elastic Stack to collect and analyze system logs. Remember that you can send just about any type of log or indexed data to Logstash using Beats, but the data becomes even more useful if it is parsed and structured with a Logstash filter, as this transforms the data into a consistent format that can be read easily by Elasticsearch.

      How to Upgrade to Ubuntu 22.04 LTS


      Although Ubuntu 20.04 LTS (Long Term Support) is still supported, users should upgrade Ubuntu to the more recent 22.04 LTS. Upgrading to the new release ensures the system can access the most recent security upgrades and application packages. This guide describes how to perform an inline upgrade from Ubuntu 20.xx or 21.xx to 22.04.

      New Features in Ubuntu 22.04

      Ubuntu 22.04 LTS from Canonical is also referred to as “Jammy Jellyfish”. Ubuntu generally supports their LTS releases for five years, which means Ubuntu 22.04 is supported until April 2027. This is superior to the Ubuntu 20.04 schedule, where support ends in April 2025. In addition, most application developers test their programs more thoroughly against the latest LTS release.

      In addition to the longer support period, Ubuntu 22.04 includes these other features and improvements:

      • Enhanced performance and better power efficiency.
      • A better display, featuring double the frame rate.
      • New power management options.
      • Updated security patches.
      • GNOME 42, which includes a streamlined user interface, on-screen notifications, and better multi-monitor support.
      • Increased customization options.
      • Firefox availability through a Snap package.
      • New releases of applications and toolchains. Updates and enhancements are available for Apache, MySQL, Perl, PHP, PostgreSQL, Python, and Ruby.
      • A new version (5.15) of the Linux Kernel.

      Inline Upgrade versus Clean Install

      There are two ways to upgrade a node. These are the inline upgrade method and the clean install approach. This guide only explains how to perform an inline upgrade. However, it is important to understand both methods to make an informed choice.

      The Inline Upgrade Method

      In an inline upgrade, the primary node is upgraded in place using either the GUI or command line directives. Ubuntu downloads and installs the new release of the operating system on the same system. The files and applications on the system are left unchanged and the node can immediately resume operations after the upgrade. Some of the advantages and disadvantages of this method, and other factors to consider, are as follows:

      • This is the easiest and fastest method of upgrading a node. Depending on the size of the new release, the node might be fully operational in as little as 15 minutes.
      • The Ubuntu upgrade procedure is well tested and generally reliable.
      • Users do not have to remember to reinstall a critical program or import data from the original node.
      • It is impossible to avoid some downtime while the updates are installed and the system reboots and initializes. During this period, any websites or applications hosted on the node are inaccessible. It is crucial to declare a maintenance window or switch to a backup system for the duration of the upgrade.
      • There is a greater chance of complications. Some applications might not work properly after the upgrade and might have to be reinstalled. There is also a greater chance of data corruption.
      • This method of upgrading tends to retain “digital residue”. This includes unnecessary or outdated packages, patches, and data.
      • This method is recommended if the system is only one release behind and is mainly running a widely used and tested configuration such as a LAMP stack. An inline upgrade might run into more problems when the system configuration is complicated or includes in-house applications.

      Caution

      Although this process upgrades the Ubuntu operating system and most common programs, it does not necessarily upgrade every application. It is difficult to predict how the upgrade might affect these programs.

      The Clean Install Method

      The clean install method takes the opposite approach to the inline upgrade. This approach deploys a brand new Linode running the Ubuntu 22.04 release. All necessary applications are reinstalled and the backed-up data from the old node is copied over. After the new node is fully configured and operational, the old node is decommissioned.

      The pros and cons of a clean install are as follows:

      • It is easier to troubleshoot any problems because there are fewer dependencies.
      • The configuration tends to be cleaner. Old packages and unwanted files are not copied over.
      • This method takes a lot more effort and is more error prone. It is easy to forget to port over important applications and application data.
      • This method is a better choice if the system is running a much older release of Ubuntu or if the configuration is very convoluted. It is also a good choice for systems under the control of an Infrastructure as Code (IaC) application, like Terraform or Chef. These applications allow administrators to automatically provision a new remote node with a standard configuration.

      For an in-depth explanation of the clean install method, see the
      Linode guide to manually upgrading a node.

      Before You Begin

      1. If you have not already done so, create a Linode account and Compute Instance. See our
        Getting Started with Linode and
        Creating a Compute Instance guides.

      2. Follow our
        Setting Up and Securing a Compute Instance guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access.

      3. Ensure there is at least 20 GB of disk space available. Verify the amount of disk space availability using the df -Th command.

      Note

      This guide is written for a non-root user. Commands that require elevated privileges are prefixed with sudo. If you are not familiar with the sudo command, see the
      Users and Groups guide.

      How to Upgrade to Ubuntu 22.04 LTS

      This guide is designed for users who want to upgrade from Ubuntu 20.04 LTS to Ubuntu 22.04 LTS. However, it is generally applicable for upgrades to Ubuntu 22.04 from any release of Ubuntu 20.xx or 21.xx.

      If the Linode is running Ubuntu 18.xx or any earlier release, first upgrade it to Ubuntu 20.04 LTS. Then perform the steps in this guide to upgrade from Ubuntu 20.04 LTS to the 22.04 LTS. See the
      Linode guide to Upgrade to Ubuntu 20.04 for more information. Alternatively, if the Ubuntu software and applications are very old, it might make more sense to perform a clean install instead.

      Caution

      This operation cannot be canceled after it is started. Ensure there is a stable connection to the Linode and backup power is available.

      How to Prepare the Linode for the Upgrade

      To increase the chance of a successful upgrade, ensure the operating system and all applications are up to date. All data should be backed up and active user applications shut down.

      To prepare the Ubuntu system for the upgrade, follow these steps.

      1. Upgrade the Linode and ensure it is up to date.

        sudo apt update -y && sudo apt upgrade -y && sudo apt dist-upgrade -y
      2. To simplify the upgrade, remove unused packages and files.

        sudo apt autoremove -y && sudo apt autoclean -y
      3. Reboot the node to ensure any new kernel upgrades are installed. Linode makes new kernels available through the Linode cloud manager. Any updates are automatically applied to the node upon a reboot. For more information, see the
        Linode guide to monitoring and maintaining a system.

      4. Make a backup copy of the system configuration and all application data. The easiest way to do this is to back up the entire system. Subscribing to the
        Linode Backup Service allows you to take a manual snapshot before the upgrade.

      5. Stop as many non-critical user applications services as possible, including web and database servers. Focus on applications that might be subject to data corruption. To see a list of the active services, use the command systemctl | grep running.

        sudo systemctl | grep running
        ...
        apache2.service loaded active running The Apache HTTP Server
        ...
        mysql.service loaded active running MySQL Community Server
            
      6. Use the command sudo systemctl stop <application_name> to stop a service. The following example demonstrates how to stop the Apache web server instance.

        Caution

        Do not stop any essential system services such as ssh or any systemd entry.

        sudo systemctl stop apache2
      7. Allow connections on TCP port 1022 through the ufw firewall. This permits Ubuntu to use a fallback port if the main connection drops. After adding the rule, reload the firewall.

        sudo ufw allow 1022/tcp
        sudo ufw reload
        Firewall reloaded
            
      8. Confirm connections on TCP port 1022 are now allowed.

        Status: active
        
        To                         Action      From
        --                         ------      ----
        OpenSSH                    ALLOW       Anywhere
        Apache Full                ALLOW       Anywhere
        1022/tcp                   ALLOW       Anywhere
        OpenSSH (v6)               ALLOW       Anywhere (v6)
        Apache Full (v6)           ALLOW       Anywhere (v6)
        1022/tcp (v6)              ALLOW       Anywhere (v6)
            

      How to Install Ubuntu Release 22.04

      The node is now ready for the upgrade. Ensure the update manager is installed, then initiate the upgrade. The upgrade might take some time, depending on the configuration, and must not be interrupted. Ensure there is enough time to complete the entire upgrade before proceeding.

      Note

      The upgrade operation can be performed using either a LISH session or an SSH connection. A LISH session is safer, but if SSH is used, the upgrade manager opens a second port for redundancy. This guide uses SSH for the procedure to demonstrate the additional steps required.

      1. Ensure the update-manager-core package is installed. On many systems, this package might already be available.

        sudo apt install update-manager-core
      2. Confirm the release-upgrader is set to the correct release update mode. The file /etc/update-manager/release-upgrades must include the line Prompt=lts.

        sudo cat /etc/update-manager/release-upgrades
        File: /etc/update-manager/release-upgrades
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        16
        
        # Default behavior for the release upgrader.
        
        [DEFAULT]
        # Default prompting and upgrade behavior, valid options:
        #
        #  never  - Never check for, or allow upgrading to, a new release.
        #  normal - Check to see if a new release is available.  If more than one new
        #           release is found, the release upgrader will attempt to upgrade to
        #           the supported release that immediately succeeds the
        #           currently-running release.
        #  lts    - Check to see if a new LTS release is available.  The upgrader
        #           will attempt to upgrade to the first LTS release available after
        #           the currently-running one.  Note that if this option is used and
        #           the currently-running release is not itself an LTS release the
        #           upgrader will assume prompt was meant to be normal.
        Prompt=lts
      3. Run the do-release-upgrade command to start the upgrade.

        Note

        To force an upgrade from the latest supported release to a development release, use the command do-release-upgrade -d. This guide focuses on upgrading to the latest supported release and does not use this flag.

      4. If the operation is performed using a SSH connection, Ubuntu verifies the SSH connection details and asks whether to continue. Answer y to continue.

        Continue running under SSH?
        
        This session appears to be running under ssh. It is not recommended
        to perform a upgrade over ssh currently because in case of failure it
        is harder to recover.
        
        If you continue, an additional ssh daemon will be started at port
        '1022'.
        Do you want to continue?
            
      5. Ubuntu asks the user to confirm the new SSH port is allowed through the firewall. The port should already be open. Press the Enter key to continue.

        Starting additional sshd
        
        To make recovery in case of failure easier, an additional sshd will
        be started on port '1022'. If anything goes wrong with the running
        ssh you can still connect to the additional one.
        If you run a firewall, you may need to temporarily open this port. As
        this is potentially dangerous it's not done automatically. You can
        open the port with e.g.:
        'iptables -I INPUT -p tcp --dport 1022 -j ACCEPT'
        
        To continue please press [ENTER]
            
      6. Ubuntu reads through the list of packages, builds the dependencies, and searches for internal package mirrors. If no mirrors are available, it prompts for approval to rewrite the sources.list file. Enter y to continue.

        Fetched 336 kB in 0s (0 B/s)
        Reading package lists... Done
        ...
        No valid mirror found
        
        While scanning your repository information no mirror entry for the
        upgrade was found. This can happen if you run an internal mirror or
        if the mirror information is out of date.
        
        Do you want to rewrite your 'sources.list' file anyway? If you choose
        'Yes' here it will update all 'focal' to 'jammy' entries.
        If you select 'No' the upgrade will cancel.
        
        Continue [yN]
            
      7. Ubuntu downloads the new packages and files. It determines which packages are no longer supported and requests approval to proceed. It also calculates how long the upgrade might take. To continue with the upgrade, answer y.

        Note

        To see details about the packages to be removed, installed, and upgraded, enter d. Enter q to exit the details screen. Then enter y to continue with the upgrade.

        Do you want to start the upgrade?
        
        
        14 installed packages are no longer supported by Canonical. You can
        still get support from the community.
        
        5 packages are going to be removed. 91 new packages are going to be
        installed. 571 packages are going to be upgraded.
        
        You have to download a total of 552 M. This download will take about
        2 minutes with you connection.
        
        Installing the upgrade can take several hours. Once the download has
        finished, the process cannot be canceled.
        
        Continue [yN]  Details [d]
            
      8. Ubuntu displays a pop-up asking whether to restart the services after the upgrade. Select either the <Yes> button to automatically restart them or <No> to restart them manually.

        Ubuntu Services Pop-up

      9. Ubuntu continues downloading the new packages. This can take a considerable length of time, especially if many applications have been installed. However, Ubuntu echoes the package names when it installs and processes them, allowing users to monitor the progress. During the upgrade, Ubuntu uses a pop-up to ask users how to handle the sshd_config file. Select the keep the local version currently installed option, and then choose <OK>.

        Ubuntu SSH Configuration Pop-up

      10. Ubuntu locates any obsolete packages and asks the user whether to remove them. Enter y to delete the outdated packages.

        Searching for obsolete software
        Reading state information... Done
        
        Remove obsolete packages?
        
        
        41 packages are going to be removed.
        
        Continue [yN]  Details [d]
            
      11. Ubuntu removes the packages and finalizes the upgrade. This stage might also take some length of time. Ubuntu informs the user that the upgrade is complete and prompts them to reboot the system. Select y to reboot and finalize the upgrade.

        System upgrade is complete.
        
        Restart required
        
        To finish the upgrade, a restart is required.
        If you select 'y' the system will be restarted.
        
        Continue [yN]
            

      How to Perform Post-Upgrade Clean-Up Activities

      Ubuntu has now been upgraded to version 22.04 LTS. After the Linode reboots, it is ready to resume operations. However, it is important to validate the upgrade. There are also still a few security concerns to fix and clean-up activities to perform. Log in to the Linode and perform the following steps.

      1. Use the lsb_release -a command to verify the correct release of Ubuntu is now installed. The Release attribute should be 22.04.

        No LSB modules are available.
        Distributor ID: Ubuntu
        Description:    Ubuntu 22.04.1 LTS
        Release:        22.04
        Codename:       jammy
            
      2. Optional: To validate the kernel version, use the uname command.

        Linux 5.15.0-53-generic x86_64
            
      3. To increase security, close port 1022 in the ufw firewall. Reload the firewall.

        sudo ufw delete allow 1022/tcp
        sudo ufw reload
      4. Confirm the firewall rules are updated.

        Status: active
        
        To                         Action      From
        --                         ------      ----
        OpenSSH                    ALLOW       Anywhere
        Apache Full                ALLOW       Anywhere
        OpenSSH (v6)               ALLOW       Anywhere (v6)
        Apache Full (v6)           ALLOW       Anywhere (v6)
            
      5. Ubuntu disables any third-party repositories during the upgrade. To search for disabled repositories, switch to the sources.list.d directory and list the entries.

        cd /etc/apt/sources.list.d
        ls -l
      6. Edit each list using a text editor. Remove the # symbol at the start of the affected entries, and save the file. In the following example, remove the # symbol in front of deb [arch=amd64].

        nano archive-application.list
        File: archive-application.list
        1
        2
        
        [DEFAULT]
        deb [arch=amd64] https://apt.releases.application-name.com jammy main
      7. Update any third-party repositories and remove unnecessary packages using apt commands.

        sudo apt update -y && sudo apt upgrade -y && sudo apt dist-upgrade -y && sudo apt autoremove -y && sudo apt autoclean -y

      Conclusion

      Users can access security updates and new features by upgrading to the new Ubuntu 22.04 LTS release. The two methods of updating a Linode are the inline update or the clean install. This guide explains how to perform an Ubuntu inline upgrade, which is the quickest and easiest method.

      To prepare to upgrade a Linode to Ubuntu 22.04 LTS, update and upgrade the node, back up the data, and stop all services. Use the do-release-upgrade command to initiate the upgrade and follow all Ubuntu prompts. After the upgrade, tighten up security, enable third-party archives, and perform some clean-up tasks. For more information about the Ubuntu 22.04 LTS release, see the
      Ubuntu server documentation.

      More Information

      You may wish to consult the following resources for additional information
      on this topic. While these are provided in the hope that they will be
      useful, please note that we cannot vouch for the accuracy or timeliness of
      externally hosted materials.



      Source link

      Installing Countly Community Edition on Ubuntu 20.04


      The Countly analytics platform offers an alternative to the ubiquitous Google Analytics. In contrast to Google Analytics, Countly puts more emphasis on privacy and an all-in-one feature set. Countly’s data gathering offers compliance with GDPR, HIPAA, and other privacy standards. Meanwhile, it provides not just visitor analytics, but also a wider range of analytics related to marketing.

      This tutorial shows you how to start using Countly for your analytics needs. Countly Community Edition is free to use, and it runs in a self-hosted server environment. Through this guide, you can learn all the steps needed to get your own Countly server up and tracking activity on your applications.

      Before You Begin

      1. Familiarize yourself with our
        Getting Started with Linode guide, and complete the steps for setting your Linode’s hostname and timezone.

      2. This guide uses sudo wherever possible. Complete the sections of our
        How to Secure Your Server guide to create a standard user account, harden SSH access, and remove unnecessary network services.

      3. Update your system.

        Debian / Ubuntu

        sudo apt update && sudo apt upgrade

        AlmaLinux / CentOS Stream / Fedora / Rocky Linux

      Note

      This guide is written for a non-root user. Commands that require elevated privileges are prefixed with sudo. If you’re not familiar with the sudo command, see the
      Users and Groups guide.

      Countly provides several installation options, which you can review in the link provided at the end of this tutorial.

      This guide covers the method using the Countly server GitHub repository, which tends to be straightforward as a result of the included installation script.

      These instructions are intended for and have been tested on Ubuntu systems. However, they may work on Debian and CentOS as well. Just be sure to make the necessary substitutions where relevant.

      After the steps on installing Countly, the tutorial includes instructions for two optional setup features to potentially improve your Countly experience: DNS and SSL.

      Installing Countly from the GitHub Repository

      These steps show you how to download the Git repository for Countly and use the included installation script. It also includes the steps you need to configure NGINX to properly serve your Countly interface.

      1. Clone the Countly server GitHub repository. This example clones the repository to the current user’s home directory. The process creates a new subdirectory there, countly-server:

        git clone https://github.com/Countly/countly-server.git
      2. The included installation script requires root access for the installation, so you should first switch to a superuser shell:

      3. Navigate to the subdirectory where the script is held:

        cd /home/example-user/countly-server/bin
      4. Run the installation script:

      5. Afterward, you can exit the superuser shell:

      6. Replace the default NGINX configuration with Countly’s
        own NGINX configuration file. Typically, you can find the default configuration file at /etc/nginx/sites-available/default. However, if this is a brand-new installation, you may have to create it:

        sudo mkdir /etc/nginx/sites-available
        sudo nano /etc/nginx/sites-available/default

        Additionally, you should extend the server_name property with the domain name and/or remote IP address you intend to use to access your Countly server. For instance, this example adds the domain name example.com and the remote IP address 192.0.2.0:

        File: /etc/nginx/sites-available/default
        1
        2
        3
        4
        5
        6
        
        server {
            listen   80;
            listen   [::]:80 ipv6only=on;
            server_name  localhost example.com 192.0.2.0;
        
            access_log  off;
      7. Open the HTTP port (80) on your server’s firewall. Typically, the firewalls on Ubuntu and Debian systems are managed with UFW. Using it, you can open the HTTP port with:

        sudo ufw allow http
        sudo ufw reload
      8. Access your Countly instance by navigating to one of the enabled addresses (that is, the server_name values from the NGINX configuration) in your web browser.

      (Optional) Assign Countly DNS

      Countly does not require you to use DNS for your server. However, doing so can make your Countly instance easier to access. It gives you access to your instance via a custom domain name, rather than just the remote IP address.

      To set up DNS on a Linode server, refer to our collection of guides on the
      Linode DNS manager. The process there is straightforward and can have your server running through a DNS quickly.

      (Optional) Assigning Countly TLS via Let’s Encrypt

      Another optional step is giving your Countly instance an SSL certificate. Doing so secures and encrypts its traffic using HTTPS.

      The following steps show you how to apply an SSL certificate to Countly using
      Certbot. Certbot allows you to easily request and download free certificates from
      Let’s Encrypt.

      1. Open the HTTPS port on your system’s firewall. Like above, you can do this using UFW with the HTTPS keyword:

        sudo ufw allow https
        sudo ufw reload
      2. Update the
        Snap app store. Snap provides application bundles that work across major Linux distributions and comes by default with all Ubuntu releases since 16.04:

        sudo snap install core && sudo snap refresh core
      3. Remove any existing Certbot installation:

      4. Install Certbot:

        sudo snap install --classic certbot
      5. Download a certificate using standalone verification. When prompted, accept the terms of service, enter an email address for notifications about certificate renewals, and enter your Countly server’s domain name:

        sudo certbot certonly --standalone

        Certbot outputs the location from which the new certificate can be accessed. Typically, it stores the required files in the following directory, replacing example.com with your domain name: /etc/letsencrypt/live/example.com.

      6. Access the NGINX site configuration again, and make the following modifications to the beginning of the file.

        These changes first add a server for port 80 that redirects traffic to the HTTPS URL. Then they alter the existing server definition to listen on port 443, the HTTPS port, and to use the SSL certificate created above.

        Replace example.com in this example with your server’s domain name:

        File: /etc/nginx/sites-available/default
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        
        server {
                listen      80;
                server_name localhost;
                access_log  off;
                rewrite ^ https://$host$request_uri? permanent;
        }
        
        server {
                listen   443 ssl;
                server_name  localhost example.com;
        
                ssl_certificate      /etc/letsencrypt/live/example.com/fullchain.pem;
                ssl_certificate_key  /etc/letsencrypt/live/example.com/privkey.pem;
        
                access_log  off;

      Now, when navigating to your Countly instance in a web browser, you should be redirected to the HTTPS URL.

      You can optionally also add your server’s remote IP address to the NGINX configuration above and use that as well to access Countly. However, you may receive a certificate warning in your browser. This is because the certificate was issued for your server’s domain name, not its IP address.

      How to Navigate the Countly Server Interface

      With your Countly instance up and running, you are ready to start setting it up for use. This next series of sections first covers the initial setup within the Countly interface.

      Further on, you can set up a Countly client SDK within your application and see it begin gathering your analytics.

      Creating an Administrator Account and Logging In

      When you first access Countly, you’re presented with a form to register an administrator user for your instance. Keep track of the login information you create here, as this user has administrative control within the Countly instance.

      Countly registration page

      Accessing the address for your Countly instance after this initial setup directs you to the login page.

      Countly login page

      Adding an Application to the Countly Dashboard

      Submitting the form to create your administrator account automatically directs you to a page to create a new application for your Countly instance. Here, you are entering the name and some descriptive information about the application.

      Creating a new application in Countly

      You can also reach this form later from the Countly dashboard by selecting the Add new app button in the upper right.

      Later, you can use the application key created by this process to associate a Countly client with your Countly server instance. Doing so then directs analytics from that client to Countly’s dashboard for the application.

      Accessing the Countly Dashboard

      From there you are directed to your Countly dashboard, the same page you land on for subsequent logins.

      Countly dashboard

      Here, you can survey the analytics generated by your Countly instance and manage all aspects of your Countly operations. You can navigate between and create application entries, and within each, view analytics for visits, events, and more.

      How to Set Up the Countly Client for Analytics

      To have Countly start collecting analytics, you need to embed one of its client SDKs within your application.

      Countly has numerous client SDKs available to fit your needs, from web and mobile apps, to the desktop, server, and beyond. You can see Countly’s
      full list of client SDKs for more information on how to download and operate each.

      To get you started and to demonstrate, the rest of this section walks you through an example using Countly’s web application SDK. It covers how you can make the client available for your web application and even includes example code to embed it. If you don’t have a web application ready, follow our guide
      Deploy a Static Site using Hugo and Object Storage.

      1. Ensure your web application’s client-side code includes or has access to the Countly web SDK file. This can be done multiple ways:

        • The SDK is automatically hosted alongside your Countly instance. Assuming your server’s domain is example.com, you can find the SDK at: example.com/sdk/web/countly.min.js.

        • Additionally, the file itself can be found among the Countly server files. Starting from the base Countly server directory, the SDK file is located at frontend/express/public/sdk/web/countly.min.js. You can then copy that file to an appropriate directory with your web application’s client-side code.

        • The Countly SDK can also be accessed from Countly’s own CDN, which you can learn about in their
          web SDK documentation.

        For these steps, it is assumed that you made a copy of the countly.min.js file from Countly’s server files. The steps also assume that you have added that file to a lib subdirectory within your client-side code.

      2. Add the following code to the head section of one of your application’s web pages.

        Replace EXAMPLE_COUNTLY_APP_KEY with the App Key found in your Countly instance. Likewise, replace https://example.com with your Countly server’s URL or IP address (preceded by http://, or ‘https:// if you set up SSL).

        File: index.html
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        16
        17
        18
        19
        20
        21
        22
        23
        24
        25
        26
        27
        28
        29
        30
        31
        32
        33
        
        <!-- [...] -->
        <script type='text/javascript'>
        
        // Initialize variables to be used by the Countly client.
        var Countly = Countly || {};
        Countly.q = Countly.q || [];
        
        // Provide the application key from the Countly dashboard.
        Countly.app_key = 'EXAMPLE_COUNTLY_APP_KEY';
        
        // Provide the URL for your Countly server instance.
        Countly.url = 'https://example.com';
        
        // These next two start pushing function calls to queue. Both
        // are recommended configurations.
        
        // Track sessions automatically.
        Countly.q.push(['track_sessions']);
        // Track web page views automatically.
        Countly.q.push(['track_pageview']);
        
        // Load the Countly script asynchronously.
        (function() {
            var cly = document.createElement('script'); cly.type = 'text/javascript';
            cly.async = true;
        
            // Replace the URL here with the location of your Countly client SDK file.
            cly.src = 'lib/countly.min.js';
            cly.onload = function(){Countly.init()};
            var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(cly, s);
        })();
        </script>
        <!-- [...] -->

        Note

        Alternatively, you can have Countly automatically generate this code specifically for your applications. Click Management (the wrench icon) in the left-hand toolstrip, then Applications. Choose your application and scroll all the way down to the blue box titled “Need some help with SDK integration?”. Click the Web button and you should be redirected to an address that starts with code.count.ly/integration-web.html. From here, choose where you want to retrieve the countly.min.js file and what features you want to use. When done, click Generate code for code that’s custom-tailored for your application.

      3. Begin incorporating Countly event calls into your application. Here is an example of one such call, used for a button on the pages:

        File: index.html
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        16
        17
        18
        19
        20
        21
        
        <html>
        <head>
        <!-- [...] -->
        <script type='text/javascript'>
        function exampleButtonClicked(ob){
          Countly.q.push(['add_event',{
            key:"asyncButtonClick",
            segmentation: {
              "id": ob.id
            }
          }]);
        }
        </script>
        <!-- [...] -->
        </head>
        <body>
        <!-- [...] -->
        <input type="button" id="exampleButton" onclick="exampleButtonClicked(this)" value="Click This Button">
        <!-- [...] -->
        </body>
        </html>

      Navigating to your web application should now generate page views in Countly. Activating an event, like clicking the button in the above example, similarly now shows in Countly.

      Countly page visitors

      Countly page events

      Conclusion

      You are now ready to run your application’s analytics with Countly. With your own Countly server set up and the client embedded, you can begin diving deeper into your Countly configuration. Take a look through the
      Countly documentation to learn all the possibilities and see more of what Countly is capable of.

      More Information

      You may wish to consult the following resources for additional information
      on this topic. While these are provided in the hope that they will be
      useful, please note that we cannot vouch for the accuracy or timeliness of
      externally hosted materials.



      Source link