One place for hosting & domains

      Install

      How To Install Elasticsearch, Logstash, and Kibana (Elastic Stack) on CentOS 7


      The author selected Software in the Public Interest to receive a donation as part of the Write for DOnations program.

      Introduction

      The Elastic Stack — formerly known as the ELK Stack — is a collection of open-source software produced by Elastic which allows you to search, analyze, and visualize logs generated from any source in any format, a practice known as centralized logging. Centralized logging can be very useful when attempting to identify problems with your servers or applications, as it allows you to search through all of your logs in a single place. It’s also useful because it allows you to identify issues that span multiple servers by correlating their logs during a specific time frame.

      The Elastic Stack has four main components:

      • Elasticsearch: a distributed RESTful search engine which stores all of the collected data.
      • Logstash: the data processing component of the Elastic Stack which sends incoming data to Elasticsearch.
      • Kibana: a web interface for searching and visualizing logs.
      • Beats: lightweight, single-purpose data shippers that can send data from hundreds or thousands of machines to either Logstash or Elasticsearch.

      In this tutorial, you will install the Elastic Stack on a CentOS 7 server. You will learn how to install all of the components of the Elastic Stack — including Filebeat, a Beat used for forwarding and centralizing logs and files — and configure them to gather and visualize system logs. Additionally, because Kibana is normally only available on the localhost, you will use Nginx to proxy it so it will be accessible over a web browser. At the end of this tutorial, you will have all of these components installed on a single server, referred to as the Elastic Stack server.

      Note: When installing the Elastic Stack, you should use the same version across the entire stack. This tutorial uses the latest versions of each component, which are, at the time of this writing, Elasticsearch 6.5.2, Kibana 6.5.2, Logstash 6.5.2, and Filebeat 6.5.2.

      Prerequisites

      To complete this tutorial, you will need the following:

      • One CentOS 7 server set up by following Initial Server Setup with CentOS 7, including a non-root user with sudo privileges and a firewall. The amount of CPU, RAM, and storage that your Elastic Stack server will require depends on the volume of logs that you intend to gather. For this tutorial, you will be using a VPS with the following specifications for our Elastic Stack server:

        • OS: CentOS 7.5
        • RAM: 4GB
        • CPU: 2
      • Java 8 — which is required by Elasticsearch and Logstash — installed on your server. Note that Java 9 is not supported. To install this, follow the “Install OpenJDK 8 JRE” section of our guide on how to install Java on CentOS.

      • Nginx installed on your server, which you will configure later in this guide as a reverse proxy for Kibana. Follow our guide on How To Install Nginx on CentOS 7 to set this up.

      Additionally, because the Elastic Stack is used to access valuable information about your server that you would not want unauthorized users to access, it’s important that you keep your server secure by installing a TLS/SSL certificate. This is optional but strongly encouraged. Because you will ultimately make changes to your Nginx server block over the course of this guide, we suggest putting this security in place by completing the Let’s Encrypt on CentOS 7 guide immediately after this tutorial’s second step.

      If you do plan to configure Let’s Encrypt on your server, you will need the following in place before doing so:

      • A fully qualified domain name (FQDN). This tutorial will use example.com throughout. You can purchase a domain name on Namecheap, get one for free on Freenom, or use the domain registrar of your choice.

      • Both of the following DNS records set up for your server. You can follow this introduction to DigitalOcean DNS for details on how to add them.

        • An A record with example.com pointing to your server’s public IP address.
        • An A record with www.example.com pointing to your server’s public IP address.

      Step 1 — Installing and Configuring Elasticsearch

      The Elastic Stack components are not available through the package manager by default, but you can install them with yum by adding Elastic’s package repository.

      All of the Elastic Stack’s packages are signed with the Elasticsearch signing key in order to protect your system from package spoofing. Packages which have been authenticated using the key will be considered trusted by your package manager. In this step, you will import the Elasticsearch public GPG key and add the Elastic repository in order to install Elasticsearch.

      Run the following command to download and install the Elasticsearch public signing key:

      • sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

      Next, add the Elastic repository. Use your preferred text editor to create the file elasticsearch.repo in the /etc/yum.repos.d/ directory. Here, we’ll use the vi text editor:

      • sudo vi /etc/yum.repos.d/elasticsearch.repo

      To provide yum with the information it needs to download and install the components of the Elastic Stack, enter insert mode by pressing i and add the following lines to the file.

      /etc/yum.repos.d/elasticsearch.repo

      [elasticsearch-6.x]
      name=Elasticsearch repository for 6.x packages
      baseurl=https://artifacts.elastic.co/packages/6.x/yum
      gpgcheck=1
      gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
      enabled=1
      autorefresh=1
      type=rpm-md
      

      Here you have included the human-readable name of the repo, the baseurl of the repo’s data directory, and the gpgkey required to verify Elastic packages.

      When you’re finished, press ESC to leave insert mode, then :wq and ENTER to save and exit the file. To learn more about the text editor vi and its successor vim, check out our Installing and Using the Vim Text Editor on a Cloud Server tutorial.

      With the repo added, you can now install the Elastic Stack. According to the official documentation, you should install Elasticsearch before the other components. Installing in this order ensures that the components each product depends on are correctly in place.

      Install Elasticsearch with the following command:

      • sudo yum install elasticsearch

      Once Elasticsearch is finished installing, open its main configuration file, elasticsearch.yml, in your editor:

      sudo vi /etc/elasticsearch/elasticsearch.yml
      

      Note: Elasticsearch’s configuration file is in YAML format, which means that indentation is very important! Be sure that you do not add any extra spaces as you edit this file.

      Elasticsearch listens for traffic from everywhere on port 9200. You will want to restrict outside access to your Elasticsearch instance to prevent outsiders from reading your data or shutting down your Elasticsearch cluster through the REST API. Find the line that specifies network.host, uncomment it, and replace its value with localhost so it looks like this:

      /etc/elasticsearch/elasticsearch.yml

      . . .
      network.host: localhost
      . . .
      

      Save and close elasticsearch.yml. Then, start the Elasticsearch service with systemctl:

      • sudo systemctl start elasticsearch

      Next, run the following command to enable Elasticsearch to start up every time your server boots:

      • sudo systemctl enable elasticsearch

      You can test whether your Elasticsearch service is running by sending an HTTP request:

      • curl -X GET "localhost:9200"

      You will see a response showing some basic information about your local node, similar to this:

      Output

      { "name" : "8oSCBFJ", "cluster_name" : "elasticsearch", "cluster_uuid" : "1Nf9ZymBQaOWKpMRBfisog", "version" : { "number" : "6.5.2", "build_flavor" : "default", "build_type" : "rpm", "build_hash" : "9434bed", "build_date" : "2018-11-29T23:58:20.891072Z", "build_snapshot" : false, "lucene_version" : "7.5.0", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" }

      Now that Elasticsearch is up and running, let’s install Kibana, the next component of the Elastic Stack.

      Step 2 — Installing and Configuring the Kibana Dashboard

      According to the installation order in the official documentation, you should install Kibana as the next component after Elasticsearch. After setting Kibana up, we will be able to use its interface to search through and visualize the data that Elasticsearch stores.

      Because you already added the Elastic repository in the previous step, you can just install the remaining components of the Elastic Stack using yum:

      Then enable and start the Kibana service:

      • sudo systemctl enable kibana
      • sudo systemctl start kibana

      Because Kibana is configured to only listen on localhost, we must set up a reverse proxy to allow external access to it. We will use Nginx for this purpose, which should already be installed on your server.

      First, use the openssl command to create an administrative Kibana user which you'll use to access the Kibana web interface. As an example, we will name this account kibanaadmin, but to ensure greater security we recommend that you choose a non-standard name for your user that would be difficult to guess.

      The following command will create the administrative Kibana user and password, and store them in the htpasswd.users file. You will configure Nginx to require this username and password and read this file momentarily:

      • echo "kibanaadmin:`openssl passwd -apr1`" | sudo tee -a /etc/nginx/htpasswd.users

      Enter and confirm a password at the prompt. Remember or take note of this login, as you will need it to access the Kibana web interface.

      Next, we will create an Nginx server block file. As an example, we will refer to this file as example.com.conf, although you may find it helpful to give yours a more descriptive name. For instance, if you have a FQDN and DNS records set up for this server, you could name this file after your FQDN:

      • sudo vi /etc/nginx/conf.d/example.com.conf

      Add the following code block into the file, being sure to update example.com and www.example.com to match your server's FQDN or public IP address. This code configures Nginx to direct your server's HTTP traffic to the Kibana application, which is listening on localhost:5601. Additionally, it configures Nginx to read the htpasswd.users file and require basic authentication.

      Note that if you followed the prerequisite Nginx tutorial through to the end, you may have already created this file and populated it with some content. In that case, delete all the existing content in the file before adding the following:

      example.com.conf'>/etc/nginx/conf.d/example.com.conf

      server {
          listen 80;
      
          server_name example.com www.example.com;
      
          auth_basic "Restricted Access";
          auth_basic_user_file /etc/nginx/htpasswd.users;
      
          location / {
              proxy_pass http://localhost:5601;
              proxy_http_version 1.1;
              proxy_set_header Upgrade $http_upgrade;
              proxy_set_header Connection 'upgrade';
              proxy_set_header Host $host;
              proxy_cache_bypass $http_upgrade;
          }
      }
      

      When you're finished, save and close the file.

      Then check the configuration for syntax errors:

      If any errors are reported in your output, go back and double check that the content you placed in your configuration file was added correctly. Once you see syntax is ok in the output, go ahead and restart the Nginx service:

      • sudo systemctl restart nginx

      By default, SELinux security policy is set to be enforced. Run the following command to allow Nginx to access the proxied service:

      • sudo setsebool httpd_can_network_connect 1 -P

      You can learn more about SELinux in the tutorial An Introduction to SELinux on CentOS 7.

      Kibana is now accessible via your FQDN or the public IP address of your Elastic Stack server. You can check the Kibana server's status page by navigating to the following address and entering your login credentials when prompted:

      http://your_server_ip/status
      

      This status page displays information about the server’s resource usage and lists the installed plugins.

      |Kibana status page

      Note: As mentioned in the Prerequisites section, it is recommended that you enable SSL/TLS on your server. You can follow this tutorial now to obtain a free SSL certificate for Nginx on CentOS 7. After obtaining your SSL/TLS certificates, you can come back and complete this tutorial.

      Now that the Kibana dashboard is configured, let's install the next component: Logstash.

      Step 3 — Installing and Configuring Logstash

      Although it's possible for Beats to send data directly to the Elasticsearch database, we recommend using Logstash to process the data first. This will allow you to collect data from different sources, transform it into a common format, and export it to another database.

      Install Logstash with this command:

      • sudo yum install logstash

      After installing Logstash, you can move on to configuring it. Logstash's configuration files are written in the JSON format and reside in the /etc/logstash/conf.d directory. As you configure it, it's helpful to think of Logstash as a pipeline which takes in data at one end, processes it in one way or another, and sends it out to its destination (in this case, the destination being Elasticsearch). A Logstash pipeline has two required elements, input and output, and one optional element, filter. The input plugins consume data from a source, the filter plugins process the data, and the output plugins write the data to a destination.

      Logstash pipeline

      Create a configuration file called 02-beats-input.conf where you will set up your Filebeat input:

      • sudo vi /etc/logstash/conf.d/02-beats-input.conf

      Insert the following input configuration. This specifies a beats input that will listen on TCP port 5044.

      /etc/logstash/conf.d/02-beats-input.conf

      input {
        beats {
          port => 5044
        }
      }
      

      Save and close the file. Next, create a configuration file called 10-syslog-filter.conf, which will add a filter for system logs, also known as syslogs:

      • sudo vi /etc/logstash/conf.d/10-syslog-filter.conf

      Insert the following syslog filter configuration. This example system logs configuration was taken from official Elastic documentation. This filter is used to parse incoming system logs to make them structured and usable by the predefined Kibana dashboards:

      /etc/logstash/conf.d/10-syslog-filter.conf

      filter {
        if [fileset][module] == "system" {
          if [fileset][name] == "auth" {
            grok {
              match => { "message" => ["%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:[%{POSINT:[system][auth][pid]}])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?",
                        "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:[%{POSINT:[system][auth][pid]}])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}",
                        "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:[%{POSINT:[system][auth][pid]}])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped_ip]}",
                        "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:[%{POSINT:[system][auth][pid]}])?: s*%{DATA:[system][auth][user]} 🙁 %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}",
                        "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:[%{POSINT:[system][auth][pid]}])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}",
                        "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:[%{POSINT:[system][auth][pid]}])?: new user: name=%{DATA:[system][auth][user][add][name]}, UID=%{NUMBER:[system][auth][user][add][uid]}, GID=%{NUMBER:[system][auth][user][add][gid]}, home=%{DATA:[system][auth][user][add][home]}, shell=%{DATA:[system][auth][user][add][shell]}$",
                        "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:[%{POSINT:[system][auth][pid]}])?: %{GREEDYMULTILINE:[system][auth][message]}"] }
              pattern_definitions => {
                "GREEDYMULTILINE"=> "(.|n)*"
              }
              remove_field => "message"
            }
            date {
              match => [ "[system][auth][timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
            }
            geoip {
              source => "[system][auth][ssh][ip]"
              target => "[system][auth][ssh][geoip]"
            }
          }
          else if [fileset][name] == "syslog" {
            grok {
              match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:[%{POSINT:[system][syslog][pid]}])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }
              pattern_definitions => { "GREEDYMULTILINE" => "(.|n)*" }
              remove_field => "message"
            }
            date {
              match => [ "[system][syslog][timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
            }
          }
        }
      }
      

      Save and close the file when finished.

      Lastly, create a configuration file called 30-elasticsearch-output.conf:

      • sudo vi /etc/logstash/conf.d/30-elasticsearch-output.conf

      Insert the following output configuration. This output configures Logstash to store the Beats data in Elasticsearch, which is running at localhost:9200, in an index named after the Beat used. The Beat used in this tutorial is Filebeat:

      /etc/logstash/conf.d/30-elasticsearch-output.conf

      output {
        elasticsearch {
          hosts => ["localhost:9200"]
          manage_template => false
          index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
        }
      }
      

      Save and close the file.

      If you want to add filters for other applications that use the Filebeat input, be sure to name the files so they're sorted between the input and the output configuration, meaning that the file names should begin with a two-digit number between 02 and 30.

      Test your Logstash configuration with this command:

      • sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t

      If there are no syntax errors, your output will display Configruation OK after a few seconds. If you don't see this in your output, check for any errors that appear in your output and update your configuration to correct them.

      If your configuration test is successful, start and enable Logstash to put the configuration changes into effect:

      • sudo systemctl start logstash
      • sudo systemctl enable logstash

      Now that Logstash is running correctly and is fully configured, let's install Filebeat.

      Step 4 — Installing and Configuring Filebeat

      The Elastic Stack uses several lightweight data shippers called Beats to collect data from various sources and transport them to Logstash or Elasticsearch. Here are the Beats that are currently available from Elastic:

      • Filebeat: collects and ships log files.
      • Metricbeat: collects metrics from your systems and services.
      • Packetbeat: collects and analyzes network data.
      • Winlogbeat: collects Windows event logs.
      • Auditbeat: collects Linux audit framework data and monitors file integrity.
      • Heartbeat: monitors services for their availability with active probing.

      In this tutorial, we will use Filebeat to forward local logs to our Elastic Stack.

      Install Filebeat using yum:

      • sudo yum install filebeat

      Next, configure Filebeat to connect to Logstash. Here, we will modify the example configuration file that comes with Filebeat.

      Open the Filebeat configuration file:

      • sudo vi /etc/filebeat/filebeat.yml

      Note: As with Elasticsearch, Filebeat's configuration file is in YAML format. This means that proper indentation is crucial, so be sure to use the same number of spaces that are indicated in these instructions.

      Filebeat supports numerous outputs, but you’ll usually only send events directly to Elasticsearch or to Logstash for additional processing. In this tutorial, we'll use Logstash to perform additional processing on the data collected by Filebeat. Filebeat will not need to send any data directly to Elasticsearch, so let's disable that output. To do so, find the output.elasticsearch section and comment out the following lines by preceding them with a #:

      /etc/filebeat/filebeat.yml

      ...
      #output.elasticsearch:
        # Array of hosts to connect to.
        #hosts: ["localhost:9200"]
      ...
      

      Then, configure the output.logstash section. Uncomment the lines output.logstash: and hosts: ["localhost:5044"] by removing the #. This will configure Filebeat to connect to Logstash on your Elastic Stack server at port 5044, the port for which we specified a Logstash input earlier:

      /etc/filebeat/filebeat.yml

      output.logstash:
        # The Logstash hosts
        hosts: ["localhost:5044"]
      

      Save and close the file.

      You can now extend the functionality of Filebeat with Filebeat modules. In this tutorial, you will use the system module, which collects and parses logs created by the system logging service of common Linux distributions.

      Let's enable it:

      • sudo filebeat modules enable system

      You can see a list of enabled and disabled modules by running:

      • sudo filebeat modules list

      You will see a list similar to the following:

      Output

      Enabled: system Disabled: apache2 auditd elasticsearch haproxy icinga iis kafka kibana logstash mongodb mysql nginx osquery postgresql redis suricata traefik

      By default, Filebeat is configured to use default paths for the syslog and authorization logs. In the case of this tutorial, you do not need to change anything in the configuration. You can see the parameters of the module in the /etc/filebeat/modules.d/system.yml configuration file.

      Next, load the index template into Elasticsearch. An Elasticsearch index is a collection of documents that have similar characteristics. Indexes are identified with a name, which is used to refer to the index when performing various operations within it. The index template will be automatically applied when a new index is created.

      To load the template, use the following command:

      • sudo filebeat setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

      This will give the following output:

      Output

      Loaded index template

      Filebeat comes packaged with sample Kibana dashboards that allow you to visualize Filebeat data in Kibana. Before you can use the dashboards, you need to create the index pattern and load the dashboards into Kibana.

      As the dashboards load, Filebeat connects to Elasticsearch to check version information. To load dashboards when Logstash is enabled, you need to manually disable the Logstash output and enable Elasticsearch output:

      • sudo filebeat setup -e -E output.logstash.enabled=false -E output.elasticsearch.hosts=['localhost:9200'] -E setup.kibana.host=localhost:5601

      You will see output that looks like this:

      Output

      . . . 2018-12-05T21:23:33.806Z INFO elasticsearch/client.go:163 Elasticsearch url: http://localhost:9200 2018-12-05T21:23:33.811Z INFO elasticsearch/client.go:712 Connected to Elasticsearch version 6.5.2 2018-12-05T21:23:33.815Z INFO template/load.go:129 Template already exists and will not be overwritten. Loaded index template Loading dashboards (Kibana must be running and reachable) 2018-12-05T21:23:33.816Z INFO elasticsearch/client.go:163 Elasticsearch url: http://localhost:9200 2018-12-05T21:23:33.819Z INFO elasticsearch/client.go:712 Connected to Elasticsearch version 6.5.2 2018-12-05T21:23:33.819Z INFO kibana/client.go:118 Kibana url: http://localhost:5601 2018-12-05T21:24:03.981Z INFO instance/beat.go:717 Kibana dashboards successfully loaded. Loaded dashboards 2018-12-05T21:24:03.982Z INFO elasticsearch/client.go:163 Elasticsearch url: http://localhost:9200 2018-12-05T21:24:03.984Z INFO elasticsearch/client.go:712 Connected to Elasticsearch version 6.5.2 2018-12-05T21:24:03.984Z INFO kibana/client.go:118 Kibana url: http://localhost:5601 2018-12-05T21:24:04.043Z WARN fileset/modules.go:388 X-Pack Machine Learning is not enabled 2018-12-05T21:24:04.080Z WARN fileset/modules.go:388 X-Pack Machine Learning is not enabled Loaded machine learning job configurations

      Now you can start and enable Filebeat:

      • sudo systemctl start filebeat
      • sudo systemctl enable filebeat

      If you've set up your Elastic Stack correctly, Filebeat will begin shipping your syslog and authorization logs to Logstash, which will then load that data into Elasticsearch.

      To verify that Elasticsearch is indeed receiving this data, query the Filebeat index with this command:

      • curl -X GET 'http://localhost:9200/filebeat-*/_search?pretty'

      You will see an output that looks similar to this:

      Output

      { "took" : 1, "timed_out" : false, "_shards" : { "total" : 3, "successful" : 3, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 3225, "max_score" : 1.0, "hits" : [ { "_index" : "filebeat-6.5.2-2018.12.05", "_type" : "doc", "_id" : "vf5GgGcB_g3p-PRo_QOw", "_score" : 1.0, "_source" : { "@timestamp" : "2018-12-05T19:00:34.000Z", "source" : "/var/log/secure", "meta" : { "cloud" : { . . .

      If your output shows 0 total hits, Elasticsearch is not loading any logs under the index you searched for, and you will need to review your setup for errors. If you received the expected output, continue to the next step, in which you'll become familiar with some of Kibana's dashboards.

      Step 5 — Exploring Kibana Dashboards

      Let's look at Kibana, the web interface that we installed earlier.

      In a web browser, go to the FQDN or public IP address of your Elastic Stack server. After entering the login credentials you defined in Step 2, you will see the Kibana homepage:

      Kibana Homepage

      Click the Discover link in the left-hand navigation bar. On the Discover page, select the predefined filebeat-* index pattern to see Filebeat data. By default, this will show you all of the log data over the last 15 minutes. You will see a histogram with log events, and some log messages below:

      Discover page

      Here, you can search and browse through your logs and also customize your dashboard. At this point, though, there won't be much in there because you are only gathering syslogs from your Elastic Stack server.

      Use the left-hand panel to navigate to the Dashboard page and search for the Filebeat System dashboards. Once there, you can search for the sample dashboards that come with Filebeat's system module.

      For example, you can view detailed stats based on your syslog messages:

      Syslog Dashboard

      You can also view which users have used the sudo command and when:

      Sudo Dashboard

      Kibana has many other features, such as graphing and filtering, so feel free to explore.

      Conclusion

      In this tutorial, you installed and configured the Elastic Stack to collect and analyze system logs. Remember that you can send just about any type of log or indexed data to Logstash using Beats, but the data becomes even more useful if it is parsed and structured with a Logstash filter, as this transforms the data into a consistent format that can be read easily by Elasticsearch.



      Source link

      Install Plex Media Server on Ubuntu 18.04 Using Salt Masterless


      Updated by Linode Contributed by Linode

      Plex is a media server that allows you to stream video and audio content that you own to many different types of devices. In this guide you will learn how to use a masterless Salt minion to set up a Plex server, attach and use a Block Storage Volume, and how to connect to your media server to stream content to your devices.

      Before You Begin

      1. Familiarize yourself with our Getting Started guide and complete the steps for setting your Linode’s hostname and timezone.

      2. Follow the steps in the How to Secure Your Server guide.

      3. Update your system:

        sudo apt-get update && sudo apt-get upgrade
        
      4. You will need to create a Block Storage Volume and attach it to your Linode. You will format and mount the drive as part of this guide. This volume will be used to store your media, so you should pick a size that’s appropriate for your media collection, though you can resize the volume later if you need more storage. For more on Block Storage, see our How to Use Block Storage guide.

      5. Plex requires an account to use their service. Visit the Plex website to sign up for an account if you do not already have one.

      Note

      The steps in this guide require root privileges. Be sure to run the steps below with the sudo prefix. For more information on privileges, see our Users and Groups guide.

      Prepare the Salt Minion

      1. On your Linode, create the /srv/salt and /srv/pillar directories. These are where the Salt state files and Pillar files will be housed.

        mkdir /srv/salt && mkdir /srv/pillar
        
      2. Install salt-minion via the Salt bootstrap script:

        curl -L https://bootstrap.saltstack.com -o bootstrap_salt.sh
        sudo sh bootstrap_salt.sh
        
      3. The Salt minion will use the official Plex Salt Formula, which is hosted on the SaltStack GitHub repository. In order to use a Salt formula hosted on an external repository, you will need GitPython installed. Install GitPython:

        sudo apt-get install python-git
        

      Modify the Salt Minion Configuration

      1. Because the Salt minion is running in masterless mode, you will need to modify the minion configuration file (/etc/salt/minion) to instruct Salt to look for state files locally. Open the minion configuration file in a text editor, uncomment the line #file_client: remote, and set it to local:

        /etc/salt/minion
        1
        2
        3
        4
        5
        6
        7
        8
        9
        
        ...
        
        # Set the file client. The client defaults to looking on the master server for
        # files, but can be directed to look at the local file directory setting
        # defined below by setting it to "local". Setting a local file_client runs the
        # minion in masterless mode.
        file_client: local
        
        ...
      2. There are some configuration values that do not normally exist in /etc/salt/minion which you will need to add in order to run your minion in masterless mode. Copy the following lines into the end of /etc/salt/minion:

        /etc/salt/minion
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        
        ...
        
        fileserver_backend:
          - roots
          - gitfs
        
        gitfs_remotes:
          - https://github.com/saltstack-formulas/plex-formula.git
        
        gitfs_provider: gitpython

        The fileserver_backend block instructs the Salt minion to look for Salt configuration files in two places. First, it tells Salt to look for Salt state files in our minion’s roots backend (/srv/salt). Secondly, it instructs Salt to use the Git Fileserver (gitfs) to look for Salt configuration files in any Git remote repositories that have been named in the gitfs_remotes section. The address for the Plex Salt formula’s Git repository is included in the gitfs_remotes section.

        Note

        It is best practice to create a fork of the Plex formula’s Git repository on GitHub and to add your fork’s Git repository address in the gitfs_remotes section. This will ensure that any further changes to the upstream Plex formula which might break your current configuration can be reviewed and handled accordingly, before applying them.

        Lastly, GitPython is specified as the gitfs_provider.

      Create the Salt State Tree

      1. Create a Salt state top file at /srv/salt/top.sls and copy in the following configuration. This file tells Salt to look for state files in the plex folder of the Plex formula’s Git repository, and for a state files named disk.sls and directory.sls, which you will create in the next steps.

        /srv/salt/top.sls
        1
        2
        3
        4
        5
        
        base:
          '*':
            - plex
            - disk
            - directory
      2. Create the disk.sls file in /srv/salt:

        /srv/salt/disk.sls
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        
        disk.format:
          module.run:
            - device: /dev/disk/by-id/scsi-0Linode_Volume_{{ pillar['volume_name'] }}
            - fs_type: ext4
        
        /mnt/plex:
          mount.mounted:
            - device: /dev/disk/by-id/scsi-0Linode_Volume_{{ pillar['volume_name'] }}
            - fstype: ext4
            - mkmnt: True
            - persist: True

        This file instructs Salt to prepare your Block Storage Volume for use with Plex. It first formats your Block Storage Volume with the ext4 filesystem type by using the disk.format Salt module, which can be run in a state file using module.run. Then disk.sls instructs Salt to mount your volume at /mnt/plex, creating the mount target if it does not already exist with mkmnt, and persisting the mount to /etc/fstab so that the volume is always mounted at boot.

      3. Create the directory.sls file in /srv/salt:

        /srv/salt/directory.sls
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        16
        17
        18
        19
        20
        
        /mnt/plex/plex-media:
          file.directory:
            - require:
              - mount: /mnt/plex
            - user: username
            - group: plex
        
        /mnt/plex/plex-media/movies:
          file.directory:
            - require:
              - mount: /mnt/plex
            - user: username
            - group: plex
        
        /mnt/plex/plex-media/television:
          file.directory:
            - require:
              - mount: /mnt/plex
            - user: username
            - group: plex

        The directories that are created during this step are for organizational purposes, and will house your media. Make sure you replace username with the name of the limited user account you created when following the How to Secure Your Server guide. The location of the directories is the volume you mounted in the previous step. If you wish to add more directories, perhaps one for your music media, you can do so here, just be sure to include the - require block, as this prevents Salt from trying to create the directory before the Block Storage Volume has been mounted.

      4. Go to the Plex Media Server download page and note the most recent version of their Linux distribution. At the time of writing, the most recent version is 1.13.9.5456-ecd600442. Create the plex.sls Pillar file in /srv/pillar and change the Plex version number and the name of your Block Storage Volume as necessary:

        /srv/pillar/plex.sls
        1
        2
        3
        
        plex:
          version: 1.13.9.5456-ecd600442
        volume_name: plex
      5. Create the Salt Pillar top file in /srv/pillar. This file will instruct Salt to look for the plex.sls Pillar file you created in the previous step.

        /srv/pillar/top.sls
      6. Apply your Salt state locally using salt-call:

        salt-call --local state.apply
        

        You should see a list of the changes Salt applied to your system. You have successfully installed Plex using Salt.

      Set Up Plex

      Initial Configuration

      1. You’ll need to create an SSH tunnel to your Linode to connect to Plex’s web UI. On your local computer, run the following command, replacing <your_ip_address> with your Plex server’s IP address.:

        ssh username@<your_ip_address> -L 8888:localhost:32400
        
      2. In a browser, navigate to http://localhost:8888/web/.

      3. Sign in with your Plex username and password.

      4. Name your media server. This example uses the name linode-plex. Be sure to check the box that reads Allow me to access my media outside my home and then click Next.

        Name your media server

      Organize Your Media

      1. Click on the Add Library button:

        Click on Add Media

      2. Select Movies and click Next:

        Select Movies and click next

      3. Click Browse for Media Folder and select the appropriate folder at /mnt/plex/plex-media/movies. Then click Add:

        Select the appropriate folder

      4. Repeat the process to add your ‘Television’ folder.

      5. When you are done adding your libraries, click Add Library.

      6. To continue the configuration process, click Next.

      7. Click on Get Plex Apps to download the appropriate Plex client for your device. Then click Done.

        Download the appropriate client for your device

      8. In the future you can add more libraries by hovering over the menu and clicking the plus sign (+) next to LIBRARIES.

        Add more libraries

      DLNA is a protocol that incorporates Universal Plug and Play (or UPnP) standards for digital media sharing across devices. If you do not wish to make use of it, it’s recommended that you disable this feature, as it is openly connectable on port 1900. From the Plex web interface, click the wrench icon in the upper right corner, and navigate to the DLNA section under SETTINGS. Uncheck Enable the DLNA server, and click Save Changes:

      Disable DLNA

      Connect to Your Plex Server

      1. Visit the Plex Apps download page or the app store on your device to download Plex Media Player if you have not already done so.

      2. Open your Plex app. The example provided here will use the Plex Media Player for macOS.

      3. Sign in to Plex.

      4. On the left there’s a dropdown menu where you can select your server by the name you chose. Select your server.

        Connect to your Plex Server

      5. You are now able to stream your content with Plex.

        Plex's macOS App

      1. You can use SCP to transfer media to your server from your local computer. Replace your username and 123.456.7.8 with the IP address of your Linode.

        scp example_video.mp4 username@123.456.7.8:/mnt/plex/plex-media/movies
        
      2. Once you’ve transferred files to your Plex media server, you may need to scan for new files before they show up in your Library. Click on the ellipsis next to a Library and select Scan Library Files.

        Scan your Library for new files

      More Information

      You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

      Find answers, ask questions, and help others.

      This guide is published under a CC BY-ND 4.0 license.



      Source link

      How to Install Nginx on FreeBSD 11.2


      Introduction

      Nginx is one of the most popular web servers in the world, known for its frequent use as a load balancer and reverse proxy. It’s more resource-friendly than Apache, and many of the largest and most highly trafficked sites on the internet depend on Nginx to serve their content.

      In this guide, we will demonstrate how to install Nginx on a FreeBSD 11.2 server.

      Prerequisites

      Before beginning this guide, you will need access to a server running FreeBSD. This guide was written specifically with a server running FreeBSD 11.2 in mind, although it should also work on older, supported versions of the operating system.

      If you’re new to working with FreeBSD, you may find it helpful to customize this server by following these instructions.

      Additionally, if you plan to set up a domain name for this server, you will need the following:

      • A registered domain name that you own and control. If you do not already have a registered domain name, you may register one with one of the many domain name registrars out there (e.g. Namecheap, GoDaddy, etc.).
      • A DNS A Record that points your domain to the public IP address of your server. You can follow this hostname tutorial for details on how to add them.

      Step 1 — Installing Nginx

      To begin, we will install Nginx using pkg, FreeBSD’s built-in binary package management tool. The following command will update your local copy of the available packages and then install the nginx package:

      If this is your first time running pkg, it will prompt you to confirm that you allow it to bootstrap itself. To do this, press y and then ENTER. Then, when prompted, confirm that you approve the installation of the nginx package by pressing y and then ENTER once again.

      If you are using either the csh or tcsh shell, make sure to run the rehash command. This makes the shell aware of the new applications you installed:

      Note: If you aren't sure which shell you're currently using, you can check with the following command:

      The output will show the file path of the shell program currently in use:

      Output

      /bin/tcsh

      Nginx is now installed on your server, but it is not yet running. In the next step, we'll enable Nginx to start running every time the server boots up and start it for this session, as well as walk through the process of securing the server by setting up a firewall with IPFW.

      Step 2 — Enabling the Nginx Service and Setting Up a Firewall with IPFW

      In order for FreeBSD to start Nginx as a conventional service, you have to tell FreeBSD that you want to enable it. This will allow you to manage it like any other service, rather than as a standalone application, and will also configure FreeBSD to start it up automatically at boot.

      To do this, you first need to know the correct rc parameter to set for the nginx service. rc is a FreeBSD utility that controls the system's automatic boot processes. Scripts for every service available on the system are located in the /usr/local/etc/rc.d directory. These define the parameters that are used to enable each service using the rcvar variable. We can see what each service's rcvar is set to by typing:

      • grep rcvar /usr/local/etc/rc.d/*

      After has been installed, this command will output a listing similar to this:

      Output

      /usr/local/etc/rc.d/cloudconfig:rcvar="cloudinit_enable" /usr/local/etc/rc.d/cloudfinal:rcvar="cloudinit_enable" /usr/local/etc/rc.d/cloudinit:rcvar="cloudinit_enable" /usr/local/etc/rc.d/cloudinitlocal:rcvar="cloudinit_enable" /usr/local/etc/rc.d/nginx:rcvar=nginx_enable /usr/local/etc/rc.d/rsyncd:rcvar=rsyncd_enable

      The parameter that you need to set for the nginx service is highlighted here in this output. The name of the script itself — the last component of the path before the colon — is also helpful to know, as that's the name that FreeBSD uses to refer to the service.

      To enable the nginx service you must add its rcvar to the rc.conf file, which holds the global system configuration information referenced by the startup scripts. Use your preferred editor to open the /etc/rc.conf file with sudo privileges. Here, we'll use ee:

      At the top of the file, there will be a few rcvar parameters already in place. Add the nginx_enable rcvar parameter below these and set it to "YES":

      /etc/rc.conf

      . . .
      sshd_enable="YES"
      nginx_enable="YES"
      
      . . .
      

      While still in the rc.conf file, we will add a few more directives to enable and configure an IPFW firewall. IPFW is a stateful firewall written for FreeBSD. It provides a powerful syntax that allows you to customize security rules for most use cases.

      Directly below the nginx_enable parameter you just added, add the following highlighted lines:

      /etc/rc.conf

      . . .
      nginx_enable="YES"
      firewall_enable="YES"
      firewall_type="workstation"
      firewall_myservices="22/tcp 80/tcp"
      firewall_allowservices="any"
      

      Here’s what each of these directives and their settings do:

      • firewall_enable="YES" — Setting this directive to "YES" enables the firewall to start up whenever the server boots.
      • firewall_type="workstation" — FreeBSD provides several default types of firewalls, each of which have slightly different configurations. By declaring the workstation type, the firewall will only protect this server using stateful rules.
      • firewall_myservices="22/tcp 80/tcp" — The firewall_myservices directive is where you can list the TCP ports you want to allow through the firewall. In this example, we’re specifying ports 22 and 80 to allow SSH and HTTP access to the server, respectively.
      • firewall_allowservices="any" — This allows a machine from any IP address to communicate over the ports specified in the firewall_myservices directive.

      After adding these lines, save the file and close the editor by pressing CTRL + C, typing exit, and then pressing ENTER.

      Then, start the ipfw firewall service. Because this is the first time you're starting the firewall on this server, there's a chance that doing so will cause your server to stall, making it inaccessible over SSH. The following nohup command — which stands for "no hangups" — will start the firewall while preventing stalling and also redirect the standard output and error to a temporary log file:

      • sudo nohup service ipfw start >/tmp/ipfw.log 2>&1

      If you're using either the csh or tcsh shells, though, this redirect will cause Ambiguous output redirect. to appear in your output. If you're using either of these shells, run the following instead to start ipfw:

      • sudo nohup service ipfw start >&/tmp/ipfw.log

      Note: In the future, you can manage the ipfw firewall as you would any other service. For example, to stop, start, and then restart the service, you would run the following commands:

      • sudo service ipfw stop
      • sudo service ipfw start
      • sudo service ipfw restart

      Next, start the nginx service

      Then, to test that Nginx is able to serve content correctly, enter your server's public IP address into the URL bar of your preferred web browser:

      http://your_server_ip
      

      Note: If you aren't sure of your server's public IP address, you can run the following command which will print your server's IP address, as seen from another location on the internet:

      If everything is working correctly, you will see the default Nginx landing page:

      Nginx default page

      This shows that Nginx is installed and running correctly and that it's being allowed through the firewall as expected. There are still a few configuration changes that need to be made, though, in order for it to work with non-default settings or serve content using a domain name.

      Step 3 — Setting Up a Server Block

      When using the Nginx web server, server blocks (similar to virtual hosts in Apache) can be used to encapsulate configuration details and host more than one domain from a single server. We will set up a domain called example.com, but you should replace this with your own domain name. To learn more about setting up a domain name with DigitalOcean, see our Introduction to DigitalOcean DNS.

      Nginx on FreeBSD 11.2 has one server block enabled by default that is configured to serve documents out of a directory at /usr/local/www/nginx. While this works well for a single site, it can become unwieldy if you are hosting multiple sites. Instead of modifying /usr/local/www/nginx, let's create a directory structure within /usr/local/www for our example.com site.

      Create the directory for example.com as follows, using the -p flag to create any necessary parent directories:

      • sudo mkdir -p /usr/local/www/example.com/html

      Next, assign ownership of the directory to the www user, the default Nginx runtime user profile:

      • sudo chown -R www:www /usr/local/www/example.com

      The permissions of your web root should be correct if you haven't modified your umask value, but you can make sure by typing:

      • sudo chmod -R 755 /usr/local/www/example.com

      Next, create a sample index.html page using ee:

      • sudo ee /usr/local/www/example.com/html/index.html

      Inside, add the following sample HTML:

      /usr/local/www/example.com/html/index.html

      <html>
          <head>
              <title>Welcome to Example.com!</title>
          </head>
          <body>
              <h1>Success!  The example.com server block is working!</h1>
          </body>
      </html>
      

      Save and close the file when you are finished.

      In order for Nginx to serve this content, it's necessary to create a server block with the correct directives. Open the main Nginx configuration file. By default, this is held in the /usr/local/etc/nginx/ directory:

      • sudo ee /usr/local/etc/nginx/nginx.conf

      Note: Generally, you want to avoid editing the default nginx.conf file. However, within this same directory, there's a file called nginx.conf-dist, which is identical to the default nginx.conf file. If you ever find that you need to revert with these configuration changes, you can just copy over this file with the following command:

      • sudo cp /usr/local/etc/nginx/nginx.conf-dist /usr/local/etc/nginx/nginx.conf

      When you first open the file, you'll see the following at the very top:

      /usr/local/etc/nginx/nginx.conf

      #user  nobody;
      worker_processes  1;
      
      . . .
      

      Uncomment the user directive by removing the pound sign (#) and then change the user from nobody to www. Then update the worker_processes directive which allows you to select how many worker processes Nginx will use. The optimal value to enter here isn't always obvious or easy to find. Setting it to auto tells Nginx sets it to one worker per CPU core, which will be sufficient in most cases:

      /usr/local/etc/nginx/nginx.conf

      user  www;
      worker_processes  auto;
      . . .
      

      Then scroll down to the server block. With all comments removed, it will look like this:

      /usr/local/etc/nginx/nginx.conf

      . . .
          server {
              listen       80;
              server_name  localhost;
      
              location / {
                  root   /usr/local/www/nginx;
                  index  index.html index.htm;
              }
      
              error_page   500 502 503 504  /50x.html;
      
              location = /50x.html {
                  root   /usr/local/www/nginx-dist;
              }
          }
      

      Delete this entire server block, including all the commented-out lines, and replace it with the following content:

      /usr/local/etc/nginx/nginx.conf

      . . .
          server {
              access_log /var/log/nginx/example.com.access.log;
              error_log /var/log/nginx/example.com.error.log;
              listen       80;
              server_name  example.com www.example.com;
      
              location / {
                  root   /usr/local/www/example.com/html;
                  index  index.html index.htm;
              }
          }
      . . .
      

      Here's what the directives in this server block do:

      • access_log: This directive defines the location of the server's access logs.
      • error_log: This defines the file where Nginx will write its error logs.
      • listen: The listen directive declares what port Nginx should listen in on. In this case, we set it to port 80 so it can listen for HTTP traffic.
      • server_name: Here, point Nginx to your domain name and any aliases you have for it. If you don't have a domain name, point Nginx to your server's public IP address.
      • root: This defines the website document root, which you created earlier in this step.
      • index: This directive defines the files that will be used as an index, and in which order they should be checked.

      All together, with comments removed, the file will look like this:

      /usr/local/letc/nginx/nginx.conf

      user  www;
      worker_processes  1;
      
      events {
          worker_connections  1024;
      }
      
      http {
          include       mime.types;
          default_type  application/octet-stream;
          sendfile        on;
          keepalive_timeout  65;
      
          server {
              access_log /var/log/nginx/example.com.access.log;
              error_log /var/log/nginx/example.com.error.log;
              listen       80;
              server_name  example.com www.example.com;
      
              location / {
                  root   /usr/local/www/example.com;
                  index  index.html index.htm;
              }
      
          }
      
      }
      

      Save and close the file when you are finished. Then, test your configuration file for syntax errors by typing:

      If your configuration file has no detectable syntax errors, you'll see the following output:

      Output

      nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful

      If the above command returns with errors, re-open the Nginx configuration file to the location where the error was found and try to fix the problem.

      When your configuration checks out correctly, go ahead and reload the nginx service to enable your changes:

      • sudo service nginx reload

      Nginx should now be serving the content you set up in the index.html file. Test this by navigating to http://example.com, where you should see something like this:

      Nginx first server block

      As a final step, we will go over some commands that will allow you to manage the Nginx service.

      Step 4 — Managing the Nginx Process

      Now that you have your web server up and running, let's review some basic management commands.

      To stop your web server, type:

      To start the web server when it is stopped, type:

      To stop and then start the service again, type:

      • sudo service nginx restart

      If you are simply making configuration changes, you can reload Nginx without dropping any connections. To do this, type:

      • sudo service nginx reload

      Lastly, recall how in Step 2 we enabled the nginx service by adding the nginx_enable="YES" directive to the rc.conf file. If you'd like to disable the nginx service to keep it from starting up when the server boots, you would need to reopen that file and remove that line.

      Conclusion

      You now have a fully functional Nginx web server installed on your machine. From here, you could encrypt your server’s web traffic by enabling HTTPS. To learn how to do this, consult How To Secure Nginx with Let's Encrypt on FreeBSD. You could also install and configure MySQL and PHP which, along with Nginx, would give you a complete FEMP stack.



      Source link