One place for hosting & domains

      Elasticsearch

      How To Install Elasticsearch, Logstash, and Kibana (Elastic Stack) on Ubuntu 20.04


      Not using Ubuntu 20.04?


      Choose a different version or distribution.

      Introduction

      The Elastic Stack — formerly known as the ELK Stack — is a collection of open-source software produced by Elastic which allows you to search, analyze, and visualize logs generated from any source in any format, a practice known as centralized logging. Centralized logging can be useful when attempting to identify problems with your servers or applications as it allows you to search through all of your logs in a single place. It’s also useful because it allows you to identify issues that span multiple servers by correlating their logs during a specific time frame.

      The Elastic Stack has four main components:

      • Elasticsearch: a distributed RESTful search engine which stores all of the collected data.
      • Logstash: the data processing component of the Elastic Stack which sends incoming data to Elasticsearch.
      • Kibana: a web interface for searching and visualizing logs.
      • Beats: lightweight, single-purpose data shippers that can send data from hundreds or thousands of machines to either Logstash or Elasticsearch.

      In this tutorial, you will install the Elastic Stack on an Ubuntu 20.04 server. You will learn how to install all of the components of the Elastic Stack — including Filebeat, a Beat used for forwarding and centralizing logs and files — and configure them to gather and visualize system logs. Additionally, because Kibana is normally only available on the localhost, we will use Nginx to proxy it so it will be accessible over a web browser. We will install all of these components on a single server, which we will refer to as our Elastic Stack server.

      Note: When installing the Elastic Stack, you must use the same version across the entire stack. In this tutorial we will install the latest versions of the entire stack which are, at the time of this writing, Elasticsearch 7.7.1, Kibana 7.7.1, Logstash 7.7.1, and Filebeat 7.7.1.

      Prerequisites

      To complete this tutorial, you will need the following:

      Additionally, because the Elastic Stack is used to access valuable information about your server that you would not want unauthorized users to access, it’s important that you keep your server secure by installing a TLS/SSL certificate. This is optional but strongly encouraged.

      However, because you will ultimately make changes to your Nginx server block over the course of this guide, it would likely make more sense for you to complete the Let’s Encrypt on Ubuntu 20.04 guide at the end of this tutorial’s second step. With that in mind, if you plan to configure Let’s Encrypt on your server, you will need the following in place before doing so:

      Step 1 — Installing and Configuring Elasticsearch

      The Elasticsearch components are not available in Ubuntu’s default package repositories. They can, however, be installed with APT after adding Elastic’s package source list.

      All of the packages are signed with the Elasticsearch signing key in order to protect your system from package spoofing. Packages which have been authenticated using the key will be considered trusted by your package manager. In this step, you will import the Elasticsearch public GPG key and add the Elastic package source list in order to install Elasticsearch.

      To begin, use cURL, the command line tool for transferring data with URLs, to import the Elasticsearch public GPG key into APT. Note that we are using the arguments -fsSL to silence all progress and possible errors (except for a server failure) and to allow cURL to make a request on a new location if redirected. Pipe the output of the cURL command into the apt-key program, which adds the public GPG key to APT.

      • curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

      Next, add the Elastic source list to the sources.list.d directory, where APT will search for new sources:

      • echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

      Next, update your package lists so APT will read the new Elastic source:

      Then install Elasticsearch with this command:

      • sudo apt install elasticsearch

      Elasticsearch is now installed and ready to be configured. Use your preferred text editor to edit Elasticsearch’s main configuration file, elasticsearch.yml. Here, we’ll use nano:

      • sudo nano /etc/elasticsearch/elasticsearch.yml

      Note: Elasticsearch’s configuration file is in YAML format, which means that we need to maintain the indentation format. Be sure that you do not add any extra spaces as you edit this file.

      The elasticsearch.yml file provides configuration options for your cluster, node, paths, memory, network, discovery, and gateway. Most of these options are preconfigured in the file but you can change them according to your needs. For the purposes of our demonstration of a single-server configuration, we will only adjust the settings for the network host.

      Elasticsearch listens for traffic from everywhere on port 9200. You will want to restrict outside access to your Elasticsearch instance to prevent outsiders from reading your data or shutting down your Elasticsearch cluster through its REST API. To restrict access and therefore increase security, find the line that specifies network.host, uncomment it, and replace its value with localhost like this:

      /etc/elasticsearch/elasticsearch.yml

      . . .
      # ---------------------------------- Network -----------------------------------
      #
      # Set the bind address to a specific IP (IPv4 or IPv6):
      #
      network.host: localhost
      . . .
      

      We have specified localhost so that Elasticsearch listens on all interfaces and bound IPs. If you want it to listen only on a specific interface, you can specify its IP in place of localhost. Save and close elasticsearch.yml. If you’re using nano, you can do so by pressing CTRL+X, followed by Y and then ENTER .

      These are the minimum settings you can start with in order to use Elasticsearch. Now you can start Elasticsearch for the first time.

      Start the Elasticsearch service with systemctl. Give Elasticsearch a few moments to start up. Otherwise, you may get errors about not being able to connect.

      • sudo systemctl start elasticsearch

      Next, run the following command to enable Elasticsearch to start up every time your server boots:

      • sudo systemctl enable elasticsearch

      You can test whether your Elasticsearch service is running by sending an HTTP request:

      • curl -X GET "localhost:9200"

      You will see a response showing some basic information about your local node, similar to this:

      Output

      { "name" : "Elasticsearch", "cluster_name" : "elasticsearch", "cluster_uuid" : "qqhFHPigQ9e2lk-a7AvLNQ", "version" : { "number" : "7.7.1", "build_flavor" : "default", "build_type" : "deb", "build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f", "build_date" : "2020-03-26T06:34:37.794943Z", "build_snapshot" : false, "lucene_version" : "8.5.1", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_version" : "6.0.0-beta1" }, "tagline" : "You Know, for Search" }

      Now that Elasticsearch is up and running, let’s install Kibana, the next component of the Elastic Stack.

      Step 2 — Installing and Configuring the Kibana Dashboard

      According to the official documentation, you should install Kibana only after installing Elasticsearch. Installing in this order ensures that the components each product depends on are correctly in place.

      Because you’ve already added the Elastic package source in the previous step, you can just install the remaining components of the Elastic Stack using apt:

      Then enable and start the Kibana service:

      • sudo systemctl enable kibana
      • sudo systemctl start kibana

      Because Kibana is configured to only listen on localhost, we must set up a reverse proxy to allow external access to it. We will use Nginx for this purpose, which should already be installed on your server.

      First, use the openssl command to create an administrative Kibana user which you’ll use to access the Kibana web interface. As an example we will name this account kibanaadmin, but to ensure greater security we recommend that you choose a non-standard name for your user that would be difficult to guess.

      The following command will create the administrative Kibana user and password, and store them in the htpasswd.users file. You will configure Nginx to require this username and password and read this file momentarily:

      • echo "kibanaadmin:`openssl passwd -apr1`" | sudo tee -a /etc/nginx/htpasswd.users

      Enter and confirm a password at the prompt. Remember or take note of this login, as you will need it to access the Kibana web interface.

      Next, we will create an Nginx server block file. As an example, we will refer to this file as your_domain, although you may find it helpful to give yours a more descriptive name. For instance, if you have a FQDN and DNS records set up for this server, you could name this file after your FQDN.

      Using nano or your preferred text editor, create the Nginx server block file:

      • sudo nano /etc/nginx/sites-available/your_domain

      Add the following code block into the file, being sure to update your_domain to match your server’s FQDN or public IP address. This code configures Nginx to direct your server’s HTTP traffic to the Kibana application, which is listening on localhost:5601. Additionally, it configures Nginx to read the htpasswd.users file and require basic authentication.

      Note that if you followed the prerequisite Nginx tutorial through to the end, you may have already created this file and populated it with some content. In that case, delete all the existing content in the file before adding the following:

      /etc/nginx/sites-available/your_domain

      server {
          listen 80;
      
          server_name your_domain;
      
          auth_basic "Restricted Access";
          auth_basic_user_file /etc/nginx/htpasswd.users;
      
          location / {
              proxy_pass http://localhost:5601;
              proxy_http_version 1.1;
              proxy_set_header Upgrade $http_upgrade;
              proxy_set_header Connection 'upgrade';
              proxy_set_header Host $host;
              proxy_cache_bypass $http_upgrade;
          }
      }
      

      When you’re finished, save and close the file.

      Next, enable the new configuration by creating a symbolic link to the sites-enabled directory. If you already created a server block file with the same name in the Nginx prerequisite, you do not need to run this command:

      • sudo ln -s /etc/nginx/sites-available/your_domain /etc/nginx/sites-enabled/your_domain

      Then check the configuration for syntax errors:

      If any errors are reported in your output, go back and double check that the content you placed in your configuration file was added correctly. Once you see syntax is ok in the output, go ahead and restart the Nginx service:

      • sudo systemctl reload nginx

      If you followed the initial server setup guide, you should have a UFW firewall enabled. To allow connections to Nginx, we can adjust the rules by typing:

      • sudo ufw allow 'Nginx Full'

      Note: If you followed the prerequisite Nginx tutorial, you may have created a UFW rule allowing the Nginx HTTP profile through the firewall. Because the Nginx Full profile allows both HTTP and HTTPS traffic through the firewall, you can safely delete the rule you created in the prerequisite tutorial. Do so with the following command:

      • sudo ufw delete allow 'Nginx HTTP'

      Kibana is now accessible via your FQDN or the public IP address of your Elastic Stack server. You can check the Kibana server’s status page by navigating to the following address and entering your login credentials when prompted:

      http://your_domain/status
      

      This status page displays information about the server’s resource usage and lists the installed plugins.

      |Kibana status page

      Note: As mentioned in the Prerequisites section, it is recommended that you enable SSL/TLS on your server. You can follow the Let’s Encrypt guide now to obtain a free SSL certificate for Nginx on Ubuntu 20.04. After obtaining your SSL/TLS certificates, you can come back and complete this tutorial.

      Now that the Kibana dashboard is configured, let’s install the next component: Logstash.

      Step 3 — Installing and Configuring Logstash

      Although it’s possible for Beats to send data directly to the Elasticsearch database, it is common to use Logstash to process the data. This will allow you more flexibility to collect data from different sources, transform it into a common format, and export it to another database.

      Install Logstash with this command:

      • sudo apt install logstash

      After installing Logstash, you can move on to configuring it. Logstash’s configuration files reside in the /etc/logstash/conf.d directory. For more information on the configuration syntax, you can check out the configuration reference that Elastic provides. As you configure the file, it’s helpful to think of Logstash as a pipeline which takes in data at one end, processes it in one way or another, and sends it out to its destination (in this case, the destination being Elasticsearch). A Logstash pipeline has two required elements, input and output, and one optional element, filter. The input plugins consume data from a source, the filter plugins process the data, and the output plugins write the data to a destination.

      Logstash pipeline

      Create a configuration file called 02-beats-input.conf where you will set up your Filebeat input:

      • sudo nano /etc/logstash/conf.d/02-beats-input.conf

      Insert the following input configuration. This specifies a beats input that will listen on TCP port 5044.

      /etc/logstash/conf.d/02-beats-input.conf

      input {
        beats {
          port => 5044
        }
      }
      

      Save and close the file.

      Next, create a configuration file called 30-elasticsearch-output.conf:

      • sudo nano /etc/logstash/conf.d/30-elasticsearch-output.conf

      Insert the following output configuration. Essentially, this output configures Logstash to store the Beats data in Elasticsearch, which is running at localhost:9200, in an index named after the Beat used. The Beat used in this tutorial is Filebeat:

      /etc/logstash/conf.d/30-elasticsearch-output.conf

      output {
        if [@metadata][pipeline] {
          elasticsearch {
          hosts => ["localhost:9200"]
          manage_template => false
          index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
          pipeline => "%{[@metadata][pipeline]}"
          }
        } else {
          elasticsearch {
          hosts => ["localhost:9200"]
          manage_template => false
          index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
          }
        }
      }
      
      
      

      Save and close the file.

      Test your Logstash configuration with this command:

      • sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t

      If there are no syntax errors, your output will display Config Validation Result: OK. Exiting Logstash after a few seconds. If you don’t see this in your output, check for any errors noted in your output and update your configuration to correct them. Note that you will receive warnings from OpenJDK, but they should not cause any problems and can be ignored.

      If your configuration test is successful, start and enable Logstash to put the configuration changes into effect:

      • sudo systemctl start logstash
      • sudo systemctl enable logstash

      Now that Logstash is running correctly and is fully configured, let’s install Filebeat.

      Step 4 — Installing and Configuring Filebeat

      The Elastic Stack uses several lightweight data shippers called Beats to collect data from various sources and transport them to Logstash or Elasticsearch. Here are the Beats that are currently available from Elastic:

      • Filebeat: collects and ships log files.
      • Metricbeat: collects metrics from your systems and services.
      • Packetbeat: collects and analyzes network data.
      • Winlogbeat: collects Windows event logs.
      • Auditbeat: collects Linux audit framework data and monitors file integrity.
      • Heartbeat: monitors services for their availability with active probing.

      In this tutorial we will use Filebeat to forward local logs to our Elastic Stack.

      Install Filebeat using apt:

      • sudo apt install filebeat

      Next, configure Filebeat to connect to Logstash. Here, we will modify the example configuration file that comes with Filebeat.

      Open the Filebeat configuration file:

      • sudo nano /etc/filebeat/filebeat.yml

      Note: As with Elasticsearch, Filebeat’s configuration file is in YAML format. This means that proper indentation is crucial, so be sure to use the same number of spaces that are indicated in these instructions.

      Filebeat supports numerous outputs, but you’ll usually only send events directly to Elasticsearch or to Logstash for additional processing. In this tutorial, we’ll use Logstash to perform additional processing on the data collected by Filebeat. Filebeat will not need to send any data directly to Elasticsearch, so let’s disable that output. To do so, find the output.elasticsearch section and comment out the following lines by preceding them with a #:

      /etc/filebeat/filebeat.yml

      ...
      #output.elasticsearch:
        # Array of hosts to connect to.
        #hosts: ["localhost:9200"]
      ...
      

      Then, configure the output.logstash section. Uncomment the lines output.logstash: and hosts: ["localhost:5044"] by removing the #. This will configure Filebeat to connect to Logstash on your Elastic Stack server at port 5044, the port for which we specified a Logstash input earlier:

      /etc/filebeat/filebeat.yml

      output.logstash:
        # The Logstash hosts
        hosts: ["localhost:5044"]
      

      Save and close the file.

      The functionality of Filebeat can be extended with Filebeat modules. In this tutorial we will use the system module, which collects and parses logs created by the system logging service of common Linux distributions.

      Let’s enable it:

      • sudo filebeat modules enable system

      You can see a list of enabled and disabled modules by running:

      • sudo filebeat modules list

      You will see a list similar to the following:

      Output

      Enabled: system Disabled: apache2 auditd elasticsearch icinga iis kafka kibana logstash mongodb mysql nginx osquery postgresql redis traefik ...

      By default, Filebeat is configured to use default paths for the syslog and authorization logs. In the case of this tutorial, you do not need to change anything in the configuration. You can see the parameters of the module in the /etc/filebeat/modules.d/system.yml configuration file.

      Next, we need to set up the Filebeat ingest pipelines, which parse the log data before sending it through logstash to Elasticsearch. To load the ingest pipeline for the system module, enter the following command:

      • sudo filebeat setup --pipelines --modules system

      Next, load the index template into Elasticsearch. An Elasticsearch index is a collection of documents that have similar characteristics. Indexes are identified with a name, which is used to refer to the index when performing various operations within it. The index template will be automatically applied when a new index is created.

      To load the template, use the following command:

      • sudo filebeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

      Output

      Index setup finished.

      Filebeat comes packaged with sample Kibana dashboards that allow you to visualize Filebeat data in Kibana. Before you can use the dashboards, you need to create the index pattern and load the dashboards into Kibana.

      As the dashboards load, Filebeat connects to Elasticsearch to check version information. To load dashboards when Logstash is enabled, you need to disable the Logstash output and enable Elasticsearch output:

      • sudo filebeat setup -E output.logstash.enabled=false -E output.elasticsearch.hosts=['localhost:9200'] -E setup.kibana.host=localhost:5601

      You should receive output similar to this:

      Output

      Overwriting ILM policy is disabled. Set `setup.ilm.overwrite:true` for enabling. Index setup finished. Loading dashboards (Kibana must be running and reachable) Loaded dashboards Setting up ML using setup --machine-learning is going to be removed in 8.0.0. Please use the ML app instead. See more: https://www.elastic.co/guide/en/elastic-stack-overview/current/xpack-ml.html Loaded machine learning job configurations Loaded Ingest pipelines

      Now you can start and enable Filebeat:

      • sudo systemctl start filebeat
      • sudo systemctl enable filebeat

      If you’ve set up your Elastic Stack correctly, Filebeat will begin shipping your syslog and authorization logs to Logstash, which will then load that data into Elasticsearch.

      To verify that Elasticsearch is indeed receiving this data, query the Filebeat index with this command:

      • curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'

      You should receive output similar to this:

      Output

      ... { { "took" : 4, "timed_out" : false, "_shards" : { "total" : 2, "successful" : 2, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 4040, "relation" : "eq" }, "max_score" : 1.0, "hits" : [ { "_index" : "filebeat-7.7.1-2020.06.04", "_type" : "_doc", "_id" : "FiZLgXIB75I8Lxc9ewIH", "_score" : 1.0, "_source" : { "cloud" : { "provider" : "digitalocean", "instance" : { "id" : "194878454" }, "region" : "nyc1" }, "@timestamp" : "2020-06-04T21:45:03.995Z", "agent" : { "version" : "7.7.1", "type" : "filebeat", "ephemeral_id" : "cbcefb9a-8d15-4ce4-bad4-962a80371ec0", "hostname" : "june-ubuntu-20-04-elasticstack", "id" : "fbd5956f-12ab-4227-9782-f8f1a19b7f32" }, ...

      If your output shows 0 total hits, Elasticsearch is not loading any logs under the index you searched for, and you will need to review your setup for errors. If you received the expected output, continue to the next step, in which we will see how to navigate through some of Kibana’s dashboards.

      Step 5 — Exploring Kibana Dashboards

      Let’s return to the Kibana web interface that we installed earlier.

      In a web browser, go to the FQDN or public IP address of your Elastic Stack server. If your session has been interrupted, you will need to re-enter entering the credentials you defined in Step 2. Once you have logged in, you should receive the Kibana homepage:

      Kibana Homepage

      Click the Discover link in the left-hand navigation bar (you may have to click the the Expand icon at the very bottom left to see the navigation menu items). On the Discover page, select the predefined filebeat-* index pattern to see Filebeat data. By default, this will show you all of the log data over the last 15 minutes. You will see a histogram with log events, and some log messages below:

      Discover page

      Here, you can search and browse through your logs and also customize your dashboard. At this point, though, there won’t be much in there because you are only gathering syslogs from your Elastic Stack server.

      Use the left-hand panel to navigate to the Dashboard page and search for the Filebeat System dashboards. Once there, you can select the sample dashboards that come with Filebeat’s system module.

      For example, you can view detailed stats based on your syslog messages:

      Syslog Dashboard

      You can also view which users have used the sudo command and when:

      Sudo Dashboard

      Kibana has many other features, such as graphing and filtering, so feel free to explore.

      Conclusion

      In this tutorial, you’ve learned how to install and configure the Elastic Stack to collect and analyze system logs. Remember that you can send just about any type of log or indexed data to Logstash using Beats, but the data becomes even more useful if it is parsed and structured with a Logstash filter, as this transforms the data into a consistent format that can be read easily by Elasticsearch.



      Source link

      Cómo instalar y configurar Elasticsearch en Ubuntu 18.04


      Introducción

      Elasticsearch es una plataforma para la búsqueda y el análisis distribuidos de datos en tiempo real.  Es una opción popular debido a su facilidad de uso, sus funciones potentes y su escalabilidad.

      En este artículo, se explicará cómo instalar Elasticsearch, configurarla para su caso de uso, proteger su instalación y comenzar a trabajar con su servidor de Elasticsearch.

      Requisitos previos

      Antes de comenzar con este tutorial, necesitará lo siguiente:

      En este tutorial, trabajaremos con la cantidad mínima de CPU y RAM requerida para ejecutar Elasticsearch. Tenga en cuenta que la cantidad de CPU, RAM y almacenamiento que su servidor de Elasticsearch requiera dependerá del volumen de registros que prevé.

      Paso 1: Instalar y configurar Elasticsearch

      Los componentes de Elasticsearch no están disponibles en los repositorios de paquetes predeterminados de Ubuntu. Sin embargo, pueden instalarse con APT una vez que agregue la lista de fuentes de paquetes de Elastic.

      Todos los paquetes de Elasticsearch están firmados con la clave de firma de Elasticsearch para proteger su sistema contra la suplantación de paquetes. Su administrador de paquetes considerará confiables los paquetes autenticados con la clave. En este paso, importará la clave GPG pública de Elasticsearch y agregará la lista de fuentes de paquetes de Elastic para instalar Elasticsearch.

      Para comenzar, utilice cURL, la herramienta de línea de comandos para transferir datos con URL, para importar la clave GPG pública de Elasticsearch a APT. Tenga en cuenta que estamos usando los argumentos -fsSL para silenciar todos los progresos y posibles errores (excepto los de errores del servidor) y para permitir a cURL hacer una solicitud en una ubicación nueva si se redirige.  Canalice el resultado del comando cURL al programa apt-key, que añade la clave GPG pública a APT.

      • curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

      A continuación, agregue la lista de fuentes de Elastic al directorio sources.list.d, donde APT buscará nuevas fuentes:

      • echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

      A continuación, actualice sus listas de paquetes para que APT lea la nueva fuente de Elastic:

      Luego, instale Elasticsearch con este comando:

      • sudo apt install elasticsearch

      Ahora, Elasticsearch está instalada y lista para usarse.

      Paso 2: Configurar Elasticsearch

      Para configurar Elasticsearch, editaremos sus archivos de configuración. Elasticsearch tiene tres archivos de configuración:

      • elasticsearch.yml se utiliza para configurar Elasticsearch, el archivo de configuración principal
      • jvm.options se usa para configurar los ajustes de la máquina virtual de Java (JVM) de Elasticsearch
      • log4j2.properties se utiliza para configurar el registro de Elasticsearch

      En este tutorial, nos interesa el archivo elasticsearch.yml, donde se almacenan la mayoría de las opciones de configuración. Este archivo se encuentra en el directorio /etc/elasticsearch.

      Utilice su editor de texto preferido para editar el archivo de configuración de Elasticsearch. En este caso, utilizaremos nano:

      • sudo nano /etc/elasticsearch/elasticsearch.yml

      Nota: El archivo de configuración de Elasticsearch se encuentra en formato YAML, lo que significa que debemos mantener el formato de sangrías. Asegúrese de no añadir espacios adicionales al editar este archivo.

      El archivo elasticsearch.yml ofrece opciones de configuración para su clúster, nodo, rutas, memoria, red, descubrimiento y puerta de enlace. La mayoría de estas opciones están preconfiguradas en el archivo, pero las puede cambiar según sus necesidades. Para los fines de nuestra demostración de una configuración de un solo servidor, modificaremos únicamente la configuración del host de red.

      Elasticsearch escucha el tráfico de todos los lugares en el puerto 9200. Es conveniente restringir el acceso externo a su instancia de Elasticsearch para evitar que terceros lean sus datos o cierren su clúster de Elasticsearch a través de su API REST.  Para restringir el acceso y, por lo tanto, aumentar la seguridad, busque la línea que especifica network.host, elimine los comentarios reemplace su valor por localhost para que tenga el siguiente aspecto:

      /etc/elasticsearch/elasticsearch.yml

      . . .
      # ---------------------------------- Network -----------------------------------
      #
      # Set the bind address to a specific IP (IPv4 or IPv6):
      #
      network.host: localhost
      . . .
      

      Hemos especificado localhost para que Elasticsearch escuche en todas las interfaces y las IP vinculadas. Si desea que escuche únicamente en una interfaz específica, puede especificar su IP en lugar de localhost. Guarde y cierre elasticsearch.yml. Si utiliza nano, puede hacerlo al pulsar CTRL+X, seguido de Y y, luego, ENTER.

      Estos son los ajustes mínimos con los que puede comenzar para usar Elasticsearch. Ahora, puede iniciar Elasticsearch por primera vez.

      Inicie el servicio de Elasticsearch con systemctl. Elasticsearch tardará unos minutos en iniciarse. Espere, de lo contrario, es posible que reciba errores de que no se puede conectar.

      • sudo systemctl start elasticsearch

      Luego, ejecute el siguiente comando para permitir que Elasticsearch se cargue cada vez que su servidor se inicie:

      • sudo systemctl enable elasticsearch

      Ahora que Elasticsearch está habilitado en el inicio, pasemos al siguiente paso para hablar sobre la seguridad.

      Paso 3: Proteger Elasticsearch

      Por defecto, cualquier persona que pueda acceder a la API HTTP puede controlar Elasticsearch. Esto no siempre representa un riesgo de seguridad, dado que Elasticsearch escucha únicamente en la interfaz de bucle invertido (es decir, 127.0.0.1), a la que solo se puede acceder de forma local. Por lo tanto, el acceso público no es posible, y siempre y cuando todos los usuarios del servidor sean confiables, probablemente, la seguridad no sea una preocupación importante.

      Si necesita permitir el acceso remoto a la API HTTP, puede limitar la exposición de la red con el firewall predeterminado de Ubuntu, UFW. Ya debería tener este firewall habilitado si siguió los pasos del tutorial Configuración inicial de servidores para Ubuntu 18.04 indicado en los requisitos previos.

      Ahora, configuraremos el firewall para que permita el acceso al puerto de la API HTTP predeterminada de Elasticsearch (TCP 9200) para el host remoto de confianza que, en general, es el servidor que utiliza en una configuración de un solo servidor, como 198.51.100.0. Para permitir el acceso, escriba el siguiente comando:

      • sudo ufw allow from 198.51.100.0 to any port 9200

      Cuando se complete, puede habilitar UFW con el siguiente comando:

      Por último, revise el estado de UFW con el comando que se indica a continuación:

      Si especificó las reglas correctamente, el resultado debería tener el siguiente aspecto:

      Output

      Status: active To Action From -- ------ ---- 9200 ALLOW 198.51.100.0 22 ALLOW Anywhere 22 (v6) ALLOW Anywhere (v6)

      Ahora, el UFW debería estar habilitado y configurado para proteger el puerto 9200 de Elasticsearch.

      Si desea invertir en protección adicional, Elasticsearch ofrece el complemento Shield comercial.

      Paso 4: Probar Elasticsearch

      Ahora, Elasticsearch debe estar en ejecución en el puerto 9200. Puede probarla con cURL y una solicitud GET.

      • curl -X GET 'http://localhost:9200'

      Debería ver la siguiente respuesta:

      Output

      { "node.name" : "My First Node", "cluster.name" : "mycluster1", "version" : { "number" : "2.3.1", "build_hash" : "bd980929010aef404e7cb0843e61d0665269fc39", "build_timestamp" : "2020-04-04T12:25:05Z", "build_snapshot" : false, "lucene_version" : "5.5.0" }, "tagline" : "You Know, for Search" }

      Si ve una respuesta similar a la anterior, Elasticsearch funciona correctamente. De lo contrario, asegúrese de haber seguido correctamente las instrucciones de instalación y haber esperado un tiempo para que Elasticsearch se inicie por completo.

      Para realizar una verificación más completa de Elasticsearch, ejecute el siguiente comando:

      • curl -XGET 'http://localhost:9200/_nodes?pretty'

      En el resultado del comando anterior, puede verificar todos los ajustes actuales del nodo, el clúster, las rutas de las aplicaciones, los módulos y más.

      Paso 5: Usar Elasticsearch

      Para comenzar a usar Elasticsearch, primero, añadiremos algunos datos. Elasticsearch utiliza una API RESTful, que responde a los comandos habituales de CRUD: create, read, update y delete. Para usarlo, utilizaremos el comando cURL de nuevo.

      Puede añadir su primera entrada de la siguiente manera:

      • curl -XPOST -H "Content-Type: application/json" 'http://localhost:9200/tutorial/helloworld/1' -d '{ "message": "Hello World!" }'

      Debería recibir la siguiente respuesta:

      Output

      {"_index":"tutorial","_type":"helloworld","_id":"1","_version":2,"result":"updated","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":1,"_primary_term":1}

      Con cURL, enviamos una solicitud HTTP POST al servidor de Elasticsearch. La URI de la solicitud era /tutorial/helloworld/1 con varios parámetros:

      • tutorial es el índice de los datos de Elasticsearch.
      • helloworld es el tipo.
      • 1 es la ID de nuestra entrada bajo el índice y el tipo anteriores.

      Puede recuperar esta primera entrada con una solicitud HTTP GET.

      • curl -X GET -H "Content-Type: application/json" 'http://localhost:9200/tutorial/helloworld/1' -d '{ "message": "Hello World!" }'

      El resultado debería ser el siguiente:

      Output

      {"_index":"tutorial","_type":"helloworld","_id":"1","_version":1,"found":true,"_source":{ "message": "Hello, World!" }}

      Para modificar una entrada existente, puede usar una solicitud HTTP PUT.

      • curl -X PUT -H "Content-Type: application/json" 'localhost:9200/tutorial/helloworld/1?pretty' -d '
      • {
      • "message": "Hello, People!"
      • }'

      Elasticsearch debe reconocer la modificación correcta de esta manera:

      Output

      { "_index" : "tutorial", "_type" : "helloworld", "_id" : "1", "_version" : 2, "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "created" : false }

      En el ejemplo anterior, modificamos message de la primera entrada a “Hello, People!”. Con eso, el número de versión se aumentó automáticamente a 2.

      Tal vez haya observado el argumento adicional pretty en la solicitud anterior. Habilita un formato de lenguaje natural para que pueda escribir cada campo de datos en una fila nueva. También puede aplicarlo a sus resultados cuando obtiene datos para que sean más legibles al ingresar el siguiente comando:

      • curl -X GET -H "Content-Type: application/json" 'http://localhost:9200/tutorial/helloworld/1?pretty'

      Ahora, la respuesta tendrá un formato que permite que un humano la analice:

      Output

      { "_index" : "tutorial", "_type" : "helloworld", "_id" : "1", "_version" : 2, "found" : true, "_source" : { "message" : "Hello, People!" } }

      En este punto, añadimos y consultamos datos en Elasticsearch. Para obtener más información sobre las demás operaciones, consulte la documentación de la API.

      Conclusión

      Instaló, configuró y comenzó a usar Elasticsearch. Desde la versión original de Elasticsearch, Elastic desarrolló tres herramientas adicionales, Logstash, Kabana y Beats, que se utilizan en conjunto con Elasticsearch como parte de la pila de Elastic. Utilizadas en conjunto, estas herramientas le permiten buscar, analizar y visualizar registros generados desde cualquier fuente y en cualquier formato en lo que se conoce como “registro centralizado”. Para comenzar a usar la pila de Elastic en Ubuntu 18.04, consulte nuestra guía Cómo instalar Elasticsearch, Logstash y Kibana (la pila de Elastic) en Ubuntu 18.04.



      Source link

      Установка и настройка Elasticsearch в Ubuntu 20.04


      Введение

      Elasticsearch — это платформа для распределенного поиска и анализа данных в режиме реального времени. Она пользуется популярностью благодаря удобству в использовании, наличию мощных характеристик и возможности масштабирования.

      Эта статья расскажет вам о том, как установить Elasticsearch, настроить платформу под ваш вариант использования, обеспечить безопасность установки и начать работу с вашим сервером Elasticsearch.

      Предварительные требования

      Для работы с этим обучающим руководством вам потребуется следующее:

      • Сервер Ubuntu 20.04 с 4 ГБ оперативной памяти и 2 процессорами, а также настроенный пользователь без прав root с привилегиями sudo. Вы можете это сделать, воспользовавшись рекомендациями по начальной настройке сервера с Ubuntu 20.04.

      • Установленный OpenJDK 11.

      В этом обучающем руководстве мы будем использовать минимальное количество процессоров и оперативной памяти, необходимое для работы с Elasticsearch. Обратите внимание, что требования сервера Elasticsearch к количеству процессоров, оперативной памяти и системе хранения данных зависят от ожидаемого объема журналов.

      Шаг 1 — Установка и настройка Elasticsearch

      Компоненты Elasticsearch отсутствуют в репозиториях пакетов Ubuntu по умолчанию. Однако их можно установить с помощью APT после добавления списка источников пакетов Elastic.

      Все пакеты подписаны ключом подписи Elasticsearch для защиты вашей системы от поддельных пакетов. Ваш диспетчер пакетов будет считать надежными пакеты, для которых проведена аутентификация с помощью ключа. На этом шаге вы импортируете открытый ключ Elasticsearch GPG и добавить список источников пакетов Elastic для установки Elasticsearch.

      Для начала используйте cURL, инструмент командной строки для передачи данных с помощью URL, для импорта открытого ключа Elasticsearch GPG в APT. Обратите внимание, что мы используем аргументы -fsSL для подавления всех текущих и возможных ошибок (кроме сбоя сервера), а также чтобы разрешить cURL подать запрос на другой локации при переадресации. Выведите результаты команды cURL в программу apt-key, которая добавит открытый ключ GPG в APT.

      • curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

      Затем добавьте список источников Elastic в директорию sources.list.d, где APT будет искать новые источники:

      • echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

      Затем обновите списки пакетов, чтобы APT мог прочитать новый источник Elastic:

      Установите Elasticsearch с помощью следующей команды:

      • sudo apt install elasticsearch

      Теперь система Elasticsearch установлена и готова к настройке.

      Шаг 2 — Настройка Elasticsearch

      Для настройки Elasticsearch мы изменим ее основной файл конфигурации elasticsearch.yml, где хранится большинство вариантов конфигурации. Этот файл находится в директории /etc/elasticsearch.

      Используйте текстовый редактор на ваш выбор для изменения файла конфигурации Elasticsearch. Мы будем использовать nano:

      • sudo nano /etc/elasticsearch/elasticsearch.yml

      Примечание. Файл конфигурации Elasticsearch представлен в формате YAML. Это означает, что нам нужно сохранить формат отступов. Не добавляйте никакие дополнительные пробелы при редактировании этого файла.

      Файл elasticsearch.yml предоставляет варианты конфигурации для вашего кластера, узла, пути, памяти, сети, обнаружения и шлюза. Большинство из этих вариантов уже настроены в файле, но вы можете изменить их в соответствии с вашими потребностями. В нашем случае для демонстрации односерверной конфигурации мы будем регулировать настройки только для хоста сети.

      Elasticsearch прослушивает весь трафик порта 9200. По желанию вы можете ограничить внешний доступ к вашему экземпляру Elasticsearch, чтобы посторонние не смогли прочесть ваши данные или отключить ваш кластер Elasticsearch через REST API. Для ограничения доступа и повышения безопасности найдите строку с указанием network.host, уберите с нее значок комментария и замените значение на localhost, чтобы она выглядела следующим образом:

      /etc/elasticsearch/elasticsearch.yml

      . . .
      # ---------------------------------- Network -----------------------------------
      #
      # Set the bind address to a specific IP (IPv4 or IPv6):
      #
      network.host: localhost
      . . .
      

      Мы указали localhost, и теперь Elasticsearch прослушивает все интерфейсы и связанные IP-адреса. Если вы хотите, чтобы прослушивался только конкретный интерфейс, вы можете указать его IP-адрес вместо localhost. Сохраните и закройте elasticsearch.yml. Если вы используете nano, вы можете сделать это, нажав CTRL+X, затем Y, а затем ENTER.

      Это минимальные настройки, с которыми вы можете начинать использовать Elasticsearch. Теперь вы можете запустить Elasticsearch в первый раз.

      Запустите службу Elasticsearch с помощью systemctl. Запуск Elasticsearch может занять некоторое время. В другом случае вы можете увидеть сообщение об ошибке подключения.

      • sudo systemctl start elasticsearch

      Затем запустите следующую команду, чтобы активировать Elasticsearch при каждой загрузке сервера:

      • sudo systemctl enable elasticsearch

      После активации Elasticsearch можно переходить к следующему шагу, где будет обсуждаться вопрос защиты.

      Шаг 3 — Защита Elasticsearch

      По умолчанию любой пользователь, который имеет доступ к HTTP API может контролировать Elasticsearch. Это не всегда связано с риском для безопасности, так как Elasticsearch прослушивает только циклический интерфейс (имеется в виду 127.0.0.1), доступ к которому только локальный. Таким образом, невозможно получить публичный доступ к серверу, и, пока все пользователи сервера являются проверенными, вопрос безопасности не будет для вас серьезной проблемой.

      Если вам потребуется разрешить удаленный доступ к HTTP API, вы можете ограничить открытость сети с помощью настроек брандмауэра Ubuntu по умолчанию, UFW. Этот брандмауэр уже должен быть активирован, если вы выполнили все предварительные шаги по начальной настройке сервера с Ubuntu 20.04.

      Теперь мы настроим брандмауэр для доступа к порту HTTP API Elasticsearch по умолчанию (TCP 9200) для доверенного удаленного хоста. Как правило, это сервер, который вы используете при настройке на одном сервере, например 198.51.100.0. Для доступа введите следующую команду:

      • sudo ufw allow from 198.51.100.0 to any port 9200

      После этого вы можете активировать UFW с помощью команды:

      В заключение проверьте статус UFW с помощью следующей команды:

      Если вы правильно указали правила, вы должны получить следующий результат:

      Output

      Status: active To Action From -- ------ ---- 9200 ALLOW 198.51.100.0 22 ALLOW Anywhere 22 (v6) ALLOW Anywhere (v6)

      Теперь UFW должен быть активирован и настроен на защиту порта Elasticsearch 9200.

      Если вы хотите инвестировать в дополнительную защиту, Elasticsearch предлагает к покупке платный плагин Shield.

      Шаг 4 — Тестирование Elasticsearch

      Сейчас система Elasticsearch должна работать на порту 9200. Вы можете протестировать ее с помощью cURL и запроса GET.

      • curl -X GET 'http://localhost:9200'

      Вы должны получить следующий ответ:

      Output

      { "name" : "elasticsearch-ubuntu20-04", "cluster_name" : "elasticsearch", "cluster_uuid" : "qqhFHPigQ9e2lk-a7AvLNQ", "version" : { "number" : "7.6.2", "build_flavor" : "default", "build_type" : "deb", "build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f", "build_date" : "2020-03-26T06:34:37.794943Z", "build_snapshot" : false, "lucene_version" : "8.4.0", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_version" : "6.0.0-beta1" }, "tagline" : "You Know, for Search" }

      Если вы получите ответ, аналогичный вышеуказанному, значит Elasticsearch работает корректно. Если нет, убедитесь, что вы правильно выполнили инструкции по установке и дали время системе Elasticsearch для полного запуска.

      Для более тщательной проверки Elasticsearch выполните следующую команду:

      • curl -XGET 'http://localhost:9200/_nodes?pretty'

      В выводе для команды, указанной выше, вы можете проверить все текущие настройки для узла, кластера, путей приложения, модулей и т. д.

      Шаг 5 — Использование Elasticsearch

      Чтобы начать использовать Elasticsearch, в первую очередь нужно добавить некоторые данные. Elasticsearch использует RESTful API, который соответствует обычным командам CRUD: create, read, update и delete. Для работы мы снова используем команду cURL.

      Ваша первая запись может выглядеть так:

      • curl -XPOST -H "Content-Type: application/json" 'http://localhost:9200/tutorial/helloworld/1' -d '{ "message": "Hello World!" }'

      Вы должны получить следующий ответ:

      Output

      {"_index":"tutorial","_type":"helloworld","_id":"1","_version":2,"result":"updated","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":1,"_primary_term":1}

      С помощью cURL мы отправили запрос HTTP POST на сервер Elasticsearch. URI запроса — /tutorial/helloworld/1​​ с несколькими параметрами:

      • tutorial — это индекс данных в Elasticsearch.
      • helloworld — это тип.
      • 1 — это ID нашей записи по индексу и типу.

      Вы можете получить эту первую запись по запросу HTTP GET.

      • curl -X GET -H "Content-Type: application/json" 'http://localhost:9200/tutorial/helloworld/1' -d '{ "message": "Hello World!" }'

      Вывод должен выглядеть следующим образом:

      Output

      {"_index":"tutorial","_type":"helloworld","_id":"1","_version":1,"found":true,"_source":{ "message": "Hello, World!" }}

      Для изменения существующей записи вы можете использовать запрос HTTP PUT.

      • curl -X PUT -H "Content-Type: application/json" 'localhost:9200/tutorial/helloworld/1?pretty' -d '
      • {
      • "message": "Hello, People!"
      • }'

      Elasticsearch должна признать успешное изменение следующим образом:

      Output

      { "_index" : "tutorial", "_type" : "helloworld", "_id" : "1", "_version" : 2, "result" : "updated", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 1, "_primary_term" : 1 }

      В примере, представленном выше, мы изменили message первой записи на «Hello, People!». При этом номер версии автоматически увеличился до 2.

      Возможно, вы заметили дополнительный аргумент pretty в представленном выше запросе. Он обеспечивает удобный для восприятия человеком формат, и вы можете для написания каждого поля данных использовать новый ряд. Вы также можете «приукрасить» ваши результаты при получении данных, чтобы получить более читабельный вывод, путем введения следующей команды:

      • curl -X GET -H "Content-Type: application/json" 'http://localhost:9200/tutorial/helloworld/1?pretty'

      Теперь ответ отформатирован так, чтобы синтаксис был удобен для человека:

      Output

      { "_index" : "tutorial", "_type" : "helloworld", "_id" : "1", "_version" : 2, "_seq_no" : 1, "_primary_term" : 1, "found" : true, "_source" : { "message" : "Hello, People!" } } }

      Мы добавили и запросили данные в Elasticsearch. Информацию о других операциях можно найти в документации API.

      Заключение

      Вы установили, настроили и начали использовать Elasticsearch. Чтобы продолжить изучение функций Elasticsearch, ознакомьтесь с официальной документацией Elasticsearch.



      Source link