One place for hosting & domains

      Elastic

      How To Analyze Managed PostgreSQL Database Statistics Using the Elastic Stack on Ubuntu 18.04


      The author selected the Free and Open Source Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      Database monitoring is the continuous process of systematically tracking various metrics that show how the database is performing. By observing the performance data, you can gain valuable insights and identify possible bottlenecks, as well as find additional ways of improving database performance. Such systems often implement alerting, which notifies administrators when things go wrong. Gathered statistics can be used to not only improve the configuration and workflow of the database, but also those of client applications.

      The benefit of using the Elastic Stack (ELK stack) for monitoring your managed database is its excellent support for searching and the ability to ingest new data very quickly. It does not excel at updating the data, but this trade off is acceptable for monitoring and logging purposes, where past data is almost never changed. Elasticsearch offers powerful means of querying the data, which you can use through Kibana to get a better understanding of how the database fares through different time periods. This will allow you to correlate database load with real-life events to gain insight into how the database is being used.

      In this tutorial, you’ll import database metrics, generated by the PostgreSQL statistics collector, into Elasticsearch via Logstash. This entails configuring Logstash to pull data from the database using the PostgreSQL JDBC connector to send it to Elasticsearch for indexing immediately afterward. The imported data can later be analyzed and visualized in Kibana. Then, if your database is brand new, you’ll use pgbench, a PostgreSQL benchmarking tool, to create more interesting visualizations. In the end, you’ll have an automated system pulling in PostgreSQL statistics for later analysis.

      Prerequisites

      Step 1 — Setting up Logstash and the PostgreSQL JDBC Driver

      In this section, you will install Logstash and download the PostgreSQL JDBC driver so that Logstash will be able to connect to your managed database.

      Start off by installing Logstash with the following command:

      • sudo apt install logstash -y

      Once Logstash is installed, enable the service to automatically start on boot:

      • sudo systemctl enable logstash

      Logstash is written in Java, so in order to connect to PostgreSQL it requires the PostgreSQL JDBC (Java Database Connectivity) library to be available on the system it is running on. Because of an internal limitation, Logstash will properly load the library only if it is found under the /usr/share/logstash/logstash-core/lib/jars directory, where it stores third-party libraries it uses.

      Head over to the download page of the JDBC library and copy the link to latest version. Then, download it using curl by running the following command:

      • sudo curl https://jdbc.postgresql.org/download/postgresql-42.2.6.jar -o /usr/share/logstash/logstash-core/lib/jars/postgresql-jdbc.jar

      At the time of writing, the latest version of the library was 42.2.6, with Java 8 as the supported runtime version. Ensure you download the latest version; pairing it with the correct Java version that both JDBC and Logstash support.

      Logstash stores its configuration files under /etc/logstash/conf.d, and is itself stored under /usr/share/logstash/bin. Before you create a configuration that will pull statistics from your database, you’ll need to enable the JDBC plugin in Logstash by running the following command:

      • sudo /usr/share/logstash/bin/logstash-plugin install logstash-input-jdbc

      You’ve installed Logstash using apt and downloaded the PostgreSQL JDBC library so that Logstash can use it to connect to your managed database. In the next step, you will configure Logstash to pull statistical data from it.

      Step 2 — Configuring Logstash To Pull Statistics

      In this section, you will configure Logstash to pull metrics from your managed PostgreSQL database.

      You’ll configure Logstash to watch over three system databases in PostgreSQL, namely:

      • pg_stat_database: provides statistics about each database, including its name, number of connections, transactions, rollbacks, rows returned by querying the database, deadlocks, and so on. It has a stats_reset field, which specifies when the statistics were last reset.
      • pg_stat_user_tables: provides statistics about each table created by the user, such as the number of inserted, deleted, and updated rows.
      • pg_stat_user_indexes: collects data about all indexes in user-created tables, such as the number of times a particular index has been scanned.

      You’ll store the configuration for indexing PostgreSQL statistics in Elasticsearch in a file named postgresql.conf under the /etc/logstash/conf.d directory, where Logstash stores configuration files. When started as a service, it will automatically run them in the background.

      Create postgresql.conf using your favorite editor (for example, nano):

      • sudo nano /etc/logstash/conf.d/postgresql.conf

      Add the following lines:

      /etc/logstash/conf.d/postgresql.conf

      input {
              # pg_stat_database
              jdbc {
                      jdbc_driver_library => ""
                      jdbc_driver_class => "org.postgresql.Driver"
                      jdbc_connection_string => "jdbc:postgresql://host:port/defaultdb"
                      jdbc_user => "username"
                      jdbc_password => "password"
                      statement => "SELECT * FROM pg_stat_database"
                      schedule => "* * * * *"
                      type => "pg_stat_database"
              }
      
              # pg_stat_user_tables
              jdbc {
                      jdbc_driver_library => ""
                      jdbc_driver_class => "org.postgresql.Driver"
                      jdbc_connection_string => "jdbc:postgresql://host:port/defaultdb"
                      jdbc_user => "username"
                      jdbc_password => "password"
                      statement => "SELECT * FROM pg_stat_user_tables"
                      schedule => "* * * * *"
                      type => "pg_stat_user_tables"
              }
      
              # pg_stat_user_indexes
              jdbc {
                      jdbc_driver_library => ""
                      jdbc_driver_class => "org.postgresql.Driver"
                      jdbc_connection_string => "jdbc:postgresql://host:port/defaultdb"
                      jdbc_user => "username"
                      jdbc_password => "password"
                      statement => "SELECT * FROM pg_stat_user_indexes"
                      schedule => "* * * * *"
                      type => "pg_stat_user_indexes"
              }
      }
      
      output {
              elasticsearch {
                      hosts => "http://localhost:9200"
                      index => "%{type}"
              }
      }
      

      Remember to replace host with your host address, port with the port to which you can connect to your database, username with the database user username, and password with its password. All these values can be found in the Control Panel of your managed database.

      In this configuration, you define three JDBC inputs and one Elasticsearch output. The three inputs pull data from the pg_stat_database, pg_stat_user_tables, and pg_stat_user_indexes databases, respectively. They all set the jdbc_driver_library parameter to an empty string, because the PostgreSQL JDBC library is in a folder that Logstash automatically loads.

      Then, they set the jdbc_driver_class, whose value is specific to the JDBC library, and provide a jdbc_connection_string, which details how to connect to the database. The jdbc: part signifies that it is a JDBC connection, while postgres:// indicates that the target database is PostgreSQL. Next come the host and port of the database, and after the forward slash you also specify a database to connect to; this is because PostgreSQL requires you to be connected to a database to be able to issue any queries. Here, it is set to the default database that always exists and can not be deleted, aptly named defaultdb.

      Next, they set a username and password of the user through which the database will be accessed. The statement parameter contains a SQL query that should return the data you wish to process—in this configuration, it selects all rows from the appropriate database.

      The schedule parameter accepts a string in cron syntax that defines when Logstash should run this input; omitting it completely will make Logstash run it only once. Specifying * * * * *, as you have done so here, will tell Logstash to run it every minute. You can specify your own cron string if you want to collect data at different intervals.

      There is only one output, which accepts data from three inputs. They all send data to Elasticsearch, which is running locally and is reachable at http://localhost:9200. The index parameter defines to which Elasticsearch index it will send the data, and its value is passed in from the type field of the input.

      When you are done with editing, save and close the file.

      You’ve configured Logstash to gather data from various PostgreSQL statistical tables and send them to Elasticsearch for storage and indexing. Next, you’ll run Logstash to test the configuration.

      Step 3 — Testing the Logstash Configuration

      In this section, you will test the configuration by running Logstash to verify it will properly pull the data. Then, you will make this configuration run in the background by configuring it as a Logstash pipeline.

      Logstash supports running a specific configuration by passing its file path to the -f parameter. Run the following command to test your new configuration from the last step:

      • sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/postgresql.conf

      It may take some time before it shows any output, which will look similar to this:

      Output

      Thread.exclusive is deprecated, use Thread::Mutex WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2019-08-02 18:29:15.123 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO ] 2019-08-02 18:29:15.154 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.3.0"} [INFO ] 2019-08-02 18:29:18.209 [Converge PipelineAction::Create<main>] Reflections - Reflections took 77 ms to scan 1 urls, producing 19 keys and 39 values [INFO ] 2019-08-02 18:29:20.195 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}} [WARN ] 2019-08-02 18:29:20.667 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://localhost:9200/"} [INFO ] 2019-08-02 18:29:21.221 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7} [WARN ] 2019-08-02 18:29:21.230 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7} [INFO ] 2019-08-02 18:29:21.274 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]} [INFO ] 2019-08-02 18:29:21.337 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}} [WARN ] 2019-08-02 18:29:21.369 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://localhost:9200/"} [INFO ] 2019-08-02 18:29:21.386 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7} [WARN ] 2019-08-02 18:29:21.386 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7} [INFO ] 2019-08-02 18:29:21.409 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]} [INFO ] 2019-08-02 18:29:21.430 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}} [WARN ] 2019-08-02 18:29:21.444 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://localhost:9200/"} [INFO ] 2019-08-02 18:29:21.465 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7} [WARN ] 2019-08-02 18:29:21.466 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7} [INFO ] 2019-08-02 18:29:21.468 [Ruby-0-Thread-7: :1] elasticsearch - Using default mapping template [INFO ] 2019-08-02 18:29:21.538 [Ruby-0-Thread-5: :1] elasticsearch - Using default mapping template [INFO ] 2019-08-02 18:29:21.545 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]} [INFO ] 2019-08-02 18:29:21.589 [Ruby-0-Thread-9: :1] elasticsearch - Using default mapping template [INFO ] 2019-08-02 18:29:21.696 [Ruby-0-Thread-5: :1] elasticsearch - Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}} [INFO ] 2019-08-02 18:29:21.769 [Ruby-0-Thread-7: :1] elasticsearch - Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}} [INFO ] 2019-08-02 18:29:21.771 [Ruby-0-Thread-9: :1] elasticsearch - Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}} [WARN ] 2019-08-02 18:29:21.871 [[main]-pipeline-manager] LazyDelegatingGauge - A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team. [INFO ] 2019-08-02 18:29:21.878 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, :thread=>"#<Thread:0x470bf1ca run>"} [INFO ] 2019-08-02 18:29:22.351 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"} [INFO ] 2019-08-02 18:29:22.721 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [INFO ] 2019-08-02 18:29:23.798 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600} /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/cronline.rb:77: warning: constant ::Fixnum is deprecated /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/cronline.rb:77: warning: constant ::Fixnum is deprecated /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/cronline.rb:77: warning: constant ::Fixnum is deprecated [INFO ] 2019-08-02 18:30:02.333 [Ruby-0-Thread-22: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:284] jdbc - (0.042932s) SELECT * FROM pg_stat_user_indexes [INFO ] 2019-08-02 18:30:02.340 [Ruby-0-Thread-23: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:331] jdbc - (0.043178s) SELECT * FROM pg_stat_user_tables [INFO ] 2019-08-02 18:30:02.340 [Ruby-0-Thread-24: :1] jdbc - (0.036469s) SELECT * FROM pg_stat_database ...

      If Logstash does not show any errors and logs that it has successfully SELECTed rows from the three databases, your database metrics will be shipped to Elasticsearch. If you get an error, double check all the values in the configuration file to ensure that the machine you’re running Logstash on can connect to the managed database.

      Logstash will continue importing the data at specified times. You can safely stop it by pressing CTRL+C.

      As previously mentioned, when started as a service, Logstash automatically runs all configuration files it finds under /etc/logstash/conf.d in the background. Run the following command to start it as a service:

      • sudo systemctl start logstash

      In this step, you ran Logstash to check if it can connect to your database and gather data. Next, you’ll visualize and explore some of the statistical data in Kibana.

      Step 4 — Exploring Imported Data in Kibana

      In this section, you’ll see how you can explore the statistical data describing your database’s performance in Kibana.

      In your browser, navigate to the Kibana installation you set up as a prerequisite. You’ll see the default welcome page.

      Kibana - Default Welcome Page

      To interact with Elasticsearch indexes in Kibana, you’ll need to create an index pattern. Index patterns specify on which indexes Kibana should operate. To create one, press on the last icon (wrench) from the left-hand vertical sidebar to open the Management page. Then, from the left menu, press on Index Patterns under Kibana. You’ll see a dialog box for creating an index pattern.

      Kibana - Add Index Pattern

      Listed are the three indexes where Logstash has been sending statistics. Type in pg_stat_database in the Index Pattern input box and then press Next step. You’ll be asked to select a field that stores time, so you’ll be able to later narrow your data by a time range. From the dropdown, select @timestamp.

      Kibana - Index Pattern Timestamp Field

      Press on Create index pattern to finish creating the index pattern. You’ll now be able to explore it using Kibana. To create a visualization, press on the second icon in the sidebar, and then on Create new visualization. Select the Line visualization when the form pops up, and then choose the index pattern you have just created (pg_stat_database). You’ll see an empty visualization.

      Kibana - Empty Visualisation

      On the central part of the screen is the resulting plot—the left-side panel governs its generation from which you can set the data for X and Y axis. In the upper right-hand side of the screen is the date range picker. Unless you specifically choose another range when configuring the data, that range will be shown on the plot.

      You’ll now visualize the average number of data tuples INSERTed on minutes in the given interval. Press on Y-Axis under Metrics in the panel on the left to unfold it. Select Average as the Aggregation and select tup_inserted as the Field. This will populate the Y axis of the plot with the average values.

      Next, press on X-Axis under Buckets. For the Aggregation, choose Date Histogram. @timestamp should be automatically selected as the Field. Then, press on the blue play button on the top of the panel to generate your graph. If your database is brand new and not used, you won’t see anything yet. In all cases, however, you will see an accurate portrayal of database usage.

      Kibana supports many other visualization forms—you can explore other forms in the Kibana documentation. You can also add the two remaining indexes, mentioned in Step 2, into Kibana to be able to visualize them as well.

      In this step, you have learned how to visualize some of the PostgreSQL statistical data, using Kibana.

      Step 5 — (Optional) Benchmarking Using pgbench

      If you haven’t yet worked in your database outside of this tutorial, you can complete this step to create more interesting visualizations by using pgbench to benchmark your database. pgbench will run the same SQL commands over and over, simulating real-world database use by an actual client.

      You’ll first need to install pgbench by running the following command:

      • sudo apt install postgresql-contrib -y

      Because pgbench will insert and update test data, you’ll need to create a separate database for it. To do so, head over to the Users & Databases tab in the Control Panel of your managed database, and scroll down to the Databases section. Type in pgbench as the name of the new database, and then press on Save. You’ll pass this name, as well as the host, port, and username information to pgbench.

      Accessing Databases section in DO control panel

      Before actually running pgbench, you’ll need to run it with the -i flag to initialize its database:

      • pgbench -h host -p port -U username -i pgbench

      You’ll need to replace host with your host address, port with the port to which you can connect to your database, and username with the database user username. You can find all these values in the Control Panel of your managed database.

      Notice that pgbench does not have a password argument; instead, you’ll be asked for it every time you run it.

      The output will look like the following:

      Output

      NOTICE: table "pgbench_history" does not exist, skipping NOTICE: table "pgbench_tellers" does not exist, skipping NOTICE: table "pgbench_accounts" does not exist, skipping NOTICE: table "pgbench_branches" does not exist, skipping creating tables... 100000 of 100000 tuples (100%) done (elapsed 0.16 s, remaining 0.00 s) vacuum... set primary keys... done.

      pgbench created four tables, which it will use for benchmarking, and populated them with some example rows. You’ll now be able to run benchmarks.

      The two most important arguments that limit for how long the benchmark will run are -t, which specifies the number of transactions to complete, and -T, which defines for how many seconds the benchmark should run. These two options are mutually exclusive. At the end of each benchmark, you’ll receive statistics, such as the number of transactions per second (tps).

      Now, start a benchmark that will last for 30 seconds by running the following command:

      • pgbench -h host -p port -U username pgbench -T 30

      The output will look like:

      Output

      starting vacuum...end. transaction type: <builtin: TPC-B (sort of)> scaling factor: 1 query mode: simple number of clients: 1 number of threads: 1 duration: 30 s number of transactions actually processed: 7602 latency average = 3.947 ms tps = 253.382298 (including connections establishing) tps = 253.535257 (excluding connections establishing)

      In this output, you see the general info about the benchmark, such as the total number of transactions executed. The effect of these benchmarks is that the statistics Logstash ships to Elasticsearch will reflect that number, which will in turn make visualizations in Kibana more interesting and closer to real-world graphs. You can run the preceding command a few more times, and possibly alter the duration.

      When you are done, head over to Kibana and press on Refresh in the upper right corner. You’ll now see a different line than before, which shows the number of INSERTs. Feel free to change the time range of the data shown by changing the values in the picker positioned above the refresh button. Here is how the graph may look after multiple benchmarks of varying duration:

      Kibana - Visualization After Benchmarks

      You’ve used pgbench to benchmark your database, and evaluated the resulting graphs in Kibana.

      Conclusion

      You now have the Elastic stack installed on your server and configured to pull statistics data from your managed PostgreSQL database on a regular basis. You can analyze and visualize the data using Kibana, or some other suitable software, which will help you gather valuable insights and real-world correlations into how your database is performing.

      For more information about what you can do with your PostgreSQL Managed Database, visit the product docs.



      Source link

      Como Instalar Elasticsearch, Logstash e Kibana (Elastic Stack) no Ubuntu 18.04


      O autor selecionou o Internet Archive para receber uma doação como parte do programa Write for DOnations

      Introdução

      O Elastic Stack — anteriormente conhecido como ELK Stack — é uma coleção de software open-source produzido pela Elastic que lhe permite pesquisar, analisar e visualizar logs gerados a partir de qualquer fonte, em qualquer formato, em uma prática conhecida como centralização de logs. A centralização de logs pode ser muito útil ao tentar identificar problemas com seus servidores ou aplicativos, pois permite que você pesquise todos os seus logs em um único local. Ela é útil também porque permite identificar problemas que envolvem vários servidores, correlacionando seus logs durante um período de tempo específico.

      O Elastic Stack possui quatro componentes principais:

      • Elasticsearch: um mecanismo de pesquisa RESTful distribuído que armazena todos os dados coletados.

      • Logstash: o componente de processamento de dados do Elastic Stack que envia dados de entrada para o Elasticsearch.

      • Kibana: uma interface web para a pesquisa e a visualização de logs.

      • Beats: carregadores de dados leves e de propósito único que podem enviar dados de centenas ou milhares de máquinas para o Logstash ou para o Elasticsearch.

      Neste tutorial você irá instalar o Elastic Stack em um servidor Ubuntu 18.04. Você irá aprender como instalar todos os componentes do Elastic Stack — incluindo o Filebeat, um Beat usado para encaminhar e centralizar logs e arquivos — e configurá-los para reunir e visualizar os logs do sistema. Além disso, como o Kibana normalmente está disponível apenas no localhost, usaremos o Nginx para fazer proxy dele, de modo que ele seja acessível em um navegador web. Instalaremos todos esses componentes em um único servidor, o qual nos referiremos como nosso servidor Elastic Stack.

      Nota: Ao instalar o Elastic Stack, você deve usar a mesma versão em toda a pilha ou stack. Neste tutorial vamos instalar as versões mais recentes de toda a stack, que são, no momento desta publicação, Elasticsearch 6.4.3, Kibana 6.4.3, Logstash 6.4.3 e Filebeat 6.4.3.

      Pré-requisitos

      Para completar este tutorial, você irá precisar do seguinte:

      • Um servidor Ubuntu configurado seguindo nosso Guia de Configuração Inicial de servidor com Ubuntu 18.04, incluindo um usuário não-root com privilégios sudo e um firewall configurado com ufw. A quantidade de CPU, RAM e armazenamento que o seu servidor Elastic Stack exigirá depende do volume de logs que você pretende reunir. Para este tutorial, usaremos um VPS com as seguintes especificações para o nosso servidor Elastic Stack:

        • SO: Ubuntu 18.04
        • RAM: 4GB
        • CPU: 2
      • Java 8 — que é exigido pelo Elasticsearch e pelo Logstash — instalado em seu servidor. Observe que o Java 9 não é suportado. Para istalar isso, siga a seção “Installing the Oracle JDK” do nosso guia sobre como instalar o Java 8 no Ubuntu 18.04.

      • Nginx instalado em seu servidor, que vamos configurar mais tarde neste guia como um proxy reverso para o Kibana. Siga nosso guia sobre Como Instalar o Nginx no Ubuntu 18.04 para configurar isso.

      Além disso, como o Elastic Stack é usado para acessar informações valiosas sobre seu servidor, as quais você não deseja que usuários não autorizados acessem, é importante manter seu servidor seguro instalando um certificado TLS/SSL. Isso é opcional mas é fortemente recomendado.

      No entanto, como você acabará fazendo alterações no bloco do servidor Nginx ao longo deste guia, provavelmente faria mais sentido para você concluir o guia Como Proteger o Nginx com o Let’s Encrypt no Ubuntu 18.04 no final do segundo passo deste tutorial. Com isso em mente, se você planeja configurar o Let’s Encrypt no seu servidor, você precisará do seguinte, antes de fazer isso:

      • Um domínio completamente qualificado (FQDN). Este tutorial irá utilizar example.com durante todo o processo. Você pode comprar um nome de domínio em Namecheap, obter um gratuitamente em Freenom, ou utilizar o registrador de domínio de sua escolha.

      • Ambos os registros DNS a seguir configurados para o seu servidor. Você pode seguir esta introdução ao DigitalOcean DNS para detalhes sobre como adicioná-los.

        • Um registro A com example.com apontando para o endereço IP público do seu servidor.
        • Um registro A com www.example.com apontando para o endereço IP público do seu servidor.

      Passo 1 — Instalando e Configurando o Elasticsearch

      Os componentes do Elastic Stack não estão disponíveis nos repositórios de pacotes padrão do Ubuntu. Eles podem, no entanto, ser instalados com o APT após adicionar a lista de origens de pacotes da Elastic.

      Todos os pacotes do Elastic Stack são assinados com a chave de assinatura do Elasticsearch para proteger seu sistema contra falsificação de pacotes. Os pacotes que foram autenticados usando a chave serão considerados confiáveis pelo seu gerenciador de pacotes. Nesta etapa, você importará a chave GPG pública do Elasticsearch e adicionará a lista de origens de pacotes da Elastic para instalar o Elasticsearch.

      Para começar, execute o seguinte comando para importar a chave GPG pública do Elasticsearch para o APT:

      • wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

      Em seguida, adicione a lista de origens da Elastic ao diretório sources.list.d, onde o APT irá procurar por novas origens:

      • echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list

      Em seguida, atualize suas listas de pacotes para que o APT leia a nova origem da Elastic:

      Em seguida, instale o Elasticsearch com este comando:

      • sudo apt install elasticsearch

      Quando o Elasticsearch terminar a instalação, use seu editor de texto preferido para editar o arquivo de configuração principal do Elasticsearch, elasticsearch.yml. Aqui, usaremos o nano:

      • sudo nano /etc/elasticsearch/elasticsearch.yml

      Nota: O arquivo de configuração do Elasticsearch está no formato YAML, o que significa que a identação é muito importante! Certifique-se de não adicionar espaços extras ao editar esse arquivo.

      O Elasticsearch escuta o tráfego vindo de qualquer lugar na porta 9200. Você vai querer restringir o acesso externo à sua instância do Elasticsearch para impedir que pessoas de fora leiam seus dados ou desliguem o cluster do Elasticsearch por meio da API REST. Encontre a linha que especifica network.host, descomente-a e substitua seu valor por localhost para que fique assim:

      /etc/elasticsearch/elasticsearch.yml

      . . .
      network.host: localhost
      . . .
      

      Salve e feche o elasticsearch.yml pressionando CTRL+X, seguido de Y e depoisENTER se você estiver usando o nano. Em seguida, inicie o serviço Elasticsearch com o systemctl:

      • sudo systemctl start elasticsearch

      Depois, execute o seguinte comando para permitir que o Elasticsearch seja iniciado toda vez que o servidor for inicializado:

      • sudo systemctl enable elasticsearch

      Você pode testar se o serviço do Elasticsearch está sendo executado enviando uma solicitação HTTP:

      • curl -X GET "localhost:9200"

      Você verá uma resposta mostrando algumas informações básicas sobre o seu nó local, semelhante a esta:

      Output

      { "name" : "ZlJ0k2h", "cluster_name" : "elasticsearch", "cluster_uuid" : "beJf9oPSTbecP7_i8pRVCw", "version" : { "number" : "6.4.2", "build_flavor" : "default", "build_type" : "deb", "build_hash" : "04711c2", "build_date" : "2018-09-26T13:34:09.098244Z", "build_snapshot" : false, "lucene_version" : "7.4.0", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" }

      Agora que o Elasticsearch está instalado e funcionando, vamos instalar o Kibana, o próximo componente do Elastic Stack.

      Passo 2 — Instalando e Configurando o Painel do Kibana

      De acordo com a documentação oficial, você deve instalar o Kibana somente após instalar o Elasticsearch. A instalação nesta ordem garante que os componentes dos quais cada produto depende estão corretamente posicionados.

      Como você já adicionou a origem de pacotes do Elastic no passo anterior, você pode simplesmente instalar os componentes restantes do Elastic Stack usando o apt:

      Em seguida, ative e inicie o serviço Kibana:

      • sudo systemctl enable kibana
      • sudo systemctl start kibana

      Como o Kibana está configurado para escutar somente no localhost, devemos configurar um proxy reverso para permitir acesso externo a ele. Utilizaremos o Nginx para esse propósito, que já deve estar instalado no seu servidor.

      Primeiro, use o comando openssl para criar um usuário administrativo do Kibana que será usado para acessar a interface web do mesmo. Como exemplo, nomearemos essa conta como kibanaadmin, mas para garantir maior segurança, recomendamos que você escolha um nome que não seja óbvio para seu usuário e que seja difícil de adivinhar.

      O comando a seguir criará o usuário e a senha do usuário administrativo do Kibana e os armazenará no arquivo htpasswd.users. Você irá configurar o Nginx para requerer este nome de usuário e senha e ler este arquivo momentaneamente:

      • echo "kibanaadmin:`openssl passwd -apr1`" | sudo tee -a /etc/nginx/htpasswd.users

      Digite e confirme uma senha no prompt. Lembre-se ou anote este login, pois você precisará dele para acessar a interface web do Kibana.

      Em seguida, criaremos um arquivo de bloco do servidor Nginx. Como exemplo, vamos nos referir a este arquivo como example.com, embora você possa achar útil dar um nome mais descritivo ao seu. Por exemplo, se você tiver um FQDN e registros DNS configurados para este servidor, poderá nomear esse arquivo após seu FQDN:

      • sudo nano /etc/nginx/sites-available/example.com

      Adicione o seguinte bloco de código ao arquivo, certificando-se de atualizar example.com para corresponder ao FQDN do seu servidor ou ao seu endereço IP público. Este código configura o Nginx para direcionar o tráfego HTTP do seu servidor para o aplicativo Kibana, que está escutando em localhost:5601. Além disso, configura o Nginx para ler o arquivo htpasswd.users e requerer autenticação básica.

      Observe que, se você seguiu o o tutorial de pré-requisitos do Nginx até o final, você já deve ter criado esse arquivo e preenchido com algum conteúdo. Nesse caso, exclua todo o conteúdo existente no arquivo antes de adicionar o seguinte:

      /etc/nginx/sites-available/example.com

      
      server {
          listen 80;
      
          server_name example.com;
      
          auth_basic "Restricted Access";
          auth_basic_user_file /etc/nginx/htpasswd.users;
      
          location / {
              proxy_pass http://localhost:5601;
              proxy_http_version 1.1;
              proxy_set_header Upgrade $http_upgrade;
              proxy_set_header Connection 'upgrade';
              proxy_set_header Host $host;
              proxy_cache_bypass $http_upgrade;
          }
      }
      

      Quando terminar, salve e feche o arquivo.

      Em seguida, ative a nova configuração criando um link simbólico para o diretório sites-enabled. Se você já criou um arquivo de bloco do servidor com o mesmo nome no pré-requisito do Nginx, não será necessário executar este comando:

      • sudo ln -s /etc/nginx/sites-available/example.com /etc/nginx/sites-enabled/example.com

      Em seguida, verifique a configuração para erros de sintaxe:

      Se algum erro for relatado em sua saída, volte e verifique se o conteúdo que você colocou no seu arquivo de configuração foi adicionado corretamente. Uma vez que você veja syntax is ok na saída, vá em frente e reinicie o serviço Nginx:

      • sudo systemctl restart nginx

      Se você seguiu o guia de configuração inicial do servidor, você deverá ter um firewall UFW ativado. Para permitir conexões ao Nginx, podemos ajustar as regras digitando:

      • sudo ufw allow 'Nginx Full'

      Nota: Se você seguiu o tutorial de pré-requisitos do Nginx, você pode ter criado uma regra UFW permitindo o perfil Nginx HTTP através do firewall. Como o perfil Nginx Full permite o tráfego HTTP e HTTPS através do firewall, você pode excluir com segurança a regra criada no tutorial de pré-requisitos. Faça isso com o seguinte comando:

      • sudo ufw delete allow 'Nginx HTTP'

      O Kibana agora pode ser acessado pelo seu FQDN ou pelo endereço IP público do seu servidor Elastic Stack. Você pode verificar a página de status do servidor Kibana, navegando até o seguinte endereço e digitando suas credenciais de login quando solicitado:

      • http://ip_do_seu_servidor/status

      Essa página de status exibe informações sobre o uso de recursos do servidor e lista os plug-ins instalados.

      Note: Conforme mencionado na seção de Pré-requisitos, é recomendável ativar o SSL/TLS em seu servidor. Você pode seguir este tutorial agora para obter um certificado SSL grátis para o Nginx no Ubuntu 18.04. Depois de obter seus certificados SSL/TLS, você pode voltar e concluir este tutorial.

      Agora que o painel do Kibana está configurado, vamos instalar o próximo componente: Logstash.

      Passo 3 — Instalando e Configurando o Logstash

      Embora seja possível que o Beats envie dados diretamente para o banco de dados do Elasticsearch, recomendamos o uso do Logstash para processar os dados. Isso permitirá coletar dados de diferentes origens, transformá-los em um formato comum e exportá-los para outro banco de dados.

      Instale o Logstash com este comando:

      • sudo apt install logstash

      Depois de instalar o Logstash, você pode continuar a configurá-lo. Os arquivos de configuração do Logstash são escritos no formato JSON e residem no diretório /etc/logstash/conf.d. Ao configurá-lo, é útil pensar no Logstash como um pipeline que coleta dados em uma extremidade, os processa de uma forma ou de outra e os envia para o destino (nesse caso, o destino é o Elasticsearch). Um pipeline do Logstash tem dois elementos obrigatórios, input e output, e um elemento opcional, filter. Os plugins de input ou de entrada consomem dados de uma fonte, os plug-ins filter ou de filtro processam os dados, e os plugins de output ou de saída gravam os dados em um destino.

      Crie um arquivo de configuração chamado 02-beats-input.conf onde você irá configurar sua entrada para o Filebeat:

      • sudo nano /etc/logstash/conf.d/02-beats-input.conf

      Insira a seguinte configuração de input. Isto especifica uma entrada beats que irá escutar na porta TCP 5044.

      /etc/logstash/conf.d/02-beats-input.conf

      
      input {
        beats {
          port => 5044
        }
      }
      

      Salve e feche o arquivo. Em seguida, crie um arquivo de configuração chamado 10-syslog-filter.conf, onde adicionaremos um filtro para logs do sistema, também conhecido como syslogs:

      • sudo nano /etc/logstash/conf.d/10-syslog-filter.conf

      Insira a seguinte configuração do filtro syslog. Este exemplo de configuração de logs do sistema foi retirado da documentação oficial do Elastic. Esse filtro é usado para analisar os logs de entrada do sistema para torná-los estruturados e utilizáveis pelos painéis predefinidos do Kibana:

      /etc/logstash/conf.d/10-syslog-filter.conf

      
      filter {
        if [fileset][module] == "system" {
          if [fileset][name] == "auth" {
            grok {
              match => { "message" => ["%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:[%{POSINT:[system][auth][pid]}])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?",
                        "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:[%{POSINT:[system][auth][pid]}])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}",
                        "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:[%{POSINT:[system][auth][pid]}])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped_ip]}",
                        "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:[%{POSINT:[system][auth][pid]}])?: s*%{DATA:[system][auth][user]} 🙁 %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}",
                        "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:[%{POSINT:[system][auth][pid]}])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}",
                        "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:[%{POSINT:[system][auth][pid]}])?: new user: name=%{DATA:[system][auth][user][add][name]}, UID=%{NUMBER:[system][auth][user][add][uid]}, GID=%{NUMBER:[system][auth][user][add][gid]}, home=%{DATA:[system][auth][user][add][home]}, shell=%{DATA:[system][auth][user][add][shell]}$",
                        "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:[%{POSINT:[system][auth][pid]}])?: %{GREEDYMULTILINE:[system][auth][message]}"] }
              pattern_definitions => {
                "GREEDYMULTILINE"=> "(.|n)*"
              }
              remove_field => "message"
            }
            date {
              match => [ "[system][auth][timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
            }
            geoip {
              source => "[system][auth][ssh][ip]"
              target => "[system][auth][ssh][geoip]"
            }
          }
          else if [fileset][name] == "syslog" {
            grok {
              match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:[%{POSINT:[system][syslog][pid]}])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }
              pattern_definitions => { "GREEDYMULTILINE" => "(.|n)*" }
              remove_field => "message"
            }
            date {
              match => [ "[system][syslog][timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
            }
          }
        }
      }
      

      Salve e feche o arquivo quando terminar.

      Por fim, crie um arquivo de configuração chamado 30-elasticsearch-output.conf:

      • sudo nano /etc/logstash/conf.d/30-elasticsearch-output.conf

      Insira a seguinte configuração para output. Essencialmente, esta saída configura o Logstash para armazenar os dados do Beats no Elasticsearch, que está sendo executado em localhost:9200, em um índice ou index nomeado após o uso do Beat. O Beat usado neste tutorial é o Filebeat:

      /etc/logstash/conf.d/30-elasticsearch-output.conf

      
      output {
        elasticsearch {
          hosts => ["localhost:9200"]
          manage_template => false
          index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
        }
      }
      

      Salve e feche o arquivo.

      Se você quiser adicionar filtros para outros aplicativos que usam a entrada Filebeat, certifique-se de nomear os arquivos de forma que eles sejam classificados entre a configuração de entrada e a de saída, o que significa que os nomes dos arquivos devem começar com um número de dois dígitos entre 02 e 30.

      Teste sua configuração do Logstash com este comando:

      • sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t

      Se não houver erros de sintaxe, sua saída exibirá Configuration OK após alguns segundos. Se você não vir isso na sua saída, verifique se há erros que apareçam na sua saída e atualize sua configuração para corrigi-los.

      Se seu teste de configuração for bem-sucedido, inicie e ative o Logstash para colocar as mudanças de configuração em vigor:

      • sudo systemctl start logstash
      • sudo systemctl enable logstash

      Agora que o Logstash está sendo executado corretamente e está totalmente configurado, vamos instalar o Filebeat.

      Passo 4 — Instalando e Configurando o Filebeat

      O Elastic Stack usa vários carregadores de dados leves chamados Beats para coletar dados de várias fontes e transportá-los para o Logstash ou para o Elasticsearch. Aqui estão os Beats que estão atualmente disponíveis na Elastic:

      • Filebeat: coleta e envia arquivos de log.

      • Metricbeat: coleta métricas de seus sistemas e serviços.

      • Packetbeat: coleta e analisa dados da rede.

      • Winlogbeat: coleta logs de eventos do Windows.

      • Auditbeat: coleta dados da estrutura de auditoria do Linux e monitora a integridade dos arquivos.

      • Heartbeat: monitora serviços para verificar sua disponibilidade com sondagem ativa.

      Neste tutorial, usaremos o Filebeat para encaminhar logs locais para o nosso Elastic Stack.

      Instale o Filebeat usando o apt:

      • sudo apt install filebeat

      Em seguida, configure o Filebeat para se conectar ao Logstash. Aqui, vamos modificar o arquivo de configuração de exemplo que vem com o Filebeat.

      Abra o arquivo de configuração do Filebeat:

      • sudo nano /etc/filebeat/filebeat.yml

      Nota: Assim como no Elasticsearch, o arquivo de configuração do Filebeat está no formato YAML. Isso significa que a identação adequada é crucial, portanto, certifique-se de usar o mesmo número de espaços indicados nestas instruções.

      O Filebeat suporta várias saídas, mas normalmente você só envia eventos diretamente para o Elasticsearch ou para o Logstash para processamento adicional. Neste tutorial, usaremos o Logstash para executar processamento adicional nos dados coletados pelo Filebeat. O Filebeat não precisará enviar nenhum dado diretamente para o Elasticsearch, então vamos desativar essa saída. Para fazer isso, encontre a seção output.elasticsearch e comente as seguintes linhas, precedendo-as com um #:

      /etc/filebeat/filebeat.yml

      
      ...
      #output.elasticsearch:
        # Array of hosts to connect to.
        #hosts: ["localhost:9200"]
      ...
      

      Em seguida, configure a seção output.logstash. Descomente as linhas output.logstash: e hosts: ["localhost:5044"] removendo o #. Isto irá configurar o Filebeat para se conectar ao Logstash no seu servidor Elastic Stack na porta 5044, a porta para a qual especificamos uma entrada do Logstash anteriormente:

      /etc/filebeat/filebeat.yml

      
      output.logstash:
        # The Logstash hosts
        hosts: ["localhost:5044"]
      

      Salve e feche o arquivo.

      A funcionalidade do Filebeat pode ser estendida com os módulos do Filebeat. Neste tutorial vamos usar o módulo system, que coleta e analisa logs criados pelo serviço de logs do sistema em distribuições comuns do Linux.

      Vamos habilitar isso:

      • sudo filebeat modules enable system

      Você pode ver uma lista de módulos ativados e desativados executando:

      • sudo filebeat modules list

      Você verá uma lista semelhante à seguinte:

      Output

      Enabled: system Disabled: apache2 auditd elasticsearch icinga iis kafka kibana logstash mongodb mysql nginx osquery postgresql redis traefik

      Por padrão, o Filebeat é configurado para usar os caminhos padrão para os logs de syslog e de autorização. No caso deste tutorial, você não precisa alterar nada na configuração. Você pode ver os parâmetros do módulo no arquivo de configuração /etc/filebeat/modules.d/system.yml.

      Em seguida, carregue o modelo de index do Elasticsearch. Um index do Elasticsearch é uma coleção de documentos que possuem características semelhantes. Os index são identificados com um nome, que é usado para se referir ao index ao executar várias operações dentro dele. O modelo de index será aplicado automaticamente quando um novo index for criado.

      Para carregar o modelo, use o seguinte comando:

      • sudo filebeat setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

      Output

      Loaded index template

      O Filebeat vem com painéis de amostra do Kibana que lhe permitem visualizar dados do Filebeat no Kibana. Antes de poder usar os painéis, você precisa criar o padrão de index e carregar os painéis no Kibana.

      À medida que os painéis são carregados, o Filebeat se conecta ao Elasticsearch para verificar as informações da versão. Para carregar painéis quando o Logstash está ativado, é necessário desativar a saída do Logstash e ativar a saída do Elasticsearch:

      • sudo filebeat setup -e -E output.logstash.enabled=false -E output.elasticsearch.hosts=['localhost:9200'] -E setup.kibana.host=localhost:5601

      Você verá uma saída que se parece com isto:

      Output

      2018-09-10T08:39:15.844Z INFO instance/beat.go:273 Setup Beat: filebeat; Version: 6.4.2 2018-09-10T08:39:15.845Z INFO elasticsearch/client.go:163 Elasticsearch url: http://localhost:9200 2018-09-10T08:39:15.845Z INFO pipeline/module.go:98 Beat name: elk 2018-09-10T08:39:15.845Z INFO elasticsearch/client.go:163 Elasticsearch url: http://localhost:9200 2018-09-10T08:39:15.849Z INFO elasticsearch/client.go:708 Connected to Elasticsearch version 6.4.2 2018-09-10T08:39:15.856Z INFO template/load.go:129 Template already exists and will not be overwritten. Loaded index template Loading dashboards (Kibana must be running and reachable) 2018-09-10T08:39:15.857Z INFO elasticsearch/client.go:163 Elasticsearch url: http://localhost:9200 2018-09-10T08:39:15.865Z INFO elasticsearch/client.go:708 Connected to Elasticsearch version 6.4.2 2018-09-10T08:39:15.865Z INFO kibana/client.go:113 Kibana url: http://localhost:5601 2018-09-10T08:39:45.357Z INFO instance/beat.go:659 Kibana dashboards successfully loaded. Loaded dashboards 2018-09-10T08:39:45.358Z INFO elasticsearch/client.go:163 Elasticsearch url: http://localhost:9200 2018-09-10T08:39:45.361Z INFO elasticsearch/client.go:708 Connected to Elasticsearch version 6.4.2 2018-09-10T08:39:45.361Z INFO kibana/client.go:113 Kibana url: http://localhost:5601 2018-09-10T08:39:45.455Z WARN fileset/modules.go:388 X-Pack Machine Learning is not enabled Loaded machine learning job configurations

      Agora você pode iniciar e ativar o Filebeat:

      • sudo systemctl start filebeat
      • sudo systemctl enable filebeat

      Se você configurou seu Elastic Stack corretamente, o Filebeat começará a enviar seus registros de log e autorização para o Logstash, que então carregará esses dados no Elasticsearch.

      Para verificar se o Elasticsearch está realmente recebendo esses dados, consulte o index do Filebeat com este comando:

      • curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'

      Você verá uma saída semelhante a esta:

      Output

      ... { "took" : 32, "timed_out" : false, "_shards" : { "total" : 3, "successful" : 3, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 1641, "max_score" : 1.0, "hits" : [ { "_index" : "filebeat-6.4.2-2018.10.10", "_type" : "doc", "_id" : "H_bZ62UBB4D0uxFRu_h3", "_score" : 1.0, "_source" : { "@version" : "1", "message" : "Oct 10 06:22:36 elk systemd[1]: Reached target Local File Systems (Pre).", "@timestamp" : "2018-10-10T08:43:56.969Z", "host" : { "name" : "elk" }, "source" : "/var/log/syslog", "input" : { "type" : "log" }, "tags" : [ "beats_input_codec_plain_applied" ], "offset" : 296, "prospector" : { "type" : "log" }, "beat" : { "version" : "6.4.2", "hostname" : "elk", "name" : "elk" } } }, ...

      Se a sua saída mostrar 0 total hits, o Elasticsearch não está carregando nenhum registro sob o index que você pesquisou, e você precisará revisar sua configuração para verificar erros. Se você recebeu a saída esperada, continue para a próxima etapa, na qual veremos como navegar em alguns dos painéis do Kibana.

      Passo 5 — Explorando os Painéis do Kibana

      Vamos dar uma olhada no Kibana, a interface web que instalamos anteriormente.

      Em um navegador web, vá para o FQDN ou endereço IP público do seu servidor Elastic Stack. Depois de inserir as credenciais de login que você definiu no Passo 2, você verá a página inicial do Kibana:

      Clique no link Discover na barra de navegação à esquerda. Na página Discover, selecione o padrão de index predefinido filebeat-* para ver os dados do Filebeat. Por padrão, isso mostrará todos os dados do log nos últimos 15 minutos. Você verá um histograma com eventos de log, e algumas mensagens de log como abaixo:

      Aqui, você pode pesquisar e navegar pelos seus logs e também personalizar seu painel. Neste ponto, porém, não haverá muita coisa porque você está apenas coletando syslogs do seu servidor Elastic Stack.

      Use o painel esquerdo para navegar até a página Dashboard e pesquise pelos painéis do Filebeat System. Uma vez lá, você pode procurar os painéis de amostra que vêm com o módulo system do Filebeat.

      Por exemplo, você pode visualizar estatísticas detalhadas com base em suas mensagens do syslog:

      Você também pode ver quais usuários usaram o comando sudo e quando:

      Kibana tem muitos outros recursos, como gráficos e filtragem, então sinta-se livre para explorar.

      Conclusão

      Neste tutorial, você aprendeu como instalar e configurar o Elastic Stack para coletar e analisar logs do sistema. Lembre-se de que você pode enviar praticamente qualquer tipo de log ou dados indexados para o Logstash usando o Beats, mas os dados se tornam ainda mais úteis se forem analisados e estruturados com um filtro Logstash, pois isso transforma os dados em um formato consistente que pode ser lido facilmente pelo Elasticsearch.

      Por Justin Ellingwood e Vadym Kalsin



      Source link

      How To Install Elasticsearch, Logstash, and Kibana (Elastic Stack) on CentOS 7


      The author selected Software in the Public Interest to receive a donation as part of the Write for DOnations program.

      Introduction

      The Elastic Stack — formerly known as the ELK Stack — is a collection of open-source software produced by Elastic which allows you to search, analyze, and visualize logs generated from any source in any format, a practice known as centralized logging. Centralized logging can be very useful when attempting to identify problems with your servers or applications, as it allows you to search through all of your logs in a single place. It’s also useful because it allows you to identify issues that span multiple servers by correlating their logs during a specific time frame.

      The Elastic Stack has four main components:

      • Elasticsearch: a distributed RESTful search engine which stores all of the collected data.
      • Logstash: the data processing component of the Elastic Stack which sends incoming data to Elasticsearch.
      • Kibana: a web interface for searching and visualizing logs.
      • Beats: lightweight, single-purpose data shippers that can send data from hundreds or thousands of machines to either Logstash or Elasticsearch.

      In this tutorial, you will install the Elastic Stack on a CentOS 7 server. You will learn how to install all of the components of the Elastic Stack — including Filebeat, a Beat used for forwarding and centralizing logs and files — and configure them to gather and visualize system logs. Additionally, because Kibana is normally only available on the localhost, you will use Nginx to proxy it so it will be accessible over a web browser. At the end of this tutorial, you will have all of these components installed on a single server, referred to as the Elastic Stack server.

      Note: When installing the Elastic Stack, you should use the same version across the entire stack. This tutorial uses the latest versions of each component, which are, at the time of this writing, Elasticsearch 6.5.2, Kibana 6.5.2, Logstash 6.5.2, and Filebeat 6.5.2.

      Prerequisites

      To complete this tutorial, you will need the following:

      • One CentOS 7 server set up by following Initial Server Setup with CentOS 7, including a non-root user with sudo privileges and a firewall. The amount of CPU, RAM, and storage that your Elastic Stack server will require depends on the volume of logs that you intend to gather. For this tutorial, you will be using a VPS with the following specifications for our Elastic Stack server:

        • OS: CentOS 7.5
        • RAM: 4GB
        • CPU: 2
      • Java 8 — which is required by Elasticsearch and Logstash — installed on your server. Note that Java 9 is not supported. To install this, follow the “Install OpenJDK 8 JRE” section of our guide on how to install Java on CentOS.

      • Nginx installed on your server, which you will configure later in this guide as a reverse proxy for Kibana. Follow our guide on How To Install Nginx on CentOS 7 to set this up.

      Additionally, because the Elastic Stack is used to access valuable information about your server that you would not want unauthorized users to access, it’s important that you keep your server secure by installing a TLS/SSL certificate. This is optional but strongly encouraged. Because you will ultimately make changes to your Nginx server block over the course of this guide, we suggest putting this security in place by completing the Let’s Encrypt on CentOS 7 guide immediately after this tutorial’s second step.

      If you do plan to configure Let’s Encrypt on your server, you will need the following in place before doing so:

      • A fully qualified domain name (FQDN). This tutorial will use example.com throughout. You can purchase a domain name on Namecheap, get one for free on Freenom, or use the domain registrar of your choice.

      • Both of the following DNS records set up for your server. You can follow this introduction to DigitalOcean DNS for details on how to add them.

        • An A record with example.com pointing to your server’s public IP address.
        • An A record with www.example.com pointing to your server’s public IP address.

      Step 1 — Installing and Configuring Elasticsearch

      The Elastic Stack components are not available through the package manager by default, but you can install them with yum by adding Elastic’s package repository.

      All of the Elastic Stack’s packages are signed with the Elasticsearch signing key in order to protect your system from package spoofing. Packages which have been authenticated using the key will be considered trusted by your package manager. In this step, you will import the Elasticsearch public GPG key and add the Elastic repository in order to install Elasticsearch.

      Run the following command to download and install the Elasticsearch public signing key:

      • sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

      Next, add the Elastic repository. Use your preferred text editor to create the file elasticsearch.repo in the /etc/yum.repos.d/ directory. Here, we’ll use the vi text editor:

      • sudo vi /etc/yum.repos.d/elasticsearch.repo

      To provide yum with the information it needs to download and install the components of the Elastic Stack, enter insert mode by pressing i and add the following lines to the file.

      /etc/yum.repos.d/elasticsearch.repo

      [elasticsearch-6.x]
      name=Elasticsearch repository for 6.x packages
      baseurl=https://artifacts.elastic.co/packages/6.x/yum
      gpgcheck=1
      gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
      enabled=1
      autorefresh=1
      type=rpm-md
      

      Here you have included the human-readable name of the repo, the baseurl of the repo’s data directory, and the gpgkey required to verify Elastic packages.

      When you’re finished, press ESC to leave insert mode, then :wq and ENTER to save and exit the file. To learn more about the text editor vi and its successor vim, check out our Installing and Using the Vim Text Editor on a Cloud Server tutorial.

      With the repo added, you can now install the Elastic Stack. According to the official documentation, you should install Elasticsearch before the other components. Installing in this order ensures that the components each product depends on are correctly in place.

      Install Elasticsearch with the following command:

      • sudo yum install elasticsearch

      Once Elasticsearch is finished installing, open its main configuration file, elasticsearch.yml, in your editor:

      sudo vi /etc/elasticsearch/elasticsearch.yml
      

      Note: Elasticsearch’s configuration file is in YAML format, which means that indentation is very important! Be sure that you do not add any extra spaces as you edit this file.

      Elasticsearch listens for traffic from everywhere on port 9200. You will want to restrict outside access to your Elasticsearch instance to prevent outsiders from reading your data or shutting down your Elasticsearch cluster through the REST API. Find the line that specifies network.host, uncomment it, and replace its value with localhost so it looks like this:

      /etc/elasticsearch/elasticsearch.yml

      . . .
      network.host: localhost
      . . .
      

      Save and close elasticsearch.yml. Then, start the Elasticsearch service with systemctl:

      • sudo systemctl start elasticsearch

      Next, run the following command to enable Elasticsearch to start up every time your server boots:

      • sudo systemctl enable elasticsearch

      You can test whether your Elasticsearch service is running by sending an HTTP request:

      • curl -X GET "localhost:9200"

      You will see a response showing some basic information about your local node, similar to this:

      Output

      { "name" : "8oSCBFJ", "cluster_name" : "elasticsearch", "cluster_uuid" : "1Nf9ZymBQaOWKpMRBfisog", "version" : { "number" : "6.5.2", "build_flavor" : "default", "build_type" : "rpm", "build_hash" : "9434bed", "build_date" : "2018-11-29T23:58:20.891072Z", "build_snapshot" : false, "lucene_version" : "7.5.0", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" }

      Now that Elasticsearch is up and running, let’s install Kibana, the next component of the Elastic Stack.

      Step 2 — Installing and Configuring the Kibana Dashboard

      According to the installation order in the official documentation, you should install Kibana as the next component after Elasticsearch. After setting Kibana up, we will be able to use its interface to search through and visualize the data that Elasticsearch stores.

      Because you already added the Elastic repository in the previous step, you can just install the remaining components of the Elastic Stack using yum:

      Then enable and start the Kibana service:

      • sudo systemctl enable kibana
      • sudo systemctl start kibana

      Because Kibana is configured to only listen on localhost, we must set up a reverse proxy to allow external access to it. We will use Nginx for this purpose, which should already be installed on your server.

      First, use the openssl command to create an administrative Kibana user which you'll use to access the Kibana web interface. As an example, we will name this account kibanaadmin, but to ensure greater security we recommend that you choose a non-standard name for your user that would be difficult to guess.

      The following command will create the administrative Kibana user and password, and store them in the htpasswd.users file. You will configure Nginx to require this username and password and read this file momentarily:

      • echo "kibanaadmin:`openssl passwd -apr1`" | sudo tee -a /etc/nginx/htpasswd.users

      Enter and confirm a password at the prompt. Remember or take note of this login, as you will need it to access the Kibana web interface.

      Next, we will create an Nginx server block file. As an example, we will refer to this file as example.com.conf, although you may find it helpful to give yours a more descriptive name. For instance, if you have a FQDN and DNS records set up for this server, you could name this file after your FQDN:

      • sudo vi /etc/nginx/conf.d/example.com.conf

      Add the following code block into the file, being sure to update example.com and www.example.com to match your server's FQDN or public IP address. This code configures Nginx to direct your server's HTTP traffic to the Kibana application, which is listening on localhost:5601. Additionally, it configures Nginx to read the htpasswd.users file and require basic authentication.

      Note that if you followed the prerequisite Nginx tutorial through to the end, you may have already created this file and populated it with some content. In that case, delete all the existing content in the file before adding the following:

      example.com.conf'>/etc/nginx/conf.d/example.com.conf

      server {
          listen 80;
      
          server_name example.com www.example.com;
      
          auth_basic "Restricted Access";
          auth_basic_user_file /etc/nginx/htpasswd.users;
      
          location / {
              proxy_pass http://localhost:5601;
              proxy_http_version 1.1;
              proxy_set_header Upgrade $http_upgrade;
              proxy_set_header Connection 'upgrade';
              proxy_set_header Host $host;
              proxy_cache_bypass $http_upgrade;
          }
      }
      

      When you're finished, save and close the file.

      Then check the configuration for syntax errors:

      If any errors are reported in your output, go back and double check that the content you placed in your configuration file was added correctly. Once you see syntax is ok in the output, go ahead and restart the Nginx service:

      • sudo systemctl restart nginx

      By default, SELinux security policy is set to be enforced. Run the following command to allow Nginx to access the proxied service:

      • sudo setsebool httpd_can_network_connect 1 -P

      You can learn more about SELinux in the tutorial An Introduction to SELinux on CentOS 7.

      Kibana is now accessible via your FQDN or the public IP address of your Elastic Stack server. You can check the Kibana server's status page by navigating to the following address and entering your login credentials when prompted:

      http://your_server_ip/status
      

      This status page displays information about the server’s resource usage and lists the installed plugins.

      |Kibana status page

      Note: As mentioned in the Prerequisites section, it is recommended that you enable SSL/TLS on your server. You can follow this tutorial now to obtain a free SSL certificate for Nginx on CentOS 7. After obtaining your SSL/TLS certificates, you can come back and complete this tutorial.

      Now that the Kibana dashboard is configured, let's install the next component: Logstash.

      Step 3 — Installing and Configuring Logstash

      Although it's possible for Beats to send data directly to the Elasticsearch database, we recommend using Logstash to process the data first. This will allow you to collect data from different sources, transform it into a common format, and export it to another database.

      Install Logstash with this command:

      • sudo yum install logstash

      After installing Logstash, you can move on to configuring it. Logstash's configuration files are written in the JSON format and reside in the /etc/logstash/conf.d directory. As you configure it, it's helpful to think of Logstash as a pipeline which takes in data at one end, processes it in one way or another, and sends it out to its destination (in this case, the destination being Elasticsearch). A Logstash pipeline has two required elements, input and output, and one optional element, filter. The input plugins consume data from a source, the filter plugins process the data, and the output plugins write the data to a destination.

      Logstash pipeline

      Create a configuration file called 02-beats-input.conf where you will set up your Filebeat input:

      • sudo vi /etc/logstash/conf.d/02-beats-input.conf

      Insert the following input configuration. This specifies a beats input that will listen on TCP port 5044.

      /etc/logstash/conf.d/02-beats-input.conf

      input {
        beats {
          port => 5044
        }
      }
      

      Save and close the file. Next, create a configuration file called 10-syslog-filter.conf, which will add a filter for system logs, also known as syslogs:

      • sudo vi /etc/logstash/conf.d/10-syslog-filter.conf

      Insert the following syslog filter configuration. This example system logs configuration was taken from official Elastic documentation. This filter is used to parse incoming system logs to make them structured and usable by the predefined Kibana dashboards:

      /etc/logstash/conf.d/10-syslog-filter.conf

      filter {
        if [fileset][module] == "system" {
          if [fileset][name] == "auth" {
            grok {
              match => { "message" => ["%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:[%{POSINT:[system][auth][pid]}])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?",
                        "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:[%{POSINT:[system][auth][pid]}])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}",
                        "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:[%{POSINT:[system][auth][pid]}])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped_ip]}",
                        "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:[%{POSINT:[system][auth][pid]}])?: s*%{DATA:[system][auth][user]} 🙁 %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}",
                        "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:[%{POSINT:[system][auth][pid]}])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}",
                        "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:[%{POSINT:[system][auth][pid]}])?: new user: name=%{DATA:[system][auth][user][add][name]}, UID=%{NUMBER:[system][auth][user][add][uid]}, GID=%{NUMBER:[system][auth][user][add][gid]}, home=%{DATA:[system][auth][user][add][home]}, shell=%{DATA:[system][auth][user][add][shell]}$",
                        "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:[%{POSINT:[system][auth][pid]}])?: %{GREEDYMULTILINE:[system][auth][message]}"] }
              pattern_definitions => {
                "GREEDYMULTILINE"=> "(.|n)*"
              }
              remove_field => "message"
            }
            date {
              match => [ "[system][auth][timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
            }
            geoip {
              source => "[system][auth][ssh][ip]"
              target => "[system][auth][ssh][geoip]"
            }
          }
          else if [fileset][name] == "syslog" {
            grok {
              match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:[%{POSINT:[system][syslog][pid]}])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }
              pattern_definitions => { "GREEDYMULTILINE" => "(.|n)*" }
              remove_field => "message"
            }
            date {
              match => [ "[system][syslog][timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
            }
          }
        }
      }
      

      Save and close the file when finished.

      Lastly, create a configuration file called 30-elasticsearch-output.conf:

      • sudo vi /etc/logstash/conf.d/30-elasticsearch-output.conf

      Insert the following output configuration. This output configures Logstash to store the Beats data in Elasticsearch, which is running at localhost:9200, in an index named after the Beat used. The Beat used in this tutorial is Filebeat:

      /etc/logstash/conf.d/30-elasticsearch-output.conf

      output {
        elasticsearch {
          hosts => ["localhost:9200"]
          manage_template => false
          index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
        }
      }
      

      Save and close the file.

      If you want to add filters for other applications that use the Filebeat input, be sure to name the files so they're sorted between the input and the output configuration, meaning that the file names should begin with a two-digit number between 02 and 30.

      Test your Logstash configuration with this command:

      • sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t

      If there are no syntax errors, your output will display Configruation OK after a few seconds. If you don't see this in your output, check for any errors that appear in your output and update your configuration to correct them.

      If your configuration test is successful, start and enable Logstash to put the configuration changes into effect:

      • sudo systemctl start logstash
      • sudo systemctl enable logstash

      Now that Logstash is running correctly and is fully configured, let's install Filebeat.

      Step 4 — Installing and Configuring Filebeat

      The Elastic Stack uses several lightweight data shippers called Beats to collect data from various sources and transport them to Logstash or Elasticsearch. Here are the Beats that are currently available from Elastic:

      • Filebeat: collects and ships log files.
      • Metricbeat: collects metrics from your systems and services.
      • Packetbeat: collects and analyzes network data.
      • Winlogbeat: collects Windows event logs.
      • Auditbeat: collects Linux audit framework data and monitors file integrity.
      • Heartbeat: monitors services for their availability with active probing.

      In this tutorial, we will use Filebeat to forward local logs to our Elastic Stack.

      Install Filebeat using yum:

      • sudo yum install filebeat

      Next, configure Filebeat to connect to Logstash. Here, we will modify the example configuration file that comes with Filebeat.

      Open the Filebeat configuration file:

      • sudo vi /etc/filebeat/filebeat.yml

      Note: As with Elasticsearch, Filebeat's configuration file is in YAML format. This means that proper indentation is crucial, so be sure to use the same number of spaces that are indicated in these instructions.

      Filebeat supports numerous outputs, but you’ll usually only send events directly to Elasticsearch or to Logstash for additional processing. In this tutorial, we'll use Logstash to perform additional processing on the data collected by Filebeat. Filebeat will not need to send any data directly to Elasticsearch, so let's disable that output. To do so, find the output.elasticsearch section and comment out the following lines by preceding them with a #:

      /etc/filebeat/filebeat.yml

      ...
      #output.elasticsearch:
        # Array of hosts to connect to.
        #hosts: ["localhost:9200"]
      ...
      

      Then, configure the output.logstash section. Uncomment the lines output.logstash: and hosts: ["localhost:5044"] by removing the #. This will configure Filebeat to connect to Logstash on your Elastic Stack server at port 5044, the port for which we specified a Logstash input earlier:

      /etc/filebeat/filebeat.yml

      output.logstash:
        # The Logstash hosts
        hosts: ["localhost:5044"]
      

      Save and close the file.

      You can now extend the functionality of Filebeat with Filebeat modules. In this tutorial, you will use the system module, which collects and parses logs created by the system logging service of common Linux distributions.

      Let's enable it:

      • sudo filebeat modules enable system

      You can see a list of enabled and disabled modules by running:

      • sudo filebeat modules list

      You will see a list similar to the following:

      Output

      Enabled: system Disabled: apache2 auditd elasticsearch haproxy icinga iis kafka kibana logstash mongodb mysql nginx osquery postgresql redis suricata traefik

      By default, Filebeat is configured to use default paths for the syslog and authorization logs. In the case of this tutorial, you do not need to change anything in the configuration. You can see the parameters of the module in the /etc/filebeat/modules.d/system.yml configuration file.

      Next, load the index template into Elasticsearch. An Elasticsearch index is a collection of documents that have similar characteristics. Indexes are identified with a name, which is used to refer to the index when performing various operations within it. The index template will be automatically applied when a new index is created.

      To load the template, use the following command:

      • sudo filebeat setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

      This will give the following output:

      Output

      Loaded index template

      Filebeat comes packaged with sample Kibana dashboards that allow you to visualize Filebeat data in Kibana. Before you can use the dashboards, you need to create the index pattern and load the dashboards into Kibana.

      As the dashboards load, Filebeat connects to Elasticsearch to check version information. To load dashboards when Logstash is enabled, you need to manually disable the Logstash output and enable Elasticsearch output:

      • sudo filebeat setup -e -E output.logstash.enabled=false -E output.elasticsearch.hosts=['localhost:9200'] -E setup.kibana.host=localhost:5601

      You will see output that looks like this:

      Output

      . . . 2018-12-05T21:23:33.806Z INFO elasticsearch/client.go:163 Elasticsearch url: http://localhost:9200 2018-12-05T21:23:33.811Z INFO elasticsearch/client.go:712 Connected to Elasticsearch version 6.5.2 2018-12-05T21:23:33.815Z INFO template/load.go:129 Template already exists and will not be overwritten. Loaded index template Loading dashboards (Kibana must be running and reachable) 2018-12-05T21:23:33.816Z INFO elasticsearch/client.go:163 Elasticsearch url: http://localhost:9200 2018-12-05T21:23:33.819Z INFO elasticsearch/client.go:712 Connected to Elasticsearch version 6.5.2 2018-12-05T21:23:33.819Z INFO kibana/client.go:118 Kibana url: http://localhost:5601 2018-12-05T21:24:03.981Z INFO instance/beat.go:717 Kibana dashboards successfully loaded. Loaded dashboards 2018-12-05T21:24:03.982Z INFO elasticsearch/client.go:163 Elasticsearch url: http://localhost:9200 2018-12-05T21:24:03.984Z INFO elasticsearch/client.go:712 Connected to Elasticsearch version 6.5.2 2018-12-05T21:24:03.984Z INFO kibana/client.go:118 Kibana url: http://localhost:5601 2018-12-05T21:24:04.043Z WARN fileset/modules.go:388 X-Pack Machine Learning is not enabled 2018-12-05T21:24:04.080Z WARN fileset/modules.go:388 X-Pack Machine Learning is not enabled Loaded machine learning job configurations

      Now you can start and enable Filebeat:

      • sudo systemctl start filebeat
      • sudo systemctl enable filebeat

      If you've set up your Elastic Stack correctly, Filebeat will begin shipping your syslog and authorization logs to Logstash, which will then load that data into Elasticsearch.

      To verify that Elasticsearch is indeed receiving this data, query the Filebeat index with this command:

      • curl -X GET 'http://localhost:9200/filebeat-*/_search?pretty'

      You will see an output that looks similar to this:

      Output

      { "took" : 1, "timed_out" : false, "_shards" : { "total" : 3, "successful" : 3, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 3225, "max_score" : 1.0, "hits" : [ { "_index" : "filebeat-6.5.2-2018.12.05", "_type" : "doc", "_id" : "vf5GgGcB_g3p-PRo_QOw", "_score" : 1.0, "_source" : { "@timestamp" : "2018-12-05T19:00:34.000Z", "source" : "/var/log/secure", "meta" : { "cloud" : { . . .

      If your output shows 0 total hits, Elasticsearch is not loading any logs under the index you searched for, and you will need to review your setup for errors. If you received the expected output, continue to the next step, in which you'll become familiar with some of Kibana's dashboards.

      Step 5 — Exploring Kibana Dashboards

      Let's look at Kibana, the web interface that we installed earlier.

      In a web browser, go to the FQDN or public IP address of your Elastic Stack server. After entering the login credentials you defined in Step 2, you will see the Kibana homepage:

      Kibana Homepage

      Click the Discover link in the left-hand navigation bar. On the Discover page, select the predefined filebeat-* index pattern to see Filebeat data. By default, this will show you all of the log data over the last 15 minutes. You will see a histogram with log events, and some log messages below:

      Discover page

      Here, you can search and browse through your logs and also customize your dashboard. At this point, though, there won't be much in there because you are only gathering syslogs from your Elastic Stack server.

      Use the left-hand panel to navigate to the Dashboard page and search for the Filebeat System dashboards. Once there, you can search for the sample dashboards that come with Filebeat's system module.

      For example, you can view detailed stats based on your syslog messages:

      Syslog Dashboard

      You can also view which users have used the sudo command and when:

      Sudo Dashboard

      Kibana has many other features, such as graphing and filtering, so feel free to explore.

      Conclusion

      In this tutorial, you installed and configured the Elastic Stack to collect and analyze system logs. Remember that you can send just about any type of log or indexed data to Logstash using Beats, but the data becomes even more useful if it is parsed and structured with a Logstash filter, as this transforms the data into a consistent format that can be read easily by Elasticsearch.



      Source link