One place for hosting & domains

      Servers

      How To Configure a Galera Cluster with MariaDB on Ubuntu 18.04 Servers


      The author selected the Free and Open Source Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      Clustering adds high availability to your database by distributing changes to different servers. In the event that one of the instances fails, others are quickly available to continue serving.

      Clusters come in two general configurations, active-passive and active-active. In active-passive clusters, all writes are done on a single active server and then copied to one or more passive servers that are poised to take over only in the event of an active server failure. Some active-passive clusters also allow SELECT operations on passive nodes. In an active-active cluster, every node is read-write and a change made to one is replicated to all.

      MariaDB is an open source relational database system that is fully compatible with the popular MySQL RDBMS system. You can read the official documentation for MariaDB at this page. Galera is a database clustering solution that enables you to set up multi-master clusters using synchronous replication. Galera automatically handles keeping the data on different nodes in sync while allowing you to send read and write queries to any of the nodes in the cluster. You can learn more about Galera at the official documentation page.

      In this guide, you will configure an active-active MariaDB Galera cluster. For demonstration purposes, you will configure and test three Ubuntu 18.04 Droplets that will act as nodes in the cluster. This is the smallest configurable cluster.

      Prerequisites

      To follow along, you will need a DigitalOcean account, in addition to the following:

      • Three Ubuntu 18.04 Droplets with private networking enabled, each with a non-root user with sudo privileges.

      While the steps in this tutorial have been written for and tested against DigitalOcean Droplets, much of them should also be applicable to non-DigitalOcean servers with private networking enabled.

      Step 1 — Adding the MariaDB Repositories to All Servers

      In this step, you will add the relevant MariaDB package repositories to each of your three servers so that you will be able to install the right version of MariaDB used in this tutorial. Once the repositories are updated on all three servers, you will be ready to install MariaDB.

      One thing to note about MariaDB is that it originated as a drop-in replacement for MySQL, so in many configuration files and startup scripts, you’ll see mysql rather than mariadb. For consistency’s sake, we will use mysql in this guide where either could work.

      In this tutorial, you will use MariaDB version 10.4. Since this version isn’t included in the default Ubuntu repositories, you’ll start by adding the external Ubuntu repository maintained by the MariaDB project to all three of your servers.

      Note: MariaDB is a well-respected provider, but not all external repositories are reliable. Be sure to install only from trusted sources.

      First, you’ll add the MariaDB repository key with the apt-key command, which the APT package manager will use to verify that the package is authentic:

      • sudo apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xF1656F24C74CD1D8

      Once you have the trusted key in the database, you can add the repository with the following command:

      • sudo add-apt-repository 'deb [arch=amd64] http://nyc2.mirrors.digitalocean.com/mariadb/repo/10.4/ubuntu bionic main'

      After adding the repository, run apt update in order to include package manifests from the new repository:

      Once you have completed this step on your first server, repeat for your second and third servers.

      Now that you have successfully added the package repository on all three of your servers, you're ready to install MariaDB in the next section.

      Step 2 — Installing MariaDB on All Servers

      In this step, you will install the actual MariaDB packages on your three servers.

      Beginning with version 10.1, the MariaDB Server and MariaDB Galera Server packages are combined, so installing mariadb-server will automatically install Galera and several dependencies:

      • sudo apt install mariadb-server

      You will be asked to confirm whether you would like to proceed with the installation. Enter yes to continue with the installation.

      From MariaDB version 10.4 onwards, the root MariaDB user does not have a password by default. To set a password for the root user, start by logging into MariaDB:

      Once you're inside the MariaDB shell, change the password by executing the following statement:

      • set password = password("your_password");

      You will see the following output indicating that the password was set correctly:

      Output

      Query OK, 0 rows affected (0.001 sec)

      Exit the MariaDB shell by running the following command:

      If you would like to learn more about SQL or need a quick refresher, check out our MySQL tutorial.

      You now have all of the pieces necessary to begin configuring the cluster, but since you'll be relying on rsync in later steps, make sure it's installed:

      This will confirm that the newest version of rsync is already available or prompt you to upgrade or install it.

      Once you have installed MariaDB and set the root password on your first server, repeat these steps for your other two servers.

      Now that you have installed MariaDB successfully on each of the three servers, you can proceed to the configuration step in the next section.

      Step 3 — Configuring the First Node

      In this step you will configure your first node. Each node in the cluster needs to have a nearly identical configuration. Because of this, you will do all of the configuration on your first machine, and then copy it to the other nodes.

      By default, MariaDB is configured to check the /etc/mysql/conf.d directory to get additional configuration settings from files ending in .cnf. Create a file in this directory with all of your cluster-specific directives:

      • sudo nano /etc/mysql/conf.d/galera.cnf

      Add the following configuration into the file. The configuration specifies different cluster options, details about the current server and the other servers in the cluster, and replication-related settings. Note that the IP addresses in the configuration are the private addresses of your respective servers; replace the highlighted lines with the appropriate IP addresses.

      /etc/mysql/conf.d/galera.cnf

      [mysqld]
      binlog_format=ROW
      default-storage-engine=innodb
      innodb_autoinc_lock_mode=2
      bind-address=0.0.0.0
      
      # Galera Provider Configuration
      wsrep_on=ON
      wsrep_provider=/usr/lib/galera/libgalera_smm.so
      
      # Galera Cluster Configuration
      wsrep_cluster_name="test_cluster"
      wsrep_cluster_address="gcomm://First_Node_IP,Second_Node_IP,Third_Node_IP"
      
      # Galera Synchronization Configuration
      wsrep_sst_method=rsync
      
      # Galera Node Configuration
      wsrep_node_address="This_Node_IP"
      wsrep_node_name="This_Node_Name"
      
      • The first section modifies or re-asserts MariaDB/MySQL settings that will allow the cluster to function correctly. For example, Galera won’t work with MyISAM or similar non-transactional storage engines, and mysqld must not be bound to the IP address for localhost. You can learn about the settings in more detail on the Galera Cluster system configuration page.
      • The "Galera Provider Configuration" section configures the MariaDB components that provide a WriteSet replication API. This means Galera in your case, since Galera is a wsrep (WriteSet Replication) provider. You specify the general parameters to configure the initial replication environment. This doesn't require any customization, but you can learn more about Galera configuration options.
      • The "Galera Cluster Configuration" section defines the cluster, identifying the cluster members by IP address or resolvable domain name and creating a name for the cluster to ensure that members join the correct group. You can change the wsrep_cluster_name to something more meaningful than test_cluster or leave it as-is, but you must update wsrep_cluster_address with the private IP addresses of your three servers.
      • The "Galera Synchronization Configuration" section defines how the cluster will communicate and synchronize data between members. This is used only for the state transfer that happens when a node comes online. For your initial setup, you are using rsync, because it's commonly available and does what you'll need for now.
      • The "Galera Node Configuration" section clarifies the IP address and the name of the current server. This is helpful when trying to diagnose problems in logs and for referencing each server in multiple ways. The wsrep_node_address must match the address of the machine you're on, but you can choose any name you want in order to help you identify the node in log files.

      When you are satisfied with your cluster configuration file, copy the contents into your clipboard, save and close the file. With the nano text editor, you can do this by pressing CTRL+X, typing y, and pressing ENTER.

      Now that you have configured your first node successfully, you can move on to configuring the remaining nodes in the next section.

      Step 4 — Configuring the Remaining Nodes

      In this step, you will configure the remaining two nodes. On your second node, open the configuration file:

      • sudo nano /etc/mysql/conf.d/galera.cnf

      Paste in the configuration you copied from the first node, then update the Galera Node Configuration to use the IP address or resolvable domain name for the specific node you're setting up. Finally, update its name, which you can set to whatever helps you identify the node in your log files:

      /etc/mysql/conf.d/galera.cnf

      . . .
      # Galera Node Configuration
      wsrep_node_address="This_Node_IP"
      wsrep_node_name="This_Node_Name"
      . . .
      

      Save and exit the file.

      Once you have completed these steps, repeat them on the third node.

      You're almost ready to bring up the cluster, but before you do, make sure that the appropriate ports are open in your firewall.

      Step 5 — Opening the Firewall on Every Server

      In this step, you will configure your firewall so that the ports required for inter-node communication are open. On every server, check the status of the firewall by running:

      In this case, only SSH is allowed through:

      Output

      Status: active To Action From -- ------ ---- OpenSSH ALLOW Anywhere OpenSSH (v6) ALLOW Anywhere (v6)

      Since only SSH traffic is permitted in this case, you’ll need to add rules for MySQL and Galera traffic. If you tried to start the cluster, it would fail because of firewall rules.

      Galera can make use of four ports:

      • 3306 For MySQL client connections and State Snapshot Transfer that use the mysqldump method.
      • 4567 For Galera Cluster replication traffic. Multicast replication uses both UDP transport and TCP on this port.
      • 4568 For Incremental State Transfer.
      • 4444 For all other State Snapshot Transfer.

      In this example, you’ll open all four ports while you do your setup. Once you've confirmed that replication is working, you'd want to close any ports you're not actually using and restrict traffic to just servers in the cluster.

      Open the ports with the following command:

      • sudo ufw allow 3306,4567,4568,4444/tcp
      • sudo ufw allow 4567/udp

      Note: Depending on what else is running on your servers you might want to restrict access right away. The UFW Essentials: Common Firewall Rules and Commands guide can help with this.

      After you have configured your firewall on the first node, create the same firewall settings on the second and third node.

      Now that you have configured the firewalls successfully, you're ready to start the cluster in the next step.

      Step 6 — Starting the Cluster

      In this step, you will start your MariaDB cluster. To begin, you need to stop the running MariaDB service so that you can bring your cluster online.

      Stop MariaDB on All Three Servers

      Use the following command on all three servers to stop MariaDB so that you can bring them back up in a cluster:

      • sudo systemctl stop mysql

      systemctl doesn't display the outcome of all service management commands, so to be sure you succeeded, use the following command:

      • sudo systemctl status mysql

      If the last line looks something like the following, the command was successful:

      Output

      . . . Apr 26 03:34:23 galera-node-01 systemd[1]: Stopped MariaDB 10.4.4 database server.

      Once you've shut down mysql on all of the servers, you're ready to proceed.

      Bring Up the First Node

      To bring up the first node, you'll need to use a special startup script. The way you've configured your cluster, each node that comes online tries to connect to at least one other node specified in its galera.cnf file to get its initial state. Without using the galera_new_cluster script that allows systemd to pass the --wsrep-new-cluster parameter, a normal systemctl start mysql would fail because there are no nodes running for the first node to connect with.

      This command will not display any output on successful execution. When this script succeeds, the node is registered as part of the cluster, and you can see it with the following command:

      • mysql -u root -p -e "SHOW STATUS LIKE 'wsrep_cluster_size'"

      You will see the following output indicating that there is one node in the cluster:

      Output

      +--------------------+-------+ | Variable_name | Value | +--------------------+-------+ | wsrep_cluster_size | 1 | +--------------------+-------+

      On the remaining nodes, you can start mysql normally. They will search for any member of the cluster list that is online, so when they find one, they will join the cluster.

      Bring Up the Second Node

      Now you can bring up the second node. Start mysql:

      • sudo systemctl start mysql

      No output will be displayed on successful execution. You will see your cluster size increase as each node comes online:

      • mysql -u root -p -e "SHOW STATUS LIKE 'wsrep_cluster_size'"

      You will see the following output indicating that the second node has joined the cluster and that there are two nodes in total.

      Output

      +--------------------+-------+ | Variable_name | Value | +--------------------+-------+ | wsrep_cluster_size | 2 | +--------------------+-------+

      Bring Up the Third Node

      It's now time to bring up the third node. Start mysql:

      • sudo systemctl start mysql

      Run the following command to find the cluster size:

      • mysql -u root -p -e "SHOW STATUS LIKE 'wsrep_cluster_size'"

      You will see the following output, which indicates that the third node has joined the cluster and that the total number nodes in the cluster is three.

      Output

      +--------------------+-------+ | Variable_name | Value | +--------------------+-------+ | wsrep_cluster_size | 3 | +--------------------+-------+

      At this point, the entire cluster is online and communicating successfully. Next, you can ensure the working setup by testing replication in the next section.

      Step 7 — Testing Replication

      You've gone through the steps up to this point so that your cluster can perform replication from any node to any other node, known as active-active replication. Follow the steps below to test and see if the replication is working as expected.

      Write to the First Node

      You'll start by making database changes on your first node. The following commands will create a database called playground and a table inside of this database called equipment.

      • mysql -u root -p -e 'CREATE DATABASE playground;
      • CREATE TABLE playground.equipment ( id INT NOT NULL AUTO_INCREMENT, type VARCHAR(50), quant INT, color VARCHAR(25), PRIMARY KEY(id));
      • INSERT INTO playground.equipment (type, quant, color) VALUES ("slide", 2, "blue");'

      In the previous command, the CREATE DATABASE statement creates a database named playground. The CREATE statement creates a table named equipment inside the playground database having an auto-incrementing identifier column called id and other columns. The type column, quant column, and color column are defined to store the type, quantity, and color of the equipment respectively. The INSERT statement inserts an entry of type slide, quantity 2 and color blue.

      You now have one value in your table.

      Read and Write on the Second Node

      Next, look at the second node to verify that replication is working:

      • mysql -u root -p -e 'SELECT * FROM playground.equipment;'

      If replication is working, the data you entered on the first node will be visible here on the second:

      Output

      +----+-------+-------+-------+ | id | type | quant | color | +----+-------+-------+-------+ | 1 | slide | 2 | blue | +----+-------+-------+-------+

      From this same node, you can write data to the cluster:

      • mysql -u root -p -e 'INSERT INTO playground.equipment (type, quant, color) VALUES ("swing", 10, "yellow");'

      Read and Write on the Third Node

      From the third node, you can read all of this data by querying the table again:

      • mysql -u root -p -e 'SELECT * FROM playground.equipment;'

      You will see the following output showing the two rows:

      Output

      +----+-------+-------+--------+ | id | type | quant | color | +----+-------+-------+--------+ | 1 | slide | 2 | blue | | 2 | swing | 10 | yellow | +----+-------+-------+--------+

      Again, you can add another value from this node:

      • mysql -u root -p -e 'INSERT INTO playground.equipment (type, quant, color) VALUES ("seesaw", 3, "green");'

      Read on the First Node:

      Back on the first node, you can verify that your data is available everywhere:

      • mysql -u root -p -e 'SELECT * FROM playground.equipment;'

      You will see the following output which indicates that the rows are available on the first node.

      Output

      +----+--------+-------+--------+ | id | type | quant | color | +----+--------+-------+--------+ | 1 | slide | 2 | blue | | 2 | swing | 10 | yellow | | 3 | seesaw | 3 | green | +----+--------+-------+--------+

      You've verified successfully that you can write to all of the nodes and that replication is being performed properly.

      Conclusion

      At this point, you have a working three-node Galera test cluster configured. If you plan on using a Galera cluster in a production situation, it’s recommended that you begin with no fewer than five nodes.

      Before production use, you may want to take a look at some of the other state snapshot transfer (sst) agents like xtrabackup, which allows you to set up new nodes very quickly and without large interruptions to your active nodes. This does not affect the actual replication, but is a concern when nodes are being initialized.

      You might also be interested in other clustering solutions such as MySQL cluster, in which case you can check out our tutorial How To Create a Multi-Node MySQL Cluster on Ubuntu 18.04 tutorial.



      Source link

      How To Install and Configure Zabbix to Securely Monitor Remote Servers on Ubuntu 18.04


      The author selected the Open Source Initiative to receive a donation as part of the Write for DOnations program.

      Introduction

      Zabbix is open-source monitoring software for networks and applications. It offers real-time monitoring of thousands of metrics collected from servers, virtual machines, network devices, and web applications. These metrics can help you determine the current health of your IT infrastructure and detect problems with hardware or software components before customers complain. Useful information is stored in a database so you can analyze data over time and improve the quality of provided services, or plan upgrades of your equipment.

      Zabbix uses several options for collecting metrics, including agentless monitoring of user services and client-server architecture. To collect server metrics, it uses a small agent on the monitored client to gather data and send it to the Zabbix server. Zabbix supports encrypted communication between the server and connected clients, so your data is protected while it travels over insecure networks.

      The Zabbix server stores its data in a relational database powered by MySQL, PostgreSQL, or Oracle. You can also store historical data in nosql databases like Elasticsearch and TimescaleDB. Zabbix provides a web interface so you can view data and configure system settings.

      In this tutorial, you will configure two machines. One will be configured as the server, and the other as a client that you’ll monitor. The server will use a MySQL database to record monitoring data and use Apache to serve the web interface.

      Prerequisites

      To follow this tutorial, you will need:

      • Two Ubuntu 18.04 servers set up by following the Initial Server Setup Guide for Ubuntu 18.04, including a non-root user with sudo privileges and a firewall configured with ufw. On one server, you will install Zabbix; this tutorial will refer to this as the Zabbix server. It will monitor your second server; this second server will be referred to as the second Ubuntu server.

      • The server that will run the Zabbix server needs Apache, MySQL, and PHP installed. Follow this guide to configure those on your Zabbix server.

      Additionally, because the Zabbix Server is used to access valuable information about your infrastructure that you would not want unauthorized users to access, it’s important that you keep your server secure by installing a TLS/SSL certificate. This is optional but strongly encouraged. You can follow the Let’s Encrypt on Ubuntu 18.04 guide to obtain the free TLS/SSL certificate.

      Step 1 — Installing the Zabbix Server

      First, you need to install Zabbix on the server where you installed MySQL, Apache, and PHP. Log into this machine as your non-root user:

      • ssh sammy@zabbix_server_ip_address

      Zabbix is available in Ubuntu’s package manager, but it’s outdated, so use the official Zabbix repository to install the latest stable version. Download and install the repository configuration package:

      • wget https://repo.zabbix.com/zabbix/4.2/ubuntu/pool/main/z/zabbix-release/zabbix-release_4.2-1+bionic_all.deb
      • sudo dpkg -i zabbix-release_4.2-1+bionic_all.deb

      You will see the following output:

      Output

      Selecting previously unselected package zabbix-release. (Reading database ... 61483 files and directories currently installed.) Preparing to unpack zabbix-release_4.2-1+bionic_all.deb ... Unpacking zabbix-release (4.2-1+bionicc) ... Setting up zabbix-release (4.2-1+bionicc) ...

      Update the package index so the new repository is included:

      Then install the Zabbix server and web frontend with MySQL database support:

      • sudo apt install zabbix-server-mysql zabbix-frontend-php

      Also, install the Zabbix agent, which will let you collect data about the Zabbix server status itself.

      • sudo apt install zabbix-agent

      Before you can use Zabbix, you have to set up a database to hold the data that the Zabbix server will collect from its agents. You can do this in the next step.

      Step 2 — Configuring the MySQL Database for Zabbix

      You need to create a new MySQL database and populate it with some basic information in order to make it suitable for Zabbix. You'll also create a specific user for this database so Zabbix isn't logging into MySQL with the root account.

      Log into MySQL as the root user using the root password that you set up during the MySQL server installation:

      Create the Zabbix database with UTF-8 character support:

      • create database zabbix character set utf8 collate utf8_bin;

      Then create a user that the Zabbix server will use, give it access to the new database, and set the password for the user:

      • grant all privileges on zabbix.* to zabbix@localhost identified by 'your_zabbix_mysql_password';

      Then apply these new permissions:

      That takes care of the user and the database. Exit out of the database console.

      Next you have to import the initial schema and data. The Zabbix installation provided you with a file that sets this up.

      Run the following command to set up the schema and import the data into the zabbix database. Use zcat since the data in the file is compressed.

      • zcat /usr/share/doc/zabbix-server-mysql/create.sql.gz | mysql -uzabbix -p zabbix

      Enter the password for the zabbix MySQL user that you configured when prompted.

      This command will not output any errors if it was successful. If you see the error ERROR 1045 (28000): Access denied for userzabbix@'localhost' (using password: YES) then make sure you used the password for the zabbix user and not the root user.

      In order for the Zabbix server to use this database, you need to set the database password in the Zabbix server configuration file. Open the configuration file in your preferred text editor. This tutorial will use nano:

      • sudo nano /etc/zabbix/zabbix_server.conf

      Look for the following section of the file:

      /etc/zabbix/zabbix_server.conf

      ### Option: DBPassword                           
      #       Database password. Ignored for SQLite.   
      #       Comment this line if no password is used.
      #                                                
      # Mandatory: no                                  
      # Default:                                       
      # DBPassword=
      

      These comments in the file explain how to connect to the database. You need to set the DBPassword value in the file to the password for your database user. Add this line below those comments to configure the database:

      /etc/zabbix/zabbix_server.conf

      ...
      DBPassword=your_zabbix_mysql_password
      

      Save and close zabbix_server.conf by pressing CTRL+X, followed by Y and then ENTER if you're using nano.

      That takes care of the Zabbix server configuration. Next, you will make some modifications to your PHP setup in order for the Zabbix web interface to work properly.

      Step 3 — Configuring PHP for Zabbix

      The Zabbix web interface is written in PHP and requires some special PHP server settings. The Zabbix installation process created an Apache configuration file that contains these settings. It is located in the directory /etc/zabbix and is loaded automatically by Apache. You need to make a small change to this file, so open it up with the following:

      • sudo nano /etc/zabbix/apache.conf

      The file contains PHP settings that meet the necessary requirements for the Zabbix web interface. However, the timezone setting is commented out by default. To make sure that Zabbix uses the correct time, you need to set the appropriate timezone.

      /etc/zabbix/apache.conf

      ...
      <IfModule mod_php7.c>
          php_value max_execution_time 300
          php_value memory_limit 128M
          php_value post_max_size 16M
          php_value upload_max_filesize 2M
          php_value max_input_time 300
          php_value always_populate_raw_post_data -1
          # php_value date.timezone Europe/Riga
      </IfModule>
      

      Uncomment the timezone line, highlighted in the preceding code block, and change it to your timezone. You can use this list of supported time zones to find the right one for you. Then save and close the file.

      Now restart Apache to apply these new settings.

      • sudo systemctl restart apache2

      You can now start the Zabbix server.

      • sudo systemctl start zabbix-server

      Then check whether the Zabbix server is running properly:

      • sudo systemctl status zabbix-server

      You will see the following status:

      Output

      ● zabbix-server.service - Zabbix Server Loaded: loaded (/lib/systemd/system/zabbix-server.service; disabled; vendor preset: enabled) Active: active (running) since Fri 2019-04-05 08:50:54 UTC; 3s ago Process: 16497 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=0/SUCCESS) ...

      Finally, enable the server to start at boot time:

      • sudo systemctl enable zabbix-server

      The server is set up and connected to the database. Next, set up the web frontend.

      Note: As mentioned in the Prerequisites section, it is recommended that you enable SSL/TLS on your server. You can follow this tutorial now to obtain a free SSL certificate for Apache on Ubuntu 18.04. After obtaining your SSL/TLS certificates, you can come back and complete this tutorial.

      Step 4 — Configuring Settings for the Zabbix Web Interface

      The web interface lets you see reports and add hosts that you want to monitor, but it needs some initial setup before you can use it. Launch your browser and go to the address http://zabbix_server_name/zabbix/. On the first screen, you will see a welcome message. Click Next step to continue.

      On the next screen, you will see the table that lists all of the prerequisites to run Zabbix.

      Prerequisites

      All of the values in this table must be OK, so verify that they are. Be sure to scroll down and look at all of the prerequisites. Once you've verified that everything is ready to go, click Next step to proceed.

      The next screen asks for database connection information.

      DB Connection

      You told the Zabbix server about your database, but the Zabbix web interface also needs access to the database to manage hosts and read data. Therefore enter the MySQL credentials you configured in Step 2 and click Next step to proceed.

      On the next screen, you can leave the options at their default values.

      Zabbix Server Details

      The Name is optional; it is used in the web interface to distinguish one server from another in case you have several monitoring servers. Click Next step to proceed.

      The next screen will show the pre-installation summary so you can confirm everything is correct.

      Summary

      Click Next step to proceed to the final screen.

      The web interface setup is complete! This process creates the configuration file /usr/share/zabbix/conf/zabbix.conf.php which you could back up and use in the future. Click Finish to proceed to the login screen. The default user is Admin and the password is zabbix.

      Before you log in, set up the Zabbix agent on your second Ubuntu server.

      Step 5 — Installing and Configuring the Zabbix Agent

      Now you need to configure the agent software that will send monitoring data to the Zabbix server.

      Log in to the second Ubuntu server:

      • ssh sammy@second_ubuntu_server_ip_address

      Then, just like on the Zabbix server, run the following commands to install the repository configuration package:

      • wget https://repo.zabbix.com/zabbix/4.2/ubuntu/pool/main/z/zabbix-release/zabbix-release_4.2-1+bionic_all.deb
      • sudo dpkg -i zabbix-release_4.2-1+bionic_all.deb

      Next, update the package index:

      Then install the Zabbix agent:

      • sudo apt install zabbix-agent

      While Zabbix supports certificate-based encryption, setting up a certificate authority is beyond the scope of this tutorial, but you can use pre-shared keys (PSK) to secure the connection between the server and agent.

      First, generate a PSK:

      • sudo sh -c "openssl rand -hex 32 > /etc/zabbix/zabbix_agentd.psk"

      Show the key so you can copy it somewhere. You will need it to configure the host.

      • cat /etc/zabbix/zabbix_agentd.psk

      The key will look something like this:

      Output

      12eb854dea38ac9ee7d1ded2d74cee6262b0a56710f6946f7913d674ab82cdd4

      Now edit the Zabbix agent settings to set up its secure connection to the Zabbix server. Open the agent configuration file in your text editor:

      • sudo nano /etc/zabbix/zabbix_agentd.conf

      Each setting within this file is documented via informative comments throughout the file, but you only need to edit some of them.

      First you have to edit the IP address of the Zabbix server. Find the following section:

      /etc/zabbix/zabbix_agentd.conf

      ...
      ### Option: Server
      #       List of comma delimited IP addresses (or hostnames) of Zabbix servers.
      #       Incoming connections will be accepted only from the hosts listed here.
      #       If IPv6 support is enabled then '127.0.0.1', '::127.0.0.1', '::ffff:127.0.0.1' are treated equally.
      #
      # Mandatory: no
      # Default:
      # Server=
      
      Server=127.0.0.1
      ...
      

      Change the default value to the IP of your Zabbix server:

      /etc/zabbix/zabbix_agentd.conf

      ...
      Server=zabbix_server_ip_address
      ...
      

      Next, find the section that configures the secure connection to the Zabbix server and enable pre-shared key support. Find the TLSConnect section, which looks like this:

      /etc/zabbix/zabbix_agentd.conf

      ...
      ### Option: TLSConnect
      #       How the agent should connect to server or proxy. Used for active checks.
      #       Only one value can be specified:
      #               unencrypted - connect without encryption
      #               psk         - connect using TLS and a pre-shared key
      #               cert        - connect using TLS and a certificate
      #
      # Mandatory: yes, if TLS certificate or PSK parameters are defined (even for 'unencrypted' connection)
      # Default:
      # TLSConnect=unencrypted
      ...
      

      Then add this line to configure pre-shared key support:

      /etc/zabbix/zabbix_agentd.conf

      ...
      TLSConnect=psk
      ...
      

      Next, locate the TLSAccept section, which looks like this:

      /etc/zabbix/zabbix_agentd.conf

      ...
      ### Option: TLSAccept
      #       What incoming connections to accept.
      #       Multiple values can be specified, separated by comma:
      #               unencrypted - accept connections without encryption
      #               psk         - accept connections secured with TLS and a pre-shared key
      #               cert        - accept connections secured with TLS and a certificate
      #
      # Mandatory: yes, if TLS certificate or PSK parameters are defined (even for 'unencrypted' connection)
      # Default:
      # TLSAccept=unencrypted
      ...
      

      Configure incoming connections to support pre-shared keys by adding this line:

      /etc/zabbix/zabbix_agentd.conf

      ...
      TLSAccept=psk
      ...
      

      Next, find the TLSPSKIdentity section, which looks like this:

      /etc/zabbix/zabbix_agentd.conf

      ...
      ### Option: TLSPSKIdentity
      #       Unique, case sensitive string used to identify the pre-shared key.
      #
      # Mandatory: no
      # Default:
      # TLSPSKIdentity=
      ...
      

      Choose a unique name to identify your pre-shared key by adding this line:

      /etc/zabbix/zabbix_agentd.conf

      ...
      TLSPSKIdentity=PSK 001
      ...
      

      You'll use this as the PSK ID when you add your host through the Zabbix web interface.

      Then set the option that points to your previously created pre-shared key. Locate the TLSPSKFile option:

      /etc/zabbix/zabbix_agentd.conf

      ...
      ### Option: TLSPSKFile
      #       Full pathname of a file containing the pre-shared key.
      #
      # Mandatory: no
      # Default:
      # TLSPSKFile=
      ...
      

      Add this line to point the Zabbix agent to your PSK file you created:

      /etc/zabbix/zabbix_agentd.conf

      ...
      TLSPSKFile=/etc/zabbix/zabbix_agentd.psk
      ...
      

      Save and close the file. Now you can restart the Zabbix agent and set it to start at boot time:

      • sudo systemctl restart zabbix-agent
      • sudo systemctl enable zabbix-agent

      For good measure, check that the Zabbix agent is running properly:

      • sudo systemctl status zabbix-agent

      You will see the following status, indicating the agent is running:

      Output

      ● zabbix-agent.service - Zabbix Agent Loaded: loaded (/lib/systemd/system/zabbix-agent.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2019-04-05 09:03:04 UTC; 1s ago ...

      The agent will listen on port 10050 for connections from the server. Configure UFW to allow connections to this port:

      You can learn more about UFW in How To Set Up a Firewall with UFW on Ubuntu 18.04.

      Your agent is now ready to send data to the Zabbix server. But in order to use it, you have to link to it from the server's web console. In the next step, you will complete the configuration.

      Step 6 — Adding the New Host to the Zabbix Server

      Installing an agent on a server you want to monitor is only half of the process. Each host you want to monitor needs to be registered on the Zabbix server, which you can do through the web interface.

      Log in to the Zabbix Server web interface by navigating to the address http://zabbix_server_name/zabbix/.

      The Zabbix login screen

      When you have logged in, click on Configuration, and then Hosts in the top navigation bar. Then click the Create host button in the top right corner of the screen. This will open the host configuration page.

      Creating a host

      Adjust the Host name and IP address to reflect the host name and IP address of your second Ubuntu server, then add the host to a group. You can select an existing group, for example Linux servers, or create your own group. The host can be in multiple groups. To do this, enter the name of an existing or new group in the Groups field and select the desired value from the proposed list.

      Once you've added the group, click the Templates tab.

      Adding a template to the host

      Type Template OS Linux in the Search field and then click Add to add this template to the host.

      Next, navigate to the Encryption tab. Select PSK for both Connections to host and Connections from host. Then set PSK identity to PSK 001, which is the value of the TLSPSKIdentity setting of the Zabbix agent you configured previously. Then set PSK value to the key you generated for the Zabbix agent. It's the one stored in the file /etc/zabbix/zabbix_agentd.psk on the agent machine.

      Setting up the encryption

      Finally, click the Add button at the bottom of the form to create the host.

      You will see your new host in the list. Wait for a minute and reload the page to see green labels indicating that everything is working fine and the connection is encrypted.

      Zabbix shows your new host

      If you have additional servers you need to monitor, log in to each host, install the Zabbix agent, generate a PSK, configure the agent, and add the host to the web interface following the same steps you followed to add your first host.

      The Zabbix server is now monitoring your second Ubuntu server. Now, set up email notifications to be notified about problems.

      Step 7 — Configuring Email Notifications

      Zabbix automatically supports several types of notifications: email, Jabber, SMS, etc. You can also use alternative notification methods, such as Telegram or Slack. You can see the full list of integrations here.

      The simplest communication method is email, and this tutorial will configure notifications for this media type.

      Click on Administration, and then Media types in the top navigation bar. You will see the list of all media types. Click on Email.

      Adjust the SMTP options according to the settings provided by your email service. This tutorial uses Gmail's SMTP capabilities to set up email notifications; if you would like more information about setting this up, see How To Use Google's SMTP Server.


      Note: If you use 2-Step Verification with Gmail, you need to generate an App Password for Zabbix. You don't need to remember it, you’ll only have to enter an App password once during setup. You will find instructions on how to generate this password in the Google Help Center.

      You can also choose the message format—html or plain text. Finally, click the Update button at the bottom of the form to update the email parameters.

      Setting up email

      Now, create a new user. Click on Administration, and then Users in the top navigation bar. You will see the list of users. Then click the Create user button in the top right corner of the screen. This will open the user configuration page.

      Creating a user

      Enter the new username in the Alias field and set up a new password. Next, add the user to the administrator's group. Type Zabbix administrators in the Groups field and select it from the proposed list.

      Once you've added the group, click the Media tab and click on the Add underlined link. You will see a pop-up window.

      Adding an email

      Enter your email address in the Send to field. You can leave the rest of the options at the default values. Click the Add button at the bottom to submit.

      Now navigate to the Permissions tab. Select Zabbix Super Admin from the User type drop-down menu.

      Finally, click the Add button at the bottom of the form to create the user.

      Now you need to enable notifications. Click on the Configuration tab, and then Actions in the top navigation bar. You will see a pre-configured action, which is responsible for sending notifications to all Zabbix administrators. You can review and change the settings by clicking on its name. For the purposes of this tutorial, use the default parameters. To enable the action, click on the red Disabled link in the Status column.

      Now you are ready to receive alerts. In the next step, you will generate one to test your notification setup.

      Step 8 — Generating a Test Alert

      In this step, you will generate a test alert to ensure everything is connected. By default, Zabbix keeps track of the amount of free disk space on your server. It automatically detects all disk mounts and adds the corresponding checks. This discovery is executed every hour, so you need to wait a while for the notification to be triggered.

      Create a temporary file that's large enough to trigger Zabbix's file system usage alert. To do this, log in to your second Ubuntu server if you're not already connected.

      • ssh sammy@second_ubuntu_server_ip_address

      Next, determine how much free space you have on the server. You can use the df command to find out:

      The command df will report the disk space usage of your file system, and the -h will make the output human-readable. You'll see output like the following:

      Output

      Filesystem Size Used Avail Use% Mounted on /dev/vda1 25G 1.2G 23G 5% /

      In this case, the free space is 23GB. Your free space may differ.

      Use the fallocate command, which allows you to pre-allocate or de-allocate space to a file, to create a file that takes up more than 80% of the available disk space. This will be enough to trigger the alert:

      • fallocate -l 20G /tmp/temp.img

      After around an hour, Zabbix will trigger an alert about the amount of free disk space and will run the action you configured, sending the notification message. You can check your inbox for the message from the Zabbix server. You will see a message like:

      Output

      Problem started at 10:37:54 on 2019.04.05 Problem name: Free disk space is less than 20% on volume / Host: Second Ubuntu server Severity: Warning Original problem ID: 34

      You can also navigate to the Monitoring tab, and then Dashboard to see the notification and its details.

      Main dashboard

      Now that you know the alerts are working, delete the temporary file you created so you can reclaim your disk space:

      After a minute Zabbix will send the recovery message and the alert will disappear from main dashboard.

      Conclusion

      In this tutorial, you learned how to set up a simple and secure monitoring solution which will help you monitor the state of your servers. It can now warn you of problems, and you have the opportunity to analyze the processes occurring in your IT infrastructure.

      To learn more about setting up monitoring infrastructure, check out How To Install Elasticsearch, Logstash, and Kibana (Elastic Stack) on Ubuntu 18.04 and How To Gather Infrastructure Metrics with Metricbeat on Ubuntu 18.04.



      Source link

      Troubleshooting Web Servers, Databases, and Other Services


      Updated by Linode Written by Linode

      This guide presents troubleshooting strategies for when you can’t connect to your web server, database, or other services running on your Linode. This guide assumes that you have access to SSH. If you can’t log in with SSH, review Troubleshooting SSH and then return to this guide.

      Where to go for help outside this guide

      This guide explains how to use different troubleshooting commands on your Linode. These commands can produce diagnostic information and logs that may expose the root of your connection issues. For some specific examples of diagnostic information, this guide also explains the corresponding cause of the issue and presents solutions for it.

      If the information and logs you gather do not match a solution outlined here, consider searching the Linode Community Site for posts that match your system’s symptoms. Or, post a new question in the Community Site and include your commands’ output.

      Linode is not responsible for the configuration or installation of software on your Linode. Refer to Linode’s Scope of Support for a description of which issues Linode Support can help with.

      General Troubleshooting Strategies

      This section highlights troubleshooting strategies that apply to every service.

      Check if the Service is Running

      The service may not be running. Check the status of the service:

      Distribution Command                                                               
      systemd systems (Arch, Ubuntu 16.04+, Debian 8+, CentOS 7+, etc) sudo systemctl status <service name> -l
      sysvinit systems (CentOS 6, Ubuntu 14.04, Debian 7, etc) sudo service <service name> status

      Restart the Service

      If the service isn’t running, try restarting it:

      Distribution Command
      systemd systems sudo systemctl restart <service name>
      sysVinit systems sudo service <service name> restart

      Enable the Service

      If your system was recently rebooted, and the service didn’t start automatically at boot, then it may not be enabled. Enable the service to prevent this from happening in the future:

      Distribution Command
      systemd systems sudo systemctl enable <service name>
      sysVinit systems sudo chkconfig <service name> on

      Check your Service’s Bound IP Address and Ports

      Your service may be listening on an unexpected port, or it may not be bound to your public IP address (or whatever address is desirable). To view which address and ports a service is bound on, run the ss command with these options:

      sudo ss -atpu
      

      Review the application’s documentation for help determining the address and port your service should bind to.

      Note

      One notable example is if a service is only bound to a public IPv4 address and not to an IPv6 address. If a user connects to your Linode over IPv6, they will not be able to access the service.

      Analyze Service Logs

      If your service doesn’t start normally, review your system logs for the service. Your system logs may be in the following locations:

      Distribution System Logs
      systemd systems Run journalctl
      Ubuntu 14.04, Debian 7 /var/log/syslog
      CentOS 6 /var/log/messages

      Your service’s log location will vary by the application, but they are often stored in /var/log. The less command is a useful tool for browsing through your logs.

      Try pasting your log messages into a search engine or searching for your messages in the Linode Community Site to see if anyone else has run into similar issues. If you don’t find any results, you can try asking about your issues in a new post on the Linode Community Site. If it becomes difficult to find a solution, you may need to rebuild your Linode.

      Review Firewall Rules

      If your service is running but your connections still fail, your firewall (which is likely implemented by the iptables software) may be blocking the connections. To review your current firewall ruleset, run:

      sudo iptables -L # displays IPv4 rules
      sudo ip6tables -L # displays IPv6 rules
      

      Note

      Your deployment may be running FirewallD or UFW, which are frontends used to more easily manage your iptables rules. Run these commands to find out if you are running either package:

      sudo ufw status
      sudo firewall-cmd --state
      

      Review How to Configure a Firewall with UFW and Introduction to FirewallD on CentOS to learn how to manage and inspect your firewall rules with those packages.

      Firewall rulesets can vary widely. Review the Control Network Traffic with iptables guide to analyze your rules and determine if they are blocking connections. For example, a rule which allows incoming HTTP traffic could look like this:

        
      -A INPUT -p tcp -m tcp --dport 80 -m conntrack --ctstate NEW -j ACCEPT
      
      

      Disable Firewall Rules

      In addition to analyzing your firewall ruleset, you can also temporarily disable your firewall to test if it is interfering with your connections. Leaving your firewall disabled increases your security risk, so we recommend re-enabling it afterward with a modified ruleset that will accept your connections. Review Control Network Traffic with iptables for help with this subject.

      1. Create a temporary backup of your current iptables:

        sudo iptables-save > ~/iptables.txt
        
      2. Set the INPUT, FORWARD and OUTPUT packet policies as ACCEPT:

        sudo iptables -P INPUT ACCEPT
        sudo iptables -P FORWARD ACCEPT
        sudo iptables -P OUTPUT ACCEPT
        
      3. Flush the nat table that is consulted when a packet that creates a new connection is encountered:

        sudo iptables -t nat -F
        
      4. Flush the mangle table that is used for specialized packet alteration:

        sudo iptables -t mangle -F
        
      5. Flush all the chains in the table:

        sudo iptables -F
        
      6. Delete every non-built-in chain in the table:

        sudo iptables -X
        
      7. Repeat these steps with the ip6tables command to flush your IPv6 rules. Be sure to assign a different name to the IPv6 rules file (e.g. ~/ip6tables.txt).

      Troubleshoot Web Servers

      If your web server is not running or if connections are timing out, review the general troubleshooting strategies.

      Note

      If your web server is responding with an error code, your troubleshooting will vary by what code is returned. For more detailed information about each request that’s failing, read your web server’s logs. Here are some commands that can help you find your web server’s logs:

      • Apache:

        grep ErrorLog -r /etc/apache2  # On Ubuntu, Debian
        grep ErrorLog -r /etc/httpd    # On CentOS, Fedora, RHEL
        
      • NGINX:

        grep error_log -r /etc/nginx
        

      Frequent Error Codes

      • HTTP 401 Unauthorized, HTTP 403 Forbidden

        The requesting user did not have sufficient permission or access to the requested URL. Review your web server authorization and access control configuration:

      • HTTP 404 Not Found

        The URL that a user requested could not be found by the web server. Review your web server configuration and make sure your website files are stored in the right location on your filesystem:

      • HTTP 500, 502, 503, 504

        The web server requested a resource from a process it depends on, but the process did not respond as expected. For example, if a database query needs to be performed for a web request, but the database isn’t running, then a 50X code will be returned. To troubleshoot these issues, investigate the service that the web server depends on.

      Troubleshoot Databases

      Is your Disk Full?

      One common reason that a database may not start is if your disk is full. To check how much disk space you are using, run:

      df -h
      

      Note

      This reported disk usage is not the same as the reported storage usage in the Linode Manager. The storage usage in the Linode Manager refers to how much of the the disk space you pay for is allocated to your Linode’s disks. The output of df -h shows how full those disks are.

      You have several options for resolving disk space issues:

      • Free up space on your disk by locating and removing files you don’t need, using a tool like ncdu.

      • If you have any unallocated space on your Linode (storage that you pay for already but which isn’t assigned to your disk), resize your disk to take advantage of the space.

      • Upgrade your Linode to a higher-tier resource plan and then resize your disk to use the newly available space. If your Linode has a pending free upgrade for your storage space, you can choose to take this free upgrade to solve the issue.

      Database Performance Troubleshooting

      If your database is running but returning slowly, research how to optimize the database software for the resources your Linode has. If you run MySQL or MariaDB, read How to Optimize MySQL Performance Using MySQLTuner.

      Find answers, ask questions, and help others.

      This guide is published under a CC BY-ND 4.0 license.



      Source link