One place for hosting & domains

      How To Centralize Logs With Journald on Ubuntu 20.04


      The author selected the Free and Open Source Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      System logs are an extremely important component of managing Linux systems. They provide an invaluable insight into how the systems are working and also how they are being used because, in addition to errors, they record operational information such as security events. The standard configuration for Linux systems is to store their logs locally on the same system where they occurred. This works for standalone systems but quickly becomes a problem as the number of systems increases. The solution to managing all these logs is to create a centralized logging server where each Linux host sends its logs, in real-time, to a dedicated log management server.

      A centralized logging solution offers several benefits compared with storing logs on each host:

      • Reduces the amount of disk space needed on each host to store log files.
      • Logs can be retained for longer as the dedicated log server can be configured with more storage capacity.
      • Advanced log analysis can be carried out that requires logs from multiple systems and also more compute resources than may be available on the hosts.
      • Systems administrators can access the logs for all their systems that they may not be able to log in to directly for security reasons.

      In this guide, you will configure a component of the systemd suite of tools to relay log messages from client systems to a centralized log collection server. You will configure the server and client to use TLS certificates to encrypt the log messages as they are transmitted across insecure networks such as the internet and also to authenticate each other.

      Prerequisites

      Before you begin this guide you’ll need the following:

      • Two Ubuntu 20.04 servers.
      • A non-root user with sudo privileges on both servers. Follow the Initial Server Setup with Ubuntu 20.04 guide for instructions on how to do this. You should also configure the UFW firewall on both servers as explained in the guide.
      • Two hostnames that point to your servers. One hostname for the client system that generates the logs and another one for the log collection server. Learn how to point hostnames to DigitalOcean Droplets by consulting the Domains and DNS documentation.

      This guide will use the following two example hostnames:

      • client.your-domain: The client system that generates the logs.
      • server.your-domain: The log collection server.

      Log in to both the client and server in separate terminals via SSH as the non-root sudo user to begin this tutorial.

      Note: Throughout the tutorial command blocks are labeled with the server name (client or server) that the command should be run on.

      Step 1 — Installing systemd-journal-remote

      In this step, you will install the systemd-journal-remote package on the client and the server. This package contains the components that the client and server use to relay log messages.

      First, on both the client and server, run a system update to ensure that the package database and the system is current:

      Client and Server

      • sudo apt update
      • sudo apt upgrade

      Next, install the systemd-journal-remote package:

      Client and Server

      • sudo apt install systemd-journal-remote

      On the server, enable and start the two systemd components that it needs to receive log messages with the following command:

      Server

      • sudo systemctl enable --now systemd-journal-remote.socket
      • sudo systemctl enable systemd-journal-remote.service

      The --now option in the first command starts the services immediately. You did not use it in the second command because this service will not start until it has TLS certificates, which you will create in the next step.

      On the client, enable the component that systemd uses to send the log messages to the server:

      Client

      • sudo systemctl enable systemd-journal-upload.service

      Next, on the server, open ports 19532 and 80 in the UFW firewall. This will allow the server to receive the log messages from the client. Port 80 is the port that certbot will use to generate the TLS certificate. The following commands will open these ports:

      Server

      • sudo ufw allow in 19532/tcp
      • sudo ufw allow in 80/tcp

      On the client, you only need to open port 80 with this command:

      Client

      You have now installed the required components and completed the base system configuration on the client and server. Before you can configure these components to start relaying log messages you will register the Let’s Encrypt TLS certificates for the client and server using the certbot utility.

      Step 2 — Installing Certbot and Registering Certificates

      Let’s Encrypt is a Certificate Authority that issues free TLS certificates. These certificates allow computers to both encrypt the data that they send between them and also verify each other’s identity. These certificates are what allow you to secure your internet browsing with HTTPS. The same certificates can be used by any other application that wants the same level of security. The process of registering the certificate is the same no matter what you use them for.

      In this step, you will install the certbot utility and use it to register the certificates. It will also automatically take care of renewing the certificates when they expire. The registration process here is the same on the client and server. You only need to change the hostname to match the host where you are running the registration command.

      First, enable Ubuntu’s universe repository as the certbot utility resides in the universe repository. If you already have the universe repository enabled, running these commands will not do anything to your system and are safe to run:

      Client and Server

      • sudo apt install software-properties-common
      • sudo add-apt-repository universe
      • sudo apt update

      Next, install certbot on both hosts:

      Client and Server

      Now you’ve installed certbot, run the following command to register the certificates on the client and server:

      Client and Server

      • sudo certbot certonly --standalone --agree-tos --email sammy@your-domain -d your-domain

      The options in this command mean as follows:

      • certonly: Register the certificate and make no other changes on the system.
      • --standalone: Use certbot’s built-in web server to verify the certificate request.
      • --agree-tos: Automatically agree to the Let’s Encrypt Terms of Service.
      • --email your-email: This is the email address that Let’s Encrypt will use to notify you about certificate expiry and other important information.
      • -d your-domain: The hostname that the certificate will be registered for. This must match the system where you run it.

      When you run this command you will be asked if you want to share the email address with Let’s Encrypt so they can email you news and other information about their work. Doing this is optional, if you do not share your email address the certificate registration will still complete normally.

      When the certificate registration process completes it will place the certificate and key files in /etc/letsencrypt/live/your-domain/ where your-domain is the hostname that you registered the certificate for.

      Finally you need to download a copy of Let’s Encrypt’s CA and intermediate certificates and put them into the same file. journald will use this file to verify the authenticity of the certificates on the client and server when they communicate with each other.

      The following command will download the two certificates from Let’s Encrypt’s website and put them into a single file called letsencrypt-combined-certs.pem in your user’s home directory.

      Run this command on the client and server to download the certificates and create the combined file:

      Client and Server

      • curl -s https://letsencrypt.org/certs/{isrgrootx1.pem.txt,letsencryptauthorityx3.pem.txt} > ~/letsencrypt-combined-certs.pem

      Next, move this file into the Let’s Encrypt directory containing the certificates and keys:

      Client and Server

      • sudo cp ~/letsencrypt-combined-certs.pem /etc/letsencrypt/live/your-domain/

      You’ve now registered the certificates and keys. In the next step, you will configure the log collection server to start listening for and storing log messages from the client.

      Step 3 — Configuring the Server

      In this step, you will configure the server to use the certificate and key files that you generated in the last step so that it can start accepting log messages from the client.

      systemd-journal-remote is the component that listens for log messages. Open its configuration file at /etc/systemd/journal-remote.conf with a text editor to start configuring it on the server:

      • sudo nano /etc/systemd/journal-remote.conf

      Next, uncomment all the lines under the [Remote] section and set the paths to point to the TLS files you just created:

      /etc/systemd/journal-remote.conf

      [Remote]
      Seal=false
      SplitMode=host
      ServerKeyFile=/etc/letsencrypt/live/server.your-domain/privkey.pem
      ServerCertificateFile=/etc/letsencrypt/live/server.your-domain/fullchain.pem
      TrustedCertificateFile=/etc/letsencrypt/live/server.your-domain/letsencrypt-combined-certs.pem
      

      Here are the options you’ve used here:

      • Seal=false: Sign the log data in the journal. Enable this if you need maximum security; otherwise, you can leave it as false.
      • SplitMode=host: The logs from the remote clients will be split by host in /var/log/journal/remote. If you would prefer all the logs to be added to a single file set this to SplitMode=false.
      • ServerKeyFile: The server’s private key file.
      • ServerCertificateFile: The server’s certificate file.
      • TrustedCertificateFile: The file containing the Let’s Encrypt CA certificates.

      Now, you need to change the permissions on the Let’s Encrypt directories that contain the certificates and key so that the systemd-journal-remote can read and use them.

      First, change the permissions so that the certificate and private key are readable:

      • sudo chmod 0755 /etc/letsencrypt/{live,archive}
      • sudo chmod 0640 /etc/letsencrypt/live/server.your-domain/privkey.pem

      Next, change the group ownership of the private key to systemd-journal-remote’s group:

      • sudo chgrp systemd-journal-remote /etc/letsencrypt/live/server.your-domain/privkey.pem

      You can now start systemd-journal-remote:

      • sudo systemctl start systemd-journal-remote.service

      Your log collection server is now running and ready to start accepting log messages from a client. In the next step, you will configure the client to relay the logs to your collection server.

      Step 4 — Configuring the Client

      In this step, you will configure the component that relays the log messages to the log collection server. This component is called systemd-journal-upload.

      The default configuration for systemd-journal-upload is that it uses a temporary user that only exists while the process is running. This makes allowing systemd-journal-upload to read the TLS certificates and keys more complicated. To resolve this you will create a new system user with the same name as the temporary user that will get used in its place.

      First, create the new user called systemd-journal-upload on the client with the following adduser command:

      • sudo adduser --system --home /run/systemd --no-create-home --disabled-login --group systemd-journal-upload

      These options to the command are:

      • --system: Create the new user as a system user. This gives the user a UID (User Identifier) number under 1000. UID’s over 1000 are usually given to user accounts that a human will use to log in with.
      • --home /run/systemd: Set /run/systemd as the home directory for this user.
      • --no-create-home: Don’t create the home directory set, as it already exists.
      • --disabled-login: The user cannot log in to the server via, for example, SSH.
      • --group: Create a group with the same name as the user.

      Next, set the permissions and ownership of the Let’s Encrypt certificate files:

      • sudo chmod 0755 /etc/letsencrypt/{live,archive}
      • sudo chmod 0640 /etc/letsencrypt/live/client.your-domain/privkey.pem
      • sudo chgrp systemd-journal-upload /etc/letsencrypt/live/client.your-domain/privkey.pem

      Now, edit the configuration for systemd-journal-upload, which is at /etc/systemd/journal-upload.conf. Open this file with a text editor:

      • sudo nano /etc/systemd/journal-upload.conf

      Edit this file so that it looks like the following:

      /etc/systemd/journal-upload.conf

      [Upload]
      URL=https://server.your-domain:19532
      ServerKeyFile=/etc/letsencrypt/live/client.your-domain/privkey.pem
      ServerCertificateFile=/etc/letsencrypt/live/client.your-domain/fullchain.pem
      TrustedCertificateFile=/etc/letsencrypt/live/client.your-domain/letsencrypt-combined-certs.pem
      

      Finally, restart the systemd-journal-upload service so it uses the new configuration:

      • sudo systemctl restart systemd-journal-upload.service

      Your client is now set up and running and is sending its log messages to the log collection server. In the next step, you will check that the logs are being sent and recorded correctly.

      Step 5 — Testing the Client and Server

      In this step, you will test that the client is relaying log messages to the server and that the server is storing them correctly.

      The log collection server stores the logs from the clients in a directory at /var/log/journal/remote/. When you restarted the client at the end of the last step it began sending log messages so there is now a log file in /var/log/journal/remote/. The file will be named after the hostname you used for the TLS certificate.

      Use the ls command to check that the client’s log file is present on the server:

      Server

      • sudo ls -la /var/log/journal/remote/

      This will print the directory contents showing the log file:

      Output

      total 16620 drwxr-xr-x 2 systemd-journal-remote systemd-journal-remote 4096 Jun 30 16:17 . drwxr-sr-x+ 4 root systemd-journal 4096 Jun 30 15:55 .. -rw-r----- 1 systemd-journal-remote systemd-journal-remote 8388608 Jul 1 10:46 'remote-CN=client.your-domain'

      Next, write a log message on the client to check that the server is receiving the client’s messages as you expect. You will use the logger utility to create a custom log message on the client. If everything is working systemd-journal-upload will relay this message to the server.

      On the client run the following logger command:

      Client

      • sudo logger -p syslog.debug "### TEST MESSAGE from client.your-domain ###"

      The -p syslog.debug in this command sets the facility and severity of the message. Setting this to syslog.debug will make clear it’s a test message. This command will record the message ### TEST MESSAGE from client.your-domain ### to the client’s journal, which systemd-journal-upload then relays to the server.

      Next, read the client’s journal file on the server to check that the log messages are arriving from the client. This file is a binary log file so you will not be able to read it with tools like less. Instead, read the file using journalctl with the --file= option that allows you to specify a custom journal file:

      Server

      • sudo journalctl --file=/var/log/journal/remote/remote-CN=client.your-domain.journal

      The log message will appear as follows:

      Test log message

      . . . Jun 29 13:10:09 client root[3576]: ### TEST MESSAGE from client.your-domain ###

      Your log centralization server is now successfully collecting logs from your client system.

      Conclusion

      In this article, you set up a log central collection server and configured a client to relay a copy of its system logs to the server. You can configure as many clients as you need to relay messages to the log collection server using the client configuration steps you used here.



      Source link

      How to Access MySQL Error Logs



      Part of the Series:
      How To Troubleshoot Issues in MySQL

      This guide is intended to serve as a troubleshooting resource and starting point as you diagnose your MySQL setup. We’ll go over some of the issues that many MySQL users encounter and provide guidance for troubleshooting specific problems. We will also include links to DigitalOcean tutorials and the official MySQL documentation that may be useful in certain cases.

      Oftentimes, the root cause of slowdowns, crashes, or other unexpected behavior in MySQL can be determined by analyzing its error logs. On Ubuntu systems, the default location for the MySQL is /var/log/mysql/error.log. In many cases, the error logs are most easily read with the less program, a command line utility that allows you to view files but not edit them:

      • sudo less /var/log/mysql/error.log

      If MySQL isn’t behaving as expected, you can obtain more information about the source of the trouble by running this command and diagnosing the error based on the log’s contents.



      Source link

      Use journalctl to View Your System's Logs


      Updated by Linode

      Written by Linode


      Use promo code DOCS10 for $10 credit on a new account.

      What is journalctl?

      journalctl is a command for viewing logs collected by systemd. The systemd-journald service is responsible for systemd’s log collection, and it retrieves messages from the kernel, systemd services, and other sources.

      These logs are gathered in a central location, which makes them easy to review. The log records in the journal are structured and indexed, and as a result journalctl is able to present your log information in a variety of useful formats.

      Using journalctl for the First Time

      Run the journalctl command without any arguments to view all the logs in your journal:

      journalctl
      

      If you do not see output, try running it with sudo:

      sudo journalctl
      

      If your Linux user does not have sudo privileges, add your user to the sudo group.

      Default Log Format and Ordering

      journalctl will display your logs in a format similar to the traditional syslog format. Each line starts with the date (in the server’s local time), followed by the server’s hostname, the process name, and the message for the log.

        
      Aug 31 12:00:25 debian sshd[15844]: pam_unix(sshd:session): session opened for user example_user by (uid=0)
      
      

      Your logs will be displayed from oldest to newest. To reverse this order and display the newest messages at the top, use the -r flag:

      journalctl -r
      

      Paging through Your Logs

      journalctl pipes its output to the less command, which shows your logs one page at a time in your terminal. If a log line exceeds the horizontal width of your terminal window, you can use the left and right arrow keys to scroll horizontally and see the rest of the line:

      Furthermore, your logs can be navigated and searched by using all the same key commands available in less:

      Key commandAction
      down arrow key, enter, e, or jMove down one line.
      up arrow key, y, or kMove up one line.
      space barMove down one page.
      bMove up one page.
      right arrow keyScroll horizontally to the right.
      left arrow keyScroll horizontally to the left.
      gGo to the first line.
      GGo to the last line.
      10gGo to the 10th line. Enter a different number to go to other lines.
      50p or 50%Go to the line half-way through the output. Enter a different number to go to other percentage positions.
      /search termSearch forward from the current position for the search term string.
      ?search termSearch backward from the current position for the search term string.
      nWhen searching, go to the next occurrence.
      NWhen searching, go to the previous occurrence.
      m<c>Set a mark, which saves your current position. Enter a single character in place of <c> to label the mark with that character.
      '<c>Return to a mark, where <c> is the single character label for the mark. Note that ' is the single-quote.
      qQuit less

      View journalctl without Paging

      To send your logs to standard output and avoid paging them, use the --no-pager option:

      journalctl --no-pager
      

      It’s not recommended that you do this without first filtering down the number of logs shown.

      Monitor New Log Messages

      Run journalctl with the -f option to view a live log of new messages as they are collected:

      journalctl -f
      

      The key commands from less are not available while in this mode. Enter Control-C on your keyboard to return to your command prompt from this mode.

      Filter journalctl Output

      In addition to searching your logs with the less key commands, you can invoke journalctl with options that filter your log messages before they are displayed.

      These filters can be used with the normal paged display, and with the --no-pager and -f options. Filters of different types can also be combined together to further narrow the output.

      Show Logs within a Time Range

      Use the --since option to show logs after a specified date and time:

      journalctl --since "2018-08-30 14:10:10"
      

      Use the --until option to show logs up to a specified date and time:

      journalctl --until "2018-09-02 12:05:50"
      

      Combine these to show logs between the two times:

      journalctl --since "2018-08-30 14:10:10" --until "2018-09-02 12:05:50"
      

      Dates and times should be specified in the YYYY-MM-DD HH:MM:SS format. If the time is omitted (i.e. only the YYYY-MM-DD date is specified), then the time is assumed to be 00:00:00.

      journalctl can also accept some alternative terms when specifying dates:

      • The terms yesterday, today, and tomorrow are recognized. When using one of these terms, the time is assumed to be 00:00:00.

      • Terms like 1 day ago or 3 hours ago are recognized.

      • The - and + symbols can be used to specify relative dates. For example, -1h15min specifies 1 hour 15 minutes in the past, and +3h30min specifies 3 hours 30 minutes in the future.

      Show Logs for a Specific Boot

      Use the -b option to show logs for the last boot of your server:

      journalctl -b
      

      Specify an integer offset for the -b option to refer to a previous boot. For example, journalctl -b -1 show logs from the previous boot, journalctl -b -2 shows logs from the boot before the previous boot, and so on.

      List the available boots:

      journalctl --list-boots
      

      Each boot listed in the output from journalctl --list-boots command includes a 32-bit boot ID. You can supply a boot ID with the -b option; for example:

      journalctl -b a09dce7b2c1c458d861d7d0f0a7c8c65
      

      If no previous boots are listed, your journald configuration may not be set up to persist log storage. Review the Persist Your Logs section for instructions on how to change this configuration.

      Show Logs for a systemd Service

      Pass the name of a systemd unit with the -u option to show logs for that service:

      journalctl -u ssh
      

      View Kernel Messages

      Supply the -k option to show only kernel messages:

      journalctl -k
      

      Change the Log Output Format

      Because the log records for systemd’s journals are structured, journalctl can show your logs in different formats. Here are a few of the formats available:

      Format NameDescription
      shortThe default option, displays logs in the traditional syslog format.
      verboseDisplays all information in the log record structure.
      jsonDisplays logs in JSON format, with one log per line.
      json-prettyDisplays logs in JSON format across multiple lines for better readability.
      catDisplays only the message from each log without any other metadata.

      Pass the format name with the -o option to display your logs in that format. For example:

      journalctl -o json-pretty
      

      Anatomy of a Log Record

      The following is an example of the structured data of a log record, as displayed by journalctl -o verbose. For more information on this data structure, review the man page for journalctl:

        
      Fri 2018-08-31 12:00:25.543177 EDT [s=0b341b44cf194c9ca45c99101497befa;i=70d5;b=a09dce7b2c1c458d861d7d0f0a7c8c65;m=9fb524664c4;t=57517dfc5f57d;x=97097ca5ede0dfd6]
          _BOOT_ID=a09dce7b2c1c458d861d7d0f0a7c8c65
          _MACHINE_ID=1009f49fff8fe746a5111e1a062f4848
          _HOSTNAME=debian
          _TRANSPORT=syslog
          PRIORITY=6
          SYSLOG_IDENTIFIER=sshd
          _UID=0
          _GID=0
          _COMM=sshd
          _EXE=/usr/sbin/sshd
          _CAP_EFFECTIVE=3fffffffff
          _SYSTEMD_CGROUP=/system.slice/ssh.service
          _SYSTEMD_UNIT=ssh.service
          _SYSTEMD_SLICE=system.slice
          SYSLOG_FACILITY=10
          SYSLOG_PID=15844
          _PID=15844
          _CMDLINE=sshd: example_user [priv
          MESSAGE=pam_unix(sshd:session): session opened for user example_user by (uid=0)
          _AUDIT_SESSION=30791
          _AUDIT_LOGINUID=1000
          _SOURCE_REALTIME_TIMESTAMP=1536120282543177
      
      

      Note

      In addition to the types of filters listed in the previous section, you can also filter logs by specifying values for the variables in the log record structure. For example, journalctl _UID=0 will show logs for user ID 0 (i.e. the root user).

      Persist Your Logs

      systemd-journald can be configured to persist your systemd logs on disk, and it also provides controls to manage the total size of your archived logs. These settings are defined in /etc/systemd/journald.conf.

      To start persisting your logs, uncomment the Storage line in /etc/systemd/journald.conf and set its value to persistent. Your archived logs will be held in /var/log/journal. If this directory does not already exist in your file system, systemd-journald will create it.

      After updating your journald.conf, load the change:

      sudo systemctl restart systemd-journald
      

      Control the Size of Your Logs’ Disk Usage

      The following settings in journald.conf control how large your logs’ size can grow to when persisted on disk:

      SettingDescription
      SystemMaxUseThe total maximum disk space that can be used for your logs.
      SystemKeepFreeThe minimum amount of disk space that should be kept free for uses outside of systemd-journald’s logging functions.
      SystemMaxFileSizeThe maximum size of an individual journal file.
      SystemMaxFilesThe maximum number of journal files that can be kept on disk.

      systemd-journald will respect both SystemMaxUse and SystemKeepFree, and it will set your journals’ disk usage to meet whichever setting results in a smaller size.

      To view your default limits, run:

      sudo journalctl -u systemd-journald
      

      You should see a line similar to the following which describes the current limits in place:

        
      Permanent journal is using 32.0M (max allowed 2.3G, trying to leave 3.5G free of 21.2G available → current limit 2.3G).
      
      

      Note

      A parallel group of settings is used when journald.conf is set to only persist the journals in memory (instead of on disk): RuntimeMaxUse, RuntimeKeepFree, RuntimeMaxFileSize, and RuntimeMaxFiles.

      Manually Clean Up Archived Logs

      journalctl offers functions for immediately removing archived journals on disk. Run journalctl with the --vacuum-size option to remove archived journal files until the total size of your journals is less than the specified amount. For example, the following command will reduce the size of your journals to 2GiB:

      journalctl --vacuum-size=2G
      

      Run journalctl with the --vacuum-time option to remove archived journal files with dates older than the specified relative time. For example, the following command will remove journals older than one year:

      journalctl --vacuum-time=1years
      

      Run journalctl with the --vacuum-files option to remove archived journal files until the specified number of files remains. For example, the following command removes all but the 10 most recent journal files:

      journalctl --vacuum-files=10
      

      More Information

      You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

      Join our Community

      Find answers, ask questions, and help others.

      This guide is published under a CC BY-ND 4.0 license.



      Source link