One place for hosting & domains

      Logfiles

      How To Manage Logfiles with Logrotate on Ubuntu 20.04


      Introduction

      Logrotate is a system utility that manages the automatic rotation and compression of log files. If log files were not rotated, compressed, and periodically pruned, they could eventually consume all available disk space on a system.

      Logrotate is installed by default on Ubuntu 20.04, and is set up to handle the log rotation needs of all installed packages, including rsyslog, the default system log processor.

      In this article, we will explore the default Logrotate configuration, then configure log rotation for a fictional custom application.

      Prerequisites

      This tutorial assumes you have an Ubuntu 20.04 server, with a non-root sudo-enabled user, as described in Initial Server Setup with Ubuntu 20.04.

      Logrotate is available on many other Linux distributions as well, but the default configuration may be quite different. Other sections of this tutorial will still apply as long as your version of Logrotate is similar to Ubuntu 20.04’s. Follow Step 1 to determine your Logrotate version.

      Log into your server as your sudo-enabled user to begin.

      You can also use an interactive terminal that is embedded on this page to experiment with logrotate in this tutorial. Click the following Launch an Interactive Terminal! button to get started.

      Launch an Interactive Terminal!

      Step 1 — Confirming Your Logrotate Version

      Logrotate is installed by default on Ubuntu. However, if you need to install it, run the following commands to update your package list and retrieve the package:

      • sudo apt update
      • sudo apt install logrotate

      If you’re using a non-Ubuntu server, first make sure Logrotate is installed by asking for its version information:

      Output

      logrotate 3.14.0 Default mail command: /usr/bin/mail Default compress command: /bin/gzip Default uncompress command: /bin/gunzip Default compress extension: .gz Default state file path: /var/lib/logrotate/status ACL support: yes SELinux support: yes

      If Logrotate is installed but the version number is significantly different, you may have issues with some of the configuration options that are explored in this tutorial. Refer to the documentation for your specific version of Logrotate by reading its manual (man) page:

      You can also consult the online version of the Logrotate documentation. Next we’ll look at Logrotate’s default configuration structure on Ubuntu.

      Step 2 — Exploring the Logrotate Configuration

      Logrotate’s configuration information can generally be found in two places on Ubuntu:

      • /etc/logrotate.conf: this file contains some default settings and sets up rotation for a few logs that are not owned by any system packages. It also uses an include statement to pull in configuration from any file in the /etc/logrotate.d directory.
      • /etc/logrotate.d/: this is where any packages you install that need help with log rotation will place their Logrotate configuration. On a standard install you should already have files here for core system tools like apt, dpkg, rsyslog and so on.

      By default, logrotate.conf will configure weekly log rotations, with log files owned by the root user and the syslog group, with four log files being retained at a time(rotate 4), and new empty log files being created after the current one is rotated (create).

      Let’s take a look at a package’s Logrotate configuration file in /etc/logrotate.d. cat the file for the apt package utility:

      Output

      /var/log/apt/term.log { rotate 12 monthly compress missingok notifempty } /var/log/apt/history.log { rotate 12 monthly compress missingok notifempty }

      This file contains configuration blocks for two different log files in the /var/log/apt/ directory: term.log and history.log. They both have the same options. Any options not set in these configuration blocks will inherit the default values or those set in /etc/logrotate.conf. Any setting in a logrotate file will override logrotate’s default values, which are configured in /etc/logrotate.conf. The options set for the apt logs are:

      • rotate 12: keep twelve old log files. This overrides the rotate 4 default.
      • monthly: rotate once a month. This overrides the weekly default.
      • compress: compress the rotated files. this uses gzip by default and results in files ending in .gz. The compression command can be changed using the compresscmd option.
      • missingok: don’t write an error message if the log file is missing.
      • notifempty: don’t rotate the log file if it is empty.

      These configuration files also inherit the default create behaviour, which instructs Logrotate to create new logs after rotation. This could be overridden with nocreate, although that would effectively disable most of the other functionality.

      There are many more configuration options available. You can read about all of them by typing man logrotate on the command line to bring up Logrotate’s manual page.

      Next, we’ll set up a configuration file to handle logs for a fictional service called your-app.

      Step 3 — Setting Up an Example Config

      To manage log files using logrotate for applications outside of the pre-packaged and pre-configured system services, we have two options:

      1. Create a new Logrotate configuration file and place it in /etc/logrotate.d/. This will be run daily as the root user along with all the other standard Logrotate jobs.
      2. Create a new configuration file and run it outside of Ubuntu’s default Logrotate setup. This is only really necessary if you need to run Logrotate as a non-root user, or if you want to rotate logs more frequently than daily (an hourly configuration in /etc/logrotate.d/ would be ineffective, because the system’s Logrotate setup only runs once a day).

      Let’s walk through these two options with some example setups.

      Adding Configuration to /etc/logrotate.d/

      We want to configure log rotation for a fictional web server that puts an access.log and error.log into /var/log/your-app/. It runs as the www-data user and group.

      To add a configuration for the your-app log files to /etc/logrotate.d/, first open up a new file in the /etc/logrotate.d directory using nano or your preferred editor:

      • sudo nano /etc/logrotate.d/your-app

      Add the following lines to your new configuration file:

      /etc/logrotate.d/your-app

      /var/log/your-app/*.log {
          daily
          missingok
          rotate 14
          compress
          notifempty
          create 0640 www-data www-data
          sharedscripts
          postrotate
              systemctl reload your-app
          endscript
      }
      

      Some of the new configuration directives in this file are:

      • create 0640 www-data www-data: this creates a new empty log file after rotation, with the specified permissions (0640), owner (www-data), and group (also www-data).
      • sharedscripts: this flag means that any scripts added to the configuration are run only once per run, instead of for each file rotated. Since the path /var/log/your-app/*.log includes a wildcard *, this configuration would match any number of log files in the your-app directory. Without the sharedscripts option, the script specified in postrotate would run each time logrotate processes a log file without this option.
      • postrotate to endscript: this block contains a script to run after the log file is rotated. In this case we’re reloading our example app. This is sometimes necessary to get your application to switch over to the newly created log file.
        Note that postrotate runs before logs are compressed. Compression could take a long time, and your software should switch to the new log file immediately. For tasks that need to run after logs are compressed, use the lastaction block instead.

      To save and quit nano, press Ctrl+X, and when prompted, Y and then Enter. You can test the config file by doing a dry run:

      • sudo logrotate /etc/logrotate.conf --debug

      This command calls logrotate, points it to the standard configuration file, and turns on debug mode.

      Information will print out about which log files Logrotate is handling and what it would have done to them. If all looks well, you’re done. The standard Logrotate job will run once a day and include your new configuration.

      Next, we’ll try a setup that doesn’t use Ubuntu’s default configuration at all.

      Step 4 — Creating an Independent Logrotate Configuration

      In this example we have an app running as our user sammy, generating logs that are stored in /home/sammy/logs/. We want to rotate these logs hourly, so we need to set this up outside of the /etc/logrotate.d structure provided by Ubuntu.

      First, we’ll create a configuration file in our home directory. Open it in a text editor:

      • nano /home/sammy/logrotate.conf

      Then paste in the following configuration:

      /home/sammy/logrotate.conf

      /home/sammy/logs/*.log {
          hourly
          missingok
          rotate 24
          compress
          create
      }
      

      Save and close the file. We’ve encountered all these options in previous steps, but let’s summarize: this configuration will rotate the files hourly, compressing and keeping twenty-four old logs and creating a new log file to replace the rotated one.

      You’ll need to customize the configuration to suit your application, but this is a good start.

      To test that the configuration works, let’s make a log file. First cd to your user’s home directory using the cd ~ command. Then create a directory for the logs using the mkdir command. Finally create an empty file in the logs directory using the touch command. Run the following commands in order to complete these steps:

      • cd ~
      • mkdir logs
      • touch logs/access.log

      Now that we have a blank log file in the right spot, let’s run the logrotate command.

      Because the logs are owned by sammy we don’t need to use sudo. However, we do need to specify a state file. This file records what logrotate found and any actions that it took the last time it ran, so that it knows what to do the next time it runs. This state tracking is handled for us when using the default /etc/logrotate.conf configuration. The state file is stored in /var/lib/logrotate/status. Since we are not using the default configuration, we’ll need to configure the state file location manually.

      We’ll have Logrotate put the state file right in our home directory for this example. It can go anywhere that’s accessible and convenient. Run the following command to use the /home/sammy/logrotate.conf that you created and to record the state that logrotate encounters:

      logrotate /home/sammy/logrotate.conf --state /home/sammy/logrotate-state --verbose
      

      Output

      reading config file /home/sammy/logrotate.conf Handling 1 logs rotating pattern: /home/sammy/logs/*.log hourly (24 rotations) empty log files are rotated, old logs are removed considering log /home/sammy/logs/access.log log does not need rotating

      The --verbose flag will print out detailed information about what Logrotate is doing. In this case it did not rotate anything. This is Logrotate’s first time encountering this log file and it is zero hours old so shouldn’t be rotated.

      If we examine at the state file using the cat utility, we’ll note that Logrotate recorded some information about the run:

      • cat /home/sammy/logrotate-state

      Output

      logrotate state -- version 2 "/home/sammy/logs/access.log" 2021-12-3-19:0:0

      Logrotate noted the logs that it saw and when it last considered them for rotation. If we run this same command one hour later, the log will be rotated as expected.

      If you want to force Logrotate to rotate the log file when it otherwise would not have, use the --force flag:

      • logrotate /home/sammy/logrotate.conf --state /home/sammy/logrotate-state --verbose --force

      This is useful when testing postrotate and other scripts.

      Finally, we need to set up a cron job to run Logrotate every hour. Open your user’s crontab:

      This will open a text file. If it is your first time using cron, you may be prompted to choose a default text editor. If you do not have a preference, we recommend nano for new users. There may be some comments already in the file that explain the cron syntax. Move the cursor down to a new blank line at the end of the file and add the following:

      crontab

      14 * * * * /usr/sbin/logrotate /home/sammy/logrotate.conf --state /home/sammy/logrotate-state

      This task will run on the 14th minute of every hour, every day. It runs nearly the same logrotate command we ran previously, though we expanded logrotate to its full path of /usr/sbin/logrotate to be safe. It’s good practice to use full paths when writing cron jobs. To learn more about cron, you can review our other tutorials.

      Save the file and exit. This will install the crontab and our task will run on the specified schedule.

      If we revisit our log directory in about an hour we should find the rotated and compressed log file access.log.1.gz (or .2.gz if you ran Logrotate with the --force flag).

      Conclusion

      In this tutorial we verified our Logrotate version, explored the default Ubuntu Logrotate configuration, and set up two different types of custom configurations. To learn more about the command line and configuration options available for Logrotate, you can read its manual page by running man logrotate in your terminal or by visiting the online documentation.



      Source link