One place for hosting & domains

      Troubleshoot

      How To Troubleshoot Common HAProxy Errors



      Part of the Series:
      Common HAProxy Errors

      This tutorial series explains how to troubleshoot and fix some of the most common errors that you may encounter when using the HAProxy TCP and HTTP proxy server.

      Each tutorial in this series includes descriptions of common HAProxy configuration, network, filesystem, or permission errors. The series begins with an overview of the commands and log files that you can use to troubleshoot HAProxy. Subsequent tutorials examine specific errors in detail.

      Introduction

      There are three main commands, and a common log location that you can use to get started troubleshooting HAProxy errors. Generally when you are troubleshooting HAProxy, you will use these commands in the order indicated here, and then examine the log file for specific diagnostic data.

      The commands and log that you will commonly use to troubleshoot HAProxy across most Linux distributions are:

      • systemctl – Used to control and interact with Linux services via the systemd service manager.
      • journalctl – Used to query and view the logs that are generated by systemd.
      • haproxy – When troubleshooting, this command is used to check HAProxy’s configuration.
      • /var/log/haproxy.log – This file contains log entries from HAProxy itself detailing TCP and HTTP traffic that is being handled by the server.

      These commands, how to use them, and HAProxy’s logs where you can find additional information about errors are described in further detail in the following sections.

      systemctl Commands for HAProxy

      To troubleshoot common HAProxy errors using the systemd service manager, the first step is to inspect the state of the HAProxy processes on your system. The following systemctl commands will query systemd for the state of HAProxy’s processes on most Linux distributions.

      • sudo systemctl status haproxy.service -l --no-pager

      The -l flag will ensure that output is not truncated or ellipsized. The --no-pager flag will make sure that output will go directly to your terminal without requiring any interaction on your part to view it. If you omit the --no-pager flag you will be able to scroll through the output using arrow keys, or the page up and down keys. To quit from the pager use the q key. You should receive output like this:

      Output

      ● haproxy.service - HAProxy Load Balancer Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2020-08-20 19:30:11 UTC; 5s ago Docs: man:haproxy(1) file:/usr/share/doc/haproxy/configuration.txt.gz Process: 487 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS) Main PID: 488 (haproxy) Tasks: 2 (limit: 2344) . . . Aug 19 21:31:46 d6cdd0c71489 systemd[1]: Started HAProxy Load Balancer.

      Your output may be slightly different depending on which Linux distribution you are using, but in any case, make a note of the Active line in the output. If your HAProxy server does not show active (running) as highlighted in the example output but you expect it should, there may be an error. Typically if there is a problem, you will have a line like the following in your output (note the highlighted failed portion):

      Example Error Output

      Active: failed (Result: exit-code) since Thu 2020-08-20 19:32:26 UTC; 6s ago

      If there is a problem with your HAProxy process or configuration you can troubleshoot it further using the journalctl command.

      journalctl Commands for HAProxy

      To inspect the systemd logs for HAProxy, you can use the journalctl command. The systemd logs for HAProxy will usually indicate whether there is a problem with starting or managing the HAProxy process.

      These logs are separate from HAProxy’s request and error logs. journalctl displays logs from systemd that describe the HAProxy service itself, from startup to shutdown, along with any process errors that may be encountered along the way.

      • sudo journalctl -u haproxy.service --since today --no-pager

      The --since today flag will limit the output of the command to log entries beginning at 00:00:00 of the current day only. Using this option will help restrict the volume of log entries that you need to examine when checking for errors. You should receive output like the following (there may be a few extra lines between the Starting and Started lines depending on your Linux distribution):

      Output

      Aug 20 19:37:08 d6cdd0c71489 systemd[1]: Starting HAProxy Load Balancer... . . . Aug 20 19:37:08 d6cdd0c71489 systemd[1]: Started HAProxy Load Balancer.

      If there is an error, you will have a line in the output that is similar to the following, with the main difference between Linux distributions being the highlighted yourhostname portion:

      Example Error Output

      Aug 20 19:32:25 yourhostname systemd[1]: Failed to start HAProxy Load Balancer.

      If your HAProxy server has errors in the journalctl logs like the previous example, then the next step to troubleshoot possible issues is investigating HAProxy’s configuration using the haproxy command line tool.

      Troubleshooting with haproxy

      To troubleshoot HAProxy configuration issues, use the haproxy -c command. The tool will parse your HAProxy files and detect any errors or missing settings before attempting to start the server.

      Run the command like this on Ubuntu, Debian, CentOS, and Fedora based distributions. Be sure to change the path to the configuration file if you are using a different filename or location:

      • sudo haproxy -c -f /etc/haproxy/haproxy.cfg

      A working HAProxy configuration will result in output like the following:

      Output

      Configuration file is valid

      If there is an error in your HAProxy configuration, like a typo or misplaced directive, haproxy -c will detect it and attempt to notify you about the problem.

      For example, attempting to use the bind directive in haproxy.cfg in the wrong location will result in messages like the following:

      Example Error Output

      [ALERT] 232/194354 (199) : parsing [/etc/haproxy/haproxy.cfg:13] : unknown keyword 'bind' in 'global' section [ALERT] 232/194354 (199) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg [ALERT] 232/194354 (199) : Fatal errors found in configuration.

      In this example the bind directive is misplaced inside a global configuration section, so HAProxy generates the unknown keyword error. The message also includes a line number 13, so that you can edit the file and fix or remove the erroneous line without having to search through the file.

      Learning how to use haproxy -c to detect and fix errors is useful when you are troubleshooting an existing error, or before you reload HAProxy with an edited configuration that may contain errors.

      HAProxy Log Files

      HAProxy log files are a very helpful resource for troubleshooting. Generally, any error that you receive in a browser or other HTTP client will have a corresponding entry in HAProxy’s logs. Sometimes HAProxy will also output errors related to configuration and other debugging information to its log files.

      On Ubuntu and Debian based Linux distributions, the haproxy package includes scripts that configure log output in /var/log/haproxy.log.

      On CentOS, Fedora, and other RedHat-derived Linux distributions, haproxy does not output to a log file by default. To log HAProxy output logs to /var/log/haproxy.log, follow this quickstart tutorial, How To Configure HAProxy Logging with Rsyslog on CentOS 8.

      When you are troubleshooting HAProxy using its log file, examine /var/log/haproxy.log for errors using a tool like tail or less. For example, to view the last two lines of the log using tail, run the following command:

      • sudo tail -n 2 /var/log/haproxy.log

      An example error will resemble something like the following lines, regardless of which Linux distribution you are using to run your HAProxy server:

      Log Examples

      Aug 20 19:36:21 d6cdd0c71489 haproxy[19202]: [ALERT] 258/134605 (19202) : Proxy 'app', server 'app1' [/etc/haproxy/haproxy.cfg:88] verify is enabled by default but no CA file specified. If you're running on a LAN where you're certain to trust the server's certificate, please set an explicit 'verify none' statement on the 'server' line, or use 'ssl-server-verify none' in the global section to disable server-side verifications by default. Aug 20 19:36:22 d6cdd0c71489 haproxy[4451]: 203.0.113.1:54428 [20/Aug/2020:19:36:22.288] main app/<NOSRV> 0/-1/-1/-1/1 503 212 - - SC-- 1/1/0/0/0 0/0 "GET / HTTP/1.1"

      These example lines are just for illustration purposes. If you are diagnosing errors with your HAProxy server, chances are the lines in your logs will have different contents than these. Some lines will include success responses and other non-critical diagnostic entries.

      Regardless of your Linux distribution, the format of the lines in your HAProxy logs will include any HTTP status codes that are returned to clients, along with requesting IPs and the status of backend servers.

      Once you have an idea of what might be causing problems with your HAProxy server you can continue researching and troubleshooting the issue. The HTTP status code and text description are especially useful, since they give you explicit and specific terms that you can use to narrow down the range of possible causes of a problem.

      Conclusion

      Troubleshooting HAProxy errors can range from diagnosing errors with the service itself, to locating misconfigured options for modules, or to examining customized access control rules in detail. This introduction to diagnosing issues with HAProxy explained how to use a number of utilities to help narrow down the possible causes of errors. Usually, you will use these utilities in the same order, although you can always skip some, or start directly with examining logs if you have a general idea of what the problem might be.

      However, as a general sequence for troubleshooting, it helps to be methodical and use these tools in the order described. Start troubleshooting with systemctl to examine the state of the HAProxy server. If you need more information, examine the systemd logs for HAProxy using the journalctl command. If the issue is still not apparent after checking journalctl, testing HAProxy’s configuration using haproxy -c -f /etc/haproxy/haproxy.cfg is the next step. Finally, for in-depth troubleshooting, examining HAProxy’s log files will usually indicate a specific error, with helpful diagnostic messages and error codes.

      The rest of the tutorials in this series will examine some common errors that you may encounter when using HAProxy in more detail.



      Source link

      How to Troubleshoot and Fix a Brute-Force Attack in WordPress on a DigitalOcean Droplet


      Introduction

      While running a WordPress installation through a hosting service can be a convenient way to start a website, it’s not without security vulnerabilities that may sometimes be hard to troubleshoot. Brute-force attacks, cyberattacks that rapidly work to guess and access personal information like logins or passwords, happen when these vulnerabilities are exploited, and can sometimes originate from your website.

      When facing brute-force attacks from your Droplets on DigitalOcean, it’s imperative to remove the threat quickly. While there are a number of ways to identify and remove compromised files vulnerable to attack, this tutorial aims to provide you with some steps to help you detect, resolve, and secure your WordPress installation(s) across DigitalOcean Droplets from vulnerabilities in the future.

      Step 1: Identify the Source of the Brute-Force Attack

      The first step in troubleshooting an issue with a brute-force attack initiated from your Droplet is to identify the malware responsible for the malicious traffic. There are numerous tools and options available, but ClamAV ( http://www.clamav.net/ ) is a good tool to initially attempt to identify and remove the malware.

      Most Linux distributions have ClamAV in their package management system, and typically you’ll need to install ClamAV and then run it.

      • For Ubuntu, Debian, and most Debian-based distributions, you can run:
      • sudo apt-get install clamav clamav-daemon
      • For CentOS 8 you need to enable the EPEL ( https://fedoraproject.org/wiki/EPEL ) repo, which is an official repository of packages supported by the Fedora project, and then install ClamAV.

      You can do so with a single command:

      • dnf --enablerepo=epel -y install clamav clamav-update

      Once ClamAV is installed, you can scan your system with:

      • clamscan --infected --recursive /path/to/wordpress/sites

      Replace the highlighted path with the correct path for your WordPress site. The --recursive parameter will make sure that the command is configured to recurse through subdirectories, and the path we used in this example points to the root folder where all WordPress installations are located. This way, with a single command you can scan all your WordPress sites. ClamAV will then return a list of all files it finds suspicious, but will not take any action yet. After investigating which files ClamAV detected as suspicious and confirming they can be safely removed without causing further damage to your system, you might want to re-run the command with the --remove option to remove the infected files.

      Note:
      --remove will delete any files it finds suspicious with no input from you, so it is NOT RECOMMENDED to run with --remove as your first scan until you can confirm the results.

      In cases where ClamAV does not find any malware, you will need to manually investigate and find the malware. While there are several ways to do this, a good starting point is to find and identify any recently uploaded files, based on the file’s timestamp information.

      To do this, use the ‘find’ command:

      • find /path/to/wordpress/site -mtime -DAYS

      To use this command, replace the /path/to/wordpress/site with the file path to your WordPress site, and -DAYS with how many days to go back. For example, if you wanted to look back 1 day, it would be -1; to look back 10 days, it would be -10.

      Take time to investigate any files that were uploaded or modified that you’re unaware of.

      Step 2: Update your WordPress Installation

      After identifying the malware, the next step to preventing malicious attacks from reoccurring is to update your WordPress installation. It’s wise to patch WordPress and any themes or plugins installed, to ensure that, if the compromise was in a plugin or theme’s install directory, you have removed and reinstalled that plugin or theme. You may be able to remove all malicious files, but in most cases, a clean installation of a compromised component is preferred.

      You can perform these updates from within WordPress’ administration UI in most cases, which doesn’t require the use of any additional tools. WordPress also offers an automatic update option that you’re encouraged to enable in order to reduce the time your websites might be vulnerable to newly discovered security issues.

      Another helpful piece of advice in preventing malicious attacks is to update all components, even the ones that are marked as inactive. In some situations, even disabled plugins and themes may be accessible and able to be compromised if not kept updated. If you’re sure you don’t need a theme or plugin, the best course of action would be to remove it in its entirety.

      In some cases, a theme or plugin may be abandoned by the author, and while you have the most recent version installed, the plugin or theme may have an issue that has not been fixed. In this case, you may need to consider other options for substituting the abandoned component that is currently updated, but was still the source of a compromise.

      Step 3: Secure Your WordPress Installation Against Malicious Attacks

      Once you have both removed any malicious files and ensured all components are updated, it’s time to secure your WordPress installation. The next step we recommend is to change all passwords for users that have access to the administration UI, especially those that have full admin rights, or the ability to upload or modify file contents.

      Checking your filesystem permissions if you’re not aware of the current configuration is also an important step in securing your WordPress installation, as the wrong permissions can allow file read and write access you didn’t intend. WordPress provides a good outline of what the settings should be and how to update them here.

      As a step in securing your Droplet’s installation, you can also install a plugin to limit the amount of failed login attempts, which dramatically reduces the risk of brute force attacks. The wp-limit-login-attempts plugin is a popular option to use.

      Finally, consider using a WordPress security plugin like Jetpack or Wordfence. These plugins help actively combat intrusion attempts and provide a final layer of security to ensure that your site is only used for what you intend.

      An alternative to using a server-side plugin like Jetpack or Wordfence would be to investigate if Cloudflare’s caching and Web Application Firewall (WAF) service might be a good fit for your specific use case. To learn more about this option, check out CloudFlare’s documentation.

      Conclusion

      Navigating troubleshooting options when brute-force attacks originate from your Droplets can be cumbersome, but in this tutorial, we shared some steps to help you detect, resolve, and secure your WordPress installation(s) across Droplets. For more security-related information to help manage Droplets, check out our Recommended Security Measures article.



      Source link

      How To Troubleshoot Common Apache Errors



      Part of the Series:
      Common Apache Errors

      This tutorial series explains how to troubleshoot and fix some of the most common errors that you may encounter when using the Apache web server.

      Each tutorial in this series includes descriptions of common Apache configuration, network, filesystem, or permission errors. The series begins with an overview of the commands and log files that you can use to troubleshoot Apache. Subsequent tutorials examine specific errors in detail.

      Series Description

      This tutorial series explains how to troubleshoot and fix some of the most common errors that you may encounter when using the Apache web server.

      Each tutorial in this series includes descriptions of common Apache configuration, network, filesystem, or permission errors. The series begins with an overview of the commands and log files that you can use to troubleshoot Apache. Subsequent tutorials examine specific errors in detail.

      Introduction

      There are three main commands, and a set of common log locations that you can use to get started troubleshooting Apache errors. Generally when you are troubleshooting Apache, you will use these commands in the order indicated here, and then examine log files for specific diagnostic data.

      The commands that you will commonly use to troubleshoot Apache across most Linux distributions are:

      • systemctl – Used to control and interact with Linux services via the systemd service manager.
      • journalctl – Used to query and view the logs that are generated by systemd.
      • apachectl – When troubleshooting, this command is used to check Apache’s configuration.

      These commands, how to use them, and Apache’s log locations where you can find additional information about errors are described in further detail in the following sections.

      Note: On Debian and Ubuntu systems, the Apache service and process name is apache2, whereas on CentOS, Fedora, and other RedHat-derived systems, Apache’s service and process name is httpd. Apart from the differences between the service and running process names, starting, stopping, and checking Apache’s status, as well as logs with journalctl should work the same on any Linux system that uses systemd to manage the Apache service. Be sure to use the correct name for your Linux distribution.

      systermctl Commands for Apache

      To troubleshoot common Apache errors using the systemd service manager, the first step is to inspect the state of the Apache processes on your system. The following systemctl commands will query systemd for the state of Apache’s processes.

      On Ubuntu and Debian systems run:

      • sudo systemctl status apache2.service -l --no-pager

      The -l flag will ensure that output is not truncated or ellipsized. The --no-pager flag will make sure that output will go directly to your terminal without requiring any interaction on your part to view it. You should receive output like this:

      Output

      ● apache2.service - The Apache HTTP Server Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled) Drop-In: /lib/systemd/system/apache2.service.d └─apache2-systemd.conf Active: active (running) since Mon 2020-07-13 14:43:35 UTC; 1 day 4h ago Process: 929 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS) Main PID: 1346 (apache2) Tasks: 55 (limit: 4702) CGroup: /system.slice/apache2.service ├─1346 /usr/sbin/apache2 -k start . . .

      To inspect the Apache process on CentOS and Fedora systems run:

      • sudo systemctl status httpd.service -l --no-pager

      You should receive output like this:

      Output

      ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2020-07-14 19:46:52 UTC; 3s ago Docs: man:httpd.service(8) Main PID: 21217 (httpd) Status: "Started, listening on: port 80" Tasks: 213 (limit: 2881) Memory: 16.6M CGroup: /system.slice/httpd.service ├─21217 /usr/sbin/httpd -DFOREGROUND . . . Jul 14 19:46:52 localhost.localdomain httpd[21217]: Server configured, listening on: port 80

      In either case, make a note of the Active line in the output. If your Apache server does not show active (running) as highlighted in the previous examples but you expect it should, there may be an error. Typically if there is a problem, you will have a line like the following in your output (note the highlighted failed portion):

      Example Error Output

      Active: failed (Result: exit-code) since Tue 2020-07-14 20:01:29 UTC; 1s ago

      If there is a problem with your Apache process or configuration you can troubleshoot it further using the journalctl command.

      Journalctl Commands for Apache

      To inspect the systemd logs for Apache, you can use the journalctl command. The systemd logs for Apache will usually indicate whether there is a problem with starting or managing the Apache process.

      These logs are separate from Apache’s request and error logs. journalctl displays logs from systemd that describe the Apache service itself, from startup to shutdown, along with any process errors that may be encountered along the way.

      On Ubuntu and Debian systems use the following command to examine the logs:

      • sudo journalctl -u apache2.service --since today --no-pager

      The --since today flag will limit the output of the command to log entries beginning at 00:00:00 of the current day only. Using this option will help restrict the volume of log entries that you need to examine when checking for errors. You should receive output like the following:

      Output

      Jul 14 20:12:14 ubuntu2004 systemd[1]: Starting The Apache HTTP Server... Jul 14 20:12:14 ubuntu2004 systemd[1]: Started The Apache HTTP Server.

      If you are using a CentOS or Fedora based system, use this version of the command:

      • sudo journalctl -u httpd.service --since today --no-pager

      You should receive output like the following:

      Output

      Jul 14 20:13:09 centos8 systemd[1]: Starting The Apache HTTP Server... . . . Jul 14 20:13:10 centos8 httpd[21591]: Server configured, listening on: port 80

      If there is an error, you will have a line in the output that is similar to the following, with the main difference between Linux distributions being the highlighted yourhostname portion:

      Example Error Output

      Jul 14 20:13:37 yourhostname systemd[1]: Failed to start The Apache HTTP Server.

      If your Apache server has errors in the journalctl logs like the previous example, then the next step to troubleshoot possible issues is investigating Apache’s configuration using the apachectl command line tool.

      Troubleshooting with apachectl

      Most Linux distributions include the apachectl utility with Apache. apachectl is an invaluable tool to help detect and diagnose Apache configuration problems.

      To troubleshoot issues using apachectl, test your Apache configuration using the apachectl configtest command. The tool will parse your Apache files and detect any errors or missing settings before attempting to start the server.

      Run the command like this on Ubuntu, Debian, CentOS, and Fedora based distributions:

      • sudo apachectl configtest

      A working Apache configuration will result in output like the following:

      Output

      Syntax OK

      Depending on your Linux distribution, there may be other lines mixed in with the output, but the important line is the one that says Syntax OK.

      If there is an error in your Apache configuration, like a directive that references a module that is not enabled or even a single typo, apachectl will detect it and attempt to notify you about the problem.

      For example, attempting to use directives for an Apache module that is not enabled will result in apachectl configtest messages like the following:

      Example Error Output

      AH00526: Syntax error on line 232 of /etc/apache2/apache2.conf: Invalid command 'SSLEngine', perhaps misspelled or defined by a module not included in the server configuration Action 'configtest' failed. The Apache error log may have more information.

      In this example the ssl module is not enabled, so the SSLEngine directive generates an error when the configuration is tested. The last line also indicates that The Apache error log may have more information, which is the next place to look for more detailed debugging information.

      Apache Log Files

      Apache log files are a very helpful resource for troubleshooting. Generally, any error that you receive in a browser or other HTTP client will have a corresponding entry in Apache’s logs. Sometimes Apache will also output errors related to configuration, built-in modules, and other debugging information to its log files.

      To examine log files for errors while troubleshooting Apache on a Fedora, CentOS, or RedHat server, examine the /var/log/httpd/error_log file.

      If you are troubleshooting a Debian or Ubuntu derived system, examine /var/log/apache2/error.log for errors using a tool like tail or less. For example, to view the last two lines of the error log using tail, run the following command:

      • sudo tail -n 2 /var/log/apache2/error.log

      Substitute the number of lines that you would like to examine in place of the number 2 in the command. On a CentOS or Fedora system, the log file to examine is /var/log/httpd/error_log.

      An example error will resemble something like the following lines, regardless of which Linux distribution you are using to run your Apache server:

      Error Log Examples

      [Wed Jul 15 01:34:12.093005 2020] [proxy:error] [pid 13949:tid 140150453516032] (13)Permission denied: AH00957: HTTP: attempt to connect to 127.0.0.1:9090 (127.0.0.1) failed [Wed Jul 15 01:34:12.093078 2020] [proxy_http:error] [pid 13949:tid 140150453516032] [client 127.0.0.1:42480] AH01114: HTTP: failed to make connection to backend: 127.0.0.1

      The two lines in this output are distinct error messages. They both reference the module causing the error (proxy in the first line, proxy_http in the second) and include an error code that is specific to the module. The first one, AH00957, indicates that the Apache server attempted to connect to a backend server (127.0.0.1 on port 9090 in this case) using the proxy module but failed to do so.

      The second error is derived from the first: AH01114 is a proxy_http module error that also indicates that Apache was unable to connect to the configured backend server to make an HTTP request.

      These example lines are just for illustration purposes. If you are diagnosing errors with your Apache server, chances are the error lines in your logs will have different contents than these. Regardless of your Linux distribution, the format of any error lines in your logs will include the relevant Apache module and error code, as well as a text description of the error.

      Once you have an idea of what might be causing problems with your Apache server you can continue researching and troubleshooting the issue. The error code and text description are especially useful, since they give you explicit and specific terms that you can use to narrow down the range of possible causes of a problem.

      Conclusion

      Troubleshooting Apache errors can range from diagnosing errors with the service itself, to locating misconfigured options for modules, or to examining customized access control rules in detail. This introduction to diagnosing issues with Apache explained how to use a number of utilities to help narrow down the possible causes of errors. Usually, you will use these utilities in the same order, although you can always skip some, or start directly with examining logs if you have a general idea of what the problem might be.

      However, as a general sequence for troubleshooting, it helps to be methodical and use these tools in the order described. Start troubleshooting with systemctl to examine the state of the Apache server. If you need more information, examine the systemd logs for Apache using the journalctl command. If the issue is still not apparent after checking journalctl, testing Apache’s configuration using apachectl configtest is the next step. Finally, for in-depth troubleshooting, examining Apache’s log files will usually indicate a specific error, with helpful diagnostic messages and error codes.

      The rest of the tutorials in this series will examine some common errors that you may encounter when using Apache in more detail.



      Source link