One place for hosting & domains

      StandAlone

      How To Use Certbot Standalone Mode to Retrieve Let’s Encrypt SSL Certificates on Debian 9


      Introduction

      Let’s Encrypt is a service offering free SSL certificates through an automated API. The most popular Let’s Encrypt client is EFF’s Certbot.

      Certbot offers a variety of ways to validate your domain, fetch certificates, and automatically configure Apache and Nginx. In this tutorial, we’ll discuss Certbot’s standalone mode and how to use it to secure other types of services, such as a mail server or a message broker like RabbitMQ.

      We won’t discuss the details of SSL configuration, but when you are done you will have a valid certificate that is automatically renewed. Additionally, you will be able to automate reloading your service to pick up the renewed certificate.

      Prerequisites

      Before starting this tutorial, you will need:

      • A Debian 9 server with a non-root, sudo-enabled user and basic firewall set up, as detailed in this Debian 9 server setup tutorial.
      • A domain name pointed at your server, which you can accomplish by following “How to Set Up a Host Name with DigitalOcean.” This tutorial will use example.com throughout.
      • Port 80 or 443 must be unused on your server. If the service you’re trying to secure is on a machine with a web server that occupies both of those ports, you’ll need to use a different mode such as Certbot’s webroot mode or DNS-based challenge mode.

      Step 1 — Installing Certbot

      Debian 9 includes the Certbot client in their default repository, and it should be up-to-date enough for basic use. If you need to do DNS-based challenges or use other newer Certbot features, you should instead install from the stretch-backports repo as instructed by the official Certbot documentation.

      Use apt to install the certbot package:

      You may test your install by asking certbot to output its version number:

      Output

      certbot 0.28.0

      Now that we have Certbot installed, let's run it to get our certificate.

      Step 2 — Running Certbot

      Certbot needs to answer a cryptographic challenge issued by the Let's Encrypt API in order to prove we control our domain. It uses ports 80 (HTTP) or 443 (HTTPS) to accomplish this. Open up the appropriate port in your firewall:

      Substitute 443 above if that's the port you're using. ufw will output confirmation that your rule was added:

      Output

      Rule added Rule added (v6)

      We can now run Certbot to get our certificate. We'll use the --standalone option to tell Certbot to handle the challenge using its own built-in web server. The --preferred-challenges option instructs Certbot to use port 80 or port 443. If you're using port 80, you want --preferred-challenges http. For port 443 it would be --preferred-challenges tls-sni. Finally, the -d flag is used to specify the domain you're requesting a certificate for. You can add multiple -d options to cover multiple domains in one certificate.

      • sudo certbot certonly --standalone --preferred-challenges http -d example.com

      When running the command, you will be prompted to enter an email address and agree to the terms of service. After doing so, you should see a message telling you the process was successful and where your certificates are stored:

      Output

      IMPORTANT NOTES: - Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/example.com/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/example.com/privkey.pem Your cert will expire on 2019-08-28. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew *all* of your certificates, run "certbot renew" - Your account credentials have been saved in your Certbot configuration directory at /etc/letsencrypt. You should make a secure backup of this folder now. This configuration directory will also contain certificates and private keys obtained by Certbot so making regular backups of this folder is ideal. - If you like Certbot, please consider supporting our work by: Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate Donating to EFF: https://eff.org/donate-le

      We've got our certificates. Let's take a look at what we downloaded and how to use the files with our software.

      Step 3 — Configuring Your Application

      Configuring your application for SSL is beyond the scope of this article, as each application has different requirements and configuration options, but let's take a look at what Certbot has downloaded for us. Use ls to list out the directory that holds our keys and certificates:

      • sudo ls /etc/letsencrypt/live/example.com

      Output

      cert.pem chain.pem fullchain.pem privkey.pem README

      The README file in this directory has more information about each of these files. Most often you'll only need two of these files:

      • privkey.pem: This is the private key for the certificate. This needs to be kept safe and secret, which is why most of the /etc/letsencrypt directory has very restrictive permissions and is accessible by only the root user. Most software configuration will refer to this as something similar to ssl-certificate-key or ssl-certificate-key-file.
      • fullchain.pem: This is our certificate, bundled with all intermediate certificates. Most software will use this file for the actual certificate, and will refer to it in their configuration with a name like 'ssl-certificate'.

      For more information on the other files present, refer to the "Where are my certificates" section of the Certbot docs.

      Some software will need its certificates in other formats, in other locations, or with other user permissions. It is best to leave everything in the letsencrypt directory, and not change any permissions in there (permissions will just be overwritten upon renewal anyway), but sometimes that's just not an option. In that case, you'll need to write a script to move files and change permissions as needed. This script will need to be run whenever Certbot renews the certificates, which we'll talk about next.

      Step 4 — Handling Certbot Automatic Renewals

      Let's Encrypt's certificates are only valid for ninety days. This is to encourage users to automate their certificate renewal process. The certbot package we installed takes care of this for us by adding a renew script to /etc/cron.d. This script runs twice a day and will renew any certificate that's within thirty days of expiration.

      With our certificates renewing automatically, we still need a way to run other tasks after a renewal. We need to at least restart or reload our server to pick up the new certificates, and as mentioned in Step 3 we may need to manipulate the certificate files in some way to make them work with the software we're using. This is the purpose of Certbot's renew_hook option.

      To add a renew_hook, we update Certbot's renewal config file. Certbot remembers all the details of how you first fetched the certificate, and will run with the same options upon renewal. We just need to add in our hook. Open the config file with you favorite editor:

      • sudo nano /etc/letsencrypt/renewal/example.com.conf

      A text file will open with some configuration options. Add your hook on the last line:

      /etc/letsencrypt/renewal/example.com.conf

      renew_hook = systemctl reload rabbitmq
      

      Update the command above to whatever you need to run to reload your server or run your custom file munging script. Usually, on Debian, you’ll mostly be using systemctl to reload a service. Save and close the file, then run a Certbot dry run to make sure the syntax is ok:

      • sudo certbot renew --dry-run

      If you see no errors, you're all set. Certbot is set to renew when necessary and run any commands needed to get your service using the new files.

      Conclusion

      In this tutorial, we've installed the Certbot Let's Encrypt client, downloaded an SSL certificate using standalone mode, and enabled automatic renewals with renew hooks. This should give you a good start on using Let's Encrypt certificates with services other than your typical web server.

      For more information, please refer to Certbot's documentation.



      Source link

      How to Install Hadoop in Stand-Alone Mode on Debian 9


      Introduction

      Hadoop is a Java-based programming framework that supports the processing and storage of extremely large datasets on a cluster of inexpensive machines. It was the first major open source project in the big data playing field and is sponsored by the Apache Software Foundation.

      Hadoop is comprised of four main layers:

      • Hadoop Common is the collection of utilities and libraries that support other Hadoop modules.
      • HDFS, which stands for Hadoop Distributed File System, is responsible for persisting data to disk.
      • YARN, short for Yet Another Resource Negotiator, is the “operating system” for HDFS.
      • MapReduce is the original processing model for Hadoop clusters. It distributes work within the cluster or map, then organizes and reduces the results from the nodes into a response to a query. Many other processing models are available for the 3.x version of Hadoop.

      Hadoop clusters are relatively complex to set up, so the project includes a stand-alone mode which is suitable for learning about Hadoop, performing simple operations, and debugging.

      In this tutorial, you’ll install Hadoop in stand-alone mode and run one of the example example MapReduce programs it includes to verify the installation.

      Before you begin, you might also like to take a look at An Introduction to Big Data Concepts and Terminology or An Introduction to Hadoop

      Prerequisites

      To follow this tutorial, you will need:

      Step 1 — Installing Hadoop

      To install Hadoop, first visit the Apache Hadoop Releases page to find the most recent stable release.

      Navigate to binary for the release you’d like to install. In this guide, we’ll install Hadoop 3.0.3.

      Screenshot of the Hadoop releases page highlighting the link to the latest stable binary

      On the next page, right-click and copy the link to the release binary.

      Screenshot of the Hadoop mirror page

      On your server, use wget to fetch it:

      • wget http://www-us.apache.org/dist/hadoop/common/hadoop-3.0.3/hadoop-3.0.3.tar.gz

      Note: The Apache website will direct you to the best mirror dynamically, so your URL may not match the URL above.

      In order to ensure that the file you downloaded hasn’t been altered, do a quick check using SHA-256. Return to the releases page, then right-click and copy the link to the checksum file for the release binary you downloaded:

      Screenshot highlighting the .mds file

      Again, use wget on your server to download the file:

      • wget https://dist.apache.org/repos/dist/release/hadoop/common/hadoop-3.0.3/hadoop-3.0.3.tar.gz.mds

      Then run the verification:

      • sha256sum hadoop-3.0.3.tar.gz

      Output

      db96e2c0d0d5352d8984892dfac4e27c0e682d98a497b7e04ee97c3e2019277a hadoop-3.0.3.tar.gz

      Compare this value with the SHA-256 value in the .mds file:

      • cat hadoop-3.0.3.tar.gz.mds | grep SHA256

      ~/hadoop-3.0.3.tar.gz.mds

      ...
      SHA256 = DB96E2C0 D0D5352D 8984892D FAC4E27C 0E682D98 A497B7E0 4EE97C3E 2019277A
      

      You can safely ignore the difference in case and the spaces. The output of the command you ran against the file we downloaded from the mirror should match the value in the file you downloaded from apache.org.

      Now that you’ve verified that the file wasn’t corrupted or changed, use the tar command with the -x flag to extract, -z to uncompress, -v for verbose output, and -f to specify that you’re extracting the archive from a file. Use tab-completion or substitute the correct version number in the command below:

      • tar -xzvf hadoop-3.0.3.tar.gz

      Finally, move the extracted files into /usr/local, the appropriate place for locally installed software. Change the version number, if needed, to match the version you downloaded.

      • sudo mv hadoop-3.0.3 /usr/local/hadoop

      With the software in place, we’re ready to configure its environment.

      Step 3 — Running Hadoop

      Let’s make sure Hadoop runs. Execute the following command to launch Hadoop and display its help options:

      • /usr/local/hadoop/bin/hadoop

      You’ll see the following output, which lets you know you’ve successfully configured Hadoop to run in stand-alone mode.

      Output

      Usage: hadoop [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS] or hadoop [OPTIONS] CLASSNAME [CLASSNAME OPTIONS] where CLASSNAME is a user-provided Java class OPTIONS is none or any of: --config dir Hadoop config directory --debug turn on shell script debug mode --help usage information buildpaths attempt to add class files from build tree hostnames list[,of,host,names] hosts to use in slave mode hosts filename list of hosts to use in slave mode loglevel level set the log4j level for this command workers turn on worker mode SUBCOMMAND is one of: . . .

      We’ll ensure that it is functioning properly by running the example MapReduce program it ships with. To do so, create a directory called input in your home directory and copy Hadoop’s configuration files into it to use those files as our data.

      • mkdir ~/input
      • cp /usr/local/hadoop/etc/hadoop/*.xml ~/input

      Next, we’ll run the MapReduce hadoop-mapreduce-examples program, a Java archive with several options. We’ll invoke its grep program, one of the many examples included in hadoop-mapreduce-examples, followed by the input directory, input and the output directory grep_example. The MapReduce grep program will count the matches of a literal word or regular expression. Finally, we’ll supply the regular expression allowed[.]* to find occurrences of the word allowed within or at the end of a declarative sentence. The expression is case-sensitive, so we wouldn’t find the word if it were capitalized at the beginning of a sentence.

      Execute the following command:

      • /usr/local/hadoop/bin/hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.3.jar grep ~/input ~/grep_example 'allowed[.]*'

      When the task completes, it provides a summary of what has been processed and errors it has encountered, but this doesn’t contain the actual results:

      Output

      . . . File System Counters FILE: Number of bytes read=1330690 FILE: Number of bytes written=3128841 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=2 Map output records=2 Map output bytes=33 Map output materialized bytes=43 Input split bytes=115 Combine input records=0 Combine output records=0 Reduce input groups=2 Reduce shuffle bytes=43 Reduce input records=2 Reduce output records=2 Spilled Records=4 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=3 Total committed heap usage (bytes)=478150656 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=147 File Output Format Counters Bytes Written=34

      The results are stored in the ~/grep_example directory.

      If this output directory already exists, the program will fail, and rather than seeing the summary, you’ll see something like this:

      Output

      . . . at java.base/java.lang.reflect.Method.invoke(Method.java:564) at org.apache.hadoop.util.RunJar.run(RunJar.java:244) at org.apache.hadoop.util.RunJar.main(RunJar.java:158)

      Check the results by running cat on the output directory:

      You'll see this output:

      Output

      19 allowed. 1 allowed

      The MapReduce task found 19 occurrences of the word allowed followed by a period and one occurrence where it was not. Running the example program has verified that our stand-alone installation is working properly and that non-privileged users on the system can run Hadoop for exploration or debugging.

      Conclusion

      In this tutorial, we've installed Hadoop in stand-alone mode and verified it by running an example program it provided. To learn how to write your own MapReduce programs, visit Apache Hadoop's MapReduce tutorial which walks through the code behind the example you used in this tutorial. When you're ready to set up a cluster, see the Apache Foundation Hadoop Cluster Setup guide.



      Source link