One place for hosting & domains

      Apache

      How To Install and Configure an Apache ZooKeeper Cluster on Ubuntu 18.04


      The author selected Wikimedia Foundation Inc. to receive a donation as part of the Write for DOnations program.

      Introduction

      Apache ZooKeeper is open-source software that enables resilient and highly reliable distributed coordination. It is commonly used in distributed systems to manage configuration information, naming services, distributed synchronization, quorum, and state. In addition, distributed systems rely on ZooKeeper to implement consensus, leader election, and group management.

      In this guide, you will install and configure Apache ZooKeeper 3.4.13 on Ubuntu 18.04. To achieve resilience and high availability, ZooKeeper is intended to be replicated over a set of hosts, called an ensemble. First, you will create a standalone installation of a single-node ZooKeeper server and then add in details for setting up a multi-node cluster. The standalone installation is useful in development and testing environments, but a cluster is the most practical solution for production environments.

      Prerequisites

      Before you begin this installation and configuration guide, you’ll need the following:

      • The standalone installation needs one Ubuntu 18.04 server with a minimum of 4GB of RAM set up by following the Ubuntu 18.04 initial server setup guide, including a non-root user with sudo privileges and a firewall. You need two additional servers, set up by following the same steps, for the multi-node cluster.
      • OpenJDK 8 installed on your server, as ZooKeeper requires Java to run. To do this, follow the “Install Specific Versions of OpenJDK” step from the How To Install Java with `apt` on Ubuntu 18.04 guide.

      Because ZooKeeper keeps data in memory to achieve high throughput and low latency, production systems work best with 8GB of RAM. Lower amounts of RAM may lead to JVM swapping, which could cause ZooKeeper server latency. High ZooKeeper server latency could result in issues like client session timeouts that would have an adverse impact on system functionality.

      Step 1 — Creating a User for ZooKeeper

      A dedicated user should run services that handle requests over a network and consume resources. This practice creates segregation and control that will improve your environment’s security and manageability. In this step, you’ll create a non-root sudo user, named zk in this tutorial, to run the ZooKeeper service.

      First, log in as the non-root sudo user that you created in the prerequisites.

      ssh sammy@your_server_ip
      

      Create the user that will run the ZooKeeper service:

      Passing the -m flag to the useradd command will create a home directory for this user. The home directory for zk will be /home/zk by default.

      Set bash as the default shell for the zk user:

      • sudo usermod --shell /bin/bash zk

      Set a password for this user:

      Next, you will add the zk user to the sudo group so it can run commands in a privileged mode:

      In terms of security, it is recommended that you allow SSH access to as few users as possible. Logging in remotely as sammy and then using su to switch to the desired user creates a level of separation between credentials for accessing the system and running processes. You will disable SSH access for both your zk and root user in this step.

      Open your sshd_config file:

      • sudo nano /etc/ssh/sshd_config

      Locate the PermitRootLogin line and set the value to no to disable SSH access for the root user:

      /etc/ssh/sshd_config

      PermitRootLogin no
      

      Under the PermitRootLogin value, add a DenyUsers line and set the value as any user who should have SSH access disabled:

      /etc/ssh/sshd_config

      DenyUsers zk
      

      Save and exit the file and then restart the SSH daemon to activate the changes.

      • sudo systemctl restart sshd

      Switch to the zk user:

      The -l flag invokes a login shell after switching users. A login shell resets environment variables and provides a clean start for the user.

      Enter the password at the prompt to authenticate the user.

      Now that you have created, configured, and logged in as the zk user, you will create a directory to store your ZooKeeper data.

      Step 2 — Creating a Data Directory for ZooKeeper

      ZooKeeper persists all configuration and state data to disk so it can survive a reboot. In this step, you will create a data directory that ZooKeeper will use to read and write data. You can create the data directory on the local filesystem or on a remote storage drive. This tutorial will focus on creating the data directory on your local filesystem.

      Create a directory for ZooKeeper to use:

      • sudo mkdir -p /data/zookeeper

      Grant your zk user ownership to the directory:

      • sudo chown zk:zk /data/zookeeper

      chown changes the ownership and group of the /data/zookeeper directory so that the user zk, who belongs to the group zk, owns the data directory.

      You have successfully created and configured the data directory. When you move on to configure ZooKeeper, you will specify this path as the data directory that ZooKeeper will use to store its files.

      Step 3 — Downloading and Extracting the ZooKeeper Binaries

      In this step, you will manually download and extract the ZooKeeper binaries to the /opt directory. You can use the Advanced Packaging Tool, apt, to download ZooKeeper, but it may install an older version with different features. Installing ZooKeeper manually will give you full control to choose which version you would like to use.

      Since you are downloading these files manually, start by changing to the /opt directory:

      From your local machine, navigate to the Apache download page. This page will automatically provide you with the mirror closest to you for the fastest download. Click the link to the suggested mirror site, then scroll down and click zookeeper/ to view the available releases. Select the version of ZooKeeper that you would like to install. This tutorial will focus on using 3.4.13. Once you select the version, right click the binary file ending with .tar.gz and copy the link address.

      From your server, use the wget command along with the copied link to download the ZooKeeper binaries:

      • sudo wget http://apache.osuosl.org/zookeeper/zookeeper-3.4.13/zookeeper-3.4.13.tar.gz

      Extract the binaries from the compressed archive:

      • sudo tar -xvf zookeeper-3.4.13.tar.gz

      The .tar.gz extension represents a combination of TAR packaging followed by a GNU zip (gzip) compression. You will notice that you passed the flag -xvf to the command to extract the archive. The flag x stands for extract, v enables verbose mode to show the extraction progress, and f allows specifying the input, in our case zookeeper-3.4.13.tar.gz, as opposed to STDIN.

      Next, give the zk user ownership of the extracted binaries so that it can run the executables. You can change ownership like so:

      • sudo chown zk:zk -R zookeeper-3.4.13

      Next, you will configure a symbolic link to ensure that your ZooKeeper directory will remain relevant across updates. You can also use symbolic links to shorten directory names, which can lessen the time it takes to set up your configuration files.

      Create a symbolic link using the ln command.

      • sudo ln -s zookeeper-3.4.13 zookeeper

      Change the ownership of that link to zk:zk. Notice that you have passed a -h flag to change the ownership of the link itself. Not specifying -h changes the ownership of the target of the link, which you explicitly did in the previous step.

      • sudo chown -h zk:zk zookeeper

      With the symbolic links created, your directory paths in the configurations will remain relevant and unchanged through future upgrades. You can now configure ZooKeeper.

      Step 4 — Configuring ZooKeeper

      Now that you've set up your environment, you are ready to configure ZooKeeper.

      The configuration file will live in the /opt/zookeeper/conf directory. This directory contains a sample configuration file that comes with the ZooKeeper distribution. This sample file, named zoo_sample.cfg, contains the most common configuration parameter definitions and sample values for these parameters. Some of the common parameters are as follows:

      • tickTime: Sets the length of a tick in milliseconds. A tick is a time unit used by ZooKeeper to measure the length between heartbeats. Minimum session timeouts are twice the tickTime.
      • dataDir: Specifies the directory used to store snapshots of the in-memory database and the transaction log for updates. You could choose to specify a separate directory for transaction logs.
      • clientPort: The port used to listen for client connections.
      • maxClientCnxns: Limits the maximum number of client connections.

      Create a configuration file named zoo.cfg at /opt/zookeeper/conf. You can create and open a file using nano or your favorite editor:

      • nano /opt/zookeeper/conf/zoo.cfg

      Add the following set of properties and values to that file:

      /opt/zookeeper/conf/zoo.cfg

      tickTime=2000
      dataDir=/data/zookeeper
      clientPort=2181
      maxClientCnxns=60
      

      A tickTime of 2000 milliseconds is the suggested interval between heartbeats. A shorter interval could lead to system overhead with limited benefits. The dataDir parameter points to the path defined by the symbolic link you created in the previous section. Conventionally, ZooKeeper uses port 2181 to listen for client connections. In most situations, 60 allowed client connections are plenty for development and testing.

      Save the file and exit the editor.

      You have configured ZooKeeper and are ready to start the server.

      Step 5 — Starting ZooKeeper and Testing the Standalone Installation

      You've configured all the components needed to run ZooKeeper. In this step, you will start the ZooKeeper service and test your configuration by connecting to the service locally.

      Navigate back to the /opt/zookeeper directory.

      Start ZooKeeper with the zkServer.sh command.

      You will see the following on your standard output:

      Output

      ZooKeeper JMX enabled by default Using config: /opt/zookeeper/bin/../conf/zoo.cfg Starting zookeeper ... STARTED

      Connect to the local ZooKeeper server with the following command:

      • bin/zkCli.sh -server 127.0.0.1:2181

      You will get a prompt with the label CONNECTED. This confirms that you have a successful local, standalone ZooKeeper installation. If you encounter errors, you will want to verify that the configuration is correct.

      Output

      Connecting to 127.0.0.1:2181 ... ... [zk: 127.0.0.1:2181(CONNECTED) 0]

      Type help on this prompt to get a list of commands that you can execute from the client. The output will be as follows:

      Output

      [zk: 127.0.0.1:2181(CONNECTED) 0] help ZooKeeper -server host:port cmd args stat path [watch] set path data [version] ls path [watch] delquota [-n|-b] path ls2 path [watch] setAcl path acl setquota -n|-b val path history redo cmdno printwatches on|off delete path [version] sync path listquota path rmr path get path [watch] create [-s] [-e] path data acl addauth scheme auth quit getAcl path close connect host:port

      After you've done some testing, you will close the client session by typing quit on the prompt. The ZooKeeper service will continue running after you closed the client session. Shut down the ZooKeeper service, as you'll configure it as a systemd service in the next step:

      You have now installed, configured, and tested a standalone ZooKeeper service. This setup is useful to familiarize yourself with ZooKeeper, but is also helpful for developmental and testing environments. Now that you know the configuration works, you will configure systemd to simplify the management of your ZooKeeper service.

      Step 6 — Creating and Using a Systemd Unit File

      The systemd, system and service manager, is an init system used to bootstrap the user space and to manage system processes after boot. You can create a daemon for starting and checking the status of ZooKeeper using systemd.

      Systemd Essentials is a great introductory resource for learning more about systemd and its constituent components.

      Use your editor to create a .service file named zk.service at /etc/systemd/system/.

      • sudo nano /etc/systemd/system/zk.service

      Add the following lines to the file to define the ZooKeeper Service:

      /etc/systemd/system/zk.service

      [Unit]
      Description=Zookeeper Daemon
      Documentation=http://zookeeper.apache.org
      Requires=network.target
      After=network.target
      
      [Service]    
      Type=forking
      WorkingDirectory=/opt/zookeeper
      User=zk
      Group=zk
      ExecStart=/opt/zookeeper/bin/zkServer.sh start /opt/zookeeper/conf/zoo.cfg
      ExecStop=/opt/zookeeper/bin/zkServer.sh stop /opt/zookeeper/conf/zoo.cfg
      ExecReload=/opt/zookeeper/bin/zkServer.sh restart /opt/zookeeper/conf/zoo.cfg
      TimeoutSec=30
      Restart=on-failure
      
      [Install]
      WantedBy=default.target
      

      The Service section in the unit file configuration specifies the working directory, the user under which the service would run, and the executable commands to start, stop, and restart the ZooKeeper service. For additional information on all the unit file configuration options, you can read the Understanding Systemd Units and Unit Files article.

      Save the file and exit the editor.

      Now that your systemd configuration is in place, you can start the service:

      Once you've confirmed that your systemd file can successfully start the service, you will enable the service to start on boot.

      This output confirms the creation of the symbolic link:

      Output

      Created symlink /etc/systemd/system/multi-user.target.wants/zk.service → /etc/systemd/system/zk.service.

      Check the status of the ZooKeeper service using:

      Stop the ZooKeeper service using systemctl.

      Finally, to restart the daemon, use the following command:

      • sudo systemctl restart zk

      The systemd mechanism is becoming the init system of choice on many Linux distributions. Now that you've configured systemd to manage ZooKeeper, you can leverage this fast and flexible init model to start, stop, and restart the ZooKeeper service.

      Step 7 — Configuring a Multi-Node ZooKeeper Cluster

      While the standalone ZooKeeper server is useful for development and testing, every production environment should have a replicated multi-node cluster.

      Nodes in the ZooKeeper cluster that work together as an application form a quorum. Quorum refers to the minimum number of nodes that need to agree on a transaction before it's committed. A quorum needs an odd number of nodes so that it can establish a majority. An even number of nodes may result in a tie, which would mean the nodes would not reach a majority or consensus.

      In a production environment, you should run each ZooKeeper node on a separate host. This prevents service disruption due to host hardware failure or reboots. This is an important and necessary architectural consideration for building a resilient and highly available distributed system.

      In this tutorial, you will install and configure three nodes in the quorum to demonstrate a multi-node setup. Before you configure a three-node cluster, you will spin up two additional servers with the same configuration as your standalone ZooKeeper installation. Ensure that the two additional nodes meet the prerequisites, and then follow steps one through six to set up a running ZooKeeper instance.

      Once you've followed steps one through six for the new nodes, open zoo.cfg in the editor on each node.

      • sudo nano /opt/zookeeper/conf/zoo.cfg

      All nodes in a quorum will need the same configuration file. In your zoo.cfg file on each of the three nodes, add the additional configuration parameters and values for initLimit, syncLimit, and the servers in the quorum, at the end of the file.

      /opt/zookeeper/conf/zoo.cfg

      tickTime=2000
      dataDir=/data/zookeeper
      clientPort=2181
      maxClientCnxns=60
      initLimit=10
      syncLimit=5
      server.1=your_zookeeper_node_1:2888:3888
      server.2=your_zookeeper_node_2:2888:3888
      server.3=your_zookeeper_node_3:2888:3888
      

      initLimit specifies the time that the initial synchronization phase can take. This is the time within which each of the nodes in the quorum needs to connect to the leader. syncLimit specifies the time that can pass between sending a request and receiving an acknowledgment. This is the maximum time nodes can be out of sync from the leader. ZooKeeper nodes use a pair of ports, :2888 and :3888, for follower nodes to connect to the leader node and for leader election, respectively.

      Once you've updated the file on each node, you will save and exit the editor.

      To complete your multi-node configuration, you will specify a node ID on each of the servers. To do this, you will create a myid file on each node. Each file will contain a number that correlates to the server number assigned in the configuration file.

      On your_zookeeper_node_1, create the myid file that will specify the node ID:

      • sudo nano /data/zookeeper/myid

      Since your_zookeeper_node_1 is identified as server.1, you will enter 1 to define the node ID. After adding the value, your file will look like this:

      your_zookeeper_node_1 /data/zookeeper/myid

      1

      Follow the same steps for the remaining nodes. The myid file on each node should be as follows:

      your_zookeeper_node_1 /data/zookeeper/myid

      1

      your_zookeeper_node_2 /data/zookeeper/myid

      2

      your_zookeeper_node_3 /data/zookeeper/myid

      3

      You have now configured a three-node ZooKeeper cluster. Next, you will run the cluster and test your installation.

      Step 8 — Running and Testing the Multi-Node Installation

      With each node configured to work as a cluster, you are ready to start a quorum. In this step, you will start the quorum on each node and then test your cluster by creating sample data in ZooKeeper.

      To start a quorum node, first change to the /opt/zookeeper directory on each node:

      Start each node with the following command:

      • java -cp zookeeper-3.4.13.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.25.jar:lib/slf4j-api-1.7.25.jar:conf org.apache.zookeeper.server.quorum.QuorumPeerMain conf/zoo.cfg

      As nodes start up, you will intermittently see some connection errors followed by a stage where they join the quorum and elect a leader among themselves. After a few seconds of initialization, you can start testing your installation.

      Log in via SSH to your_zookeeper_node_3 as the non-root user you configured in the prerequisites:

      • ssh sammy@your_zookeeper_node_3

      Once logged in, switch to your zk user:

      your_zookeeper_node_3 /data/zookeeper/myid

      Enter the password for the zk user. Once logged in, change the directory to /opt/zookeeper:

      your_zookeeper_node_3 /data/zookeeper/myid

      You will now start a ZooKeeper command line client and connect to ZooKeeper on your_zookeeper_node_1:

      your_zookeeper_node_3 /data/zookeeper/myid

      • bin/zkCli.sh -server your_zookeeper_node_1:2181

      In the standalone installation, both the client and server were running on the same host. This allowed you to establish a client connection with the ZooKeeper server using localhost. Since the client and server are running on different nodes in your multi-node cluster, in the previous step you needed to specify the IP address of your_zookeeper_node_1 to connect to it.

      You will see the familiar prompt with the CONNECTED label, similar to what you saw in Step 5.

      Next, you will create, list, and then delete a znode. The znodes are the fundamental abstractions in ZooKeeper that are analogous to files and directories on a file system. ZooKeeper maintains its data in a hierarchical namespace, and znodes are the data registers of this namespace.

      Testing that you can successfully create, list, and then delete a znode is essential to establishing that your ZooKeeper cluster is installed and configured correctly.

      Create a znode named zk_znode_1 and associate the string sample_data with it.

      • create /zk_znode_1 sample_data

      You will see the following output once created:

      Output

      Created /zk_znode_1

      List the newly created znode:

      Get the data associated with it:

      ZooKeeper will respond like so:

      Output

      [zk: your_zookeeper_node_1:2181(CONNECTED)] ls / [zk_znode_1, zookeeper] [zk: your_zookeeper_node_1:2181(CONNECTED)] get /zk_znode_1 sample_data cZxid = 0x100000002 ctime = Tue Nov 06 19:47:41 UTC 2018 mZxid = 0x100000002 mtime = Tue Nov 06 19:47:41 UTC 2018 pZxid = 0x100000002 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 11 numChildren = 0

      The output confirms the value, sample_data, that you associated with zk_node_1. ZooKeeper also provides additional information about creation time, ctime, and modification time, mtime. ZooKeeper is a versioned data store, so it also presents you with metadata about the data version.

      Delete the zk_znode_1 znode:

      In this step, you successfully tested connectivity between two of your ZooKeeper nodes. You also learned basic znode management by creating, listing, and deleting znodes. Your multi-node configuration is complete, and you are ready to start using ZooKeeper.

      Conclusion

      In this tutorial, you configured and tested both a standalone and multi-node ZooKeeper environment. Now that your multi-node ZooKeeper deployment is ready to use, you can review the official ZooKeeper documentation for additional information and projects.



      Source link

      How to Back Up, Import, and Migrate Your Apache Kafka Data on Ubuntu 18.04


      The author selected Tech Education Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      Backing up your Apache Kafka data is an important practice that will help you recover from unintended data loss or bad data added to the cluster due to user error. Data dumps of cluster and topic data are an efficient way to perform backups and restorations.

      Importing and migrating your backed up data to a separate server is helpful in situations where your Kafka instance becomes unusable due to server hardware or networking failures and you need to create a new Kafka instance with your old data. Importing and migrating backed up data is also useful when you are moving the Kafka instance to an upgraded or downgraded server due to a change in resource usage.

      In this tutorial, you will back up, import, and migrate your Kafka data on a single Ubuntu 18.04 installation as well as on multiple Ubuntu 18.04 installations on separate servers. ZooKeeper is a critical component of Kafka’s operation. It stores information about cluster state such as consumer data, partition data, and the state of other brokers in the cluster. As such, you will also back up ZooKeeper’s data in this tutorial.

      Prerequisites

      To follow along, you will need:

      • An Ubuntu 18.04 server with at least 4GB of RAM and a non-root sudo user set up by following the tutorial.
      • An Ubuntu 18.04 server with Apache Kafka installed, to act as the source of the backup. Follow the How To Install Apache Kafka on Ubuntu 18.04 guide to set up your Kafka installation, if Kafka isn’t already installed on the source server.
      • OpenJDK 8 installed on the server. To install this version, follow these instructions on installing specific versions of OpenJDK.

      • Optional for Step 7 — Another Ubuntu 18.04 server with Apache Kafka installed, to act as the destination of the backup. Follow the article link in the previous prerequisite to install Kafka on the destination server. This prerequisite is required only if you are moving your Kafka data from one server to another. If you want to back up and import your Kafka data to a single server, you can skip this prerequisite.

      Step 1 — Creating a Test Topic and Adding Messages

      A Kafka message is the most basic unit of data storage in Kafka and is the entity that you will publish to and subscribe from Kafka. A Kafka topic is like a container for a group of related messages. When you subscribe to a particular topic, you will receive only messages that were published to that particular topic. In this section you will log in to the server that you would like to back up (the source server) and add a Kafka topic and a message so that you have some data populated for the backup.

      This tutorial assumes you have installed Kafka in the home directory of the kafka user (/home/kafka/kafka). If your installation is in a different directory, modify the ~/kafka part in the following commands with your Kafka installation’s path, and for the commands throughout the rest of this tutorial.

      SSH into the source server by executing:

      • ssh sammy@source_server_ip

      Run the following command to log in as the kafka user:

      Create a topic named BackupTopic using the kafka-topics.sh shell utility file in your Kafka installation's bin directory, by typing:

      • ~/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic BackupTopic

      Publish the string "Test Message 1" to the BackupTopic topic by using the ~/kafka/bin/kafka-console-producer.sh shell utility script.

      If you would like to add additional messages here, you can do so now.

      • echo "Test Message 1" | ~/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic BackupTopic > /dev/null

      The ~/kafka/bin/kafka-console-producer.sh file allows you to publish messages directly from the command line. Typically, you would publish messages using a Kafka client library from within your program, but since that involves different setups for different programming languages, you can use the shell script as a language-independent way of publishing messages during testing or while performing administrative tasks. The --topic flag specifies the topic that you will publish the message to.

      Next, verify that the kafka-console-producer.sh script has published the message(s) by running the following command:

      • ~/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic BackupTopic --from-beginning

      The ~/kafka/bin/kafka-console-consumer.sh shell script starts the consumer. Once started, it will subscribe to messages from the topic that you published in the "Test Message 1" message in the previous command. The --from-beginning flag in the command allows consuming messages that were published before the consumer was started. Without the flag enabled, only messages published after the consumer was started will appear. On running the command, you will see the following output in the terminal:

      Output

      Test Message 1

      Press CTRL+C to stop the consumer.

      You've created some test data and verified that it's persisted. Now you can back up the state data in the next section.

      Step 2 — Backing Up the ZooKeeper State Data

      Before backing up the actual Kafka data, you need to back up the cluster state stored in ZooKeeper.

      ZooKeeper stores its data in the directory specified by the dataDir field in the ~/kafka/config/zookeeper.properties configuration file. You need to read the value of this field to determine the directory to back up. By default, dataDir points to the /tmp/zookeeper directory. If the value is different in your installation, replace /tmp/zookeeper with that value in the following commands.

      Here is an example output of the ~/kafka/config/zookeeper.properties file:

      ~/kafka/config/zookeeper.properties

      ...
      ...
      ...
      # the directory where the snapshot is stored.
      dataDir=/tmp/zookeeper
      # the port at which the clients will connect
      clientPort=2181
      # disable the per-ip limit on the number of connections since this is a non-production config
      maxClientCnxns=0
      ...
      ...
      ...
      

      Now that you have the path to the directory, you can create a compressed archive file of its contents. Compressed archive files are a better option over regular archive files to save disk space. Run the following command:

      • tar -czf /home/kafka/zookeeper-backup.tar.gz /tmp/zookeeper/*

      The command's output tar: Removing leading / from member names you can safely ignore.

      The -c and -z flags tell tar to create an archive and apply gzip compression to the archive. The -f flag specifies the name of the output compressed archive file, which is zookeeper-backup.tar.gz in this case.

      You can run ls in your current directory to see zookeeper-backup.tar.gz as part of your output.

      You have now successfully backed up the ZooKeeper data. In the next section, you will back up the actual Kafka data.

      Step 3 — Backing Up the Kafka Topics and Messages

      In this section, you will back up Kafka's data directory into a compressed tar file like you did for ZooKeeper in the previous step.

      Kafka stores topics, messages, and internal files in the directory that the log.dirs field specifies in the ~/kafka/config/server.properties configuration file. You need to read the value of this field to determine the directory to back up. By default and in your current installation, log.dirs points to the /tmp/kafka-logs directory. If the value is different in your installation, replace /tmp/kafka-logs in the following commands with the correct value.

      Here is an example output of the ~/kafka/config/server.properties file:

      ~/kafka/config/server.properties

      ...
      ...
      ...
      ############################# Log Basics #############################
      
      # A comma separated list of directories under which to store log files
      log.dirs=/tmp/kafka-logs
      
      # The default number of log partitions per topic. More partitions allow greater
      # parallelism for consumption, but this will also result in more files across
      # the brokers.
      num.partitions=1
      
      # The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
      # This value is recommended to be increased for installations with data dirs located in RAID array.
      num.recovery.threads.per.data.dir=1
      ...
      ...
      ...
      

      First, stop the Kafka service so that the data in the log.dirs directory is in a consistent state when creating the archive with tar. To do this, return to your server's non-root user by typing exit and then run the following command:

      • sudo systemctl stop kafka

      After stopping the Kafka service, log back in as your kafka user with:

      It is necessary to stop/start the Kafka and ZooKeeper services as your non-root sudo user because in the Apache Kafka installation prerequisite you restricted the kafka user as a security precaution. This step in the prerequisite disables sudo access for the kafka user, which leads to commands failing to execute.

      Now, create a compressed archive file of the directory's contents by running the following command:

      • tar -czf /home/kafka/kafka-backup.tar.gz /tmp/kafka-logs/*

      Once again, you can safely ignore the command's output (tar: Removing leading / from member names).

      You can run ls in the current directory to see kafka-backup.tar.gz as part of the output.

      You can start the Kafka service again — if you do not want to restore the data immediately — by typing exit, to switch to your non-root sudo user, and then running:

      • sudo systemctl start kafka

      Log back in as your kafka user:

      You have successfully backed up the Kafka data. You can now proceed to the next section, where you will be restoring the cluster state data stored in ZooKeeper.

      Step 4 — Restoring the ZooKeeper Data

      In this section you will restore the cluster state data that Kafka creates and manages internally when the user performs operations such as creating a topic, adding/removing additional nodes, and adding and consuming messages. You will restore the data to your existing source installation by deleting the ZooKeeper data directory and restoring the contents of the zookeeper-backup.tar.gz file. If you want to restore data to a different server, see Step 7.

      You need to stop the Kafka and ZooKeeper services as a precaution against the data directories receiving invalid data during the restoration process.

      First, stop the Kafka service by typing exit, to switch to your non-root sudo user, and then running:

      • sudo systemctl stop kafka

      Next, stop the ZooKeeper service:

      • sudo systemctl stop zookeeper

      Log back in as your kafka user:

      You can then safely delete the existing cluster data directory with the following command:

      Now restore the data you backed up in Step 2:

      • tar -C /tmp/zookeeper -xzf /home/kafka/zookeeper-backup.tar.gz --strip-components 2

      The -C flag tells tar to change to the directory /tmp/zookeeper before extracting the data. You specify the --strip 2 flag to make tar extract the archive's contents in /tmp/zookeeper/ itself and not in another directory (such as /tmp/zookeeper/tmp/zookeeper/) inside of it.

      You have restored the cluster state data successfully. Now, you can proceed to the Kafka data restoration process in the next section.

      Step 5 — Restoring the Kafka Data

      In this section you will restore the backed up Kafka data to your existing source installation (or the destination server if you have followed the optional Step 7) by deleting the Kafka data directory and restoring the compressed archive file. This will allow you to verify that restoration works successfully.

      You can safely delete the existing Kafka data directory with the following command:

      Now that you have deleted the data, your Kafka installation resembles a fresh installation with no topics or messages present in it. To restore your backed up data, extract the files by running:

      • tar -C /tmp/kafka-logs -xzf /home/kafka/kafka-backup.tar.gz --strip-components 2

      The -C flag tells tar to change to the directory /tmp/kafka-logs before extracting the data. You specify the --strip 2 flag to ensure that the archive's contents are extracted in /tmp/kafka-logs/ itself and not in another directory (such as /tmp/kafka-logs/kafka-logs/) inside of it.

      Now that you have extracted the data successfully, you can start the Kafka and ZooKeeper services again by typing exit, to switch to your non-root sudo user, and then executing:

      • sudo systemctl start kafka

      Start the ZooKeeper service with:

      • sudo systemctl start zookeeper

      Log back in as your kafka user:

      You have restored the kafka data, you can move on to verifying that the restoration is successful in the next section.

      Step 6 — Verifying the Restoration

      To test the restoration of the Kafka data, you will consume messages from the topic you created in Step 1.

      Wait a few minutes for Kafka to start up and then execute the following command to read messages from the BackupTopic:

      • ~/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic BackupTopic --from-beginning

      If you get a warning like the following, you need to wait for Kafka to start fully:

      Output

      [2018-09-13 15:52:45,234] WARN [Consumer clientId=consumer-1, groupId=console-consumer-87747] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

      Retry the previous command in another few minutes or run sudo systemctl restart kafka as your non-root sudo user. If there are no issues in the restoration, you will see the following output:

      Output

      Test Message 1

      If you do not see this message, you can check if you missed out any commands in the previous section and execute them.

      Now that you have verified the restored Kafka data, this means you have successfully backed up and restored your data in a single Kafka installation. You can continue to Step 7 to see how to migrate the cluster and topics data to an installation in another server.

      Step 7 — Migrating and Restoring the Backup to Another Kafka Server (Optional)

      In this section, you will migrate the backed up data from the source Kafka server to the destination Kafka server. To do so, you will first use the scp command to download the compressed tar.gz files to your local system. You will then use scp again to push the files to the destination server. Once the files are present in the destination server, you can follow the steps used previously to restore the backup and verify that the migration is successful.

      You are downloading the backup files locally and then uploading them to the destination server, instead of copying it directly from your source to destination server, because the destination server will not have your source server's SSH key in its /home/sammy/.ssh/authorized_keys file and cannot connect to and from the source server. Your local machine can connect to both servers however, saving you an additional step of setting up SSH access from the source to destination server.

      Download the zookeeper-backup.tar.gz and kafka-backup.tar.gz files to your local machine by executing:

      • scp sammy@source_server_ip:/home/kafka/zookeeper-backup.tar.gz .

      You will see output similar to:

      Output

      zookeeper-backup.tar.gz 100% 68KB 128.0KB/s 00:00

      Now run the following command to download the kafka-backup.tar.gz file to your local machine:

      • scp sammy@source_server_ip:/home/kafka/kafka-backup.tar.gz .

      You will see the following output:

      Output

      kafka-backup.tar.gz 100% 1031KB 488.3KB/s 00:02

      Run ls in the current directory of your local machine, you will see both of the files:

      Output

      kafka-backup.tar.gz zookeeper.tar.gz

      Run the following command to transfer the zookeeper-backup.tar.gz file to /home/kafka/ of the destination server:

      • scp zookeeper-backup.tar.gz sammy@destination_server_ip:/home/sammy/zookeeper-backup.tar.gz

      Now run the following command to transfer the kafka-backup.tar.gz file to /home/kafka/ of the destination server:

      • scp kafka-backup.tar.gz sammy@destination_server_ip:/home/sammy/kafka-backup.tar.gz

      You have uploaded the backup files to the destination server successfully. Since the files are in the /home/sammy/ directory and do not have the correct permissions for access by the kafka user, you can move the files to the /home/kafka/ directory and change their permissions.

      SSH into the destination server by executing:

      • ssh sammy@destination_server_ip

      Now move zookeeper-backup.tar.gz to /home/kafka/ by executing:

      • sudo mv zookeeper-backup.tar.gz /home/sammy/zookeeper-backup.tar.gz

      Similarly, run the following command to copy kafka-backup.tar.gz to /home/kafka/:

      • sudo mv kafka-backup.tar.gz /home/kafka/kafka-backup.tar.gz

      Change the owner of the backup files by running the following command:

      • sudo chown kafka /home/kafka/zookeeper-backup.tar.gz /home/kafka/kafka-backup.tar.gz

      The previous mv and chown commands will not display any output.

      Now that the backup files are present in the destination server at the correct directory, follow the commands listed in Steps 4 to 6 of this tutorial to restore and verify the data for your destination server.

      Conclusion

      In this tutorial, you backed up, imported, and migrated your Kafka topics and messages from both the same installation and installations on separate servers. If you would like to learn more about other useful administrative tasks in Kafka, you can consult the operations section of Kafka's official documentation.

      To store backed up files such as zookeeper-backup.tar.gz and kafka-backup.tar.gz remotely, you can explore Digital Ocean Spaces. If Kafka is the only service running on your server, you can also explore other backup methods such as full instance backups.



      Source link

      Configure Apache with Salt Stack


      Updated by Linode Written by Linode

      Salt is a powerful configuration management tool. In this guide you will create Salt state files that are capable of installing and configuring Apache on Ubuntu 18.04, Debian 9, or CentOS 7.

      Before You Begin

      You will need at least two Linodes with Salt installed. If you have not already, read our Getting Started with Salt – Basic Installation and Setup Guide and follow the instructions for setting up a Salt master and minion.

      The following steps will be performed on your Salt master.

      Note

      The steps in this guide require root privileges. Be sure to run the steps below as root or with the sudo prefix. For more information on privileges, see our Users and Groups guide.

      Setting Up Your Salt Master and Managed Files

      Salt Master SLS Files

      1. Create the /srv/salt directory if it does not already exist:

        mkdir /srv/salt
        
      2. Create a Salt top file in /srv/salt that will be Salt’s entry point to the Apache configuration:

        /srv/salt/top.sls
        1
        2
        3
        4
        5
        6
        7
        8
        
        base:
          'G@os_family:Debian':
            - match: compound
            - apache-debian
        
          'G@os:CentOS':
            - match: compound
            - apache-centos

        This top file uses compound matching to target your minions by operating system using Salt Grains. This will allow Salt to choose the appropriate Apache configuration depending on the Linux distribution. These matchers could be extended to be even more specific. For instance, if you wanted to only target minions with the ID of web-server that are running on Ubuntu, you can type web* and G@os:Ubuntu.

      Pillar Files

      1. Create the /srv/pillar directory if it does not already exist:

        mkdir /srv/pillar
        
      2. Create a Pillar top file. This top file references the apache.sls Pillar file that you will create in the next step:

        /srv/pillar/top.sls
        1
        2
        3
        
        base:
          '*':
            - apache
      3. Create the apache.sls file that was referenced in the previous step. This file defines Pillar data that will be used inside our Apache state file in the next section, in this case your domain name. Replace example.com with your domain:

        /srv/pillar/apache.sls

      Website Files

      1. Create a directory for your website files in the /srv/salt directory. Replace example.com with your website domain name:

        mkdir /srv/salt/example.com
        

        This directory will be accessible from your Salt state files at salt://example.com.

      2. Create an index.html file for your website in the /srv/salt/example.com directory, substituting example.com for the folder name you chose in the previous step. You will use this file as a test to make sure your website is functioning correctly.

        /srv/salt/example.com/index.html
        1
        2
        3
        4
        5
        
        <html>
          <body>
            <h1>Server Up and Running!</h1>
          </body>
        </html>

      Configuration Files

      1. Create a folder for your additional configuration files at /srv/salt/files. These files will be accessible at salt://files.

        mkdir /srv/salt/files
        
      2. Create a file called tune_apache.conf in /srv/salt/files and paste in the following block:

        /srv/salt/files/tune_apache.conf
        1
        2
        3
        4
        5
        6
        7
        
        <IfModule mpm_prefork_module>
        StartServers 4
        MinSpareServers 20
        MaxSpareServers 40
        MaxClients 200
        MaxRequestsPerChild 4500
        </IfModule>

        This MPM prefork module provides additional tuning for your Apache installation. This file will be managed by Salt and installed into the appropriate configuration directory in a later step.

      3. If you will be installing Apache on a CentOS machine, create a file called include_sites_enabled.conf in /srv/salt/files and paste in the following:

        /srv/salt/files/include_sites_enabled.conf
        1
        
        IncludeOptional sites-enabled/*.conf

        This file will allow us to use file directories like those found on Debian installations to help organize the Apache configuration.

      Creating the Apache State File for Debian and Ubuntu

      Individual Steps

      This guide will be going through the process of creating the Apache for Debian and Ubuntu state file step by step. If you would like to view the entirety of the state file, you can view it at the end of this section.

      1. Create a state file named apache-debian.sls in /srv/salt and open it in a text editor of your choice.

      2. Instruct Salt to install the apache2 package and start the apache2 service:

        /srv/salt/apache-debian.sls
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        
        apache2:
          pkg.installed
        
        apache2 Service:
          service.running:
            - name: apache2
            - enable: True
            - require:
              - pkg: apache2
        
        ...

        Here Salt makes sure the apache2 package is installed with pkg.installed. Likewise, it ensures the apache2 service is running and enabled under service.running. Also under service.running, apache-debian.sls uses require to ensure that this command does not run before the apache2 package is installed. This require step will be repeated throughout apache-debain.sls.

        Lastly, a watch statement is employed to restart the apache2 service if your site’s configuration file changes. You will define that configuration file in a later step. Note that this configuration file is named using the domain you supplied when creating your Salt Pillar file in the first section. This Pillar data will be used throughout apache-debian.sls.

      3. Turn off KeepAlive:

        /srv/salt/apache-debian.sls
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        
        ...
        
        Turn Off KeepAlive:
          file.replace:
            - name: /etc/apache2/apache2.conf
            - pattern: 'KeepAlive On'
            - repl: 'KeepAlive Off'
            - show_changes: True
            - require:
              - pkg: apache2
        ...

        KeepAlive allows multiple requests to be sent over the same TCP connection. For the purpose of this guide KeepAlive will be disabled. To disable it, Salt is instructed to find the KeepAlive directive in /etc/apache2/apache2.conf by matching a pattern and replacing it with KeepAlive Off. show_changes instructs Salt to display any changes it has made during a highstate.

      4. Transfer tune_apache.conf to your minion and enable it:

        /srv/salt/apache-debian.sls
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        
        ...
        
        /etc/apache2/conf-available/tune_apache.conf:
          file.managed:
            - source: salt://files/tune_apache.conf
            - require:
              - pkg: apache2
        
        Enable tune_apache:
          apache_conf.enabled:
            - name: tune_apache
            - require:
              - pkg: apache2
        
        ...

        This step takes the tune_apache.conf file you created in the Configuration Files step and transfers it to your Salt minion. Then, Salt enables that configuration file with the apache_conf module.

      5. Create the necessary directories:

        /srv/salt/apache-debian.sls
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        
        ...
        
        /var/www/html/{{ pillar['domain'] }}:
          file.directory
        
        /var/www/html/{{ pillar['domain'] }}/log:
          file.directory
        
        /var/www/html/{{ pillar['domain'] }}/backups:
          file.directory
        
        /var/www/html/{{ pillar['domain'] }}/public_html:
          file.directory
        
        ...
      6. Disable the default virtual host configuration file:

        /srv/salt/apache-debian.sls
        1
        2
        3
        4
        5
        6
        7
        8
        
        ...
        
        000-default:
          apache_site.disabled:
            - require:
              - pkg: apache2
        
        ...

        This step uses Salt’s apache_site module to disable the default Apache virtual host configuration file, and is the same as running a2dissite on a Debian-based machine.

      7. Create your site’s virtual host configuration file:

        /srv/salt/apache-debian.sls
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        16
        
        ...
        
        /etc/apache2/sites-available/{{ pillar['domain'] }}.conf:
          apache.configfile:
            - config:
              - VirtualHost:
                  this: '*:80'
                  ServerName:
                    - {{ pillar['domain'] }}
                  ServerAlias:
                    - www.{{ pillar['domain'] }}
                  DocumentRoot: /var/www/html/{{ pillar['domain'] }}/public_html
                  ErrorLog: /var/www/html/{{ pillar['domain'] }}/log/error.log
                  CustomLog: /var/www/html/{{ pillar['domain'] }}/log/access.log combined
        
        ...

        This step uses Salt’s apache module, (not to be confused with the apache_site module used in the previous step), to create your site’s virtual host configuration file. The this variable signifies what would traditionally be include with VirtualHost within angle brackets in an Apache configuration file: <VirtualHost *:80>.

      8. Enable your new virtual host configuration file:

        /srv/salt/apache-debian.sls
        1
        2
        3
        4
        5
        6
        7
        8
        
        ...
        
        {{ pillar['domain'] }}:
          apache_site.enabled:
            - require:
              - pkg: apache2
        
        ...

        This step uses the same apache_site module you used to disable the default virtual host file to enable your newly created virtual host file. apache_site.enabled creates a symlink from /etc/apache2/sites-available/example.com.conf to /etc/apache2/sites-enabled/example.com.conf and is the same as running a2ensite on a Debian-based machine.

      9. Transfer your index.html website file to your minion:

        /srv/salt/apache-debian.sls
        1
        2
        3
        4
        5
        
        ...
        
        /var/www/html/{{ pillar['domain'] }}/public_html/index.html:
          file.managed:
            - source: salt://{{ pillar['domain'] }}/index.html

        Any changes made to your index.html file on your Salt master will be propagated to your minion.

        Note

        Since Salt is not watching configuration files for a change to trigger a restart for Apache, you may need to use the command below from your Salt master.

        salt '*' apache.signal restart
        

      Complete State File

      The complete apache-debian.sls file looks like this:

      /srv/salt/apache-debian.sls
       1
       2
       3
       4
       5
       6
       7
       8
       9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      37
      38
      39
      40
      41
      42
      43
      44
      45
      46
      47
      48
      49
      50
      51
      52
      53
      54
      55
      56
      57
      58
      59
      60
      61
      62
      63
      64
      65
      66
      67
      68
      69
      
      apache2:
        pkg.installed
      
      apache2 Service:
        service.running:
          - name: apache2
          - enable: True
          - require:
            - pkg: apache2
      
      Turn Off KeepAlive:
        file.replace:
          - name: /etc/apache2/apache2.conf
          - pattern: 'KeepAlive On'
          - repl: 'KeepAlive Off'
          - show_changes: True
          - require:
            - pkg: apache2
      
      /etc/apache2/conf-available/tune_apache.conf:
        file.managed:
          - source: salt://files/tune_apache.conf
          - require:
            - pkg: apache2
      
      Enable tune_apache:
        apache_conf.enabled:
          - name: tune_apache
          - require:
            - pkg: apache2
      
      /var/www/html/{{ pillar['domain'] }}:
        file.directory
      
      /var/www/html/{{ pillar['domain'] }}/log:
        file.directory
      
      /var/www/html/{{ pillar['domain'] }}/backups:
        file.directory
      
      /var/www/html/{{ pillar['domain'] }}/public_html:
        file.directory
      
      000-default:
        apache_site.disabled:
          - require:
            - pkg: apache2
      
      /etc/apache2/sites-available/{{ pillar['domain'] }}.conf:
        apache.configfile:
          - config:
            - VirtualHost:
                this: '*:80'
                ServerName:
                  - {{ pillar['domain'] }}
                ServerAlias:
                  - www.{{ pillar['domain'] }}
                DocumentRoot: /var/www/html/{{ pillar['domain'] }}/public_html
                ErrorLog: /var/www/html/{{ pillar['domain'] }}/log/error.log
                CustomLog: /var/www/html/{{ pillar['domain'] }}/log/access.log combined
      
      {{ pillar['domain'] }}:
        apache_site.enabled:
          - require:
            - pkg: apache2
      
      /var/www/html/{{ pillar['domain'] }}/public_html/index.html:
        file.managed:
          - source: salt://{{ pillar['domain'] }}/index.html

      Creating an Apache State File for CentOS

      Individual Steps

      1. Create a file called apache-centos.sls in /srv/salt and open it in a text editor of your choice.

      2. On CentOS Apache is named httpd. Instruct Salt to install httpd and run the httpd service:

        /srv/salt/apache-centos.sls
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        
        httpd:
          pkg.installed
        
        httpd Service:
          service.running:
            - name: httpd
            - enable: True
            - require:
              - pkg: httpd
            - watch:
              - file: /etc/httpd/sites-available/{{ pillar['domain'] }}.conf
        
        ...

        Here Salt makes sure the httpd package is installed with pkg.installed. Likewise, it ensures the httpd service is running and enabled under service.running. Also under service.running, apache-debian.sls uses require to ensure that this command does not run before the httpd package is installed. This require step will be repeated throughout apache-centos.sls.

        Lastly, a watch statement is employed to restart the httpd service if your site’s configuration file changes. You will define that configuration file in a later step. Note that this configuration file is named using the domain you supplied when creating your Salt Pillar file in the first section. This Pillar data will be used throughout apache-centos.sls.

      3. Turn off KeepAlive:

        /srv/salt/apache-centos.sls
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        
        ...
        
        Turn Off KeepAlive:
          file.replace:
            - name: /etc/httpd/conf/httpd.conf
            - pattern: 'KeepAlive On'
            - repl: 'KeepAlive Off'
            - show_changes: True
            - require:
              - pkg: httpd
        ...

        KeepAlive allows multiple requests to be sent over the same TCP connection. For the purpose of this guide KeepAlive will be disabled. To disable it, Salt is instructed to find the KeepAlive directive in /etc/httpd/conf/httpd.conf by matching a pattern and replacing it with KeepAlive Off. show_changes instructs Salt to display any changes it has made during a highstate.

      4. Change the DocumentRoot:

        /srv/salt/apache-centos.sls
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        
        ...
        
        Change DocumentRoot:
          file.replace:
            - name: /etc/httpd/conf/httpd.conf
            - pattern: 'DocumentRoot "/var/www/html"'
            - repl: 'DocumentRoot "/var/www/html/{{ pillar['domain'] }}/public_html"'
            - show_changes: True
            - require:
              - pkg: httpd
        
        ...

        Similar to the last step, in this step salt-centos.sls instructs Salt to search for the DocumentRoot directive in Apache’s httpd.conf file, and replaces that line with the new document root. This allows for the use of a Debian-style site directory architecture.

      5. Transfer the tune_apache.conf and include_sites_enabled.conf to your minion.

        /srv/salt/apache-centos.sls
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        
        ...
        
        /etc/httpd/conf.d/tune_apache.conf:
          file.managed:
            - source: salt://files/tune_apache.conf
            - require:
              - pkg: httpd
        
        /etc/httpd/conf.d/include_sites_enabled.conf:
          file.managed:
            - source: salt://files/include_sites_enabled.conf
            - require:
              - pkg: httpd
        
        ...
      6. Create the necessary directories:

        srv/salt/apache-centos.sls
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        16
        17
        18
        
        ...
        
        /etc/httpd/sites-available:
          file.directory
        
        /etc/httpd/sites-enabled:
          file.directory
        
        /var/www/html/{{ pillar['domain'] }}:
          file.directory
        
        /var/www/html/{{ pillar['domain'] }}/backups:
          file.directory
        
        /var/www/html/{{ pillar['domain'] }}/public_html:
          file.directory
        
        ...
      7. Create your site’s virtual host configuration file:

        /srv/salt/apache-centos.sls
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
        11
        12
        13
        14
        15
        16
        17
        
        ...
        
        /etc/httpd/sites-available/{{ pillar['domain'] }}.conf:
          apache.configfile:
            - config:
              - VirtualHost:
                  this: '*:80'
                  ServerName:
                    - {{ pillar['domain'] }}
                  ServerAlias:
                    - www.{{ pillar['domain'] }}
                  DocumentRoot: /var/www/html/{{ pillar['domain'] }}/public_html
          file.symlink:
            - target: /etc/httpd/sites-enabled/{{ pillar['domain'] }}.conf
            - force: True
        
        ...

        This step uses Salt’s apache module to create your site’s virtual host configuration file. The this variable signifies what would traditionally be include with VirtualHost within angle brackets in an Apache configuration file: <VirtualHost *:80>.

      8. Transfer your index.html website file to your minion:

        /srv/salt/apache-debian.sls
        1
        2
        3
        4
        5
        6
        7
        
        ...
        
        /var/www/html/{{ pillar['domain'] }}/public_html/index.html:
          file.managed:
            - source: salt://{{ pillar['domain'] }}/index.html
        
        ...

        Any changes made to your index.html file on your Salt master will be propigated to your minion.

      9. Configure your firewall to allow http and https traffic:

        /srv/salt/apache-centos.sls
        1
        2
        3
        4
        5
        6
        7
        8
        9
        
        ...
        
        Configure Firewall:
          firewalld.present:
            - name: public
            - ports:
              - 22/tcp
              - 80/tcp
              - 443/tcp

        Note

        It is imperative that you list all ports you need open to your machine in this section. Failure to list these ports will result in their closure by Salt.

      Complete State File

      The complete apache-centos.sls file looks like this:

      /srv/salt/apache-centos.sls
       1
       2
       3
       4
       5
       6
       7
       8
       9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      37
      38
      39
      40
      41
      42
      43
      44
      45
      46
      47
      48
      49
      50
      51
      52
      53
      54
      55
      56
      57
      58
      59
      60
      61
      62
      63
      64
      65
      66
      67
      68
      69
      70
      71
      72
      73
      74
      75
      76
      77
      78
      79
      80
      81
      82
      
      httpd:
        pkg.installed
      
      httpd Service:
        service.running:
          - name: httpd
          - enable: True
          - require:
            - pkg: httpd
          - watch:
            - file: /etc/httpd/sites-available/{{ pillar['domain'] }}.conf
      
      Turn off KeepAlive:
        file.replace:
          - name: /etc/httpd/conf/httpd.conf
          - pattern: 'KeepAlive On'
          - repl: 'KeepAlive Off'
          - show_changes: True
          - require:
            - pkg: httpd
      
      Change DocumentRoot:
        file.replace:
          - name: /etc/httpd/conf/httpd.conf
          - pattern: 'DocumentRoot "/var/www/html"'
          - repl: 'DocumentRoot "/var/www/html/{{ pillar['domain'] }}/public_html"'
          - show_changes: True
          - require:
            - pkg: httpd
      
      /etc/httpd/conf.d/tune_apache.conf:
        file.managed:
          - source: salt://files/tune_apache.conf
          - require:
            - pkg: httpd
      
      /etc/httpd/conf.d/include_sites_enabled.conf:
        file.managed:
          - source: salt://files/include_sites_enabled.conf
          - require:
            - pkg: httpd
      
      /etc/httpd/sites-available:
        file.directory
      
      /etc/httpd/sites-enabled:
        file.directory
      
      /var/www/html/{{ pillar['domain'] }}:
        file.directory
      
      /var/www/html/{{ pillar['domain'] }}/backups:
        file.directory
      
      /var/www/html/{{ pillar['domain'] }}/public_html:
        file.directory
      
      /etc/httpd/sites-available/{{ pillar['domain'] }}.conf:
        apache.configfile:
          - config:
            - VirtualHost:
                this: '*:80'
                ServerName:
                  - {{ pillar['domain'] }}
                ServerAlias:
                  - www.{{ pillar['domain'] }}
                DocumentRoot: /var/www/html/{{ pillar['domain'] }}/public_html
        file.symlink:
          - target: /etc/httpd/sites-enabled/{{ pillar['domain'] }}.conf
          - force: True
      
      /var/www/html/{{ pillar['domain'] }}/public_html/index.html:
        file.managed:
          - source: salt://{{ pillar['domain'] }}/index.html
      
      Configure Firewall:
        firewalld.present:
          - name: public
          - ports:
            - 22/tcp
            - 80/tcp
            - 443/tcp

      Running the Apache State File

      On your Salt master, issue a highstate command:

      salt '*' state.apply
      

      After a few moments you should see a list of Salt commands and a summary of their successes. Navigate to your website’s domain name if you have your DNS set up already, or your website’s public IP address. You should see your index.html file. You have now used Salt to configure Apache. Visit the links in the section below for more information.

      More Information

      You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

      Find answers, ask questions, and help others.

      This guide is published under a CC BY-ND 4.0 license.



      Source link