One place for hosting & domains

      An Introduction to Helm, the Package Manager for Kubernetes


      Introduction

      Deploying applications to Kubernetes – the powerful and popular container-orchestration system – can be complex. Setting up a single application can involve creating multiple interdependent Kubernetes resources – such as pods, services, deployments, and replicasets – each requiring you to write a detailed YAML manifest file.

      Helm is a package manager for Kubernetes that allows developers and operators to more easily package, configure, and deploy applications and services onto Kubernetes clusters.

      Helm is now an official Kubernetes project and is part of the Cloud Native Computing Foundation, a non-profit that supports open source projects in and around the Kubernetes ecosystem.

      In this article we will give an overview of Helm and the various abstractions it uses to simplify deploying applications to Kubernetes. If you are new to Kubernetes, it may be helpful to read An Introduction to Kubernetes first to familiarize yourself with the basics concepts.

      An Overview of Helm

      Most every programming language and operating system has its own package manager to help with the installation and maintenance of software. Helm provides the same basic feature set as many of the package managers you may already be familiar with, such as Debian’s apt, or Python’s pip.

      Helm can:

      • Install software.
      • Automatically install software dependencies.
      • Upgrade software.
      • Configure software deployments.
      • Fetch software packages from repositories.

      Helm provides this functionality through the following components:

      • A command line tool, helm, which provides the user interface to all Helm functionality.
      • A companion server component, tiller, that runs on your Kubernetes cluster, listens for commands from helm, and handles the configuration and deployment of software releases on the cluster.
      • The Helm packaging format, called charts.
      • An official curated charts repository with prepackaged charts for popular open-source software projects.

      We’ll investigate the charts format in more detail next.

      Charts

      Helm packages are called charts, and they consist of a few YAML configuration files and some templates that are rendered into Kubernetes manifest files. Here is the basic directory structure of a chart:

      Example chart directory

      package-name/
        charts/
        templates/
        Chart.yaml
        LICENSE
        README.md
        requirements.yaml
        values.yaml
      

      These directories and files have the following functions:

      • charts/: Manually managed chart dependencies can be placed in this directory, though it is typically better to use requirements.yaml to dynamically link dependencies.
      • templates/: This directory contains template files that are combined with configuration values (from values.yaml and the command line) and rendered into Kubernetes manifests. The templates use the Go programming language’s template format.
      • Chart.yaml: A YAML file with metadata about the chart, such as chart name and version, maintainer information, a relevant website, and search keywords.
      • LICENSE: A plaintext license for the chart.
      • README.md: A readme file with information for users of the chart.
      • requirements.yaml: A YAML file that lists the chart’s dependencies.
      • values.yaml: A YAML file of default configuration values for the chart.

      The helm command can install a chart from a local directory, or from a .tar.gz packaged version of this directory structure. These packaged charts can also be automatically downloaded and installed from chart repositories or repos.

      We’ll look at chart repositories next.

      Chart Repositories

      A Helm chart repo is a simple HTTP site that serves an index.yaml file and .tar.gz packaged charts. The helm command has subcommands available to help package charts and create the required index.yaml file. These files can be served by any web server, object storage service, or a static site host such as GitHub Pages.

      Helm comes preconfigured with a default chart repository, referred to as stable. This repo points to a Google Storage bucket at https://kubernetes-charts.storage.googleapis.com. The source for the stable repo can be found in the helm/charts Git repository on GitHub.

      Alternate repos can be added with the helm repo add command. Some popular alternate repositories are:

      Whether you’re installing a chart you’ve developed locally, or one from a repo, you’ll need to configure it for your particular setup. We’ll look into configs next.

      Chart Configuration

      A chart usually comes with default configuration values in its values.yaml file. Some applications may be fully deployable with default values, but you’ll typically need to override some of the configuration to meet your needs.

      The values that are exposed for configuration are determined by the author of the chart. Some are used to configure Kubernetes primitives, and some may be passed through to the underlying container to configure the application itself.

      Here is a snippet of some example values:

      values.yaml

      service:
        type: ClusterIP
        port: 3306
      

      These are options to configure a Kubernetes Service resource. You can use helm inspect values chart-name to dump all of the available configuration values for a chart.

      These values can be overridden by writing your own YAML file and using it when running helm install, or by setting options individually on the command line with the --set flag. You only need to specify those values that you want to change from the defaults.

      A Helm chart deployed with a particular configuration is called a release. We will talk about releases next.

      Releases

      During the installation of a chart, Helm combines the chart’s templates with the configuration specified by the user and the defaults in value.yaml. These are rendered into Kubernetes manifests that are then deployed via the Kubernetes API. This creates a release, a specific configuration and deployment of a particular chart.

      This concept of releases is important, because you may want to deploy the same application more than once on a cluster. For instance, you may need multiple MySQL servers with different configurations.

      You also will probably want to upgrade different instances of a chart individually. Perhaps one application is ready for an updated MySQL server but another is not. With Helm, you upgrade each release individually.

      You might upgrade a release because its chart has been updated, or because you want to update the release’s configuration. Either way, each upgrade will create a new revision of a release, and Helm will allow you to easily roll back to previous revisions in case there’s an issue.

      Creating Charts

      If you can’t find an existing chart for the software you are deploying, you may want to create your own. Helm can output the scaffold of a chart directory with helm create chart-name. This will create a folder with the files and directories we discussed in the Charts section above.

      From there, you’ll want to fill out your chart’s metadata in Chart.yaml and put your Kubernetes manifest files into the templates directory. You’ll then need to extract relevant configuration variables out of your manifests and into values.yaml, then include them back into your manifest templates using the templating system.

      The helm command has many subcommands available to help you test, package, and serve your charts. For more information, please read the official Helm documentation on developing charts.

      Conclusion

      In this article we reviewed Helm, the package manager for Kubernetes. We overviewed the Helm architecture and the individual helm and tiller components, detailed the Helm charts format, and looked at chart repositories. We also looked into how to configure a Helm chart and how configurations and charts are combined and deployed as releases on Kubernetes clusters. Finally, we touched on the basics of creating a chart when a suitable chart isn’t already available.

      For more information about Helm, take a look at the official Helm documentation. To find official charts for Helm, check out the official helm/charts Git repository on GitHub.



      Source link

      How To Install the Django Web Framework on Ubuntu 18.04


      Introduction

      Django is a full-featured Python web framework for developing dynamic websites and applications. Using Django, you can quickly create Python web applications and rely on the framework to do a good deal of the heavy lifting.

      In this guide, you will get Django up and running on an Ubuntu 18.04 server. After installation, you will start a new project to use as the basis for your site.

      Different Methods

      There are different ways to install Django, depending upon your needs and how you want to configure your development environment. These have different advantages and one method may lend itself better to your specific situation than others.

      Some of the different methods include:

      • Global install from packages: The official Ubuntu repositories contain Django packages that can be installed with the conventional apt package manager. This is simple, but not as flexible as some other methods. Also, the version contained in the repositories may lag behind the official versions available from the project.
      • Install with pip in a virtual environment: You can create a self-contained environment for your projects using tools like venv and virtualenv. A virtual environment allows you to install Django in a project directory without affecting the larger system, along with other per-project customizations and packages. This is typically the most practical and recommended approach to working with Django.
      • Development version install with git: If you wish to install the latest development version instead of the stable release, you can acquire the code from the Git repo. This is necessary to get the latest features/fixes and can be done within your virtual environment. Development versions do not have the same stability guarantees as more stable versions, however.

      Prerequisites

      Before you begin, you should have a non-root user with sudo privileges available on your Ubuntu 18.04 server. To set this up, follow our Ubuntu 18.04 initial server setup guide.

      Global Install from Packages

      If you wish to install Django using the Ubuntu repositories, the process is very straightforward.

      First, update your local package index with apt:

      Next, check which version of Python you have installed. 18.04 ships with Python 3.6 by default, which you can verify by typing:

      You should see output like this:

      Output

      Python 3.6.5

      Next, install Django:

      • sudo apt install python3-django

      You can test that the installation was successful by typing:

      Output

      1.11.11

      This means that the software was successfully installed. You may also notice that the Django version is not the latest stable version. To learn more about how to use the software, skip ahead to learn how to create sample project.

      Install with pip in a Virtual Environment

      The most flexible way to install Django on your system is within a virtual environment. We will show you how to install Django in a virtual environment that we will create with the venv module, part of the standard Python 3 library. This tool allows you to create virtual Python environments and install Python packages without affecting the rest of the system. You can therefore select Python packages on a per-project basis, regardless of conflicts with other projects' requirements.

      Let's begin by refreshing the local package index:

      Check the version of Python you have installed:

      Output

      Python 3.6.5

      Next, let's install pip from the Ubuntu repositories:

      • sudo apt install python3-pip

      Once pip is installed, you can use it to install the venv package:

      • sudo apt install python3-venv

      Now, whenever you start a new project, you can create a virtual environment for it. Start by creating and moving into a new project directory:

      • mkdir ~/newproject
      • cd ~/newproject

      Next, create a virtual environment within the project directory using the python command that's compatible with your version of Python. We will call our virtual environment my_env, but you should name it something descriptive:

      This will install standalone versions of Python and pip into an isolated directory structure within your project directory. A directory will be created with the name you select, which will hold the file hierarchy where your packages will be installed.

      To install packages into the isolated environment, you must activate it by typing:

      • source my_env/bin/activate

      Your prompt should change to reflect that you are now in your virtual environment. It will look something like (my_env)username@hostname:~/newproject$.

      In your new environment, you can use pip to install Django. Regardless of your Python version, pip should just be called pip when you are in your virtual environment. Also note that you do not need to use sudo since you are installing locally:

      You can verify the installation by typing:

      Output

      2.1

      Note that your version may differ from the version shown here.

      To leave your virtual environment, you need to issue the deactivate command from anywhere on the system:

      Your prompt should revert to the conventional display. When you wish to work on your project again, re-activate your virtual environment by moving back into your project directory and activating:

      • cd ~/newproject
      • source my_env/bin/activate

      Development Version Install with Git

      If you need a development version of Django, you can download and install Django from its Git repository. Let's do this from within a virtual environment.

      First, let's update the local package index:

      Check the version of Python you have installed:

      Output

      Python 3.6.5

      Next, install pip from the official repositories:

      • sudo apt install python3-pip

      Install the venv package to create your virtual environment:

      • sudo apt install python3-venv

      The next step is cloning the Django repository. Between releases, this repository will have more up-to-date features and bug fixes at the possible expense of stability. You can clone the repository to a directory called ~/django-dev within your home directory by typing:

      • git clone git://github.com/django/django ~/django-dev

      Change to this directory:

      Create a virtual environment using the python command that's compatible with your installed version of Python:

      Activate it:

      • source my_env/bin/activate

      Next, you can install the repository using pip. The -e option will install in "editable" mode, which is necessary when installing from version control:

      • pip install -e ~/django-dev

      You can verify that the installation was successful by typing:

      Output

      2.2.dev20180802155335

      Again, the version you see displayed may not match what is shown here.

      You now have the latest version of Django in your virtual environment.

      Creating a Sample Project

      With Django installed, you can begin building your project. We will go over how to create a project and test it on your development server using a virtual environment.

      First, create a directory for your project and change into it:

      • mkdir ~/django-test
      • cd ~/django-test

      Next, create your virtual environment:

      Activate the environment:

      • source my_env/bin/activate

      Install Django:

      To build your project, you can use django-admin with the startproject command. We will call our project djangoproject, but you can replace this with a different name. startproject will create a directory within your current working directory that includes:

      • A management script, manage.py, which you can use to administer various Django-specific tasks.
      • A directory (with the same name as the project) that includes the actual project code.

      To avoid having too many nested directories, however, let's tell Django to place the management script and inner directory in the current directory (notice the ending dot):

      • django-admin startproject djangoproject .

      To migrate the database (this example uses SQLite by default), let's use the migrate command with manage.py. Migrations apply any changes you've made to your Django models to your database schema.

      To migrate the database, type:

      You will see output like the following:

      Output

      Operations to perform: Apply all migrations: admin, auth, contenttypes, sessions Running migrations: Applying contenttypes.0001_initial... OK Applying auth.0001_initial... OK Applying admin.0001_initial... OK Applying admin.0002_logentry_remove_auto_add... OK Applying admin.0003_logentry_add_action_flag_choices... OK Applying contenttypes.0002_remove_content_type_name... OK Applying auth.0002_alter_permission_name_max_length... OK Applying auth.0003_alter_user_email_max_length... OK Applying auth.0004_alter_user_username_opts... OK Applying auth.0005_alter_user_last_login_null... OK Applying auth.0006_require_contenttypes_0002... OK Applying auth.0007_alter_validators_add_error_messages... OK Applying auth.0008_alter_user_username_max_length... OK Applying auth.0009_alter_user_last_name_max_length... OK Applying sessions.0001_initial... OK

      Finally, let's create an administrative user so that you can use the Djano admin interface. Let's do this with the createsuperuser command:

      • python manage.py createsuperuser

      You will be prompted for a username, an email address, and a password for your user.

      Modifying ALLOWED_HOSTS in the Django Settings

      To successfully test your application, you will need to modify one of the directives in the Django settings.

      Open the settings file by typing:

      • nano ~/django-test/djangoproject/settings.py

      Inside, locate the ALLOWED_HOSTS directive. This defines a whitelist of addresses or domain names that may be used to connect to the Django instance. An incoming request with a Host header that is not in this list will raise an exception. Django requires that you set this to prevent a certain class of security vulnerability.

      In the square brackets, list the IP addresses or domain names that are associated with your Django server. Each item should be listed in quotations, with separate entries separated by a comma. If you want requests for an entire domain and any subdomains, prepend a period to the beginning of the entry:

      ~/django-test/djangoproject/settings.py

      . . .
      ALLOWED_HOSTS = ['your_server_ip_or_domain', 'your_second_ip_or_domain', . . .]
      

      When you are finished, save the file and exit your editor.

      Testing the Development Server

      Once you have a user, you can start up the Django development server to see what a fresh Django project looks like. You should only use this for development purposes. When you are ready to deploy, be sure to follow Django's guidelines on deployment carefully.

      Before you try the development server, make sure you open the appropriate port in your firewall. If you followed the initial server setup guide and are using UFW, you can open port 8000 by typing:

      Start the development server:

      • python manage.py runserver your_server_ip:8000

      Visit your server's IP address followed by :8000 in your web browser:

      http://your_server_ip:8000
      

      You should see something that looks like this:

      Django public page

      To access the admin interface, add /admin/ to the end of your URL:

      http://your_server_ip:8000/admin/
      

      This will take you to a log in screen:

      Django admin login

      If you enter the admin username and password that you just created, you will have access to the main admin section of the site:

      Django admin page

      For more information about working with the Django admin interface, please see "How To Enable and Connect the Django Admin Interface."

      When you are finished looking through the default site, you can stop the development server by typing CTRL-C in your terminal.

      The Django project you've created provides the structural basis for designing a more complete site. Check out the Django documentation for more information about how to build your applications and customize your site.

      Conclusion

      You should now have Django installed on your Ubuntu 18.04 server, providing the main tools you need to create powerful web applications. You should also know how to start a new project and launch the developer server. Leveraging a complete web framework like Django can help make development faster, allowing you to concentrate only on the unique aspects of your applications.

      If you would like more information about working with Django, including in-depth discussions of things like models and views, please see our Django development series.



      Source link

      How To Install Apache Kafka on CentOS 7


      The author selected the Free and Open Source Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      Apache Kafka is a popular distributed message broker designed to efficiently handle large volumes of real-time data. A Kafka cluster is not only highly scalable and fault-tolerant, but it also has a much higher throughput compared to other message brokers such as ActiveMQ and RabbitMQ. Though it is generally used as a publish/subscribe messaging system, a lot of organizations also use it for log aggregation because it offers persistent storage for published messages.

      A publish/subscribe messaging system allows one or more producers to publish messages without considering the number of consumers or how they will process the messages. Subscribed clients are notified automatically about updates and the creation of new messages. This system is more efficient and scalable than systems where clients poll periodically to determine if new messages are available.

      In this tutorial, you will install and use Apache Kafka 1.1.0 on CentOS 7.

      Prerequisites

      To follow along, you will need:

      • One CentOS 7 server and a non-root user with sudo privileges. Follow the steps specified in this guide if you do not have a non-root user set up.
      • At least 4GB of RAM on the server. Installations without this amount of RAM may cause the Kafka service to fail, with the Java virtual machine (JVM) throwing an “Out Of Memory” exception during startup.
      • OpenJDK 8 installed on your server. To install this version, follow these instructions on installing specific versions of OpenJDK. Kafka is written in Java, so it requires a JVM; however, its startup shell script has a version detection bug that causes it to fail to start with JVM versions above 8.

      Step 1 — Creating a User for Kafka

      Since Kafka can handle requests over a network, you should create a dedicated user for it. This minimizes damage to your CentOS machine should the Kafka server be compromised. We will create a dedicated kafka user in this step, but you should create a different non-root user to perform other tasks on this server once you have finished setting up Kafka.

      Logged in as your non-root sudo user, create a user called kafka with the useradd command:

      The -m flag ensures that a home directory will be created for the user. This home directory, /home/kafka, will act as our workspace directory for executing commands in the sections below.

      Set the password using passwd:

      Add the kafka user to the wheel group with the adduser command, so that it has the privileges required to install Kafka's dependencies:

      • sudo usermod -aG wheel kafka

      Your kafka user is now ready. Log into this account using su:

      Now that we've created the Kafka-specific user, we can move on to downloading and extracting the Kafka binaries.

      Step 2 — Downloading and Extracting the Kafka Binaries

      Let's download and extract the Kafka binaries into dedicated folders in our kafka user's home directory.

      To start, create a directory in /home/kafka called Downloads to store your downloads:

      Use curl to download the Kafka binaries:

      • curl "http://www-eu.apache.org/dist/kafka/1.1.0/kafka_2.12-1.1.0.tgz" -o ~/Downloads/kafka.tgz

      Create a directory called kafka and change to this directory. This will be the base directory of the Kafka installation:

      • mkdir ~/kafka && cd ~/kafka

      Extract the archive you downloaded using the tar command:

      • tar -xvzf ~/Downloads/kafka.tgz --strip 1

      We specify the --strip 1 flag to ensure that the archive's contents are extracted in ~/kafka/ itself and not in another directory (such as ~/kafka/kafka_2.12-1.1.0/) inside of it.

      Now that we've downloaded and extracted the binaries successfully, we can move on configuring to Kafka to allow for topic deletion.

      Step 3 — Configuring the Kafka Server

      Kafka's default behavior will not allow us to delete a topic, the category, group, or feed name to which messages can be published. To modify this, let's edit the configuration file.

      Kafka's configuration options are specified in server.properties. Open this file with vi or your favorite editor:

      • vi ~/kafka/config/server.properties

      Let's add a setting that will allow us to delete Kafka topics. Press i to insert text, and add the following to the bottom of the file:

      ~/kafka/config/server.properties

      delete.topic.enable = true
      

      When you are finished, press ESC to exit insert mode and :wq to write the changes to the file and quit. Now that we've configured Kafka, we can move on to creating systemd unit files for running and enabling it on startup.

      Step 4 — Creating Systemd Unit Files and Starting the Kafka Server

      In this section, we will create systemd unit files for the Kafka service. This will help us perform common service actions such as starting, stopping, and restarting Kafka in a manner consistent with other Linux services.

      Zookeeper is a service that Kafka uses to manage its cluster state and configurations. It is commonly used in many distributed systems as an integral component. If you would like to know more about it, visit the official Zookeeper docs.

      Create the unit file for zookeeper:

      • sudo vi /etc/systemd/system/zookeeper.service

      Enter the following unit definition into the file:

      /etc/systemd/system/zookeeper.service

      [Unit]
      Requires=network.target remote-fs.target
      After=network.target remote-fs.target
      
      [Service]
      Type=simple
      User=kafka
      ExecStart=/home/kafka/kafka/bin/zookeeper-server-start.sh /home/kafka/kafka/config/zookeeper.properties
      ExecStop=/home/kafka/kafka/bin/zookeeper-server-stop.sh
      Restart=on-abnormal
      
      [Install]
      WantedBy=multi-user.target
      

      The [Unit] section specifies that Zookeeper requires networking and the filesystem to be ready before it can start.

      The [Service] section specifies that systemd should use the zookeeper-server-start.sh and zookeeper-server-stop.sh shell files for starting and stopping the service. It also specifies that Zookeeper should be restarted automatically if it exits abnormally.

      Next, create the systemd service file for kafka:

      • sudo vi /etc/systemd/system/kafka.service

      Enter the following unit definition into the file:

      /etc/systemd/system/kafka.service

      [Unit]
      Requires=zookeeper.service
      After=zookeeper.service
      
      [Service]
      Type=simple
      User=kafka
      ExecStart=/bin/sh -c '/home/kafka/kafka/bin/kafka-server-start.sh /home/kafka/kafka/config/server.properties > /home/kafka/kafka/kafka.log 2>&1'
      ExecStop=/home/kafka/kafka/bin/kafka-server-stop.sh
      Restart=on-abnormal
      
      [Install]
      WantedBy=multi-user.target
      

      The [Unit] section specifies that this unit file depends on zookeeper.service. This will ensure that zookeeper gets started automatically when the kafa service starts.

      The [Service] section specifies that systemd should use the kafka-server-start.sh and kafka-server-stop.sh shell files for starting and stopping the service. It also specifies that Kafka should be restarted automatically if it exits abnormally.

      Now that the units have been defined, start Kafka with the following command:

      • sudo systemctl start kafka

      To ensure that the server has started successfully, check the journal logs for the kafka unit:

      You should see output similar to the following:

      Output

      Jul 17 18:38:59 kafka-centos systemd[1]: Started kafka.service.

      You now have a Kafka server listening on port 9092.

      While we have started the kafka service, if we were to reboot our server, it would not be started automatically. To enable kafka on server boot, run:

      • sudo systemctl enable kafka

      Now that we've started and enabled the services, let's check the installation.

      Step 5 — Testing the Installation

      Let's publish and consume a "Hello World" message to make sure the Kafka server is behaving correctly. Publishing messages in Kafka requires:

      • A producer, which enables the publication of records and data to topics.
      • A consumer, which reads messages and data from topics.

      First, create a topic named TutorialTopic by typing:

      • ~/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic TutorialTopic

      You can create a producer from the command line using the kafka-console-producer.sh script. It expects the Kafka server's hostname, port, and a topic name as arguments.

      Publish the string "Hello, World" to the TutorialTopic topic by typing:

      • echo "Hello, World" | ~/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic TutorialTopic > /dev/null

      Next, you can create a Kafka consumer using the kafka-console-consumer.sh script. It expects the ZooKeeper server's hostname and port, along with a topic name as arguments.

      The following command consumes messages from TutorialTopic. Note the use of the --from-beginning flag, which allows the consumption of messages that were published before the consumer was started:

      • ~/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic TutorialTopic --from-beginning

      If there are no configuration issues, you should see Hello, World in your terminal:

      Output

      Hello, World

      The script will continue to run, waiting for more messages to be published to the topic. Feel free to open a new terminal and start a producer to publish a few more messages. You should be able to see them all in the consumer's output.

      When you are done testing, press CTRL+C to stop the consumer script. Now that we have tested the installation, let's move on to installing KafkaT.

      Step 6 — Installing KafkaT (Optional)

      KafkaT is a tool from Airbnb that makes it easier for you to view details about your Kafka cluster and perform certain administrative tasks from the command line. Because it is a Ruby gem, you will need Ruby to use it. You will also need ruby-devel and build-related packages such as make and gcc to be able to build the other gems it depends on. Install them using yum:

      • sudo yum install ruby ruby-devel make gcc patch

      You can now install KafkaT using the gem command:

      KafkaT uses .kafkatcfg as the configuration file to determine the installation and log directories of your Kafka server. It should also have an entry pointing KafkaT to your ZooKeeper instance.

      Create a new file called .kafkatcfg:

      Add the following lines to specify the required information about your Kafka server and Zookeeper instance:

      ~/.kafkatcfg

      {
        "kafka_path": "~/kafka",
        "log_path": "/tmp/kafka-logs",
        "zk_path": "localhost:2181"
      }
      

      You are now ready to use KafkaT. For a start, here's how you would use it to view details about all Kafka partitions:

      You will see the following output:

      Output

      Topic Partition Leader Replicas ISRs TutorialTopic 0 0 [0] [0] __consumer_offsets 0 0 [0] [0] ... ...

      You will see TutorialTopic, as well as __consumer_offsets, an internal topic used by Kafka for storing client-related information. You can safely ignore lines starting with __consumer_offsets.

      To learn more about KafkaT, refer to its GitHub repository.

      Step 7 — Setting Up a Multi-Node Cluster (Optional)

      If you want to create a multi-broker cluster using more CentOS 7 machines, you should repeat Step 1, Step 4, and Step 5 on each of the new machines. Additionally, you should make the following changes in the server.properties file for each:

      • The value of the broker.id property should be changed such that it is unique throughout the cluster. This property uniquely identifies each server in the cluster and can have any string as its value. For example, "server1", "server2", etc.

      • The value of the zookeeper.connect property should be changed such that all nodes point to the same ZooKeeper instance. This property specifies the Zookeeper instance's address and follows the <HOSTNAME/IP_ADDRESS>:<PORT> format. For example, "203.0.113.0:2181", "203.0.113.1:2181" etc.

      If you want to have multiple ZooKeeper instances for your cluster, the value of the zookeeper.connect property on each node should be an identical, comma-separated string listing the IP addresses and port numbers of all the ZooKeeper instances.

      Step 8 — Restricting the Kafka User

      Now that all of the installations are done, you can remove the kafka user's admin privileges. Before you do so, log out and log back in as any other non-root sudo user. If you are still running the same shell session you started this tutorial with, simply type exit.

      Remove the kafka user from the sudo group:

      • sudo gpasswd -d kafka wheel

      To further improve your Kafka server's security, lock the kafka user's password using the passwd command. This makes sure that nobody can directly log into the server using this account:

      At this point, only root or a sudo user can log in as kafka by typing in the following command:

      In the future, if you want to unlock it, use passwd with the -u option:

      You have now successfully restricted the kafka user's admin privileges.

      Conclusion

      You now have Apache Kafka running securely on your CentOS server. You can make use of it in your projects by creating Kafka producers and consumers using Kafka clients, which are available for most programming languages. To learn more about Kafka, you can also consult its documentation.



      Source link