One place for hosting & domains

      2019

      How To Install Apache Kafka on Debian 10


      The author selected the Free and Open Source Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      Apache Kafka is a popular distributed message broker designed to handle large volumes of real-time data. A Kafka cluster is highly scalable and fault-tolerant, and also has a much higher throughput compared to other message brokers such as ActiveMQ and RabbitMQ. Though it is generally used as a publish/subscribe messaging system, a lot of organizations also use it for log aggregation because it offers persistent storage for published messages.

      A publish/subscribe messaging system allows one or more producers to publish messages without considering the number of consumers or how they will process the messages. Subscribed clients are notified automatically about updates and the creation of new messages. This system is more efficient and scalable than systems where clients poll periodically to determine if new messages are available.

      In this tutorial, you will install and configure Apache Kafka 2.1.1 securely on a Debian 10 server, then test your setup by producing and consuming a Hello World message. You will then optionally install KafkaT to monitor Kafka and set up a Kafka multi-node cluster.

      Prerequisites

      To follow along, you will need:

      Note: Installations without 4GB of RAM may cause the Kafka service to fail, with the Java virtual machine (JVM) throwing an Out Of Memory exception during startup.

      Step 1 — Creating a User for Kafka

      Since Kafka can handle requests over a network, it is a best practice to create a dedicated user for it. This minimizes damage to your Debian machine should the Kafka server be compromised. You will create the dedicated user kafka in this step.

      Logged in as your non-root sudo user, create a user called kafka with the useradd command:

      The -m flag ensures that a home directory will be created for the user. This home directory, /home/kafka, will act as your workspace directory for executing commands later on.

      Set the password using passwd:

      Enter the password you wish to use for this user.

      Next, add the kafka user to the sudo group with the adduser command, so that it has the privileges required to install Kafka’s dependencies:

      Your kafka user is now ready. Log into this account using su:

      Now that you’ve created the Kafka-specific user, you can move on to downloading and extracting the Kafka binaries.

      Step 2 — Downloading and Extracting the Kafka Binaries

      In this step, you will download and extract the Kafka binaries into dedicated folders in your kafka user’s home directory.

      To start, create a directory in /home/kafka called Downloads to store your downloads:

      Next, install curl using apt-get so that you’ll be able to download remote files:

      • sudo apt-get update && sudo apt-get install curl

      When prompted, type Y to confirm the curl download.

      Once curl is installed, use it to download the Kafka binaries:

      • curl "https://archive.apache.org/dist/kafka/2.1.1/kafka_2.11-2.1.1.tgz" -o ~/Downloads/kafka.tgz

      Create a directory called kafka and change to this directory. This will be the base directory of the Kafka installation:

      • mkdir ~/kafka && cd ~/kafka

      Extract the archive you downloaded using the tar command:

      • tar -xvzf ~/Downloads/kafka.tgz --strip 1

      You specified the --strip 1 flag to ensure that the archive’s contents are extracted in ~/kafka/ itself and not in another directory inside of it, such as ~/kafka/kafka_2.12-2.1.1/.

      Now that you’ve downloaded and extracted the binaries successfully, you can move on to configuring Kafka to allow for topic deletion.

      Step 3 — Configuring the Kafka Server

      Kafka’s default behavior will not allow us to delete a topic, the category, group, or feed name to which messages can be published. To modify this, you will edit the configuration file.

      Kafka’s configuration options are specified in server.properties. Open this file with nano or your favorite editor:

      • nano ~/kafka/config/server.properties

      Let’s add a setting that will allow us to delete Kafka topics. Add the following highlighted line to the bottom of the file:

      ~/kafka/config/server.properties

      ...
      group.initial.rebalance.delay.ms
      
      delete.topic.enable = true
      

      Save the file, and exit nano. Now that you’ve configured Kafka, you can create systemd unit files for running and enabling Kafka on startup.

      Step 4 — Creating Systemd Unit Files and Starting the Kafka Server

      In this section, you will create systemd unit files for the Kafka service. This will help you perform common service actions such as starting, stopping, and restarting Kafka in a manner consistent with other Linux services.

      ZooKeeper is a service that Kafka uses to manage its cluster state and configurations. It is commonly used in distributed systems as an integral component. In this tutorial, you will use Zookeeper to manage these aspects of Kafka. If you would like to know more about it, visit the official ZooKeeper docs.

      First, create the unit file for zookeeper:

      • sudo nano /etc/systemd/system/zookeeper.service

      Enter the following unit definition into the file:

      /etc/systemd/system/zookeeper.service

      [Unit]
      Requires=network.target remote-fs.target
      After=network.target remote-fs.target
      
      [Service]
      Type=simple
      User=kafka
      ExecStart=/home/kafka/kafka/bin/zookeeper-server-start.sh /home/kafka/kafka/config/zookeeper.properties
      ExecStop=/home/kafka/kafka/bin/zookeeper-server-stop.sh
      Restart=on-abnormal
      
      [Install]
      WantedBy=multi-user.target
      

      The [Unit] section specifies that ZooKeeper requires networking and for the filesystem to be ready before it can start.

      The [Service] section specifies that systemd should use the zookeeper-server-start.sh and zookeeper-server-stop.sh shell files for starting and stopping the service. It also specifies that ZooKeeper should be restarted automatically if it exits abnormally.

      Next, create the systemd service file for kafka:

      • sudo nano /etc/systemd/system/kafka.service

      Enter the following unit definition into the file:

      /etc/systemd/system/kafka.service

      [Unit]
      Requires=zookeeper.service
      After=zookeeper.service
      
      [Service]
      Type=simple
      User=kafka
      ExecStart=/bin/sh -c '/home/kafka/kafka/bin/kafka-server-start.sh /home/kafka/kafka/config/server.properties > /home/kafka/kafka/kafka.log 2>&1'
      ExecStop=/home/kafka/kafka/bin/kafka-server-stop.sh
      Restart=on-abnormal
      
      [Install]
      WantedBy=multi-user.target
      

      The [Unit] section specifies that this unit file depends on zookeeper.service. This will ensure that zookeeper gets started automatically when the kafka service starts.

      The [Service] section specifies that systemd should use the kafka-server-start.sh and kafka-server-stop.sh shell files for starting and stopping the service. It also specifies that Kafka should be restarted automatically if it exits abnormally.

      Now that the units have been defined, start Kafka with the following command:

      • sudo systemctl start kafka

      To ensure that the server has started successfully, check the journal logs for the kafka unit:

      You will see output similar to the following:

      Output

      Mar 23 13:31:48 kafka systemd[1]: Started kafka.service.

      You now have a Kafka server listening on port 9092, which is the default port for Kafka.

      You have started the kafka service, but if you were to reboot your server, it would not yet be started automatically. To enable kafka on server boot, run:

      • sudo systemctl enable kafka

      Now that you’ve started and enabled the services, it’s time to check the installation.

      Step 5 — Testing the Installation

      Let’s publish and consume a Hello World message to make sure the Kafka server is behaving correctly. Publishing messages in Kafka requires:

      • A producer, which enables the publication of records and data to topics.
      • A consumer, which reads messages and data from topics.

      First, create a topic named TutorialTopic by typing:

      • ~/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic TutorialTopic

      You can create a producer from the command line using the kafka-console-producer.sh script. It expects the Kafka server’s hostname, port, and a topic name as arguments.

      Publish the string Hello, World to the TutorialTopic topic by typing:

      • echo "Hello, World" | ~/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic TutorialTopic > /dev/null

      The --broker-list flag determines the list of message brokers to send the message to, in this case localhost:9092. --topic designates the topic as TutorialTopic.

      Next, you can create a Kafka consumer using the kafka-console-consumer.sh script. It expects the ZooKeeper server’s hostname and port and a topic name as arguments.

      The following command consumes messages from TutorialTopic. Note the use of the --from-beginning flag, which allows the consumption of messages that were published before the consumer was started:

      • ~/kafka/bin/kafka-console-consumer.sh --bootstrap-server `localhost:9092` --topic TutorialTopic --from-beginning

      --bootstrap-server provides a list of ingresses into the Kafka cluster. In this case, you are using localhost:9092.

      You will see Hello, World in your terminal:

      Output

      Hello, World

      The script will continue to run, waiting for more messages to be published to the topic. Feel free to open a new terminal and start a producer to publish a few more messages. You should be able to see them all in the consumer’s output. If you’d like to learn more about how to use Kafka, see the official Kafka documentation.

      When you are done testing, press CTRL+C to stop the consumer script. Now that you have tested the installation, you can move on to installing KafkaT in order to better administer your Kafka cluster.

      Step 6 — Installing KafkaT (Optional)

      KafkaT is a tool from Airbnb that makes it easier for you to view details about your Kafka cluster and perform certain administrative tasks from the command line. Because it is a Ruby gem, you will need Ruby to use it. You will also need the build-essential package to be able to build the other gems it depends on. Install them using apt:

      • sudo apt install ruby ruby-dev build-essential

      You can now install KafkaT using the gem command:

      • sudo CFLAGS=-Wno-error=format-overflow gem install kafkat

      The CFLAGS=-Wno-error=format-overflow option disables format overflow warnings and is required for the ZooKeeper gem, which is a dependency of KafkaT.

      KafkaT uses .kafkatcfg as the configuration file to determine the installation and log directories of your Kafka server. It should also have an entry pointing KafkaT to your ZooKeeper instance.

      Create a new file called .kafkatcfg:

      Add the following lines to specify the required information about your Kafka server and Zookeeper instance:

      ~/.kafkatcfg

      {
        "kafka_path": "~/kafka",
        "log_path": "/tmp/kafka-logs",
        "zk_path": "localhost:2181"
      }
      

      You are now ready to use KafkaT. For a start, here’s how you would use it to view details about all Kafka partitions:

      You will see the following output:

      Output

      Topic Partition Leader Replicas ISRs TutorialTopic 0 0 [0] [0] __consumer_offsets 0 0 [0] [0] ...

      This output shows TutorialTopic, as well as __consumer_offsets, an internal topic used by Kafka for storing client-related information. You can safely ignore lines starting with __consumer_offsets.

      To learn more about KafkaT, refer to its GitHub repository.

      Now that you have installed KafkaT, you can optionally set up Kafka on a cluster of Debian 10 servers to make a multi-node cluster.

      Step 7 — Setting Up a Multi-Node Cluster (Optional)

      If you want to create a multi-broker cluster using more Debian 10 servers, repeat Step 1, Step 4, and Step 5 on each of the new machines. Additionally, make the following changes in the ~/kafka/config/server.properties file for each:

      • Change the value of the broker.id property such that it is unique throughout the cluster. This property uniquely identifies each server in the cluster and can have any string as its value. For example, "server1", "server2", etc., would be useful as identifiers.

      • Change the value of the zookeeper.connect property such that all nodes point to the same ZooKeeper instance. This property specifies the ZooKeeper instance’s address and follows the <HOSTNAME/IP_ADDRESS>:<PORT> format. For this tutorial, you would use your_first_server_IP:2181, replacing your_first_server_IP with the IP address of the Debian 10 server you already set up.

      If you want to have multiple ZooKeeper instances for your cluster, the value of the zookeeper.connect property on each node should be an identical, comma-separated string listing the IP addresses and port numbers of all the ZooKeeper instances.

      Note: If you have a firewall activated on the Debian 10 server with Zookeeper installed, make sure to open up port 2181 to allow for incoming requests from the other nodes in the cluster.

      Step 8 — Restricting the Kafka User

      Now that all of the installations are done, you can remove the kafka user’s admin privileges. Before you do so, log out and log back in as any other non-root sudo user. If you are still running the same shell session you started this tutorial with, simply type exit.

      Remove the kafka user from the sudo group:

      To further improve your Kafka server’s security, lock the kafka user’s password using the passwd command. This makes sure that nobody can directly log into the server using this account:

      At this point, only root or a sudo user can log in as kafka by typing in the following command:

      In the future, if you want to unlock it, use passwd with the -u option:

      You have now successfully restricted the kafka user’s admin privileges.

      Conclusion

      You now have Apache Kafka running securely on your Debian server. You can make use of it in your projects by creating Kafka producers and consumers using Kafka clients, which are available for most programming languages. To learn more about Kafka, you can also consult the Apache Kafka documentation.



      Source link

      How To Set Up the code-server Cloud IDE Platform on CentOS 7


      The author selected the Free and Open Source Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      With developer tools moving to the cloud, creation and adoption of cloud IDE (Integrated Development Environment) platforms is growing. Cloud IDEs allow for real-time collaboration between developer teams to work in a unified development environment that minimizes incompatibilities and enhances productivity. Accessible through web browsers, cloud IDEs are available from every type of modern device.

      code-server is Microsoft Visual Studio Code running on a remote server and accessible directly from your browser. Visual Studio Code is a modern code editor with integrated Git support, a code debugger, smart autocompletion, and customizable and extensible features. This means that you can use various devices running different operating systems, and always have a consistent development environment on hand.

      In this tutorial, you will set up the code-server cloud IDE platform on your CentOS 7 machine and expose it at your domain, secured with free Let’s Encrypt TLS certificates. In the end, you’ll have Microsoft Visual Studio Code running on your CentOS 7 server, available at your domain and protected with a password.

      Prerequisites

      • A server running CentOS 7 with at least 2GB RAM, root access, and a sudo, non-root account. You can set this up by following this initial server setup guide.

      • Nginx installed on your server. For a guide on how to do this, see How To Install Nginx on CentOS 7.

      • Both of the following DNS records set up for your server. You can follow this introduction to DigitalOcean DNS for details on how to add them.

        • An A record with your-domain pointing to your server’s public IP address.
        • An A record with www.your-domain pointing to your server’s public IP address.
      • A fully registered domain name to host code-server, pointed to your server. This tutorial will use code-server.your-domain throughout. You can purchase a domain name on Namecheap, get one for free on Freenom, or use the domain registrar of your choice.

      Step 1 — Installing code-server

      In this section, you will set up code-server on your server. This entails downloading the latest version and creating a systemd service that will keep code-server always running in the background. You’ll also specify a restart policy for the service, so that code-server stays available after possible crashes or reboots.

      You’ll store all data pertaining to code-server in a folder named ~/code-server. Create it by running the following command:

      Navigate to it:

      You’ll need to head over to the Github releases page of code-server and pick the latest Linux build (the file will contain ‘linux’ in its name). At the time of writing, the latest version was 2.1692. Download it using curl by running the following command:

      • curl -LO https://github.com/cdr/code-server/releases/download/2.1692-vsc1.39.2/code-server2.1692-vsc1.39.2-linux-x86_64.tar.gz

      Then, unpack the archive by running:

      • tar -xzvf code-server2.1692-vsc1.39.2-linux-x86_64.tar.gz

      You’ll get a folder named exactly as the original file you downloaded, which contains the code-server executable. Navigate to it:

      • cd code-server2.1692-vsc1.39.2-linux-x86_64

      Copy the code-server executable to /usr/local/bin so you’ll be able to access it system wide by running the following command:

      • sudo cp code-server /usr/local/bin

      Next, create a folder for code-server, where it will store user data:

      • sudo mkdir /var/lib/code-server

      Now that you’ve downloaded code-server and made it available system-wide, you will create a systemd service to keep code-server running in the background at all times.

      You’ll store the service configuration in a file named code-server.service, in the /usr/lib/systemd/system directory, where systemd stores its services. Create it using the vi editor:

      • sudo vi /usr/lib/systemd/system/code-server.service

      Add the following lines:

      /usr/lib/systemd/system/code-server.service

      [Unit]
      Description=code-server
      After=nginx.service
      
      [Service]
      Type=simple
      Environment=PASSWORD=your_password
      ExecStart=/usr/local/bin/code-server --host 127.0.0.1 --user-data-dir /var/lib/code-server --auth password
      Restart=always
      
      [Install]
      WantedBy=multi-user.target
      

      Here you first specify the description of the service. Then, you state that the nginx service must be started before this one. After the [Unit] section, you define the type of the service (simple means that the process should be simply run) and provide the command that will be executed.

      You also specify that the global code-server executable should be started with a few arguments specific to code-server. --host 127.0.0.1 binds it to localhost, so it’s only directly accessible from inside of your server. --user-data-dir /var/lib/code-server sets its user data directory, and --auth password specifies that it should authenticate visitors with a password, specified in the PASSWORD environment variable declared on the line above it.

      Remember to replace your_password with your desired password. Type :wq and then ENTER to save and close the file.

      The next line tells systemd to restart code-server in all malfunction events (for example, when it crashes or the process is killed). The [Install] section orders systemd to start this service when it becomes possible to log in to your server.

      Start the code-server service by running the following command:

      • sudo systemctl start code-server

      Check that it’s started correctly by observing its status:

      • sudo systemctl status code-server

      You’ll see output similar to:

      Output

      ● code-server.service - code-server Loaded: loaded (/usr/lib/systemd/system/code-server.service; disabled; vendor preset: disabled) Active: active (running) since Thu 2019-12-19 19:24:42 UTC; 5s ago Main PID: 1668 (code-server) CGroup: /system.slice/code-server.service ├─1668 /usr/local/bin/code-server --host 127.0.0.1 --user-data-dir /var/lib/code-server --auth password └─1679 /usr/local/bin/code-server --host 127.0.0.1 --user-data-dir /var/lib/code-server --auth password Dec 19 19:24:42 code-server-centos systemd[1]: Started code-server. Dec 19 19:24:44 code-server-centos code-server[1668]: info Server listening on http://127.0.0.1:8080 Dec 19 19:24:44 code-server-centos code-server[1668]: info - Using custom password for authentication Dec 19 19:24:44 code-server-centos code-server[1668]: info - Not serving HTTPS

      To make code-server start automatically after a server reboot, enable its service by running the following command:

      • sudo systemctl enable code-server

      In this step, you’ve downloaded code-server and made it available globally. Then, you’ve created a systemd service for it and enabled it, so code-server will start at every server boot. Next, you’ll expose it at your domain by configuring Nginx to serve as a reverse proxy between the visitor and code-server.

      Step 2 — Exposing code-server at Your Domain

      In this section, you will configure Nginx as a reverse proxy for code-server.

      As you have learned in the Nginx prerequisite step, its site configuration files are stored under /etc/nginx/conf.d and will automatically be loaded when Nginx starts.

      You’ll store the configuration for exposing code-server at your domain in a file named code-server.conf, under /etc/nginx/conf.d. Start off by creating it using your editor:

      • sudo vi /etc/nginx/conf.d/code-server.conf

      Add the following lines:

      /etc/nginx/conf.d/code-server.conf

      server {
          listen 80;
          listen [::]:80;
      
          server_name code-server.your-domain;
      
          location / {
              proxy_pass http://localhost:8080/;
              proxy_set_header Upgrade $http_upgrade;
              proxy_set_header Connection upgrade;
              proxy_set_header Accept-Encoding gzip;
          }
      }
      

      Replace code-server.your-domain with your desired domain, then save and close the file.

      In this file, you define that Nginx should listen to HTTP port 80. Then, you specify a server_name that tells Nginx for which domain to accept requests and apply this particular configuration.

      In the next block, for the root location (/), you specify that requests should be passed back and forth to code-server running at localhost:8080. The next three lines (starting with proxy_set_header) order Nginx to carry over some HTTP request headers that are needed for correct functioning of WebSockets, which code-server extensively uses.

      To test the validity of the configuration, run the following command:

      You’ll see the following output:

      Output

      nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful

      For the configuration to take effect, you’ll need to restart Nginx:

      • sudo systemctl restart nginx

      CentOS 7 comes with SELinux turned on, with a strict ruleset, which by default does not permit Nginx to connect to local TCP sockets. Nginx needs to do in order to serve as a reverse proxy for code-server. Run the following command to relax the rule permanently:

      • sudo setsebool httpd_can_network_connect 1 -P

      Then, in your browser, navigate to the domain you used for code-server. You will see the code-server login prompt.

      code-server login prompt

      code-server is asking you for your password. Enter the one you set in the previous step and press Enter IDE. You’ll now enter code-server and immediately see its editor GUI.

      code-server GUI

      You now have your code-server installation accessible at your domain. In the next step, you’ll secure it by applying a free Let’s Encrypt TLS certificate.

      Step 3 — Securing Your Domain

      In this section, you will secure your domain using a Let’s Encrypt TLS certificate, which you’ll provision using Certbot.

      To install the latest version of Certbot and its Nginx plugin, run the following command:

      • sudo yum install certbot python2-certbot-nginx

      To request certificates for your domain, run the following command:

      • sudo certbot --nginx -d code-server.your-domain

      In this command, you run certbot to request certificates for your domain—you pass the domain name with the -d parameter. The --nginx flag tells it to automatically change Nginx site configuration to support HTTPS. Remember to replace code-server.your-domain with your domain name.

      If this is your first time running Certbot, you’ll be asked to provide an email address for urgent notices and to accept the EFF’s Terms of Service. Certbot will then request certificates for your domain from Let’s Encrypt. It will then ask you if you’d like to redirect all HTTP traffic to HTTPS:

      Output

      Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1: No redirect - Make no further changes to the webserver configuration. 2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for new sites, or if you're confident your site works on HTTPS. You can undo this change by editing your web server's configuration. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Select the appropriate number [1-2] then [enter] (press 'c' to cancel):

      It is recommended to select the second option in order to maximize security. After you input your selection, press ENTER.

      The output will be similar to this:

      Output

      IMPORTANT NOTES: - Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/code-server.your-domain/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/code-server.your-domain/privkey.pem Your cert will expire on ... To obtain a new or tweaked version of this certificate in the future, simply run certbot again with the "certonly" option. To non-interactively renew *all* of your certificates, run "certbot renew" - Your account credentials have been saved in your Certbot configuration directory at /etc/letsencrypt. You should make a secure backup of this folder now. This configuration directory will also contain certificates and private keys obtained by Certbot so making regular backups of this folder is ideal. - If you like Certbot, please consider supporting our work by: Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate Donating to EFF: https://eff.org/donate-le

      This means that Certbot has sucessfully generated TLS certificates and applied them to the Nginx configuration for your domain. You can now reload your code-server domain in your browser and observe a padlock to the left of the site address, which means that your connection is properly secured.

      Now you’ll make Certbot automatically renew the certificates before they expire. To run the renewal check daily, you’ll use cron, a standard system service for running periodic jobs. You direct cron by opening and editing a file called a crontab:

      This command will open the default crontab, which is currently an empty text file. Add the following line, then save and close it:

      crontab

      . . .
      15 3 * * * /usr/bin/certbot renew --quiet
      

      15 3 * * * will run the following command at 3:15 am every day—you can adapt this to any time.

      The renew command for Certbot will check all certificates installed on the system and update any that are set to expire in less than thirty days. --quiet tells Certbot not to output information or wait for user input.

      cron will now run this command daily. All installed certificates will be automatically renewed and reloaded when they have thirty days or less before they expire.

      Now that you have code-server accessible at your domain through a secured Nginx reverse proxy, you’re ready to review the user interface of code-server.

      Step 4 — Using the code-server Interface

      In this section, you’ll use some of the features of the code-server interface. Since code-server is Visual Studio Code running in the cloud, it has the same interface as the standalone desktop edition.

      On the left-hand side of the IDE, there is a vertical row of six buttons opening the most commonly used features in a side panel known as the Activity Bar.

      code-server GUI - Sidepanel

      This bar is customizable so you can move these views to a different order or remove them from the bar. By default, the first button opens the general menu in a dropdown, while the second view opens the Explorer panel that provides tree-like navigation of the project’s structure. You can manage your folders and files here—creating, deleting, moving, and renaming them as necessary. The next view provides access to a search and replace functionality.

      Following this, in the default order, is your view of the source control systems, like Git. Visual Studio code also supports other source control providers and you can find further instructions for source control work flows with the editor in this documentation.

      Git pane with context-menu open

      The debugger option on the Activity Bar provides all the common actions for debugging in the panel. Visual Studio Code comes with built-in support for the Node.js runtime debugger and any language that transpiles to Javascript. For other languages you can install extensions for the required debugger. You can save debugging configurations in the launch.json file.

      Debugger View with launch.json open

      The final view in the Activity Bar provides a menu to access available extensions on the Marketplace.

      code-server GUI - Tabs

      The central part of the GUI is your editor, which you can separate by tabs for your code editing. You can change your editing view to a grid system or to side-by-side files.

      Editor Grid View

      After creating a new file through the File menu, an empty file will open in a new tab, and once saved, the file’s name will be viewable in the Explorer side panel. Creating folders can be done by right clicking on the Explorer sidebar and clicking on New Folder. You can expand a folder by clicking on its name as well as dragging and dropping files and folders to upper parts of the hierarchy to move them to a new location.

      code-server GUI - New Folder

      You can gain access to a terminal by entering CTRL+SHIFT+`, or by clicking on Terminal in the upper menu dropdown, and selecting New Terminal. The terminal will open in a lower panel and its working directory will be set to the project’s workspace, which contains the files and folders shown in the Explorer side panel.

      You’ve explored a high-level overview of the code-server interface and reviewed some of the most commonly used features.

      Conclusion

      You now have code-server, a versatile cloud IDE, installed on your CentOS 7 server, exposed at your domain and secured using Let’s Encrypt certificates. You can now work on projects individually, as well as in a team-collaboration setting. Running a cloud IDE frees resources on your local machine and allows you to scale the resources when needed. For further information, see the Visual Studio Code documentation for additional features and detailed instructions on other components of code-server.

      If you would like to run code-server on your DigitalOcean Kubernetes cluster check out our tutorial on How To Set Up the code-server Cloud IDE Platform on DigitalOcean Kubernetes.



      Source link

      How To Set Up the code-server Cloud IDE Platform on Debian 10


      The author selected the Free and Open Source Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      With developer tools moving to the cloud, creation and adoption of cloud IDE (Integrated Development Environment) platforms is growing. Cloud IDEs allow for real-time collaboration between developer teams to work in a unified development environment that minimizes incompatibilities and enhances productivity. Accessible through web browsers, cloud IDEs are available from every type of modern device.

      code-server is Microsoft Visual Studio Code running on a remote server and accessible directly from your browser. Visual Studio Code is a modern code editor with integrated Git support, a code debugger, smart autocompletion, and customizable and extensible features. This means that you can use various devices running different operating systems, and always have a consistent development environment on hand.

      In this tutorial, you will set up the code-server cloud IDE platform on your Debian 10 machine and expose it at your domain, secured with free Let’s Encrypt TLS certificates. In the end, you’ll have Microsoft Visual Studio Code running on your Debian 10 server, available at your domain and protected with a password.

      Prerequisites

      • A server running Debian 10 with at least 2GB RAM, root access, and a sudo, non-root account. You can set this up by following this initial server setup guide.

      • Nginx installed on your server. For a guide on how to do this, complete Steps 1 to 4 of How To Install Nginx on Debian 10.

      • Both of the following DNS records set up for your server. You can follow this introduction to DigitalOcean DNS for details on how to add them.

        • An A record with your-domain pointing to your server’s public IP address.
        • An A record with www.your-domain pointing to your server’s public IP address.
      • A fully registered domain name to host code-server, pointed to your server. This tutorial will use code-server.your-domain throughout. You can purchase a domain name on Namecheap, get one for free on Freenom, or use the domain registrar of your choice.

      Step 1 — Installing code-server

      In this section, you will set up code-server on your server. This entails downloading the latest version and creating a systemd service that will keep code-server always running in the background. You’ll also specify a restart policy for the service, so that code-server stays available after possible crashes or reboots.

      You’ll store all data pertaining to code-server in a folder named ~/code-server. Create it by running the following command:

      Navigate to it:

      You’ll need to head over to the Github releases page of code-server and pick the latest Linux build (the file will contain ‘linux’ in its name). At the time of writing, the latest version was 2.1692. Download it using wget by running the following command:

      • wget https://github.com/cdr/code-server/releases/download/2.1692-vsc1.39.2/code-server2.1692-vsc1.39.2-linux-x86_64.tar.gz

      Then, unpack the archive by running:

      • tar -xzvf code-server2.1692-vsc1.39.2-linux-x86_64.tar.gz

      You’ll get a folder named exactly as the original file you downloaded, which contains the code-server executable. Navigate to it:

      • cd code-server2.1692-vsc1.39.2-linux-x86_64

      Copy the code-server executable to /usr/local/bin so you’ll be able to access it system wide by running the following command:

      • sudo cp code-server /usr/local/bin

      Next, create a folder for code-server, where it will store user data:

      • sudo mkdir /var/lib/code-server

      Now that you’ve downloaded code-server and made it available system-wide, you will create a systemd service to keep code-server running in the background at all times.

      You’ll store the service configuration in a file named code-server.service, in the /lib/systemd/system directory, where systemd stores its services. Create it using your text editor:

      • sudo nano /lib/systemd/system/code-server.service

      Add the following lines:

      /lib/systemd/system/code-server.service

      [Unit]
      Description=code-server
      After=nginx.service
      
      [Service]
      Type=simple
      Environment=PASSWORD=your_password
      ExecStart=/usr/local/bin/code-server --host 127.0.0.1 --user-data-dir /var/lib/code-server --auth password
      Restart=always
      
      [Install]
      WantedBy=multi-user.target
      

      Here you first specify the description of the service. Then, you state that the nginx service must be started before this one. After the [Unit] section, you define the type of the service (simple means that the process should be simply run) and provide the command that will be executed.

      You also specify that the global code-server executable should be started with a few arguments specific to code-server. --host 127.0.0.1 binds it to localhost, so it’s only directly accessible from inside of your server. --user-data-dir /var/lib/code-server sets its user data directory, and --auth password specifies that it should authenticate visitors with a password, specified in the PASSWORD environment variable declared on the line above it.

      Remember to replace your_password with your desired password, then save and close the file.

      The next line tells systemd to restart code-server in all malfunction events (for example, when it crashes or the process is killed). The [Install] section orders systemd to start this service when it becomes possible to log in to your server.

      Start the code-server service by running the following command:

      • sudo systemctl start code-server

      Check that it’s started correctly by observing its status:

      • sudo systemctl status code-server

      You’ll see output similar to:

      Output

      ● code-server.service - code-server Loaded: loaded (/lib/systemd/system/code-server.service; disabled; vendor preset: enabled) Active: active (running) since Thu 2019-12-19 16:56:05 UTC; 4s ago Main PID: 1790 (code-server) Tasks: 23 (limit: 1168) Memory: 74.6M CGroup: /system.slice/code-server.service ├─1790 /usr/local/bin/code-server --host 127.0.0.1 --user-data-dir /var/lib/code-server --auth password └─1801 /usr/local/bin/code-server --host 127.0.0.1 --user-data-dir /var/lib/code-server --auth password Dec 19 16:56:05 code-server-debian10 systemd[1]: Started code-server. Dec 19 16:56:06 code-server-debian10 code-server[1790]: info Server listening on http://127.0.0.1:8080 Dec 19 16:56:06 code-server-debian10 code-server[1790]: info - Using custom password for authentication Dec 19 16:56:06 code-server-debian10 code-server[1790]: info - Not serving HTTPS

      To make code-server start automatically after a server reboot, enable its service by running the following command:

      • sudo systemctl enable code-server

      In this step, you’ve downloaded code-server and made it available globally. Then, you’ve created a systemd service for it and enabled it, so code-server will start at every server boot. Next, you’ll expose it at your domain by configuring Nginx to serve as a reverse proxy between the visitor and code-server.

      Step 2 — Exposing code-server at Your Domain

      In this section, you will configure Nginx as a reverse proxy for code-server.

      As you have learned in the Nginx prerequisite step, its site configuration files are stored under /etc/nginx/sites-available and must later be symlinked to /etc/nginx/sites-enabled to become active.

      You’ll store the configuration for exposing code-server at your domain in a file named code-server.conf, under /etc/nginx/sites-available. Start off by creating it using your editor:

      • sudo nano /etc/nginx/sites-available/code-server.conf

      Add the following lines:

      /etc/nginx/sites-available/code-server.conf

      server {
          listen 80;
          listen [::]:80;
      
          server_name code-server.your-domain;
      
          location / {
              proxy_pass http://localhost:8080/;
              proxy_set_header Upgrade $http_upgrade;
              proxy_set_header Connection upgrade;
              proxy_set_header Accept-Encoding gzip;
          }
      }
      

      Replace code-server.your-domain with your desired domain, then save and close the file.

      In this file, you define that Nginx should listen to HTTP port 80. Then, you specify a server_name that tells Nginx for which domain to accept requests and apply this particular configuration. In the next block, for the root location (/), you specify that requests should be passed back and forth to code-server running at localhost:8080. The next three lines (starting with proxy_set_header) order Nginx to carry over some HTTP request headers that are needed for correct functioning of WebSockets, which code-server extensively uses.

      To make this site configuration active, you will need to create a symlink of it in the /etc/nginx/sites-enabled folder by running:

      • sudo ln -s /etc/nginx/sites-available/code-server.conf /etc/nginx/sites-enabled/code-server.conf

      To test the validity of the configuration, run the following command:

      You’ll see the following output:

      Output

      nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful

      For the configuration to take effect, you’ll need to restart Nginx:

      • sudo systemctl restart nginx

      You now have your code-server installation accessible at your domain. In the next step, you’ll secure it by applying a free Let’s Encrypt TLS certificate.

      Step 3 — Securing Your Domain

      In this section, you will secure your domain using a Let’s Encrypt TLS certificate, which you’ll provision using Certbot.

      To install the latest version of Certbot and its Nginx plugin, run the following command:

      • sudo apt install certbot python-certbot-nginx

      As part of the prerequisites, you have enabled ufw (Uncomplicated Firewall) and configured it to allow unencrypted HTTP traffic. To be able to access the secured site, you’ll need to configure it to accept encrypted traffic by running the following command:

      The output will be:

      Output

      Rule added Rule added (v6)

      Similarly to Nginx, you’ll need to reload it for the configuration to take effect:

      The output will show:

      Output

      Firewall reloaded

      Then, in your browser, navigate to the domain you used for code-server. You will see the code-server login prompt.

      code-server login prompt

      code-server is asking you for your password. Enter the one you set in the previous step and press Enter IDE. You’ll now enter code-server and immediately see its editor GUI.

      code-server GUI

      Now that you’ve checked that code-server is correctly exposed at your domain, you’ll install Let’s Encrypt TLS certificates to secure it, using Certbot.

      To request certificates for your domain, run the following command:

      • sudo certbot --nginx -d code-server.your-domain

      In this command, you run certbot to request certificates for your domain—you pass the domain name with the -d parameter. The --nginx flag tells it to automatically change Nginx site configuration to support HTTPS. Remember to replace code-server.your-domain with your domain name.

      If this is your first time running Certbot, you’ll be asked to provide an email address for urgent notices and to accept the EFF’s Terms of Service. Certbot will then request certificates for your domain from Let’s Encrypt. It will then ask you if you’d like to redirect all HTTP traffic to HTTPS:

      Output

      Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1: No redirect - Make no further changes to the webserver configuration. 2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for new sites, or if you're confident your site works on HTTPS. You can undo this change by editing your web server's configuration. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Select the appropriate number [1-2] then [enter] (press 'c' to cancel):

      It is recommended to select the second option in order to maximize security. After you input your selection, press ENTER.

      The output will be similar to this:

      Output

      IMPORTANT NOTES: - Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/code-server.your-domain/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/code-server.your-domain/privkey.pem Your cert will expire on ... To obtain a new or tweaked version of this certificate in the future, simply run certbot again with the "certonly" option. To non-interactively renew *all* of your certificates, run "certbot renew" - Your account credentials have been saved in your Certbot configuration directory at /etc/letsencrypt. You should make a secure backup of this folder now. This configuration directory will also contain certificates and private keys obtained by Certbot so making regular backups of this folder is ideal. - If you like Certbot, please consider supporting our work by: Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate Donating to EFF: https://eff.org/donate-le

      This means that Certbot has successfully generated TLS certificates and applied them to the Nginx configuration for your domain. You can now reload your code-server domain in your browser and observe a padlock to the left of the site address, which means that your connection is properly secured.

      Now that you have code-server accessible at your domain through a secured Nginx reverse proxy, you’re ready to review the user interface of code-server.

      Step 4 — Using the code-server Interface

      In this section, you’ll use some of the features of the code-server interface. Since code-server is Visual Studio Code running in the cloud, it has the same interface as the standalone desktop edition.

      On the left-hand side of the IDE, there is a vertical row of six buttons opening the most commonly used features in a side panel known as the Activity Bar.

      code-server GUI - Sidepanel

      This bar is customizable so you can move these views to a different order or remove them from the bar. By default, the first button opens the general menu in a dropdown, while the second view opens the Explorer panel that provides tree-like navigation of the project’s structure. You can manage your folders and files here—creating, deleting, moving, and renaming them as necessary. The next view provides access to a search and replace functionality.

      Following this, in the default order, is your view of the source control systems, like Git. Visual Studio code also supports other source control providers and you can find further instructions for source control work flows with the editor in this documentation.

      Git pane with context-menu open

      The debugger option on the Activity Bar provides all the common actions for debugging in the panel. Visual Studio Code comes with built-in support for the Node.js runtime debugger and any language that transpiles to Javascript. For other languages you can install extensions for the required debugger. You can save debugging configurations in the launch.json file.

      Debugger View with launch.json open

      The final view in the Activity Bar provides a menu to access available extensions on the Marketplace.

      code-server GUI - Tabs

      The central part of the GUI is your editor, which you can separate by tabs for your code editing. You can change your editing view to a grid system or to side-by-side files.

      Editor Grid View

      After creating a new file through the File menu, an empty file will open in a new tab, and once saved, the file’s name will be viewable in the Explorer side panel. Creating folders can be done by right clicking on the Explorer sidebar and clicking on New Folder. You can expand a folder by clicking on its name as well as dragging and dropping files and folders to upper parts of the hierarchy to move them to a new location.

      code-server GUI - New Folder

      You can gain access to a terminal by entering CTRL+SHIFT+`, or by clicking on Terminal in the upper menu dropdown, and selecting New Terminal. The terminal will open in a lower panel and its working directory will be set to the project’s workspace, which contains the files and folders shown in the Explorer side panel.

      You’ve explored a high-level overview of the code-server interface and reviewed some of the most commonly used features.

      Conclusion

      You now have code-server, a versatile cloud IDE, installed on your Debian 10 server, exposed at your domain and secured using Let’s Encrypt certificates. You can now work on projects individually, as well as in a team-collaboration setting. Running a cloud IDE frees resources on your local machine and allows you to scale the resources when needed. For further information, see the Visual Studio Code documentation for additional features and detailed instructions on other components of code-server.

      If you would like to run code-server on your DigitalOcean Kubernetes cluster check out our tutorial on How To Set Up the code-server Cloud IDE Platform on DigitalOcean Kubernetes.



      Source link