One place for hosting & domains

      How To Set Up Ansible Inventories


      Introduction

      Ansible is a modern configuration management tool that facilitates the task of setting up and maintaining remote servers, with a minimalist design intended to get users up and running quickly. Ansible uses an inventory file to keep track of which hosts are part of your infrastructure, and how to reach them for running commands and playbooks.

      There are multiple ways in which you can set up your Ansible inventory file, depending on your environment and project needs. In this guide, we’ll demonstrate how to create inventory files and organize servers into groups and subgroups, how to set up host variables, and how to use patterns to control the execution of Ansible commands and playbooks per host and per group.

      Prerequisites

      In order follow this guide, you’ll need:

      • One Ansible control node: an Ubuntu 20.04 machine with Ansible installed and configured to connect to your Ansible hosts using SSH keys. Make sure the control node has a regular user with sudo permissions and a firewall enabled, as explained in our Initial Server Setup guide. To set up Ansible, please follow our guide on How to Install and Configure Ansible on Ubuntu 20.04.
      • Two or more Ansible Hosts: two or more remote Ubuntu 20.04 servers.

      Step 1 — Creating a Custom Inventory File

      Upon installation, Ansible creates an inventory file that is typically located at /etc/ansible/hosts. This is the default location used by Ansible when a custom inventory file is not provided with the -i option, during a playbook or command execution.

      Even though you can use this file without problems, using per-project inventory files is a good practice to avoid mixing servers when executing commands and playbooks. Having per-project inventory files will also facilitate sharing your provisioning setup with collaborators, given you include the inventory file within the project’s code repository.

      To get started, access your home folder and create a new directory to hold your Ansible files:

      Move to that directory and open a new inventory file using your text editor of choice. Here, we’ll use nano:

      • cd ansible
      • nano inventory

      A list of your nodes, with one server per line, is enough for setting up a functional inventory file. Hostnames and IP addresses are interchangeable:

      ~/ansible/inventory

      203.0.113.111
      203.0.113.112
      203.0.113.113
      server_hostname
      

      Once you have an inventory file set up, you can use the ansible-inventory command to validate and obtain information about your Ansible inventory:

      • ansible-inventory -i inventory --list

      Output

      { "_meta": { "hostvars": {} }, "all": { "children": [ "ungrouped" ] }, "ungrouped": { "hosts": [ "203.0.113.111", "203.0.113.112", "203.0.113.113", "server_hostname" ] } }

      Even though we haven’t set up any groups within our inventory, the output shows 2 distinct groups that are automatically inferred by Ansible: all and ungrouped. As the name suggests, all is used to refer to all servers from your inventory file, no matter how they are organized. The ungrouped group is used to refer to servers that aren’t listed within a group.

      Running Commands and Playbooks with Custom Inventories

      To run Ansible commands with a custom inventory file, use the -i option as follows:

      • ansible all -i inventory -m ping

      This would execute the ping module on all hosts listed in your custom inventory file.

      Similarly, this is how you execute Ansible playbooks with a custom inventory file:

      • ansible-playbook -i inventory playbook.yml

      Note: For more information on how to connect to nodes and how to run commands and playbooks, please refer to our How to Use Ansible guide.

      So far, we’ve seen how to create a basic inventory and how to use it for running commands and playbooks. In the next step, we’ll see how to organize nodes into groups and subgroups.

      Step 2 — Organizing Servers Into Groups and Subgroups

      Within the inventory file, you can organize your servers into different groups and subgroups. Beyond helping to keep your hosts in order, this practice will enable you to use group variables, a feature that can greatly facilitate managing multiple staging environments with Ansible.

      A host can be part of multiple groups. The following inventory file in INI format demonstrates a setup with four groups: webservers, dbservers, development, and production. You’ll notice that the servers are grouped by two different qualities: their purpose (web and database), and how they’re being used (development and production).

      ~/ansible/inventory

      [webservers]
      203.0.113.111
      203.0.113.112
      
      [dbservers]
      203.0.113.113
      server_hostname
      
      [development]
      203.0.113.111
      203.0.113.113
      
      [production]
      203.0.113.112
      server_hostname
      

      If you were to run the ansible-inventory command again with this inventory file, you would see the following arrangement:

      Output

      { "_meta": { "hostvars": {} }, "all": { "children": [ "dbservers", "development", "production", "ungrouped", "webservers" ] }, "dbservers": { "hosts": [ "203.0.113.113", "server_hostname" ] }, "development": { "hosts": [ "203.0.113.111", "203.0.113.113" ] }, "production": { "hosts": [ "203.0.113.112", "server_hostname" ] }, "webservers": { "hosts": [ "203.0.113.111", "203.0.113.112" ] } }

      It is also possible to aggregate multiple groups as children under a “parent” group. The “parent” is then called a metagroup. The following example demonstrates another way to organize the previous inventory using metagroups to achieve a comparable, yet more granular arrangement:

      ~/ansible/inventory

      [web_dev]
      203.0.113.111
      
      [web_prod]
      203.0.113.112
      
      [db_dev]
      203.0.113.113
      
      [db_prod]
      server_hostname
      
      [webservers:children]
      web_dev
      web_prod
      
      [dbservers:children]
      db_dev
      db_prod
      
      [development:children]
      web_dev
      db_dev
      
      [production:children]
      web_prod
      db_prod
      

      The more servers you have, the more it makes sense to break groups down or create alternative arrangements so that you can target smaller groups of servers as needed.

      Step 3 — Setting Up Host Aliases

      You can use aliases to name servers in a way that facilitates referencing those servers later, when running commands and playbooks.

      To use an alias, include a variable named ansible_host after the alias name, containing the corresponding IP address or hostname of the server that should respond to that alias:

      ~/ansible/inventory

      server1 ansible_host=203.0.113.111
      server2 ansible_host=203.0.113.112
      server3 ansible_host=203.0.113.113
      server4 ansible_host=server_hostname
      

      If you were to run the ansible-inventory command with this inventory file, you would see output similar to this:

      Output

      { "_meta": { "hostvars": { "server1": { "ansible_host": "203.0.113.111" }, "server2": { "ansible_host": "203.0.113.112" }, "server3": { "ansible_host": "203.0.113.113" }, "server4": { "ansible_host": "server_hostname" } } }, "all": { "children": [ "ungrouped" ] }, "ungrouped": { "hosts": [ "server1", "server2", "server3", "server4" ] } }

      Notice how the servers are now referenced by their aliases instead of their IP addresses or hostnames. This makes it easier for targeting individual servers when running commands and playbooks.

      Step 4 — Setting Up Host Variables

      It is possible to use the inventory file to set up variables that will change Ansible’s default behavior when connecting and executing commands on your nodes. This is in fact what we did in the previous step, when setting up host aliases. The ansible_host variable tells Ansible where to find the remote nodes, in case an alias is used to refer to that server.

      Inventory variables can be set per host or per group. In addition to customizing Ansible’s default settings, these variables are also accessible from your playbooks, which enables further customization for individual hosts and groups.

      The following example shows how to define the default remote user when connecting to each of the nodes listed in this inventory file:

      ~/ansible/inventory

      server1 ansible_host=203.0.113.111 ansible_user=sammy
      server2 ansible_host=203.0.113.112 ansible_user=sammy
      server3 ansible_host=203.0.113.113 ansible_user=myuser
      server4 ansible_host=server_hostname ansible_user=myuser
      

      You could also create a group to aggregate the hosts with similar settings, and then set up their variables at the group level:

      ~/ansible/inventory

      [group_a]
      server1 ansible_host=203.0.113.111 
      server2 ansible_host=203.0.113.112
      
      [group_b]
      server3 ansible_host=203.0.113.113 
      server4 ansible_host=server_hostname
      
      [group_a:vars]
      ansible_user=sammy
      
      [group_b:vars]
      ansible_user=myuser
      

      This inventory arrangement would generate the following output with ansible-inventory:

      Output

      { "_meta": { "hostvars": { "server1": { "ansible_host": "203.0.113.111", "ansible_user": "sammy" }, "server2": { "ansible_host": "203.0.113.112", "ansible_user": "sammy" }, "server3": { "ansible_host": "203.0.113.113", "ansible_user": "myuser" }, "server4": { "ansible_host": "server_hostname", "ansible_user": "myuser" } } }, "all": { "children": [ "group_a", "group_b", "ungrouped" ] }, "group_a": { "hosts": [ "server1", "server2" ] }, "group_b": { "hosts": [ "server3", "server4" ] } }

      Notice that all inventory variables are listed within the _meta node in the JSON output produced by ansible-inventory.

      Step 5 — Using Patterns to Target Execution of Commands and Playbooks

      When executing commands and playbooks with Ansible, you must provide a target. Patterns allow you to target specific hosts, groups, or subgroups in your inventory file. They’re very flexible, supporting regular expressions and wildcards.

      Consider the following inventory file:

      ~/ansible/inventory

      [webservers]
      203.0.113.111
      203.0.113.112
      
      [dbservers]
      203.0.113.113
      server_hostname
      
      [development]
      203.0.113.111
      203.0.113.113
      
      [production]
      203.0.113.112
      server_hostname
      

      Now imagine you need to execute a command targeting only the database server(s) that are running on production. In this example, there’s only server_hostname matching that criteria; however, it could be the case that you have a large group of database servers in that group. Instead of individually targeting each server, you could use the following pattern:

      • ansible dbservers:&production -m ping

      This pattern would target only servers that are present both in the dbservers as well as in the production groups. If you wanted to do the opposite, targeting only servers that are present in the dbservers but not in the production group, you would use the following pattern instead:

      • ansible dbservers:!production -m ping

      The following table contains a few different examples of common patterns you can use when running commands and playbooks with Ansible:

      Pattern Result Target
      all All Hosts from your inventory file
      host1 A single host (host1)
      host1:host2 Both host1 and host2
      group1 A single group (group1)
      group1:group2 All servers in group1 and group2
      group1:&group2 Only servers that are both in group1 and group2
      group1:!group2 Servers in group1 except those also in group2

      For more advanced pattern options, such as using positional patterns and regex to define targets, please refer to the official Ansible documentation on patterns.

      Conclusion

      In this guide, we had a detailed look into Ansible inventories. We’ve seen how to organize nodes into groups and subgroups, how to set up inventory variables, and how to use patterns to target different groups of servers when running commands and playbooks.

      In the next part of this series, we’ll see how to manage multiple servers with Ansible ad-hoc commands.



      Source link

      How To Set Up a Node.js Application for Production on Ubuntu 20.04


      Not using Ubuntu 20.04?


      Choose a different version or distribution.

      Introduction

      Node.js is an open-source JavaScript runtime environment for building server-side and networking applications. The platform runs on Linux, macOS, FreeBSD, and Windows. Though you can run Node.js applications at the command line, this tutorial will focus on running them as a service. This means that they will restart on reboot or failure and are safe for use in a production environment.

      In this tutorial, you will set up a production-ready Node.js environment on a single Ubuntu 20.04 server. This server will run a Node.js application managed by PM2, and provide users with secure access to the application through an Nginx reverse proxy. The Nginx server will offer HTTPS using a free certificate provided by Let’s Encrypt.

      Prerequisites

      This guide assumes that you have the following:

      When you’ve completed the prerequisites, you will have a server serving your domain’s default placeholder page at https://example.com/.

      Step 1 — Installing Node.js

      Let’s begin by installing the latest LTS release of Node.js, using the NodeSource package archives.

      First, install the NodeSource PPA in order to get access to its contents. Make sure you’re in your home directory, and use curl to retrieve the installation script for the most recent LTS version of Node.js from its archives.

      • cd ~
      • curl -sL https://deb.nodesource.com/setup_14.x -o nodesource_setup.sh

      You can inspect the contents of this script with nano or your preferred text editor:

      When you’re done inspecting the script, run it under sudo:

      • sudo bash nodesource_setup.sh

      The PPA will be added to your configuration and your local package cache will be updated automatically. After running the setup script from Nodesource, you can install the Node.js package:

      To check which version of Node.js you have installed after these initial steps, type:

      Output

      v14.4.0

      Note: When installing from the NodeSource PPA, the Node.js executable is called nodejs, rather than node.

      The nodejs package contains the nodejs binary as well as npm, a package manager for Node modules, so you don’t need to install npm separately.

      npm uses a configuration file in your home directory to keep track of updates. It will be created the first time you run npm. Execute this command to verify that npm is installed and to create the configuration file:

      Output

      6.14.5

      In order for some npm packages to work (those that require compiling code from source, for example), you will need to install the build-essential package:

      • sudo apt install build-essential

      You now have the necessary tools to work with npm packages that require compiling code from source.

      With the Node.js runtime installed, let’s move on to writing a Node.js application.

      Step 2 — Creating a Node.js Application

      Let’s write a Hello World application that returns “Hello World” to any HTTP requests. This sample application will help you get Node.js set up. You can replace it with your own application — just make sure that you modify your application to listen on the appropriate IP addresses and ports.

      First, let’s create a sample application called hello.js:

      Insert the following code into the file:

      ~/hello.js

      const http = require('http');
      
      const hostname="localhost";
      const port = 3000;
      
      const server = http.createServer((req, res) => {
        res.statusCode = 200;
        res.setHeader('Content-Type', 'text/plain');
        res.end('Hello World!n');
      });
      
      server.listen(port, hostname, () => {
        console.log(`Server running at http://${hostname}:${port}/`);
      });
      

      Save the file and exit the editor.

      This Node.js application listens on the specified address (localhost) and port (3000), and returns “Hello World!” with a 200 HTTP success code. Since we’re listening on localhost, remote clients won’t be able to connect to our application.

      To test your application, type:

      You will receive the following output:

      Output

      Server running at http://localhost:3000/

      Note: Running a Node.js application in this manner will block additional commands until the application is killed by pressing CTRL+C.

      To test the application, open another terminal session on your server, and connect to localhost with curl:

      • curl http://localhost:3000

      If you get the following output, the application is working properly and listening on the correct address and port:

      Output

      Hello World!

      If you do not get the expected output, make sure that your Node.js application is running and configured to listen on the proper address and port.

      Once you’re sure it’s working, kill the application (if you haven’t already) by pressing CTRL+C.

      Step 3 — Installing PM2

      Next let’s install PM2, a process manager for Node.js applications. PM2 makes it possible to daemonize applications so that they will run in the background as a service.

      Use npm to install the latest version of PM2 on your server:

      • sudo npm install pm2@latest -g

      The -g option tells npm to install the module globally, so that it’s available system-wide.

      Let’s first use the pm2 start command to run your application, hello.js, in the background:

      This also adds your application to PM2’s process list, which is outputted every time you start an application:

      Output

      ... [PM2] Spawning PM2 daemon with pm2_home=/home/sammy/.pm2 [PM2] PM2 Successfully daemonized [PM2] Starting /home/sammy/hello.js in fork_mode (1 instance) [PM2] Done. ┌────┬────────────────────┬──────────┬──────┬───────────┬──────────┬──────────┐ │ id │ name │ mode │ ↺ │ status │ cpu │ memory │ ├────┼────────────────────┼──────────┼──────┼───────────┼──────────┼──────────┤ │ 0 │ hello │ fork │ 0 │ online │ 0% │ 25.2mb │ └────┴────────────────────┴──────────┴──────┴───────────┴──────────┴──────────┘

      As indicated above, PM2 automatically assigns an App name (based on the filename, without the .js extension) and a PM2 id. PM2 also maintains other information, such as the PID of the process, its current status, and memory usage.

      Applications that are running under PM2 will be restarted automatically if the application crashes or is killed, but we can take an additional step to get the application to launch on system startup using the startup subcommand. This subcommand generates and configures a startup script to launch PM2 and its managed processes on server boots:

      The last line of the resulting output will include a command to run with superuser privileges in order to set PM2 to start on boot:

      Output

      [PM2] Init System found: systemd sammy [PM2] To setup the Startup Script, copy/paste the following command: sudo env PATH=$PATH:/usr/bin /usr/lib/node_modules/pm2/bin/pm2 startup systemd -u sammy --hp /home/sammy

      Run the command from the output, with your username in place of sammy:

      • sudo env PATH=$PATH:/usr/bin /usr/lib/node_modules/pm2/bin/pm2 startup systemd -u sammy --hp /home/sammy

      As an additional step, we can save the PM2 process list and corresponding environments:

      You have now created a systemd unit that runs pm2 for your user on boot. This pm2 instance, in turn, runs hello.js.

      Start the service with systemctl:

      • sudo systemctl start pm2-sammy

      If at this point you encounter an error, you may need to reboot, which you can achieve with sudo reboot.

      Check the status of the systemd unit:

      • systemctl status pm2-sammy

      For a detailed overview of systemd, please review Systemd Essentials: Working with Services, Units, and the Journal.

      In addition to those we have covered, PM2 provides many subcommands that allow you to manage or look up information about your applications.

      Stop an application with this command (specify the PM2 App name or id):

      Restart an application:

      • pm2 restart app_name_or_id

      List the applications currently managed by PM2:

      Get information about a specific application using its App name:

      The PM2 process monitor can be pulled up with the monit subcommand. This displays the application status, CPU, and memory usage:

      Note that running pm2 without any arguments will also display a help page with example usage.

      Now that your Node.js application is running and managed by PM2, let’s set up the reverse proxy.

      Step 4 — Setting Up Nginx as a Reverse Proxy Server

      Your application is running and listening on localhost, but you need to set up a way for your users to access it. We will set up the Nginx web server as a reverse proxy for this purpose.

      In the prerequisite tutorial, you set up your Nginx configuration in the /etc/nginx/sites-available/example.com file. Open this file for editing:

      • sudo nano /etc/nginx/sites-available/example.com

      Within the server block, you should have an existing location / block. Replace the contents of that block with the following configuration. If your application is set to listen on a different port, update the highlighted portion to the correct port number:

      /etc/nginx/sites-available/example.com

      server {
      ...
          location / {
              proxy_pass http://localhost:3000;
              proxy_http_version 1.1;
              proxy_set_header Upgrade $http_upgrade;
              proxy_set_header Connection 'upgrade';
              proxy_set_header Host $host;
              proxy_cache_bypass $http_upgrade;
          }
      ...
      }
      

      This configures the server to respond to requests at its root. Assuming our server is available at example.com, accessing https://example.com/ via a web browser would send the request to hello.js, listening on port 3000 at localhost.

      You can add additional location blocks to the same server block to provide access to other applications on the same server. For example, if you were also running another Node.js application on port 3001, you could add this location block to allow access to it via https://example.com/app2:

      /etc/nginx/sites-available/example.com — Optional

      server {
      ...
          location /app2 {
              proxy_pass http://localhost:3001;
              proxy_http_version 1.1;
              proxy_set_header Upgrade $http_upgrade;
              proxy_set_header Connection 'upgrade';
              proxy_set_header Host $host;
              proxy_cache_bypass $http_upgrade;
          }
      ...
      }
      

      Once you are done adding the location blocks for your applications, save the file and exit your editor.

      Make sure you didn’t introduce any syntax errors by typing:

      Restart Nginx:

      • sudo systemctl restart nginx

      Assuming that your Node.js application is running, and your application and Nginx configurations are correct, you should now be able to access your application via the Nginx reverse proxy. Try it out by accessing your server’s URL (its public IP address or domain name).

      Conclusion

      Congratulations! You now have your Node.js application running behind an Nginx reverse proxy on an Ubuntu 20.04 server. This reverse proxy setup is flexible enough to provide your users access to other applications or static web content that you want to share.



      Source link

      How To Set Up Django with Postgres, Nginx, and Gunicorn on Ubuntu 20.04


      Not using Ubuntu 20.04?


      Choose a different version or distribution.

      Introduction

      Django is a powerful web framework that can help you get your Python application or website off the ground. Django includes a simplified development server for testing your code locally, but for anything even slightly production related, a more secure and powerful web server is required.

      In this guide, we will demonstrate how to install and configure some components on Ubuntu 20.04 to support and serve Django applications. We will be setting up a PostgreSQL database instead of using the default SQLite database. We will configure the Gunicorn application server to interface with our applications. We will then set up Nginx to reverse proxy to Gunicorn, giving us access to its security and performance features to serve our apps.

      Prerequisites and Goals

      In order to complete this guide, you should have a fresh Ubuntu 20.04 server instance with a basic firewall and a non-root user with sudo privileges configured. You can learn how to set this up by running through our initial server setup guide.

      We will be installing Django within a virtual environment. Installing Django into an environment specific to your project will allow your projects and their requirements to be handled separately.

      Once we have our database and application up and running, we will install and configure the Gunicorn application server. This will serve as an interface to our application, translating client requests from HTTP to Python calls that our application can process. We will then set up Nginx in front of Gunicorn to take advantage of its high performance connection handling mechanisms and its easy-to-implement security features.

      Let’s get started.

      Installing the Packages from the Ubuntu Repositories

      To begin the process, we’ll download and install all of the items we need from the Ubuntu repositories. We will use the Python package manager pip to install additional components a bit later.

      We need to update the local apt package index and then download and install the packages. The packages we install depend on which version of Python your project will use.

      If you are using Django with Python 3, type:

      • sudo apt update
      • sudo apt install python3-pip python3-dev libpq-dev postgresql postgresql-contrib nginx curl

      Django 1.11 is the last release of Django that will support Python 2. If you are starting new projects, it is strongly recommended that you choose Python 3. If you still need to use Python 2, type:

      • sudo apt update
      • sudo apt install python-pip python-dev libpq-dev postgresql postgresql-contrib nginx curl

      This will install pip, the Python development files needed to build Gunicorn later, the Postgres database system and the libraries needed to interact with it, and the Nginx web server.

      Creating the PostgreSQL Database and User

      We’re going to jump right in and create a database and database user for our Django application.

      By default, Postgres uses an authentication scheme called “peer authentication” for local connections. Basically, this means that if the user’s operating system username matches a valid Postgres username, that user can login with no further authentication.

      During the Postgres installation, an operating system user named postgres was created to correspond to the postgres PostgreSQL administrative user. We need to use this user to perform administrative tasks. We can use sudo and pass in the username with the -u option.

      Log into an interactive Postgres session by typing:

      You will be given a PostgreSQL prompt where we can set up our requirements.

      First, create a database for your project:

      • CREATE DATABASE myproject;

      Note: Every Postgres statement must end with a semi-colon, so make sure that your command ends with one if you are experiencing issues.

      Next, create a database user for our project. Make sure to select a secure password:

      • CREATE USER myprojectuser WITH PASSWORD 'password';

      Afterwards, we’ll modify a few of the connection parameters for the user we just created. This will speed up database operations so that the correct values do not have to be queried and set each time a connection is established.

      We are setting the default encoding to UTF-8, which Django expects. We are also setting the default transaction isolation scheme to “read committed”, which blocks reads from uncommitted transactions. Lastly, we are setting the timezone. By default, our Django projects will be set to use UTC. These are all recommendations from the Django project itself:

      • ALTER ROLE myprojectuser SET client_encoding TO 'utf8';
      • ALTER ROLE myprojectuser SET default_transaction_isolation TO 'read committed';
      • ALTER ROLE myprojectuser SET timezone TO 'UTC';

      Now, we can give our new user access to administer our new database:

      • GRANT ALL PRIVILEGES ON DATABASE myproject TO myprojectuser;

      When you are finished, exit out of the PostgreSQL prompt by typing:

      Postgres is now set up so that Django can connect to and manage its database information.

      Creating a Python Virtual Environment for your Project

      Now that we have our database, we can begin getting the rest of our project requirements ready. We will be installing our Python requirements within a virtual environment for easier management.

      To do this, we first need access to the virtualenv command. We can install this with pip.

      If you are using Python 3, upgrade pip and install the package by typing:

      • sudo -H pip3 install --upgrade pip
      • sudo -H pip3 install virtualenv

      If you are using Python 2, upgrade pip and install the package by typing:

      • sudo -H pip install --upgrade pip
      • sudo -H pip install virtualenv

      With virtualenv installed, we can start forming our project. Create and move into a directory where we can keep our project files:

      • mkdir ~/myprojectdir
      • cd ~/myprojectdir

      Within the project directory, create a Python virtual environment by typing:

      This will create a directory called myprojectenv within your myprojectdir directory. Inside, it will install a local version of Python and a local version of pip. We can use this to install and configure an isolated Python environment for our project.

      Before we install our project’s Python requirements, we need to activate the virtual environment. You can do that by typing:

      • source myprojectenv/bin/activate

      Your prompt should change to indicate that you are now operating within a Python virtual environment. It will look something like this: (myprojectenv)user@host:~/myprojectdir$.

      With your virtual environment active, install Django, Gunicorn, and the psycopg2 PostgreSQL adaptor with the local instance of pip:

      Note: When the virtual environment is activated (when your prompt has (myprojectenv) preceding it), use pip instead of pip3, even if you are using Python 3. The virtual environment’s copy of the tool is always named pip, regardless of the Python version.

      • pip install django gunicorn psycopg2-binary

      You should now have all of the software needed to start a Django project.

      Creating and Configuring a New Django Project

      With our Python components installed, we can create the actual Django project files.

      Creating the Django Project

      Since we already have a project directory, we will tell Django to install the files here. It will create a second level directory with the actual code, which is normal, and place a management script in this directory. The key to this is that we are defining the directory explicitly instead of allowing Django to make decisions relative to our current directory:

      • django-admin.py startproject myproject ~/myprojectdir

      At this point, your project directory (~/myprojectdir in our case) should have the following content:

      • ~/myprojectdir/manage.py: A Django project management script.
      • ~/myprojectdir/myproject/: The Django project package. This should contain the __init__.py, settings.py, urls.py, asgi.py, and wsgi.py files.
      • ~/myprojectdir/myprojectenv/: The virtual environment directory we created earlier.

      Adjusting the Project Settings

      The first thing we should do with our newly created project files is adjust the settings. Open the settings file in your text editor:

      • nano ~/myprojectdir/myproject/settings.py

      Start by locating the ALLOWED_HOSTS directive. This defines a list of the server’s addresses or domain names may be used to connect to the Django instance. Any incoming requests with a Host header that is not in this list will raise an exception. Django requires that you set this to prevent a certain class of security vulnerability.

      In the square brackets, list the IP addresses or domain names that are associated with your Django server. Each item should be listed in quotations with entries separated by a comma. If you wish requests for an entire domain and any subdomains, prepend a period to the beginning of the entry. In the snippet below, there are a few commented out examples used to demonstrate:

      Note: Be sure to include localhost as one of the options since we will be proxying connections through a local Nginx instance.

      ~/myprojectdir/myproject/settings.py

      . . .
      # The simplest case: just add the domain name(s) and IP addresses of your Django server
      # ALLOWED_HOSTS = [ 'example.com', '203.0.113.5']
      # To respond to 'example.com' and any subdomains, start the domain with a dot
      # ALLOWED_HOSTS = ['.example.com', '203.0.113.5']
      ALLOWED_HOSTS = ['your_server_domain_or_IP', 'second_domain_or_IP', . . ., 'localhost']
      

      Next, find the section that configures database access. It will start with DATABASES. The configuration in the file is for a SQLite database. We already created a PostgreSQL database for our project, so we need to adjust the settings.

      Change the settings with your PostgreSQL database information. We tell Django to use the psycopg2 adaptor we installed with pip. We need to give the database name, the database username, the database user’s password, and then specify that the database is located on the local computer. You can leave the PORT setting as an empty string:

      ~/myprojectdir/myproject/settings.py

      . . .
      
      DATABASES = {
          'default': {
              'ENGINE': 'django.db.backends.postgresql_psycopg2',
              'NAME': 'myproject',
              'USER': 'myprojectuser',
              'PASSWORD': 'password',
              'HOST': 'localhost',
              'PORT': '',
          }
      }
      
      . . .
      

      Next, move down to the bottom of the file and add a setting indicating where the static files should be placed. This is necessary so that Nginx can handle requests for these items. The following line tells Django to place them in a directory called static in the base project directory:

      ~/myprojectdir/myproject/settings.py

      . . .
      
      STATIC_URL = '/static/'
      STATIC_ROOT = os.path.join(BASE_DIR, 'static/')
      

      Save and close the file when you are finished.

      Completing Initial Project Setup

      Now, we can migrate the initial database schema to our PostgreSQL database using the management script:

      • ~/myprojectdir/manage.py makemigrations
      • ~/myprojectdir/manage.py migrate

      Create an administrative user for the project by typing:

      • ~/myprojectdir/manage.py createsuperuser

      You will have to select a username, provide an email address, and choose and confirm a password.

      We can collect all of the static content into the directory location we configured by typing:

      • ~/myprojectdir/manage.py collectstatic

      You will have to confirm the operation. The static files will then be placed in a directory called static within your project directory.

      If you followed the initial server setup guide, you should have a UFW firewall protecting your server. In order to test the development server, we’ll have to allow access to the port we’ll be using.

      Create an exception for port 8000 by typing:

      Finally, you can test our your project by starting up the Django development server with this command:

      • ~/myprojectdir/manage.py runserver 0.0.0.0:8000

      In your web browser, visit your server’s domain name or IP address followed by :8000:

      http://server_domain_or_IP:8000
      

      You should receive the default Django index page:

      Django index page

      If you append /admin to the end of the URL in the address bar, you will be prompted for the administrative username and password you created with the createsuperuser command:

      Django admin login

      After authenticating, you can access the default Django admin interface:

      Django admin interface

      When you are finished exploring, hit CTRL-C in the terminal window to shut down the development server.

      Testing Gunicorn’s Ability to Serve the Project

      The last thing we want to do before leaving our virtual environment is test Gunicorn to make sure that it can serve the application. We can do this by entering our project directory and using gunicorn to load the project’s WSGI module:

      • cd ~/myprojectdir
      • gunicorn --bind 0.0.0.0:8000 myproject.wsgi

      This will start Gunicorn on the same interface that the Django development server was running on. You can go back and test the app again.

      Note: The admin interface will not have any of the styling applied since Gunicorn does not know how to find the static CSS content responsible for this.

      We passed Gunicorn a module by specifying the relative directory path to Django’s wsgi.py file, which is the entry point to our application, using Python’s module syntax. Inside of this file, a function called application is defined, which is used to communicate with the application. To learn more about the WSGI specification, click here.

      When you are finished testing, hit CTRL-C in the terminal window to stop Gunicorn.

      We’re now finished configuring our Django application. We can back out of our virtual environment by typing:

      The virtual environment indicator in your prompt will be removed.

      Creating systemd Socket and Service Files for Gunicorn

      We have tested that Gunicorn can interact with our Django application, but we should implement a more robust way of starting and stopping the application server. To accomplish this, we’ll make systemd service and socket files.

      The Gunicorn socket will be created at boot and will listen for connections. When a connection occurs, systemd will automatically start the Gunicorn process to handle the connection.

      Start by creating and opening a systemd socket file for Gunicorn with sudo privileges:

      • sudo nano /etc/systemd/system/gunicorn.socket

      Inside, we will create a [Unit] section to describe the socket, a [Socket] section to define the socket location, and an [Install] section to make sure the socket is created at the right time:

      /etc/systemd/system/gunicorn.socket

      [Unit]
      Description=gunicorn socket
      
      [Socket]
      ListenStream=/run/gunicorn.sock
      
      [Install]
      WantedBy=sockets.target
      

      Save and close the file when you are finished.

      Next, create and open a systemd service file for Gunicorn with sudo privileges in your text editor. The service filename should match the socket filename with the exception of the extension:

      • sudo nano /etc/systemd/system/gunicorn.service

      Start with the [Unit] section, which is used to specify metadata and dependencies. We’ll put a description of our service here and tell the init system to only start this after the networking target has been reached. Because our service relies on the socket from the socket file, we need to include a Requires directive to indicate that relationship:

      /etc/systemd/system/gunicorn.service

      [Unit]
      Description=gunicorn daemon
      Requires=gunicorn.socket
      After=network.target
      

      Next, we’ll open up the [Service] section. We’ll specify the user and group that we want to process to run under. We will give our regular user account ownership of the process since it owns all of the relevant files. We’ll give group ownership to the www-data group so that Nginx can communicate easily with Gunicorn.

      We’ll then map out the working directory and specify the command to use to start the service. In this case, we’ll have to specify the full path to the Gunicorn executable, which is installed within our virtual environment. We will bind the process to the Unix socket we created within the /run directory so that the process can communicate with Nginx. We log all data to standard output so that the journald process can collect the Gunicorn logs. We can also specify any optional Gunicorn tweaks here. For example, we specified 3 worker processes in this case:

      /etc/systemd/system/gunicorn.service

      [Unit]
      Description=gunicorn daemon
      Requires=gunicorn.socket
      After=network.target
      
      [Service]
      User=sammy
      Group=www-data
      WorkingDirectory=/home/sammy/myprojectdir
      ExecStart=/home/sammy/myprojectdir/myprojectenv/bin/gunicorn 
                --access-logfile - 
                --workers 3 
                --bind unix:/run/gunicorn.sock 
                myproject.wsgi:application
      

      Finally, we’ll add an [Install] section. This will tell systemd what to link this service to if we enable it to start at boot. We want this service to start when the regular multi-user system is up and running:

      /etc/systemd/system/gunicorn.service

      [Unit]
      Description=gunicorn daemon
      Requires=gunicorn.socket
      After=network.target
      
      [Service]
      User=sammy
      Group=www-data
      WorkingDirectory=/home/sammy/myprojectdir
      ExecStart=/home/sammy/myprojectdir/myprojectenv/bin/gunicorn 
                --access-logfile - 
                --workers 3 
                --bind unix:/run/gunicorn.sock 
                myproject.wsgi:application
      
      [Install]
      WantedBy=multi-user.target
      

      With that, our systemd service file is complete. Save and close it now.

      We can now start and enable the Gunicorn socket. This will create the socket file at /run/gunicorn.sock now and at boot. When a connection is made to that socket, systemd will automatically start the gunicorn.service to handle it:

      • sudo systemctl start gunicorn.socket
      • sudo systemctl enable gunicorn.socket

      We can confirm that the operation was successful by checking for the socket file.

      Checking for the Gunicorn Socket File

      Check the status of the process to find out whether it was able to start:

      • sudo systemctl status gunicorn.socket

      You should receive an output like this:

      Output

      ● gunicorn.socket - gunicorn socket Loaded: loaded (/etc/systemd/system/gunicorn.socket; enabled; vendor prese> Active: active (listening) since Fri 2020-06-26 17:53:10 UTC; 14s ago Triggers: ● gunicorn.service Listen: /run/gunicorn.sock (Stream) Tasks: 0 (limit: 1137) Memory: 0B CGroup: /system.slice/gunicorn.socket

      Next, check for the existence of the gunicorn.sock file within the /run directory:

      Output

      /run/gunicorn.sock: socket

      If the systemctl status command indicated that an error occurred or if you do not find the gunicorn.sock file in the directory, it’s an indication that the Gunicorn socket was not able to be created correctly. Check the Gunicorn socket’s logs by typing:

      • sudo journalctl -u gunicorn.socket

      Take another look at your /etc/systemd/system/gunicorn.socket file to fix any problems before continuing.

      Testing Socket Activation

      Currently, if you’ve only started the gunicorn.socket unit, the gunicorn.service will not be active yet since the socket has not yet received any connections. You can check this by typing:

      • sudo systemctl status gunicorn

      Output

      ● gunicorn.service - gunicorn daemon Loaded: loaded (/etc/systemd/system/gunicorn.service; disabled; vendor preset: enabled) Active: inactive (dead)

      To test the socket activation mechanism, we can send a connection to the socket through curl by typing:

      • curl --unix-socket /run/gunicorn.sock localhost

      You should receive the HTML output from your application in the terminal. This indicates that Gunicorn was started and was able to serve your Django application. You can verify that the Gunicorn service is running by typing:

      • sudo systemctl status gunicorn

      Output

      ● gunicorn.service - gunicorn daemon Loaded: loaded (/etc/systemd/system/gunicorn.service; disabled; vendor preset: enabled) Active: active (running) since Fri 2020-06-26 18:52:21 UTC; 2s ago TriggeredBy: ● gunicorn.socket Main PID: 22914 (gunicorn) Tasks: 4 (limit: 1137) Memory: 89.1M CGroup: /system.slice/gunicorn.service ├─22914 /home/sammy/myprojectdir/myprojectenv/bin/python /home/sammy/myprojectdir/myprojectenv/bin/gunicorn --access-logfile - --workers 3 --bind unix:/run/gunico> ├─22927 /home/sammy/myprojectdir/myprojectenv/bin/python /home/sammy/myprojectdir/myprojectenv/bin/gunicorn --access-logfile - --workers 3 --bind unix:/run/gunico> ├─22928 /home/sammy/myprojectdir/myprojectenv/bin/python /home/sammy/myprojectdir/myprojectenv/bin/gunicorn --access-logfile - --workers 3 --bind unix:/run/gunico> └─22929 /home/sammy/myprojectdir/myprojectenv/bin/python /home/sammy/myprojectdir/myprojectenv/bin/gunicorn --access-logfile - --workers 3 --bind unix:/run/gunico> Jun 26 18:52:21 django-tutorial systemd[1]: Started gunicorn daemon. Jun 26 18:52:21 django-tutorial gunicorn[22914]: [2020-06-26 18:52:21 +0000] [22914] [INFO] Starting gunicorn 20.0.4 Jun 26 18:52:21 django-tutorial gunicorn[22914]: [2020-06-26 18:52:21 +0000] [22914] [INFO] Listening at: unix:/run/gunicorn.sock (22914) Jun 26 18:52:21 django-tutorial gunicorn[22914]: [2020-06-26 18:52:21 +0000] [22914] [INFO] Using worker: sync Jun 26 18:52:21 django-tutorial gunicorn[22927]: [2020-06-26 18:52:21 +0000] [22927] [INFO] Booting worker with pid: 22927 Jun 26 18:52:21 django-tutorial gunicorn[22928]: [2020-06-26 18:52:21 +0000] [22928] [INFO] Booting worker with pid: 22928 Jun 26 18:52:21 django-tutorial gunicorn[22929]: [2020-06-26 18:52:21 +0000] [22929] [INFO] Booting worker with pid: 22929

      If the output from curl or the output of systemctl status indicates that a problem occurred, check the logs for additional details:

      • sudo journalctl -u gunicorn

      Check your /etc/systemd/system/gunicorn.service file for problems. If you make changes to the /etc/systemd/system/gunicorn.service file, reload the daemon to reread the service definition and restart the Gunicorn process by typing:

      • sudo systemctl daemon-reload
      • sudo systemctl restart gunicorn

      Make sure you troubleshoot the above issues before continuing.

      Configure Nginx to Proxy Pass to Gunicorn

      Now that Gunicorn is set up, we need to configure Nginx to pass traffic to the process.

      Start by creating and opening a new server block in Nginx’s sites-available directory:

      • sudo nano /etc/nginx/sites-available/myproject

      Inside, open up a new server block. We will start by specifying that this block should listen on the normal port 80 and that it should respond to our server’s domain name or IP address:

      /etc/nginx/sites-available/myproject

      server {
          listen 80;
          server_name server_domain_or_IP;
      }
      

      Next, we will tell Nginx to ignore any problems with finding a favicon. We will also tell it where to find the static assets that we collected in our ~/myprojectdir/static directory. All of these files have a standard URI prefix of “/static”, so we can create a location block to match those requests:

      /etc/nginx/sites-available/myproject

      server {
          listen 80;
          server_name server_domain_or_IP;
      
          location = /favicon.ico { access_log off; log_not_found off; }
          location /static/ {
              root /home/sammy/myprojectdir;
          }
      }
      

      Finally, we’ll create a location / {} block to match all other requests. Inside of this location, we’ll include the standard proxy_params file included with the Nginx installation and then we will pass the traffic directly to the Gunicorn socket:

      /etc/nginx/sites-available/myproject

      server {
          listen 80;
          server_name server_domain_or_IP;
      
          location = /favicon.ico { access_log off; log_not_found off; }
          location /static/ {
              root /home/sammy/myprojectdir;
          }
      
          location / {
              include proxy_params;
              proxy_pass http://unix:/run/gunicorn.sock;
          }
      }
      

      Save and close the file when you are finished. Now, we can enable the file by linking it to the sites-enabled directory:

      • sudo ln -s /etc/nginx/sites-available/myproject /etc/nginx/sites-enabled

      Test your Nginx configuration for syntax errors by typing:

      If no errors are reported, go ahead and restart Nginx by typing:

      • sudo systemctl restart nginx

      Finally, we need to open up our firewall to normal traffic on port 80. Since we no longer need access to the development server, we can remove the rule to open port 8000 as well:

      • sudo ufw delete allow 8000
      • sudo ufw allow 'Nginx Full'

      You should now be able to go to your server’s domain or IP address to view your application.

      Note: After configuring Nginx, the next step should be securing traffic to the server using SSL/TLS. This is important because without it, all information, including passwords are sent over the network in plain text.

      If you have a domain name, the easiest way to get an SSL certificate to secure your traffic is using Let’s Encrypt. Follow this guide to set up Let’s Encrypt with Nginx on Ubuntu 20.04. Follow the procedure using the Nginx server block we created in this guide.

      Troubleshooting Nginx and Gunicorn

      If this last step does not show your application, you will need to troubleshoot your installation.

      Nginx Is Showing the Default Page Instead of the Django Application

      If Nginx displays the default page instead of proxying to your application, it usually means that you need to adjust the server_name within the /etc/nginx/sites-available/myproject file to point to your server’s IP address or domain name.

      Nginx uses the server_name to determine which server block to use to respond to requests. If you receive the default Nginx page, it is a sign that Nginx wasn’t able to match the request to a sever block explicitly, so it’s falling back on the default block defined in /etc/nginx/sites-available/default.

      The server_name in your project’s server block must be more specific than the one in the default server block to be selected.

      Nginx Is Displaying a 502 Bad Gateway Error Instead of the Django Application

      A 502 error indicates that Nginx is unable to successfully proxy the request. A wide range of configuration problems express themselves with a 502 error, so more information is required to troubleshoot properly.

      The primary place to look for more information is in Nginx’s error logs. Generally, this will tell you what conditions caused problems during the proxying event. Follow the Nginx error logs by typing:

      • sudo tail -F /var/log/nginx/error.log

      Now, make another request in your browser to generate a fresh error (try refreshing the page). You should receive a fresh error message written to the log. If you look at the message, it should help you narrow down the problem.

      You might receive the following message:

      connect() to unix:/run/gunicorn.sock failed (2: No such file or directory)

      This indicates that Nginx was unable to find the gunicorn.sock file at the given location. You should compare the proxy_pass location defined within /etc/nginx/sites-available/myproject file to the actual location of the gunicorn.sock file generated by the gunicorn.socket systemd unit.

      If you cannot find a gunicorn.sock file within the /run directory, it generally means that the systemd socket file was unable to create it. Go back to the section on checking for the Gunicorn socket file to step through the troubleshooting steps for Gunicorn.

      connect() to unix:/run/gunicorn.sock failed (13: Permission denied)

      This indicates that Nginx was unable to connect to the Gunicorn socket because of permissions problems. This can happen when the procedure is followed using the root user instead of a sudo user. While systemd is able to create the Gunicorn socket file, Nginx is unable to access it.

      This can happen if there are limited permissions at any point between the root directory (/) the gunicorn.sock file. We can review the permissions and ownership values of the socket file and each of its parent directories by passing the absolute path to our socket file to the namei command:

      • namei -l /run/gunicorn.sock

      Output

      f: /run/gunicorn.sock drwxr-xr-x root root / drwxr-xr-x root root run srw-rw-rw- root root gunicorn.sock

      The output displays the permissions of each of the directory components. By looking at the permissions (first column), owner (second column) and group owner (third column), we can figure out what type of access is allowed to the socket file.

      In the above example, the socket file and each of the directories leading up to the socket file have world read and execute permissions (the permissions column for the directories end with r-x instead of ---). The Nginx process should be able to access the socket successfully.

      If any of the directories leading up to the socket do not have world read and execute permission, Nginx will not be able to access the socket without allowing world read and execute permissions or making sure group ownership is given to a group that Nginx is a part of.

      Django Is Displaying: “could not connect to server: Connection refused”

      One message that you may receive from Django when attempting to access parts of the application in the web browser is:

      OperationalError at /admin/login/
      could not connect to server: Connection refused
          Is the server running on host "localhost" (127.0.0.1) and accepting
          TCP/IP connections on port 5432?
      

      This indicates that Django is unable to connect to the Postgres database. Make sure that the Postgres instance is running by typing:

      • sudo systemctl status postgresql

      If it is not, you can start it and enable it to start automatically at boot (if it is not already configured to do so) by typing:

      • sudo systemctl start postgresql
      • sudo systemctl enable postgresql

      If you are still having issues, make sure the database settings defined in the ~/myprojectdir/myproject/settings.py file are correct.

      Further Troubleshooting

      For additional troubleshooting, the logs can help narrow down root causes. Check each of them in turn and look for messages indicating problem areas.

      The following logs may be helpful:

      • Check the Nginx process logs by typing: sudo journalctl -u nginx
      • Check the Nginx access logs by typing: sudo less /var/log/nginx/access.log
      • Check the Nginx error logs by typing: sudo less /var/log/nginx/error.log
      • Check the Gunicorn application logs by typing: sudo journalctl -u gunicorn
      • Check the Gunicorn socket logs by typing: sudo journalctl -u gunicorn.socket

      As you update your configuration or application, you will likely need to restart the processes to adjust to your changes.

      If you update your Django application, you can restart the Gunicorn process to pick up the changes by typing:

      • sudo systemctl restart gunicorn

      If you change Gunicorn socket or service files, reload the daemon and restart the process by typing:

      • sudo systemctl daemon-reload
      • sudo systemctl restart gunicorn.socket gunicorn.service

      If you change the Nginx server block configuration, test the configuration and then Nginx by typing:

      • sudo nginx -t && sudo systemctl restart nginx

      These commands are helpful for picking up changes as you adjust your configuration.

      Conclusion

      In this guide, we’ve set up a Django project in its own virtual environment. We’ve configured Gunicorn to translate client requests so that Django can handle them. Afterwards, we set up Nginx to act as a reverse proxy to handle client connections and serve the correct project depending on the client request.

      Django makes creating projects and applications simple by providing many of the common pieces, allowing you to focus on the unique elements. By leveraging the general tool chain described in this article, you can easily serve the applications you create from a single server.



      Source link