One place for hosting & domains

      Install

      How To Install and Use ClickHouse on Ubuntu 20.04


      The author selected the Free and Open Source Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      ClickHouse is an open source, column-oriented analytics database created by Yandex for OLAP and big data use cases. ClickHouse’s support for real-time query processing makes it suitable for applications that require sub-second analytical results. ClickHouse’s query language is a dialect of SQL that enables powerful declarative querying capabilities while offering familiarity and a smaller learning curve for the end user.

      Column-oriented databases store records in blocks grouped by columns instead of rows. By not loading data for columns absent in the query, column-oriented databases spend less time reading data while completing queries. As a result, these databases can compute and return results much faster than traditional row-based systems for certain workloads, such as OLAP.

      Online Analytics Processing (OLAP) systems allow for organizing large amounts of data and performing complex queries. They are capable of managing petabytes of data and returning query results quickly. In this way, OLAP is useful for work in areas like data science and business analytics.

      In this tutorial, you’ll install the ClickHouse database server and client on your machine. You’ll use the DBMS for typical tasks and optionally enable remote access from another server so that you’ll be able to connect to the database from another machine. Then you’ll test ClickHouse by modeling and querying example website-visit data.

      Prerequisites

      • One Ubuntu 20.04 server with a sudo enabled non-root user and firewall setup. The server should have at least 2GB of RAM. You can follow the Initial Server Setup tutorial to create the user and set up the firewall.

      • (Optional) A secondary Ubuntu 20.04 server with a sudo enabled non-root user and firewall setup.

      Step 1 — Installing ClickHouse

      In this section, you will install the ClickHouse server and client programs using apt.

      First, SSH into your server by running:

      Yandex maintains an APT repository that has the latest version of ClickHouse. Add the repository’s GPG key so that you’ll be able to securely download validated ClickHouse packages:

      • sudo apt-key adv --keyserver keyserver.ubuntu.com --recv E0C56BD4

      You will see output similar to the following:

      Output

      Executing: /tmp/apt-key-gpghome.JkkcKnBAFY/gpg.1.sh --keyserver keyserver.ubuntu.com --recv E0C56BD4 gpg: key C8F1E19FE0C56BD4: public key "ClickHouse Repository Key <milovidov@yandex-team.ru>" imported gpg: Total number processed: 1 gpg: imported: 1

      The output confirms it has successfully verified and added the key.

      Add the repository to your APT repositories list by executing:

      • echo "deb http://repo.yandex.ru/clickhouse/deb/stable/ main/" | sudo tee /etc/apt/sources.list.d/clickhouse.list

      Here you’ve piped the output of echo to sudo tee so that this output can print to a root-owned file.

      Now, update your packages:

      The clickhouse-server and clickhouse-client packages will now be available for installation. Install them with:

      • sudo apt install clickhouse-server clickhouse-client

      During the installation, you will be asked to set a password for the default ClickHouse user.

      You’ve installed the ClickHouse server and client successfully. You’re now ready to start the database service and ensure that it’s running correctly.

      Step 2 — Starting the Service

      The clickhouse-server package that you installed in the previous section creates a systemd service, which performs actions such as starting, stopping, and restarting the database server. systemd is an init system for Linux to initialize and manage services. In this section you’ll start the service and verify that it is running successfully.

      Start the clickhouse-server service by running:

      • sudo service clickhouse-server start

      The previous command will not display any output. To verify that the service is running successfully, execute:

      • sudo service clickhouse-server status

      You’ll see output similar to the following:

      Output

      ● clickhouse-server.service - ClickHouse Server (analytic DBMS for big data) Loaded: loaded (/etc/systemd/system/clickhouse-server.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2020-09-16 05:18:54 UTC; 5s ago Main PID: 2697 (clickhouse-serv) Tasks: 46 (limit: 1137) Memory: 459.7M CGroup: /system.slice/clickhouse-server.service └─2697 /usr/bin/clickhouse-server --config=/etc/clickhouse-server/config.xml --pid-file=/run/clickhouse-server/clickhouse-server.pid

      The output notes that the server is running.

      You have successfully started the ClickHouse server and will now be able to use the clickhouse-client CLI program to connect to the server.

      Step 3 — Creating Databases and Tables

      In ClickHouse, you can create and delete databases by executing SQL statements directly in the interactive database prompt. Statements consist of commands following a particular syntax that tell the database server to perform a requested operation along with any data required. You create databases by using the CREATE DATABASE table_name syntax. To create a database, first start a client session by running the following command:

      • clickhouse-client --password

      You will be asked to enter the password you had set during the installation—enter it to successfully to start the client session.

      The previous command will log you in to the client prompt where you can run ClickHouse SQL statements to perform actions such as:

      • Creating, updating, and deleting databases, tables, indexes, partitions, and views.

      • Executing queries to retrieve data that is optionally filtered and grouped using various conditions.

      In this step, with the ClickHouse client ready for inserting data, you’re going to create a database and table. For the purposes of this tutorial, you’ll create a database named test, and inside that you’ll create a table named visits that tracks website-visit durations.

      Now that you’re inside the ClickHouse command prompt, create your test database by executing:

      You’ll see the following output that shows that you have created the database:

      Output

      CREATE DATABASE test Ok. 0 rows in set. Elapsed: 0.003 sec.

      A ClickHouse table is similar to tables in other relational databases; it holds a collection of related data in a structured format. You can specify columns along with their types, add rows of data, and execute different kinds of queries on tables.

      The syntax for creating tables in ClickHouse follows this example structure:

      CREATE TABLE table_name
      (
          column_name1 column_type [options],
          column_name2 column_type [options],
          ...
      ) ENGINE = engine
      

      The table_name and column_name values can be any valid ASCII identifiers. ClickHouse supports a wide range of column types; some of the most popular are:

      • UInt64: used for storing integer values in the range 0 to 18446744073709551615.

      • Float64: used for storing floating point numbers such as 2039.23, 10.5, etc.

      • String: used for storing variable length characters. It does not require a max-length attribute since it can store arbitrary lengths.

      • Date: used for storing dates that follow the YYYY-MM-DD format.

      • DateTime: used for storing dates coupled with time and follows the YYYY-MM-DD HH:MM:SS format.

      After the column definitions, you specify the engine used for the table. In ClickHouse, Engines determine the physical structure of the underlying data, the table’s querying capabilities, its concurrent access modes, and support for indexes. Different engine types are suitable for different application requirements. The most commonly used and widely applicable engine type is MergeTree.

      Now that you have an overview of table creation, you’ll create a table. Start by confirming the database you’ll be modifying:

      You will see the following output showing that you have switched to the test database from the default database:

      Output

      USE test Ok. 0 rows in set. Elapsed: 0.001 sec.

      The remainder of this guide will assume that you are executing statements within this database’s context.

      Create your visits table by running this command:

      • CREATE TABLE visits (
      • id UInt64,
      • duration Float64,
      • url String,
      • created DateTime
      • ) ENGINE = MergeTree()
      • PRIMARY KEY id
      • ORDER BY id;

      Here’s a breakdown of what the command does. You create a table named visits that has four columns:

      • id: The primary key column. Similarly to other RDBMS systems, a primary key column in ClickHouse uniquely identifies a row; each row should have a unique value for this column.

      • duration: A float column used to store the duration of each visit in seconds. float columns can store decimal values such as 12.50.

      • url: A string column that stores the URL visited, such as http://example.com.

      • created: A date and time column that tracks when the visit occurred.

      After the column definitions, you specify MergeTree as the storage engine for the table. The MergeTree family of engines is recommended for production databases due to its optimized support for large real-time inserts, overall robustness, and query support. Additionally, MergeTree engines support sorting of rows by primary key, partitioning of rows, and replicating and sampling data.

      If you intend to use ClickHouse for archiving data that is not queried often or for storing temporary data, you can use the Log family of engines to optimize for that use-case.

      After the column definitions, you’ll define other table-level options. The PRIMARY KEY clause sets id as the primary key column and the ORDER BY clause will store values sorted by the id column. A primary key uniquely identifies a row and is used for efficiently accessing a single row and efficient colocation of rows.

      On executing the create statement, you will see the following output:

      Output

      CREATE TABLE visits ( id UInt64, duration Float64, url String, created DateTime ) ENGINE = MergeTree() PRIMARY KEY id ORDER BY id Ok. 0 rows in set. Elapsed: 0.010 sec.

      In this section, you’ve created a database and a table to track website-visit data. In the next step, you’ll insert data into the table, update existing data, and delete that data.

      Step 4 — Inserting, Updating, and Deleting Data and Columns

      In this step, you’ll use your visits table to insert, update, and delete data. The following command is an example of the syntax for inserting rows into a ClickHouse table:

      INSERT INTO table_name VALUES (column_1_value, column_2_value, ....);
      

      Now, insert a few rows of example website-visit data into your visits table by running each of the following statements:

      • INSERT INTO visits VALUES (1, 10.5, 'http://example.com', '2019-01-01 00:01:01');
      • INSERT INTO visits VALUES (2, 40.2, 'http://example1.com', '2019-01-03 10:01:01');
      • INSERT INTO visits VALUES (3, 13, 'http://example2.com', '2019-01-03 12:01:01');
      • INSERT INTO visits VALUES (4, 2, 'http://example3.com', '2019-01-04 02:01:01');

      You’ll see the following output repeated for each insert statement.

      Output

      INSERT INTO visits VALUES Ok. 1 rows in set. Elapsed: 0.004 sec.

      The output for each row shows that you’ve inserted it successfully into the visits table.

      Now you’ll add an additional column to the visits table. When adding or deleting columns from existing tables, ClickHouse supports the ALTER syntax.

      For example, the basic syntax for adding a column to a table is as follows:

      ALTER TABLE table_name ADD COLUMN column_name column_type;
      

      Add a column named location that will store the location of the visits to a website by running the following statement:

      • ALTER TABLE visits ADD COLUMN location String;

      You’ll see output similar to the following:

      Output

      ALTER TABLE visits ADD COLUMN location String Ok. 0 rows in set. Elapsed: 0.014 sec.

      The output shows that you have added the location column successfully.

      As of version 19.13.3, ClickHouse doesn’t support updating and deleting individual rows of data due to implementation constraints. ClickHouse has support for bulk updates and deletes, however, and has a distinct SQL syntax for these operations to highlight their non-standard usage.

      The following syntax is an example for bulk updating rows:

      ALTER TABLE table_name UPDATE  column_1 = value_1, column_2 = value_2 ...  WHERE  filter_conditions;
      

      You’ll run the following statement to update the url column of all rows that have a duration of less than 15. Enter it into the database prompt to execute:

      • ALTER TABLE visits UPDATE url="http://example2.com" WHERE duration < 15;

      The output of the bulk update statement will be as follows:

      Output

      ALTER TABLE visits UPDATE url="http://example2.com" WHERE duration < 15 Ok. 0 rows in set. Elapsed: 0.003 sec.

      The output shows that your update query completed successfully. The 0 rows in set in the output denotes that the query did not return any rows; this will be the case for any update and delete queries.

      The example syntax for bulk deleting rows is similar to updating rows and has the following structure:

      ALTER TABLE table_name DELETE WHERE filter_conditions;
      

      To test deleting data, run the following statement to remove all rows that have a duration of less than 5:

      • ALTER TABLE visits DELETE WHERE duration < 5;

      The output of the bulk delete statement will be similar to:

      Output

      ALTER TABLE visits DELETE WHERE duration < 5 Ok. 0 rows in set. Elapsed: 0.003 sec.

      The output confirms that you have deleted the rows with a duration of less than five seconds.

      To delete columns from your table, the syntax would follow this example structure:

      ALTER TABLE table_name DROP COLUMN column_name;
      

      Delete the location column you added previously by running the following:

      • ALTER TABLE visits DROP COLUMN location;

      The DROP COLUMN output confirming that you have deleted the column will be as follows:

      Output

      ALTER TABLE visits DROP COLUMN location String Ok. 0 rows in set. Elapsed: 0.010 sec.

      Now that you’ve successfully inserted, updated, and deleted rows and columns in your visits table, you’ll move on to query data in the next step.

      Step 5 — Querying Data

      ClickHouse’s query language is a custom dialect of SQL with extensions and functions suited for analytics workloads. In this step, you’ll run selection and aggregation queries to retrieve data and results from your visits table.

      Selection queries allow you to retrieve rows and columns of data filtered by conditions that you specify, along with options such as the number of rows to return. You can select rows and columns of data using the SELECT syntax. The basic syntax for SELECT queries is:

      SELECT func_1(column_1), func_2(column_2) FROM table_name WHERE filter_conditions row_options;
      

      Execute the following statement to retrieve url and duration values for rows where the url is http://example.com.

      • SELECT url, duration FROM visits WHERE url="http://example2.com" LIMIT 2;

      You will see the following output:

      Output

      SELECT url, duration FROM visits WHERE url="http://example2.com" LIMIT 2 ┌─url─────────────────┬─duration─┐ │ http://example2.com │ 10.5 │ └─────────────────────┴──────────┘ ┌─url─────────────────┬─duration─┐ │ http://example2.com │ 13 │ └─────────────────────┴──────────┘ 2 rows in set. Elapsed: 0.013 sec.

      The output has returned two rows that match the conditions you specified. Now that you’ve selected values, you can move on to executing aggregation queries.

      Aggregation queries are queries that operate on a set of values and return single output values. In analytics databases, these queries are run frequently and are well optimized by the database. Some aggregate functions supported by ClickHouse are:

      • count: returns the count of rows matching the conditions specified.

      • sum: returns the sum of selected column values.

      • avg: returns the average of selected column values.

      Some ClickHouse-specific aggregate functions include:

      • uniq: returns an approximate number of distinct rows matched.

      • topK: returns an array of the most frequent values of a specific column using an approximation algorithm.

      To demonstrate the execution of aggregation queries, you’ll calculate the total duration of visits by running the sum query:

      • SELECT SUM(duration) FROM visits;

      You will see output similar to the following:

      Output

      SELECT SUM(duration) FROM visits ┌─SUM(duration)─┐ │ 63.7 │ └───────────────┘ 1 rows in set. Elapsed: 0.010 sec.

      Now, calculate the top two URLs by executing:

      • SELECT topK(2)(url) FROM visits;

      You will see output similar to the following:

      Output

      SELECT topK(2)(url) FROM visits ┌─topK(2)(url)──────────────────────────────────┐ │ ['http://example2.com','http://example1.com'] │ └───────────────────────────────────────────────┘ 1 rows in set. Elapsed: 0.010 sec.

      Now that you have successfully queried your visits table, you’ll delete tables and databases in the next step.

      Step 6 — Deleting Tables and Databases

      In this section, you’ll delete your visits table and test database.

      The syntax for deleting tables follows this example:

      DROP TABLE table_name;
      

      To delete the visits table, run the following statement:

      You will see the following output declaring that you’ve deleted the table successfully:

      output

      DROP TABLE visits Ok. 0 rows in set. Elapsed: 0.005 sec.

      You can delete databases using the DROP database table_name syntax. To delete the test database, execute the following statement:

      The resulting output shows that you’ve deleted the database successfully.

      Output

      DROP DATABASE test Ok. 0 rows in set. Elapsed: 0.003 sec.

      You’ve deleted tables and databases in this step. Now that you’ve created, updated, and deleted databases, tables, and data in your ClickHouse instance, you’ll enable remote access to your database server in the next section.

      Step 7 — Setting Up Firewall Rules (Optional)

      If you intend to only use ClickHouse locally with applications running on the same server, or do not have a firewall enabled on your server, you don’t need to complete this section. If instead, you’ll be connecting to the ClickHouse database server remotely, you should follow this step.

      Currently your server has a firewall enabled that disables your public IP address accessing all ports. You’ll complete the following two steps to allow remote access:

      • Modify ClickHouse’s configuration and allow it to listen on all interfaces.

      • Add a firewall rule allowing incoming connections to port 8123, which is the HTTP port that the ClickHouse server runs.

      If you are inside the database prompt, exit it by typing CTRL+D.

      Edit the configuration file by executing:

      • sudo nano /etc/clickhouse-server/config.xml

      Then uncomment the line containing <!-- <listen_host>0.0.0.0</listen_host> -->, like the following file:

      /etc/clickhouse-server/config.xml

      ...
       <interserver_http_host>example.yandex.ru</interserver_http_host>
          -->
      
          <!-- Listen specified host. use :: (wildcard IPv6 address), if you want to accept connections both with IPv4 and IPv6 from everywhere. -->
          <!-- <listen_host>::</listen_host> -->
          <!-- Same for hosts with disabled ipv6: -->
          <listen_host>0.0.0.0</listen_host>
      
          <!-- Default values - try listen localhost on ipv4 and ipv6: -->
          <!--
          <listen_host>::1</listen_host>
          <listen_host>127.0.0.1</listen_host>
          -->
      ...
      

      Save the file and exit. For the new configuration to apply restart the service by running:

      • sudo service clickhouse-server restart

      You won’t see any output from this command. ClickHouse’s server listens on port 8123 for HTTP connections and port 9000 for connections from clickhouse-client. Allow access to both ports for your second server’s IP address with the following command:

      • sudo ufw allow from second_server_ip/32 to any port 8123
      • sudo ufw allow from second_server_ip/32 to any port 9000

      You will see the following output for both commands that shows that you’ve enabled access to both ports:

      Output

      Rule added

      ClickHouse will now be accessible from the IP that you added. Feel free to add additional IPs such as your local machine’s address if required.

      To verify that you can connect to the ClickHouse server from the remote machine, first follow the steps in Step 1 of this tutorial on the second server and ensure that you have the clickhouse-client installed on it.

      Now that you have logged in to the second server, start a client session by executing:

      • clickhouse-client --host your_server_ip --password

      You will see the following output that shows that you have connected successfully to the server:

      Output

      ClickHouse client version 19.13.3.26 (official build). Password for user (default): Connecting to your_server_ip:9000 as user default. Connected to ClickHouse server version 19.13.3 revision 54425. hostname :)

      In this step, you’ve enabled remote access to your ClickHouse database server by adjusting your firewall rules.

      Conclusion

      You have successfully set up a ClickHouse database instance on your server and created a database and table, added data, performed queries, and deleted the database. Within ClickHouse’s documentation you can read about their benchmarks against other open-source and commercial analytics databases and general reference documents.

      Further features ClickHouse offers include distributed query processing across multiple servers to improve performance and protect against data loss by storing data over different shards.



      Source link

      How To Install and Configure Mahara on Ubuntu 18.04


      The author selected the COVID-19 Relief Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      Mahara is an open-source ePortfolio and social networking system that bills itself as a “learner-centered personal learning environment.” Mahara is especially popular in academic institutions where it provides learners a platform to curate and share their knowledge and growth. In a remote learning environment, Mahara can help educators foster a digital classroom and organize a student’s progress. Unlike more a traditional Learning Management Systems (LMS), Mahara is more distributed and user-focused. This amount of user creativity contrasts other, more institutional systems.

      Mahara also has many non-student applications. You can use it to build a blog, a resume-builder, a file-repository, or a competency framework.

      In this guide, you will install Mahara and configure it to use a Postgres database as an authentication source. Upon completion, you will have administrative access to a server-based ePortfolio.

      Prerequisites

      Step 1 — Creating a Database and User for Mahara

      Your first step is to create a Postgres database and user for Mahara.

      From your server’s command line, switch to the postgres user:

      The -i switch provides an interactive shell, while sudo used with the -u switch allows you to switch users.

      Now create a Postgres user; make sure to replace mahara with your preferred username:

      Postgres will prompt you to Enter password for a new role:. role here refers to the new user you are trying to create. Provide a password of your choosing, and then reenter the password when prompted.

      Now create a database for your Mahara installation. Use the following command, replacing maharadb with a preferred name for your new database and mahara with the new user you just created:

      • createdb -O mahara -EUTF8 maharadb

      You can now exit from the postgres user:

      Note: It is recommended to use Postgres for your Mahara database, but if you decide to use MySQL, you’ll need to set up a new user and database for Mahara. To do so, log in to your MySQL root account using the following command:

      Provide your MySQL root password to gain access. Now create your Mahara database. Replace maharadb with another name:

      • create database maharadb character set UTF8;

      Next, create a new MySQL user for Mahara, set a password, and grant it access to your maharadb database. Remember to provide a strong password in place of password and replace mahara with your desired username:

      • grant all on maharadb .* to 'mahara '@'localhost' identified by 'password';

      You have now created a database for Mahara and created a role. Now you can install and configure Mahara.

      Step 2 — Downloading Mahara

      Now that you have a database created and configured for Mahara, let’s download Mahara itself. You can download a standard build from Mahara’s server, and you can find the latest Mahara build here.

      Once you have a .tar.gz link to the latest stable release, download it using the wget command. Make sure to replace /19.10/19.10.2/+download/mahara-19.10.2.tar.gz with the release you selected:

      • wget https://launchpad.net/mahara/19.10/19.10.2/+download/mahara-19.10.2.tar.gz

      This will download a compressed file labeled mahara-19.10.2.tar.gz. Extract the compressed file to create the Mahara directory structure:

      • tar xzvf mahara-19.10.2.tar.gz
      • ls

      The ls command checks the extraction. You will now see the new mahara-19.10.2 directory:

      Output

      mahara-19.10.2.tar.gz mahara-19.10.2

      Let’s copy Mahara to /var/www/mahara.your_domain. mahara.your_domain should match the directory you created during the Virtual Host configuration step in the Prerequisite section.

      Note: Mahara must install into an empty directory. If you have files here already, consider changing your DocumentRoot and using a new directory.

      Now copy the Mahara directory:

      • sudo cp -a mahara-19.10.2/. /var/www/mahara.your_domain/

      The -a switch ensures that the system copies the file permissions along with the files and folders. The . at the end of the source file ensures that everything in the source directory, including hidden files, gets copied to the destination directory.

      Mahara requires additional PHP extensions to function correctly.

      Run the following commands to install all the necessary dependencies:

      • sudo apt update
      • sudo apt install php-gd php-pgsql php-xmlrpc php-xml php-curl php-mbstring

      Type y for any prompts to finish the installation.

      You have installed Mahara on your server. Next, you will create the dataroot directory.

      Step 3 — Creating the Dataroot Directory

      Now that you have your Mahara code set up let’s create a dataroot directory. This is where Mahara will write uploaded files as well as some other files it needs to run. Therefore, the web server needs to be able to write to it.

      The dataroot directory should also be outside the directory where you have your Mahara code. You should avoid having it inside your public_html directory altogether.

      Let’s create the directory in /var/maharadata:

      • sudo mkdir /var/maharadata

      Update the ownership with chown:

      • sudo chown -R www-data:www-data /var/maharadata

      This allows the web server to write in the directory. Now we can finish configuring Mahara.

      Step 4 — Setting Up the Mahara Configuration File

      In this step you will make some final adjustments to Mahara’s config.php file. These adjustments will connect your database and encrypt your password.

      Switch to the Mahara code directory:

      • cd /var/www/mahara.your_domain/htdocs

      Create your config.php file by copying the config-dist.php file in the htdocs directory.

      Change directories and copy the file:

      • cp config-dist.php config.php

      Now open config.php using nano or your preferred text editor:

      Locate the following section in the file and make the required changes. Leave $cfg->dbtype as postgres unless you’re using MySQL, in which case change it to mysql. Change maharadb, mahara, and dbpassword to match the values you used in Step 1 for the database name, user, and password, respectively:

      ./htdocs/config.php

      ...
      $cfg->dbtype="postgres";
      $cfg->dbhost="localhost";
      $cfg->dbport   = null; // Change if you are using a non-standard port number for your database
      $cfg->dbname="maharadb";
      $cfg->dbuser="mahara";
      $cfg->dbpass="dbpassword";
      ...
      

      In the same file, locate the following section:

      /var/www/mahara/htdocs/config.php

      ...
      $cfg->dataroot="/path/to/uploaddir";
      ...
      

      Change it to point to Mahara’s dataroot directory as you set it in Step 3:

      /var/www/mahara/htdocs/config.php

      ...
      $cfg->dataroot="/var/maharadata";
      ...
      

      Finally, let’s update the passwordsaltmain with a random secret string. This encrypts stored user passwords. Locate and uncomment the following line:

      /var/www/mahara/htdocs/config.php

      ...
      // $cfg->passwordsaltmain = 'some long random string here with lots of characters';
      ...
      

      You can use the OpenSSL rand function to generate a random string to use as your secret salt string. Open a new terminal, connect to your server again, and run the following command to generate this string:

      The -base64 32 option ensures a Base64 encoded string that is 32 characters long. Update the value for passwordsaltmain to your_generated_salt:

      /var/www/mahara/htdocs/config.php

      ...
      $cfg->passwordsaltmain = 'your_generated_salt';
      ...
      

      Warning: Once set, you must not lose this string or you will need to reset all user passwords. It’s good practice to keep a secure backup of your config.php file

      Note: You may wish to set productionmode to false when using Mahara as a development or test site. Setting this to false will enable the on-screen display of warnings and error messages to aid in testing.

      Save and close the file.

      With Mahara configured, you are ready to serve Mahara using the Apache web server.

      Step 5 — Configuring Apache

      In the prerequisites, you configured a domain and pointed it to this server. You also installed Apache and set up HTTPS using Let’s Encrypt. Let’s now use Apache to serve Mahara.

      If your Virtual Host configuration file is named mahara.your_domain.conf Let’s Encrypt created a new configuration file called mahara.your_domain-le-ssl.conf that handles HTTPS requests for your domain.

      Open that file. Be sure to replace mahara.your_domain with the actual name of your file:

      • sudo nano /etc/apache2/sites-available/mahara.your_domain-le-ssl.conf

      The file will look similar to this:

      /etc/apache2/sites-available/your_domain-le-ssl.conf

      ...
      <IfModule mod_ssl.c>
      <VirtualHost *:443>
              ServerAdmin webmaster@localhost
              ServerName mahara.your_domain
              DocumentRoot /var/www/mahara.your_domain
              ErrorLog ${APACHE_LOG_DIR}/error.log
              CustomLog ${APACHE_LOG_DIR}/access.log combined
      
      SSLCertificateFile /etc/letsencrypt/live/mahara.your_domain/fullchain.pem
      SSLCertificateKeyFile /etc/letsencrypt/live/mahara.your_domain/privkey.pem
      Include /etc/letsencrypt/options-ssl-apache.conf
      </VirtualHost>
      </IfModule>
      ...
      

      The ServerName directive defines the base domain that should match for this virtual host definition. This should also be the domain name you chose to secure with an SSL certificate in the Prerequisites section.

      Note: Your Apache configuration should contain no ServerAliases. Mahara expects to be accessed through one url. If you use a server alias, you’ll experience issues like having to log in twice and breaking Single Sign On (SSO) features.

      However, if you do need an alias for some reason, you can set up a separate second VirtualHost directive in addition to the one above.

      Make the following additions to the file:

      /etc/apache2/sites-available/your_domain-le-ssl.conf

      ...
      <IfModule mod_ssl.c>
      <VirtualHost *:443>
              ServerAdmin webmaster@localhost
              ServerName mahara.your_domain
              DocumentRoot /var/www/mahara.your_domain/htdocs
      
              <Directory />
                  Options FollowSymLinks
                  AllowOverride None
              </Directory>
      
              <Directory /var/www/your_domain/htdocs>
                  Options Indexes FollowSymLinks MultiViews
                  AllowOverride None
                  Order allow,deny
                  allow from all
              </Directory>
      
              ErrorLog ${APACHE_LOG_DIR}/error.log
              CustomLog ${APACHE_LOG_DIR}/access.log combined
      
      SSLCertificateFile /etc/letsencrypt/live/mahara.your_domain/fullchain.pem
      SSLCertificateKeyFile /etc/letsencrypt/live/mahara.your_domain/privkey.pem
      Include /etc/letsencrypt/options-ssl-apache.conf
      </VirtualHost>
      </IfModule>
      ...
      

      First, you update the DocumentRoot to include /htdocs. Then you add two new blocks.

      The <Directory /> block refers to the filesystem root. It specifies directives applying to all files and directories anywhere on the server. Specifying directives for this block allows us to protect your server from intrusions by placing default directives such as AllowOverride None.

      The <Directory /var/www/mahara.your_domain/htdocs> block refers to the document root for Mahara. Directives declared here override ones declared on the <Directory /> block. These directives are the fundamental requirements for an Apache configuration to serve Mahara.

      Save and close the file.

      You are now ready to configure your PHP service.

      Step 6 — Adjusting PHP Settings

      Our final step is to update Apache’s php.ini file. This will change our logging verbosity and resize maximum upload and POST sizes.

      Open the file:

      • sudo nano /etc/php/7.2/apache2/php.ini

      This is a large file. Let’s go through the updates one by one.

      Find the log_errors line and make sure it matches the highlighted code:

      /etc/php/7.2/apache2/php.ini

      . . .
      log_errors = On
      . . .
      

      Setting log_errors to On will enable PHP to log errors encountered while running Mahara to aid in debugging.

      Now find the line beginning upload_max_filesize = 2M. Replace 2M with 50M:

      7.2/apache2/php.ini’>/etc/php/7.2/apache2/php.ini

      . . .
      upload_max_filesize = 50M
      . . .
      

      This will increase the max upload file size to 50 megabytes.

      Now find the line beginning post_max_size = 8M. Replace 8M with 100M:

      7.2/apache2/php.ini’>/etc/php/7.2/apache2/php.ini

      . . .
      post_max_size = 100M
      . . .
      

      upload_max_filesize accounts for the maximum size allowed for files posted, while post_max_size is the maximum size allowed for all POST body data. So you’ll always want to have this second number higher. These settings will enable Mahara to accept file uploads at the set maximums.

      Now move to the end of the file and add the following directives:

      /etc/php/7.2/apache2/php.ini

      ...
      register_globals = Off
      magic_quotes_runtime = Off
      magic_quotes_sybase = Off
      magic_quotes_gpc = Off
      allow_call_time_pass_reference = Off
      

      register_globals = Off disables register_globals. Enabling it has security implications as highlighted on PHP’s Manual.

      Setting magic_quotes_runtime to Off stops PHP from automatically escaping quotes with a backslash from any functions coming in from an external source (POST/GET). magic_quotes_runtime, magic_quotes_sybase, and magic_quotes_gpc are deprecated.

      allow_call_time_pass_reference is also deprecated.

      Save and close the file.

      Now restart Apache:

      • sudo systemctl restart apache2

      This completes your server setup. You are now ready to run the Mahara installer and enter your ePortfolio through a web browser.

      Step 7 — Running the Mahara Installer

      With Apache and PHP properly configured, let’s proceed to finish up Mahara’s installation through its web-based installer.

      In your browser, navigate to your server’s domain name. A screen will appear displaying Mahara’s licensing information and a button to continue installation.

      Mahara Install Screen

      Click the Install Mahara button and wait until the installer finishes performing all installations. Once done, scroll down. You will see a section that says Successfully installed Mahara. Continue. Click on Continue to proceed.

      Mahara will prompt you to enter a new password and a primary email address. Go ahead and submit the form.

      Mahara Welcome Screen

      You are ready to start using Mahara. But before you start exploring your new ePortfolio, you might want to set up a few final features.

      Set up Email

      For production settings, make sure that Mahara can send out emails. This is useful for sending out notifications such as confirmation emails after users register on the site. You can specify an outgoing SMTP server with admin settings. Click on the drop-down menu on the upper-right and choose Configure Site -> Site options -> Email.

      Set up a Cron Job

      Lastly, consider setting up a cron job to hit htdocs/lib/cron.php every minute. Without setting this up, you will find that RSS feeds will not update and some email notifications won’t get sent out. Go ahead and enter the following command. You’ll get an option to select the editor of your choice. In this example, we’re proceeding with the nano editor:

      Add the following line at the bottom of the file replacing mahara.your_domain with the name of the directory where you installed Mahara on Step 2:

      * * * * * php /var/www/mahara.your_domain/htdocs/lib/cron.php
      

      With your email and cron job configured, you are now ready to use Mahara.

      Conclusion

      Mahara is a powerful and popular platform. It is a great resource for teacher and learners, particularly within remote-learning environments. Mahara is also popular outside of education, where users create blogs and resume builders.

      Users can also choose from numerous themes while admins can explore a large library of plugins.

      After familiarizing yourself with the user interface, consider following the Next Steps article on Mahara’s website for more ideas.



      Source link

      How To Install Jenkins on Kubernetes


      The author selected the COVID-19 Relief Fund to receive a donation as part of the Write for DOnations program.

      Introduction

      Continuous Integration/Continuous Deployment (CI/CD) pipelines are one of the core components of the DevOps environment. They help streamline the workflow between multiple teams and increase productivity. Jenkins is a widely-used open source automation server that can set up CI/CD pipelines.

      In this tutorial, you will install Jenkins on Kubernetes. You will then access the Jenkins UI and run a sample pipeline.

      Prerequisites

      To follow this tutorial, you will need:

      Step 1 — Installing Jenkins on Kubernetes

      Kubernetes has a declarative API and you can convey the desired state using either a YAML or JSON file. For this tutorial, you will use a YAML file to deploy Jenkins. Make sure you have the kubectl command configured for the cluster.

      First, use kubectl to create the Jenkins namespace:

      • kubectl create namespace jenkins

      Next, create the YAML file that will deploy Jenkins.

      Create and open a new file called jenkins.yaml using nano or your preferred editor:

      Now add the following code to define the Jenkins image, its port, and several more configurations:

      jenkins.yaml

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: jenkins
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: jenkins
        template:
          metadata:
            labels:
              app: jenkins
          spec:
            containers:
            - name: jenkins
              image: jenkins/jenkins:lts
              ports:
                - name: http-port
                  containerPort: 8080
                - name: jnlp-port
                  containerPort: 50000
              volumeMounts:
                - name: jenkins-vol
                  mountPath: /var/jenkins_vol
            volumes:
              - name: jenkins-vol
                emptyDir: {}
      

      This YAML file creates a deployment using the Jenkins LTS image and also opens port 8080 and 50000. You use these ports to access Jenkins and accept connections from Jenkins workers respectively.

      Now create this deployment in the jenkins namespace:

      • kubectl create -f jenkins.yaml --namespace jenkins

      Give the cluster a few minutes to pull the Jenkins image and get the Jenkins pod running.

      Use kubectl to verify the pod’s state:

      • kubectl get pods -n jenkins

      You will receive an output like this:

      NAME                       READY   STATUS    RESTARTS   AGE
      jenkins-6fb994cfc5-twnvn   1/1     Running   0          95s
      

      Note that the pod name will be different in your environment.

      Once the pod is running, you need to expose it using a Service. You will use the NodePort Service type for this tutorial. Also, you will create a ClusterIP type service for workers to connect to Jenkins.

      Create and open a new file called jenkins-service.yaml:

      • nano jenkins-service.yaml

      Add the following code to define the NodePort Service:

      jenkins-service.yaml

      apiVersion: v1
      kind: Service
      metadata:
        name: jenkins
      spec:
        type: NodePort
        ports:
          - port: 8080
            targetPort: 8080
            nodePort: 30000
        selector:
          app: jenkins
      
      ---
      
      apiVersion: v1
      kind: Service
      metadata:
        name: jenkins-jnlp
      spec:
        type: ClusterIP
        ports:
          - port: 50000
            targetPort: 50000
        selector:
          app: jenkins
      

      In the above YAML file, you define your NodePort Service and then expose port 8080 of the Jenkins pod to port 30000.

      Now create the Service in the same namespace:

      • kubectl create -f jenkins-service.yaml --namespace jenkins

      Check that the Service is running:

      • kubectl get services --namespace jenkins

      You will receive an output like this:

      Output

      NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE jenkins NodePort your_cluster_ip <none> 8080:30000/TCP 15d

      With NodePort and Jenkins operational, you are ready to access the Jenkins UI and begin exploring it.

      Step 2 — Accessing the Jenkins UI

      In this step, you will access and explore the Jenkins UI. Your NodePort service is accessible on port 30000 across the cluster nodes. You need to retrieve a node IP to access the Jenkins UI.

      Use kubectl to retrieve your node IPs:

      • kubectl get nodes -o wide

      kubectl will produce an output with your external IPs:

      Output

      NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME your_node Ready <none> 16d v1.18.8 your_internal_ip your_external_ip Debian GNU/Linux 10 (buster) 4.19.0-10-cloud-amd64 docker://18.9.9 your_node Ready <none> 16d v1.18.8 your_internal_ip your_external_ip Debian GNU/Linux 10 (buster) 4.19.0-10-cloud-amd64 docker://18.9.9 your_node Ready <none> 16d v1.18.8 your_internal_ip your_external_ip Debian GNU/Linux 10 (buster) 4.19.0-10-cloud-amd64 docker://18.9.9

      Copy one of the your_external_ip values.

      Now open a web browser and navigate to http://your_external_ip:30000.

      A page will appear asking for an administrator password and instructions on retrieving this password from the Jenkins Pod logs.

      Let’s use kubectl to pull the password from those logs.

      First, return to your terminal and retrieve your Pod name:

      • kubectl get pods -n jenkins

      You will receive an output like this:

      NAME                       READY   STATUS    RESTARTS   AGE
      jenkins-6fb994cfc5-twnvn   1/1     Running   0          9m54s
      

      Next, check the Pod’s logs for the admin password. Replace the highlighted section with your pod name:

      • kubectl logs jenkins-6fb994cfc5-twnvn -n jenkins

      You might need to scroll up or down to find the password:

      Running from: /usr/share/jenkins/jenkins.war
      webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
      . . .
      
      Jenkins initial setup is required. An admin user has been created and a password generated.
      Please use the following password to proceed to installation:
      
      your_jenkins_password
      
      This may also be found at: /var/jenkins_home/secrets/initialAdminPassword
      . . .
      

      Copy your_jenkins_password. Now return to your browser and paste it into the Jenkins UI.

      Once you enter the password, Jenkins will prompt you to install plugins. Because you are not doing anything unusual, select Install suggested plugins.

      After installation, Jenkins will load a new page and ask you to create an admin user. Fill out the fields, or skip this step by pressing the skip and continue as admin link. This will leave your username as admin and your password as your_jenkins_password.

      Another screen will appear asking about instance configuration. Click the Not now link and continue.

      After this, Jenkins will create a summary of your choices and print Jenkins is ready! Click on start using Jenkins and the Jenkins home page will appear.

      jenkins wizard

      Now that you have installed and configured Jenkins on your cluster let’s demonstrate its capabilities and run a sample pipeline.

      Step 3 — Running a Sample Pipeline

      Jenkins excels at creating pipelines and managing CI/CD workflows. In this step we will build one of Jenkins’ sample pipelines.

      From the Jenkins home page, click on the New item link on the left-hand menu.

      A new page will appear. Choose Pipeline and press OK.

      jenkins wizard

      Jenkins will redirect you to the pipeline’s configuration. Find the Pipeline section and select Hello World from the try sample pipeline dropdown menu. This menu appears on the right-hand side. After selecting Hello World, click the Save button.

      jenkins wizard

      Jenkins will redirect you to the pipeline home page. Click on build now from the left-hand menu and watch the pipeline begin to run. The #1 signifies that this is the first build. Once the task completes, you will see some stats about the build.

      jenkins wizard

      You can also check the console output to see what happened while the pipeline was running. Hover over #1 and a dropdown menu will appear. Choose console output to view the build’s details.

      Your Hello World pipeline is not very sophisticated, but it does demonstrate just how well Jenkins can create and manage CI/CD workflows.

      Conclusion

      In this tutorial, you installed and configured Jenkins on a Kubernetes cluster and then you ran a sample pipeline. Jenkins has a large repository of plugins that can help you perform very complex operations. You can also add your GitHub repositories, multiple types of worker instances, and more. To learn more about using Jenkins, explore the official Jenkins documentation.



      Source link