One place for hosting & domains

      Job

      How To Run a PHP Job Multiple Times in a Minute with Crontab on Ubuntu 20.04


      The author selected Girls Who Code to receive a donation as part of the Write for DOnations program.

      Introduction

      In Linux, you can use the versatile crontab tool to process long-running tasks in the background at specific times. While the daemon is great for running repetitive tasks, it has got one limitation: you can only execute tasks at a minimum time interval of 1 minute.

      However, in many applications, to avoid a poor user experience it’s preferable for jobs to execute more frequently. For instance, if you’re using the job-queue model to schedule file-processing tasks on your website, a significant wait will have a negative impact on end users.

      Another scenario is an application that uses the job-queue model either to send text messages or emails to clients once they’ve completed a certain task in an application (for example, sending money to a recipient). If users have to wait a minute before the delivery of a confirmation message, they might think that the transaction failed and try to repeat the same transaction.

      To overcome these challenges, you can program a PHP script that loops and processes tasks repetitively for 60 seconds as it awaits for the crontab daemon to call it again after the minute. Once the PHP script is called for the first time by the crontab daemon, it can execute tasks in a time period that matches the logic of your application without keeping the user waiting.

      In this guide, you will create a sample cron_jobs database on an Ubuntu 20.04 server. Then, you’ll set up a tasks table and a script that executes the jobs in your table in intervals of 5 seconds using the PHP while(...){...} loop and sleep() functions.

      Prerequisites

      To complete this tutorial, you require the following:

      Step 1 — Setting Up a Database

      In this step, you’ll create a sample database and table. First, SSH to your server and log in to MySQL as root:

      Enter your root password for the MySQL server and press ENTER to proceed. Then, run the following command to create a cron_jobs database.

      • CREATE DATABASE cron_jobs;

      Create a non-root user for the database. You’ll need the credentials of this user to connect to the cron_jobs database from PHP. Remember to replace EXAMPLE_PASSWORD with a strong value:

      • CREATE USER 'cron_jobs_user'@'localhost' IDENTIFIED WITH mysql_native_password BY 'EXAMPLE_PASSWORD';
      • GRANT ALL PRIVILEGES ON cron_jobs.* TO 'cron_jobs_user'@'localhost';
      • FLUSH PRIVILEGES;

      Next, switch to the cron_jobs database:

      Output

      Database changed

      Once you’ve selected the database, create a tasks table. In this table, you’ll insert some tasks that will be automatically executed by a cron job. Since the minimum time interval for running a cron job is 1 minute, you’ll later code a PHP script that overrides this setting and instead, execute the jobs in intervals of 5 seconds.

      For now, create your tasks table:

      • CREATE TABLE tasks (
      • task_id BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY,
      • task_name VARCHAR(50),
      • queued_at DATETIME,
      • completed_at DATETIME,
      • is_processed CHAR(1)
      • ) ENGINE = InnoDB;

      Insert three records to the tasks table. Use the MySQL NOW() function in the queued_at column to record the current date and time when the tasks are queued. Also for the completed_at column, use the MySQL CURDATE() function to set a default time of 00:00:00. Later, as tasks complete, your script will update this column:

      • INSERT INTO tasks (task_name, queued_at, completed_at, is_processed) VALUES ('TASK 1', NOW(), CURDATE(), 'N');
      • INSERT INTO tasks (task_name, queued_at, completed_at, is_processed) VALUES ('TASK 2', NOW(), CURDATE(), 'N');
      • INSERT INTO tasks (task_name, queued_at, completed_at, is_processed) VALUES ('TASK 3', NOW(), CURDATE(), 'N');

      Confirm the output after running each INSERT command:

      Output

      Query OK, 1 row affected (0.00 sec) ...

      Make sure the data is in place by running a SELECT statement against the tasks table:

      • SELECT task_id, task_name, queued_at, completed_at, is_processed FROM tasks;

      You will find a list of all tasks:

      Output

      +---------+-----------+---------------------+---------------------+--------------+ | task_id | task_name | queued_at | completed_at | is_processed | +---------+-----------+---------------------+---------------------+--------------+ | 1 | TASK 1 | 2021-03-06 06:27:19 | 2021-03-06 00:00:00 | N | | 2 | TASK 2 | 2021-03-06 06:27:28 | 2021-03-06 00:00:00 | N | | 3 | TASK 3 | 2021-03-06 06:27:36 | 2021-03-06 00:00:00 | N | +---------+-----------+---------------------+---------------------+--------------+ 3 rows in set (0.00 sec)

      The time for the completed_at column is set to 00:00:00, this column will update once the tasks are processed by a PHP script that you will create next.

      Exit from the MySQL command-line interface:

      Output

      Bye

      Your cron_jobs database and tasks table are now in place and you can now create a PHP script that processes the jobs.

      Step 2 — Creating a PHP Script that Runs Tasks After 5 Seconds

      In this step, you’ll create a script that uses a combination of the PHP while(...){...} loop and sleep functions to run tasks after every 5 seconds.

      Open a new /var/www/html/tasks.php file in the root directory of your web server using nano:

      • sudo nano /var/www/html/tasks.php

      Next, create a new try { block after a <?php tag and declare the database variables that you created in Step 1. Remember to replace EXAMPLE_PASSWORD with the actual password for your database user:

      /var/www/html/tasks.php

      <?php
      try {
          $db_name="cron_jobs";
          $db_user="cron_jobs_user";
          $db_password = 'EXAMPLE_PASSWORD';
          $db_host="localhost";
      

      Next, declare a new PDO (PHP Data Object) class and set the attribute ERRMODE_EXCEPTION to catch any PDO errors. Also, switch ATTR_EMULATE_PREPARES to false to let the native MySQL database engine handle emulation. Prepared statements allow you to send the SQL queries and data separately to enhance security and reduce chances of an SQL injection attack:

      /var/www/html/tasks.php

      
          $pdo = new PDO('mysql:host=" . $db_host . "; dbname=" . $db_name, $db_user, $db_password);
          $pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);  
          $pdo->setAttribute(PDO::ATTR_EMULATE_PREPARES, false);       
      

      Then, create a new variable named $loop_expiry_time and set it to the current time plus 60 seconds. Then open a new PHP while(time() < $loop_expiry_time) { statement. The idea here is to create a loop that runs until the current time (time()) matches the variable $loop_expiry_time:

      /var/www/html/tasks.php

             
          $loop_expiry_time = time() + 60;
      
          while (time() < $loop_expiry_time) { 
      

      Next, declare a prepared SQL statement that retrieves unprocessed jobs from the tasks table:

      /var/www/html/tasks.php

         
              $data = [];
              $sql  = "select 
                       task_id
                       from tasks
                       where is_processed = :is_processed
                       ";
      

      Execute the SQL command and fetch all rows from the tasks table that have the column is_processed set to N. This means the rows are not processed:

      /var/www/html/tasks.php

        
              $data["is_processed'] = 'N';  
      
              $stmt = $pdo->prepare($sql);
              $stmt->execute($data);
      

      Next, loop through the retrieved rows using a PHP while ($row = $stmt->fetch(PDO::FETCH_ASSOC)) {...} statement and create another SQL statement. This time around, the SQL command updates the is_processed and completed_at columns for each task processed. This ensures that you don’t process tasks more than one time:

      /var/www/html/tasks.php

        
              while ($row = $stmt->fetch(PDO::FETCH_ASSOC)) { 
                  $data_update   = [];         
                  $sql_update    = "update tasks set 
                                    is_processed  = :is_processed,
                                    completed_at  = :completed_at
                                    where task_id = :task_id                                 
                                    ";
      
                  $data_update   = [                
                                   'is_processed' => 'Y',                          
                                   'completed_at' => date("Y-m-d H:i:s"),
                                   'task_id'      => $row['task_id']                         
                                   ];
                  $stmt = $pdo->prepare($sql_update);
                  $stmt->execute($data_update);
              }
      

      Note: If you have a large queue to be processed (for example, 100,000 records per second), you might consider queueing jobs in a Redis Server since it is faster than MySQL when it comes to implementing the job-queue model. Nevertheless, this guide will process a smaller dataset.

      Before you close the first PHP while (time() < $loop_expiry_time) { statement, include a sleep(5); statement to pause the jobs execution for 5 seconds and free up your server resources.

      You may change the 5 seconds period depending on your business logic and how fast you want tasks to execute. For instance, if you would like the tasks to be processed 3 times in a minute, set this value to 20 seconds.

      Remember to catch any PDO error messages inside a } catch (PDOException $ex) { echo $ex->getMessage(); } block:

      /var/www/html/tasks.php

                       sleep(5); 
      
              }       
      
      } catch (PDOException $ex) {
          echo $ex->getMessage(); 
      }
      

      Your complete tasks.php file will be as follows:

      /var/www/html/tasks.php

      <?php
      try {
          $db_name="cron_jobs";
          $db_user="cron_jobs_user";
          $db_password = 'EXAMPLE_PASSWORD';
          $db_host="localhost";
      
          $pdo = new PDO('mysql:host=" . $db_host . "; dbname=" . $db_name, $db_user, $db_password);
          $pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);  
          $pdo->setAttribute(PDO::ATTR_EMULATE_PREPARES, false);               
      
          $loop_expiry_time = time() + 60;
      
          while (time() < $loop_expiry_time) { 
              $data = [];
              $sql  = "select 
                       task_id
                       from tasks
                       where is_processed = :is_processed
                       ";
      
              $data["is_processed'] = 'N';             
      
              $stmt = $pdo->prepare($sql);
              $stmt->execute($data);
      
              while ($row = $stmt->fetch(PDO::FETCH_ASSOC)) { 
                  $data_update   = [];         
                  $sql_update    = "update tasks set 
                                    is_processed  = :is_processed,
                                    completed_at  = :completed_at
                                    where task_id = :task_id                                 
                                    ";
      
                  $data_update   = [                
                                   'is_processed' => 'Y',                          
                                   'completed_at' => date("Y-m-d H:i:s"),
                                   'task_id'      => $row['task_id']                         
                                   ];
                  $stmt = $pdo->prepare($sql_update);
                  $stmt->execute($data_update);
              }   
      
              sleep(5); 
      
              }       
      
      } catch (PDOException $ex) {
          echo $ex->getMessage(); 
      }
      

      Save the file by pressing CTRL + X, Y then ENTER.

      Once you’ve completed coding the logic in the /var/www/html/tasks.php file, you’ll schedule the crontab daemon to execute the file after every 1 minute in the next step.

      Step 3 — Scheduling the PHP Script to Run After 1 Minute

      In Linux, you can schedule jobs to run automatically after a stipulated time by entering a command into the crontab file. In this step, you will instruct the crontab daemon to run your /var/www/html/tasks.php script after every minute. So, open the /etc/crontab file using nano:

      Then add the following toward the end of the file to execute the http://localhost/tasks.php after every 1 minute:

      /etc/crontab

      ...
      * * * * * root /usr/bin/wget -O - http://localhost/tasks.php
      

      Save and close the file.

      This guide assumes that you have a basic knowledge of how cron jobs work. Consider reading our guide on How to Use Cron to Automate Tasks on Ubuntu.

      As earlier indicated, although the cron daemon runs the tasks.php file after every 1 minute, once the file is executed for the first time, it will loop through the open tasks for another 60 seconds. By the time the loop time expires, the cron daemon will execute the file again and the process will continue.

      After updating and closing the /etc/crontab file, the crontab daemon should begin executing the MySQL tasks that you inserted in the tasks table immediately. To confirm whether everything is working as expected, you’ll query your cron_jobs database next.

      Step 4 — Confirming Job Execution

      In this step, you will open your database one more time to check whether the tasks.php file is processing queued jobs when executed automatically by the crontab.

      Log back in to your MySQL server as root:

      Then, enter your MySQL server’s root password and hit ENTER to proceed. Then, switch to the database:

      Output

      Database changed

      Run a SELECT statement against the tasks table:

      • SELECT task_id, task_name, queued_at, completed_at, is_processed FROM tasks;

      You will receive output similar to the following. In the completed_at column, tasks have been processed at intervals of 5 seconds. Also, the tasks have been marked as completed since the is_processed column is now set to Y, which means YES.

      Output

      +---------+-----------+---------------------+---------------------+--------------+ | task_id | task_name | queued_at | completed_at | is_processed | +---------+-----------+---------------------+---------------------+--------------+ | 1 | TASK 1 | 2021-03-06 06:27:19 | 2021-03-06 06:30:01 | Y | | 2 | TASK 2 | 2021-03-06 06:27:28 | 2021-03-06 06:30:06 | Y | | 3 | TASK 3 | 2021-03-06 06:27:36 | 2021-03-06 06:30:11 | Y | +---------+-----------+---------------------+---------------------+--------------+ 3 rows in set (0.00 sec)

      This confirms that your PHP script is working as expected; you have run tasks in a shorter time interval by overriding the limitation of the 1 minute time period set by the crontab daemon.

      Conclusion

      In this guide, you’ve set up a sample database on an Ubuntu 20.04 server. Then, you have created jobs in a table and run them at intervals of 5 seconds using the PHP while(...){...} loop and sleep() functions. Use the logic in this tutorial when you’re next implementing a job-queue-based application where tasks need to be run multiple times within a 1 minute time period.

      For more PHP tutorials, check out our PHP topic page.



      Source link

      How To Automate Jenkins Job Configuration Using Job DSL


      The author selected the Internet Archive to receive a donation as part of the Write for DOnations program.

      Introduction

      Jenkins is a popular automation server, often used to orchestrate continuous integration (CI) and continuous deployment (CD) workflows. However, the process of setting up Jenkins itself has traditionally been a manual, siloed process for the system administrator. The process typically involves installing dependencies, running the Jenkins server, configuring the server, defining pipelines, and configuring jobs.

      Then came the Everything as Code (EaC) paradigm, which allowed administrators to define these manual tasks as declarative code that can be version-controlled and automated. In previous tutorials, we covered how to define Jenkins pipelines as code using Jenkinsfiles, as well as how to install dependencies and define configuration of a Jenkins server as code using Docker and JCasC. But using only Docker, JCasC, and pipelines to set up your Jenkins instance would only get you so far—these servers would not come pre-loaded with any jobs, so someone would still have to configure them manually. The Job DSL plugin provides a solution, and allows you to configure Jenkins jobs as code.

      In this tutorial, you’ll use Job DSL to configure two demo jobs: one that prints a 'Hello World' message in the console, and one that runs a pipeline from a Git repository. If you follow the tutorial to the end, you will have a minimal Job DSL script that you can build on for your own use cases.

      Prerequisites

      To complete this tutorial, you will need:

      Step 1 — Installing the Job DSL Plugin

      The Job DSL plugin provides the Job DSL features you’ll use in this tutorial for your demo jobs. In this step, you will install the Job DSL plugin.

      First, navigate to your_jenkins_url/pluginManager/available. In the search box, type in Job DSL. Next, in the resulting plugins list, check the box next to Job DSL and click Install without restart.

      Plugin Manager page showing Job DSL checked

      Note: If searching for Job DSL returned no results, it either means the Job DSL plugin is already installed, or that your Jenkin server’s plugin list is not updated.

      You can check if the Job DSL plugin is already installed by navigating to your_jenkins_url/pluginManager/installed and searching for Job DSL.

      You can update your Jenkins server’s plugin list by navigating to your_jenkins_url/pluginManager/available and clicking on the Check Now button at the bottom of the (empty) plugins list.

      After initiating the installation process, you’ll be redirected to a page that shows the progress of the installation. Wait until you see Success next to both Job DSL and Loading plugin extensions before continuing to the next step.

      You’ve installed the Job DSL plugin. You are now ready to use Job DSL to configure jobs as code. In the next step, you will define a demo job inside a Job DSL script. You’ll then incorporate the script into a seed job, which, when executed, will create the jobs defined.

      Step 2 — Creating a Seed Job

      The seed job is a normal Jenkins job that runs the Job DSL script; in turn, the script contains instructions that create additional jobs. In short, the seed job is a job that creates more jobs. In this step, you will construct a Job DSL script and incorporate it into a seed job. The Job DSL script that you’ll define will create a single freestyle job that prints a 'Hello World!' message in the job’s console output.

      A Job DSL script consists of API methods provided by the Job DSL plugin; you can use these API methods to configure different aspects of a job, such as its type (freestyle versus pipeline jobs), build triggers, build parameters, post-build actions, and so on. You can find all supported methods on the API reference site.

      Jenkins Job DSL API Reference web page

      By default, the site shows the API methods for job configuration settings that are available as part of the core Jenkins installation, as well as settings that are enabled by 184 supported plugins (accurate as of v1.77). To get a clearer picture of what API methods the Job DSL plugin provides for only the core Jenkins installation, click on the funnel icon next to the search box, and then check and uncheck the Filter by Plugin checkbox to deselect all the plugins.

      Jenkins Job DSL API reference web page showing only the core APIs

      The list of API methods are now significantly reduced. The ones that remain would work even if the Jenkins installation had no plugins installed apart from the Job DSL plugin.

      For the ‘Hello World’ freestyle job, you need the job API method (freeStyleJob is an alias of job and would also work). Let’s navigate to the documentation for the job method.

      job API method reference

      Click the ellipsis icon () in job(String name) { … } to show the methods and blocks that are available within the job block.

      Expanded view of the job API method reference

      Let’s go over some of the most commonly used methods and blocks within the job block:

      • parameters: setting parameters for users to input when they create a new build of the job.
      • properties: static values that are to be used within the job.
      • scm: configuration for how to retrieve the source code from a source-control management provider like GitHub.
      • steps: definitions for each step of the build.
      • triggers: apart from manually creating a build, specifies in what situations the job should be run (for example, periodically like a cron job, or after some events like a push to a GitHub repository).

      You can further expand child blocks to see what methods and blocks are available within. Click on the ellipsis icon () in steps { … } to uncover the shell(String command) method, which you can use to run a shell script.

      Reference for the Job DSL steps block

      Putting the pieces together, you can write a Job DSL script like the following to create a freestyle job that, when run, will print 'Hello World!' in the output console.

      job('demo') {
          steps {
              shell('echo Hello World!')
          }
      }
      

      To run the Job DSL script, we must first incorporate it into a seed job.

      To create the seed job, go to your_jenkins_url, log in (if necessary), click the New Item link on the left of the dashboard. On the screen that follows, type in seed, select Freestyle project, and click OK.

      Part of the New Item screen where you give the item the name of 'seed' and with the 'Freestyle project' option selected

      In the screen that follows, scroll down to the Build section and click on the Add build step dropdown. Next select Process Job DSLs.

      Screen showing the Add build step dropdown expanded and the Process Job DSLs option selected

      Then, click on the radio button next to Use the provided DSL script, and paste the Job DSL script you wrote into the DSL Script text area.

      Job DSL script added to the Process Job DSLs build step

      Click Save to create the job. This will take you to the seed job page.

      Seed job page

      Then, navigate to your_jenkins_url and confirm that the seed job is there.

      Jenkins jobs list showing the seed job

      You’ve successfully created a seed job that incorporates your Job DSL script. In the next step, you will run the seed job so that new jobs are created based on your Job DSL script.

      Step 3 — Running the Seed Job

      In this step, you will run the seed job and confirm that the jobs defined within the Job DSL script are indeed created.

      First, click back into the seed job page and click on the Build Now button on the left to run the seed job.

      Refresh the page and you’ll see a new section that says Generated Items; it lists the demo job that you’ve specified in your Job DSL script.

      Seed job page showing a list of generated items from running the seed job

      Navigate to your_server_ip and you will find the demo job that you specified in the Job DSL script.

      Jenkins jobs list showing the demo and seed jobs

      Click the demo link to go to the demo job page. You’ll see Seed job: seed, indicating that this job is created by the seed job. Now, click the Build Now link to run the demo job once.

      Demo job page showing a section on seed job

      This creates an entry inside the Build History box. Hover over the date of the entry to reveal a little arrow; click on it to reveal the dropdown. From the dropdown, choose Console Output.

      Screen showing the Console Output option selected in the dropdown for Build #1 inside the Build History box

      This will bring you the logs and console output from this build. In it, you will find the line + echo Hello World! followed by Hello World!, which corresponds to the shell('echo Hello World!') step in your Job DSL script.

      Console output of build #1 showing the echo Hello World! command and output

      You’ve run the demo job and confirmed that the echo step specified in the Job DSL script was executed. In the next and final step, you will be modifying and re-applying the Job DSL script to include an additional pipeline job.

      Step 4 — Defining Pipeline Jobs

      In line with the Everything as Code paradigm, more and more developers are choosing to define their builds as pipeline jobs—those that use a pipeline script (typically named Jenkinsfile)—instead of freestyle jobs. The demo job you’ve defined so far is a small demonstration. In this step, you will define a more realistic job that pulls down a Git repository from GitHub and run a pipeline defined in one of its pipeline scripts.

      For Jenkins to pull a Git repository and build using pipeline scripts, you’ll need to install additional plugins. So, before you make any changes to the Job DSL script, first make sure that the required plugins are installed.

      Navigate to your_jenkins_url/pluginManager/installed and check the plugins lists for the presence of the Git, Pipeline: Job, and Pipeline: Groovy plugins. If any of them are not installed, go to your_jenkins_url/pluginManager/available and search for and select the plugins, then click Install without restart.

      Now that the required plugins are installed, let’s shift our focus to modifying your Job DSL script to include an additional pipeline job.

      We will be defining a pipeline job that pulls the code from the public jenkinsci/pipeline-examples Git repository and run the environmentInStage.groovy declarative pipeline script found in it.

      Once again, navigate to the Jenkins Job DSL API Reference, click the funnel icon to bring up the Filter by Plugin menu, then deselect all the plugins except Git, Pipeline: Job, and Pipeline: Groovy.

      The Jenkins Job DSL API Reference page with all plugins deselected except for Pipeline: Job, and (not shown) Git and Pipeline: Groovy

      Click on pipelineJob on the left-hand side menu and expand the pipelineJob(String name) { … } block, then, in order, the definition { … }, cpsScm { … }, and scm { … } blocks.

      Expanded view of the pipelineJob API method block

      There are comments above each API method that explain their roles. For our use case, you’d want to define your pipeline job using a pipeline script found inside a GitHub repository. So you’d need to modify your Job DSL script as follows:

      job('demo') {
          steps {
              shell('echo Hello World!')
          }
      }
      
      pipelineJob('github-demo') {
          definition {
              cpsScm {
                  scm {
                      git {
                          remote {
                              github('jenkinsci/pipeline-examples')
                          }
                      }
                  }
                  scriptPath('declarative-examples/simple-examples/environmentInStage.groovy')
              }
          }
      }
      

      To make the change, go to your_jenkins_url/job/seed/configure and find the DSL Script text area, and replace the contents with your new Job DSL script. Then press Save. In the next screen, click on Build Now to re-run the seed job.

      Then, go to the Console Output page of the new build and you’ll find Added items: GeneratedJob{name="github-demo"}, which means you’ve successfully added the new pipeline job, whilst the existing job remains unchanged.

      Console output for the modified seed job, showing that the github-demo job has been added

      You can confirm this by going to your_jenkins_url; you will find the github-demo job appear in the list of jobs.

      Job list showing the github-demo job

      Finally, confirm that your job is working as intended by navigating to your_jenkins_url/job/github-demo/ and clicking Build Now. After the build has finished, navigate to your_jenkins_url/job/github-demo/1/console and you will find the Console Output page showing that Jenkins has successfully cloned the repository and executed the pipeline script.

      Conclusion

      In this tutorial, you’ve used the Job DSL plugin to configure jobs on Jenkins servers in a consistent and repeatable way.

      But Job DSL is not the only tool in the Jenkins ecosystem that follows the Everything as Code (EaC) paradigm. You can also deploy Jenkins as Docker containers and set it up using Jenkins Configuration as Code (JCasC). Together, Docker, JCasC, Job DSL, and pipelines allow developers and administrators to deploy and configure Jenkins completely automatically, without any manual involvement.



      Source link