2 Dec 2020 · Software Engineering

    How to Run Cloud-based Cron Jobs with Semaphore

    6 min read
    Contents

    cron is a Unix job scheduler that runs commands periodically according to a schedule given by the user, known as the crontab. Scheduled jobs automate routine, repetitive tasks or tasks that you don’t want to run for every push.

    This tutorial shows how to use Semaphore’s workflow-scheduler/cron feature. The step-by-step example creates a project with a super-simple pipeline that runs a bash script to check that a website is up (returning a 200 HTTP response) every 5 minutes.

    So that it’s clear just how quick and easy Semaphore is to use, the example project in this tutorial is very, maybe excessively, simple. But don’t be fooled. Many Semaphore users implement Continuous Integration (CI) and Continuous Deployment (CD) flows for enterprise-grade software systems. For example, Semaphore builds Docker images 7x faster than Docker Hub.

    Prerequisites

    To follow this tutorial, you will need:

    • git, a GitHub account and a working knowledge of forking, branches, committing and pushing with git.
    • A Semaphore account. You can get one for free at semaphoreci.com.

    Let’s get started!

    Create your Website Healthcheck Repository in GitHub

    To get started, create a new repository in GitHub and add just one bash script, run_healthcheck.sh given below, to the repo. The script works as follows:

    • Send an HTTP request to an ENDPOINT (Google in the example) using curl.
    • Store the HTTP response code in status_code.
    • Check the response code and set the script’s exit code as follows:
    • 1 if the response code is not 200 (signifying a status of OK); or
    • 0 if the response code is 200.

    The script’s exit code is arguably the most important detail, as this is what Semaphore will check to determine the success or failure of a job that runs the script.

    #!/bin/bash
    
    # run_healthcheck.sh
    
    ENDPOINT=https://www.google.com
    
    status_code=$(curl --write-out %{http_code} --silent --output /dev/null $ENDPOINT)
    
    if [[ "$status_code" -ne 200 ]] ; then
      echo "$ENDPOINT status $status_code"
      exit 1
    else
      echo "$ENDPOINT OK"
      exit 0
    fi

    Create Your Semaphore Project and Choose a Starter Workflow

    • Log in to Semapahore
    • Use the + New Project button on the top toolbar to add a new project to your account
    • Select your repository from the list
    Repository chooser
    • Wait for Semaphore to set up the project and connect it to GitLab
    • When presented with the Invite People page, click Continue to workflow setup
    Continue to workflow setup button
    • Choose Single job as your starter workflow
    Choose workflow pane
    • In the workflow configuration pane, choose to Customize it
    Single job configuration pane

    Customize your Semaphore Workflow

    Here’s a very quick overview of the main modelling concepts used in Semaphore Workflows:

    • Ultimately, things happen when Semaphore runs your commands.
    • Commands are grouped into sequences within jobs.
    • Jobs are collected into blocks, where the jobs can run in parallel.
    • Finally, blocks are arranged into a pipeline, which sequences and triggers interdependent blocks.

    The Semaphore docs give more detail on the concepts Semaphore uses.

    For this tutorial, we want the pipeline to run a command that executes our run_healthcheck.sh script. For that, we need to create a single job in a single block.

    • Add a command to execute our script to the existing job.
      • Leave the existing checkout command in place. It is Semaphore’s pre-installed script for downloading your code from your Git repository.
      • Append ./run_healthcheck.sh to Job #1.
    Add command to job
    • Optionally, rename the job to something more meaningful
    Rename Job
    • You’ll see that, as you rename the job, Semaphore instantly updates the pipeline diagram to help you understand the entirety of the configuration
    Pipeline showing new job name

    Run the Workflow and Check the Output

    • Click Start first workflow. All workflows execute from version-controlled configuration stored in your repository, so you can always reproduce the result of a workflow. Semaphore displays a dialog where you can customise the branch and commit message that it will push to your repository.
    Run the workflow dialog
    • Clicking Looks good, Start will cause Semaphore to push the new .semaphore/semaphore.yml file to the named branch, which it will create if necessary. Semaphore runs the pipeline, and in a few seconds displays the result.
    Workflow run result

    The pipeline, with it’s single job, passed! Maybe you, like me, have had the experience where when something seems to work first time, nothing was happening at all? We can check that the job passed for the expected reason by checking the log.

    • Simply click on the Healthcheck Job to see the log.
    Healthcheck Job Log

    We can see, at line 73, the command that executes the run_healthcheck.sh script. Use the arrow to expand the output and it’s clear that the script ran successfully.

    Merge the Semaphore configuration topic branch to master

    Semaphore created a topic branch with the YAML configuration files. This allowed us to test the configuration before unleashing it on the master branch. Now that we’ve verified the pipeline behaviour, we can adopt the pipeline on our master branch, so:

    • Use GitHub or git in your shell to merge the new branch (named set-up-semaphore by default) to master.

    Schedule Your Pipeline to Run Every 5 Minutes

    The configuration we’ve done so far has created a pipeline that will be run (under Semaphore’s default settings) whenever there is a push to the repository. The next stage is for us to schedule the running of the pipeline every 5 minutes.

    • Return to your new project’s main page by clicking the project name in the Jump to project… drop-down.
    Jump to project drop-down
    • On the project’s page, click Schedulers
    Location of Schedulers
    • Click New Scheduler.
    • Fill in the Name of the Schedule field.
    • Set the Branch to master.
    • Set the Pipeline file to run field to be the file we created on the branch in the earlier steps (.semaphore/semaphore.yml by default).
    • Type a schedule into the When field for every 5 minutes using cron syntax: */5 * * * *. Follow the Crontab Guru link to get help with the syntax.
    cron schedule entry
    Scheduler
    • Click Create!

    That’s it! You’re done! To see the result of the scheduled run, return to your new project’s main page.

    You can see the history of results for a branch by clicking on the branch name.

    Conclusion

    You’ve seen how quick and simple it is to set up a Semaphore workflow from scratch and have it run according to a schedule.

    Currently, the results are visible from within your Semaphore account. An even more useful setup is to have Semaphore notify you should the workflow ever detect a failure. To find out more, take a look at this Introduction to Slack Integration and the Slack Notifications docs

    Thanks for reading this tutorial! Find out more about what Semaphore can do in the Continuous Delivery Blog!

    Leave a Reply

    Your email address will not be published. Required fields are marked *

    Avatar
    Writen by: