Docker is a container technology that enables developers to run entire applications as a unit. It offers all the benefits of virtual machines, without the high overhead:

  • Consistency: Production and development environments are equal.
  • Portability: fewer dependencies with the underlying OS; the same image can be deployed on any cloud provider.
  • No overhead: better performance than virtual machines.
  • Divide and conquer: distribute services among different containers.

However, Docker introduces a new variable to the equation: the app must be baked into the container image, then correctly deployed. In addition, setting up a test environment can prove more challenging.

Here is where a CI/CD platform can be of great value to us: by automating all tasks and giving us a fast and reliable environment to work in.

In this hands-on tutorial, we’ll learn how Semaphore can help us achieve all this in a few minutes. The tutorial is broken down into two sections:

  • Continuous integration: build and test the app image.
  • Continuous deployment: send the image to Heroku to run online.

Enter the Application

First, we will work with Semaphore’s ready-to-use Python Flask demo. The app consists of a simple task manager, written for the Flask web micro-framework, with a MongoDB acting as a database backend. To keep things nice and tidy, it has been split into two containers: one for the database and the other for the web server.

The CI/CD pipelines will:

  • Build a Docker image with our application.
  • Push the image to Docker Hub.
  • Test the application inside the container.
  • Deploy it to Heroku.

While Semaphore will do most of the heavy lifting, you need to ensure to have the following things set up:

Then, grab a copy of the demo:

  1. Navigate to semaphore-demo-python-flask.
  2. Click on the Fork button.
  3. To get the git URL, click the Clone or download button.
  4. Use the git URL to copy the repository to your machine:
$ git clone http://github.com/...URL...

Continuous Integration

A well designed Continuous Integration setup will help us to:

  • Spend less time testing and deploying.
  • Get 100% automated testing.
  • Avoid it-works-on-my-machine syndrome.

The objective of this section is to bake the app into a container. Ready to get started?

Sign up to Semaphore and Docker

Docker Hub provides free storage for images, so go ahead and get a Docker Hub account. You’ll also need a Semaphoreaccount; by signing up with GitHub, you’ll get $20 worth of service time every month. Plenty for our purposes.

In Semaphore, adding a project takes a couple of clicks:

  1. In the left side navigation, click on New… under Projects.
  2. Choose your GitHub repo and click Add Repository.

One last thing: Semaphore needs to be able to access your Docker Hub repository. The best way of storing sensitive data is by using Secrets, which are automatically encrypted and made available to jobs when called:

  1. Under Configuration, go to Secrets.
  2. Click the Create New Secret button.
  3. Name your secret pyflask-semaphore and type your Docker Hub credentials:
Screenshot of creating a secret in Semaphore CI/CD platform

Integrate

Now, everything is in place to start integrating. How about a trial run? Edit any file in your repository and push the update:

$ touch some_file
$ git add some_file
$ git commit -m "first run of the integration pipeline"
$ git push

Head back to Semaphore. in a few seconds, you’ll see the new workflow starting in your dashboard:

Screenshot of new workflow starting in Semaphore CI/CD platform
What you should see in your Semaphore dashboard.

Inside the workflow, you’ll be able to see the pipeline:

Screenshot of CI/CD pipeline on Semaphore
A continuous integration pipeline on Semaphore.

Don’t mind the Promote button; we’ll get to it in a bit. The good news is that the integration pipeline is all green. If everything went according to plan you should have a new image in your Docker Repository:

Screenshot of new image in Docker repository

We’re halfway there. But… what happened? How did Semaphore do all that?

Building an Image

Creating a custom Docker image is easily achieved with the right Dockerfile. The project already ships with a working flask.Dockerfile. Take a look at the contents:

FROM python:3.7
ADD . ./opt/
WORKDIR /opt/
EXPOSE 5000
RUN pip install --upgrade pip
RUN pip install -r requirements.txt
CMD ["python","run.py"]

The first line defines the base image used as a starting point, in this case, a basic Debian image with Python 3.7. Next, we ADDthe app files into the /opt dir. RUN invokes pip, Python’s package manager, to install all the app dependencies. The final CMDdefines how the app starts.

The container setup is completed with docker-compose.yml:

version: '3.5'
services:
  mongodb:
    image: mongo:3.4.20
    container_name: "mongodb"
    ports:
      - 27017:27017
    command: mongod --smallfiles --logpath=/dev/null
  flasksemaphore:
    image: pyflasksemaphore
    container_name: semaphore-pyflask-docker_flasksemaphore_1
    build:
      context: .
      dockerfile: ./flask.Dockerfile
    ports:
      - "5000:5000"
    volumes:
      - .:/opt/
    environment:
      - DB=mongodb://mongodb:27017/tasks
      - PORT=5000
    depends_on:
      - mongodb

With Compose, we define two services:

  • flasksemaphore: the custom-built app container.
  • mongodb: references a public MongoDB image.

Docker Compose can manage both containers as a unit:

  • depends_on: defines the container starting order.
  • environment variables: DB points to the MongoDB service.
  • build: references the Dockerfile.
  • ports and volumes: mappings between host and containers.

It only takes one command to build the image and start everything:

$ docker-compose up

The CI Pipeline

Next, let’s examine how is Semaphore implementing the integration pipeline. If you ever get lost, take a look at the Semaphore’s guided tour.

version: v1.0
name: Semaphore Python / Flask / Docker Example Pipeline
agent:
  machine:
    # Use a machine type with more RAM and CPU power for faster container
    # builds:
    type: e1-standard-4
    os_image: ubuntu1804
blocks:
  - name: Build
    task:
      # Mount a secret which defines DOCKER_USERNAME and DOCKER_PASSWORD
      # environment variables.
      # For info on creating secrets, see:
      # https://docs.semaphoreci.com/article/66-environment-variables-and-secrets
      secrets:
        - name: pyflask-semaphore
      jobs:
      - name: Docker build
        commands:
          # Authenticate with Docker Hub
          - echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
          - checkout
          - docker-compose build
          - docker tag pyflasksemaphore:latest "$DOCKER_USERNAME"/pyflasksemaphore:latest
          - docker tag pyflasksemaphore:latest "$DOCKER_USERNAME"/pyflasksemaphore:$SEMAPHORE_WORKFLOW_ID
          - docker push "$DOCKER_USERNAME"/pyflasksemaphore:latest
          - docker push "$DOCKER_USERNAME"/pyflasksemaphore:$SEMAPHORE_WORKFLOW_ID
          - docker pull "$DOCKER_USERNAME"/pyflasksemaphore:$SEMAPHORE_WORKFLOW_ID
          - docker images

  - name: Run & Test Docker image
    task:
      # Mount a secret which defines DOCKER_USERNAME and DOCKER_PASSWORD
      # environment variables.
      # For info on creating secrets, see:
      # https://docs.semaphoreci.com/article/66-environment-variables-and-secrets
      secrets:
        - name: pyflask-semaphore
      prologue:
        commands:
          # Authenticate with Docker Hub
          - echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
          - checkout
          - docker pull "$DOCKER_USERNAME"/pyflasksemaphore
          - docker-compose up -d
      jobs:
      - name: Check Running Images
        commands:
          - docker ps
      - name: Run Unit test
        commands:
          - docker exec -it semaphore-pyflask-docker_flasksemaphore_1 python -m unittest

promotions:
  - name: Deploy to Heroku
    pipeline_file: deploy-heroku.yml
    auto_promote_on:
      - result: passed

Let’s break it down in more digestible bits.

Agent definition

The agent sets the machine type and the Operating System that drives the pipeline. Semaphore offers machines in several sizes. For our needs, the e1-standard-4 with its 4 CPUs and 8GB of RAM is powerful enough to build a Docker image in a few seconds.

The super convenient Ubuntu 18.04 image includes everything you need to get started right away, including Docker, Docker Compose, Python, and Heroku. No additional components required.

Build block

Blocks define actions for the pipeline. Each block has a single task, and each task can have one or more jobs. Jobs within a block run concurrently, each one in its own fully isolated virtual machine. Once all jobs in a block complete, the next block begins.

The build block does exactly that; it builds the Docker image:

  1. secrets imports the Docker Hub variables.
  2. checkout clones GitHub repository.
  3. docker login is required for pushing the image to Docker Hub.
  4. docker-compose builds the image…
  5. …which is tagged, and finally pushed to the registry.

Test block

In this block, both containers are spun up to do integration tests. The prologue is executed before every job in a block. In this case, the prologue pulls the image and starts the app with docker-compose up.

We have two tests jobs:

  • Run unit test: start a test script inside the container.
  • Check running images: lists docker containers running.

Promotions

The final part of .semaphore/semaphore.yml defines how the workflow continues. At this point, we can chain multiple pipelines with promotions to create complex workflows. Promotions can be manual or automatic. In this case, we have a manual connection to the deployment pipeline:

promotions:
  - name: Deploy to Heroku
    pipeline_file: deploy-heroku.yml

Continuous Delivery

Once we have a working image, we’re ready to enter the continuous delivery stage. Here we’ll discuss how we can make deploy the app so our users can enjoy it.

We’re going to add two new services to our scheme:

Heroku

Sign up for a Heroku account and get an authorization token:

  1. Sign up for a Heroku account.
  2. Click on your account profile, on the top right corner.
  3. Select Account Settings.
  4. Go to the Applications tab. Press the Create Authorization button:
Screenshot of Manage Account section in Heroku

5. Set the description as: semaphore-demo-python-flask. Leave Expires after blank:

Screenshot of creating an authorization in Heroku

6. Copy the authorization token:

Screenshot of authorization token in Heroku
The authorization token will populate after you enter a description.

Finally, create an empty application. From your Heroku dashboard, click the New button and select Create New App:

  • Set your application name, or leave blank for a random one.
  • Select your preferred zone: US or Europe.

In Semaphore, create a new secret called heroku to store the authorization token:

Screenshot of creating a heroku secret in CI/CD platform Semaphore

MongoDB Atlas Account

While Heroku has a MongoDB addon, it isn’t included on the free plan. 
Meanwhile, MongoDB Atlas offers a 500MB cluster for free. Not bad, not bad at all. However, the setup process is rather lengthy, so please bear with me:

  1. Sign up for a MongoDB Atlas Account
  2. Select AWS as a provider.
  3. Choose the region that matches your Heroku app:
    • For the US: pick us-east-1
    • For Europe: pick eu-west-1
Screenshot of MongoDC Atlas account setup

4. On Cluster Tier, select the M0 Sandbox:

Screenshot of cluster tier in MongoDB Atlas

5. You may set a name to describe your cluster. You can leave the rest of the settings alone.

6. Click on Create cluster. Give it a few minutes to provision.

Now, for the database user:

  1. On the left side navigation bar, open Security.
  2. Click on the +Add New User button.
  3. Add the user as follows:
    • Username: semaphore-demo-python-flask
    • Password: choose a secure password.
    • User privileges: Read and write to any database.
Creating a new user in MongoDB Atlas

4. Back in Security, select the IP Whitelist tab.

5. Click on the +Add IP Address button.

6. Choose Allow access from anywhere and Confirm:

Creating a security whitelist entry in MongoDB Atlas

Get the connection URI:

  1. Go back to Clusters from the left navigation bar.
  2. Click the Connect button for your cluster:
Connecting your application in MongoDB Atlas

3. Select the Connect Your Application option:

4. Choose Python 3.6 or later. Copy the entire connection string as shown.

The connection string is incomplete; replace <password> with your actual password. If it has any special characters, you should first run it through URL Encode.

Head back to Semaphore to create a mongodb-atlas secret:

Creating a MongoDB Atlas secret in CI/CD platform Semaphore

The CD Pipeline

In this section, we’re going to examine all the steps that the deployment pipeline goes through. Take a look at .semaphore/deploy-heroku.yml

# .semaphore/deploy-heroku.yml

version: v1.0
name: Deploy to Heroku
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804
blocks:
  - name: Deploy to Heroku
    task:
      secrets:
        # Make credentials available for jobs for MongoDB, Docker Hub and Heroku
        # For info on creating secrets, see:
        # https://docs.semaphoreci.com/article/66-environment-variables-and-secrets
        - name: mongodb-atlas
        - name: pyflask-semaphore
        - name: heroku
      # Define environment variables for the jobs on this block.
      # For info on environment variables, see:
      # https://docs.semaphoreci.com/article/66-environment-variables-and-secrets
      env_vars:
        - name: HEROKU_APP
          value: <YOUR_APP_NAME>
      jobs:
        - name: Deploy
          commands:
            - checkout
            # login to Docker Hub and pull the image
            - echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
            - docker pull "$DOCKER_USERNAME"/pyflasksemaphore:$SEMAPHORE_WORKFLOW_ID
            # push the image to Heroku registry
            - heroku container:login
            - docker tag "$DOCKER_USERNAME"/pyflasksemaphore:$SEMAPHORE_WORKFLOW_ID registry.heroku.com/$HEROKU_APP/web
            - docker push registry.heroku.com/$HEROKU_APP/web
            # define environment variable for MongoDB connection
            - heroku config:set DB="$MONGODB_URI"
            # release the app to production
            - heroku stack:set container --app $HEROKU_APP
            - heroku container:release web --app $HEROKU_APP

Environment and Secrets

We only need to explicitly set a single variable, $HEROKU_APP, which should point to your Heroku app name. Once you’ve done this, go ahead and replace the value with it.

The deployment block needs access to all services, and the variables are imported from secrets: mongodb-atlas,pyflask-semaphoreheroku.

Deploy Block

The only new commands introduced in this block are related to Heroku Docker:

  1. Tag the image with Heroku Registry URL.
  2. Send the URI variable for the MongoDB connection.
  3. Set the stack mode to container.
  4. Release: gets the app started.

Deploy

Only one more step to go! Commit the changes to get the CI/CD started:

$ git add .semaphore
$ git commit -m "ready to deploy!"
$ git push

After a few minutes, we should have the CI pipeline completed. Click on the Promote button to launch the CD pipeline:

Screenshot of the complete CI/CD pipeline in Semaphore
The completed CI pipeline in Semaphore.

Check your Heroku app dashboard. Then, the app should be online:

Checking your app status in Heroku dashboard
What you will see if your app is online in Heroku.

Finally, click on the Open app button to access the live application. Happy task managing!

Message in Heroku showing a successfuly created application

Conclusion

We’ve explored how to run a Python application using Docker. After that, we learned how to use Semaphore to create a pipeline that automates running the tests and the necessary build commands, as well as deploying the application to Heroku.

Want to deliver continuously your applications made with Docker? Check out Semaphore’s Docker platform.

Read next: