3 Aug 2023 · Software Engineering

    Continuous Deployment of a Python Flask Application with Docker and Semaphore

    12 min read
    Contents

    Docker is a container technology that enables developers to run entire applications as a unit. It offers all the benefits of virtual machines, without the high overhead:

    • Consistency: Production and development environments are equal.
    • Portability: fewer dependencies with the underlying OS; the same image can be deployed on any cloud provider.
    • No overhead: better performance than virtual machines.
    • Divide and conquer: distribute services among different containers.

    However, Docker introduces a new variable to the equation: the app must be baked into the container image, then correctly deployed. In addition, setting up a test environment can prove more challenging.

    Here is where a CI/CD platform can be of great value to us: by automating all tasks and giving us a fast and reliable environment to work in.

    In this hands-on tutorial, we’ll learn how Semaphore can help us achieve all this in a few minutes. The tutorial is broken down into two sections:

    Enter the Application

    First, we will work with Semaphore’s ready-to-use Python Flask demo. The app consists of a simple task manager, written for the Flask web micro-framework, with a MongoDB acting as a database backend. To keep things nice and tidy, it has been split into two containers: one for the database and the other for the webserver.

    The CI/CD pipelines will:

    • Build a Docker image with our application.
    • Push the image to Docker Hub.
    • Test the application inside the container.
    • Deploy it to Heroku.

    While Semaphore will do most of the heavy lifting, you need to ensure to have the following things set up:

    Then, grab a copy of the demo:

    1. Navigate to semaphore-demo-python-flask.
    2. Click on the Fork button.
    3. To get the git URL, click the Clone or download button.
    4. Use the git URL to copy the repository to your machine:
    $ git clone http://github.com/...URL...

    Continuous Integration

    A well designed Continuous Integration setup will help us to:

    • Spend less time testing and deploying.
    • Get 100% automated testing.
    • Avoid it-works-on-my-machine syndrome.

    The objective of this section is to bake the app into a container. Ready to get started?

    Sign up to Semaphore and Docker

    Docker Hub provides free storage for images, so go ahead and get a Docker Hub account. You’ll also need a Semaphore account; by signing up with GitHub, you’ll get $10 worth of service time every month, which is enough for 1300 minutes of free build time.

    In Semaphore, adding a project takes a couple of clicks:

    1. In the left-side navigation, click on + (plus sign) under Projects.
    2. Choose your GitHub repo and click Choose.
    3. Select the option I will use the existing configuration as the repository includes some ready-to-use CI/CD pipelines.
    Adding a repository to Semaphore

    One last thing: Semaphore needs to be able to access your Docker Hub repository. The best way of storing sensitive data is by using Secrets, which are automatically encrypted and made available to jobs when called:

    1. On the left-side navigation menu, under Configuration, go to Secrets.
    2. Click the Create New Secret button.
    3. Name your secret “pyflask-semaphore” and type your Docker Hub credentials.
    4. Click on Save Changes.
    Screenshot of creating a secret in Semaphore CI/CD platform

    Integrate

    Now, everything is in place to start integrating. How about a trial run? Semaphore will pick up the existing pipeline on the first push.

    You can add an empty file from the command line with Git:

    $ touch some_file
    $ git add some_file
    $ git commit -m "first run of the integration pipeline"
    $ git push

    Or directly from GitHub:

    • Click on Create new file.
    • Type any file name. You can leave the contents of the file blank.
    • Click on the Commit new file button.
    Creating a new file in GitHub

    Head back to Semaphore. in a few seconds, you’ll see the new workflow starting in your dashboard:

    First run on Semaphore

    Inside the workflow, you’ll be able to see the pipeline:

    A continuous integration pipeline on Semaphore.

    Don’t mind the Promote button; we’ll get to it in a bit. The good news is that the integration pipeline is all green. If everything went according to plan you should have a new image in your Docker Repository:

    Screenshot of new image in Docker repository

    We’re halfway there. But… what happened? How did Semaphore do all that?

    Building an Image

    Creating a custom Docker image is easily achieved with the right Dockerfile. The project already ships with a working flask.Dockerfile. Take a look at the contents:

    FROM python:3.8.7
    ADD . ./opt/
    WORKDIR /opt/
    EXPOSE 5000
    RUN pip install --upgrade pip
    RUN pip install -r requirements.txt
    CMD ["python","run.py"]

    The first line defines the base image used as a starting point, in this case, a basic Debian image with Python 3.7. Next, we ADDthe app files into the /opt dir. RUN invokes pip, Python’s package manager, to install all the app dependencies. The final CMDdefines how the app starts.

    The container setup is completed with docker-compose.yml:

    version: '3.8'
    services:
      mongodb:
        image: mongo:4.4.3-bionic
        container_name: "mongodb"
        ports:
          - 27017:27017
        command: mongod --logpath=/dev/null
      flasksemaphore:
        image: pyflasksemaphore
        container_name: semaphore-pyflask-docker_flasksemaphore_1
        build:
          context: .
          dockerfile: ./flask.Dockerfile
        ports:
          - "5000:5000"
        volumes:
          - .:/opt/
        environment:
          - DB=mongodb://mongodb:27017/tasks
          - PORT=5000
        depends_on:
          - mongodb

    With Compose, we define two services:

    • flasksemaphore: the custom-built app container.
    • mongodb: references a public MongoDB image.

    Docker Compose can manage both containers as a unit:

    • depends_on: defines the container starting order.
    • environment variables: DB points to the MongoDB service.
    • build: references the Dockerfile.
    • ports and volumes: mappings between host and containers.

    It only takes one command to build the image and start everything:

    $ docker-compose up

    The CI Pipeline

    Next, let’s examine how is Semaphore implementing the integration pipeline. If you ever get lost, take a look at the Semaphore’s guided tour.

    Click on the Edit Workflow button on the top-right corner of the pipeline to open the Workflow Builder:

    Opening the Workflow Builder

    Click on the CI pipeline to examine how it works:

    CI Pipeline configuration: Name and Agent

    Let’s break it down in more digestible bits.

    Agent

    The agent sets the machine type and the Operating System that drives the pipeline. To modify it, select the pipeline in the Workflow Builder and its configuration will appear on the right side.

    Semaphore offers machines in several sizes. For our needs, the e1-standard-2 with its 2 CPUs and 4GB of RAM is powerful enough to build a Docker image in a few seconds.

    The super convenient Ubuntu 18.04 image includes everything you need to get started right away, including Docker, Docker Compose, Python, and Heroku. No additional components required.

    Blocks and Jobs

    Blocks define actions for the pipeline. Each block has a single task, and each task can have one or more jobs. Jobs within a block run concurrently, each one in its own fully isolated environment. Once all jobs in a block complete, the next block begins.

    The “Build” block does exactly that; it builds the Docker image:

    • Open the Secrets section to view the secrets imported in this block. pyflask-semaphore is the secret you created earlier with the Docker Hub credentials.
    • Jobs have a name and a list of commands, one per line.
      • The checkout command clones GitHub repository.
      • docker login is required for pushing the image to Docker Hub.
      • docker-compose build creates the image…
      • …which is tagged, and finally pushed to the registry with docker push.
    The Build Block

    The Test Block

    In this block, both containers are spun up to do integration tests. The prologue is executed before every job in a block. In this case, the prologue pulls the image and starts the app with docker-compose up.

    We have two tests jobs:

    • Run unit test: start a test script inside the container.
    • Check running images: lists docker containers running.
    The Run & Test Block

    Promotions

    We can create more pipelines and connect them with promotions. We can chain multiple pipelines with promotions to create complex workflows. Promotions can be manual or automatic. In this case, we have a manual connection to the deployment pipeline:

    Promotion connecting pipelines

    Optimizing the Build Job

    Let’s take a few minutes to learn how we can make the build job a bit faster. The job spends a good chunk of time pulling the base images for MongoDB and Python from Docker Hub. To reduce this time, we can use the Semaphore Docker Registry, which offers faster downloads and pulling from it doesn’t count against the Docker pull limit.

    To switch registries change:

    • In docker-compose.yml
      • mongo:4.4.3-bionic ➡️ registry.semaphoreci.com/mongo:4.4 in
    • In flask.Dockerfile
      • FROM python:3.8.7 ➡️ FROM registry.semaphoreci.com/python:3.8

    The final docker-compose.yml should look like:

    version: '3.8'
     services:
       mongodb:
         image: registry.semaphoreci.com/mongo:4.4
         container_name: "mongodb"
         ports:
           - 27017:27017
         command: mongod --logpath=/dev/null # --quiet
       flasksemaphore:
         image: pyflasksemaphore
         container_name: semaphore-pyflask-docker_flasksemaphore_1
         build:
           context: .
           dockerfile: ./flask.Dockerfile
         ports:
           - "5000:5000"
         volumes:
           - .:/opt/
         environment:
           - DB=mongodb://mongodb:27017/tasks
           - PORT=5000
         depends_on:
           - mongodb

    And Dockerfile should pull from registry.semaphoreci.com.

    FROM registry.semaphoreci.com/python:3.8
     ADD . ./opt/
     WORKDIR /opt/
     EXPOSE 5000
     RUN pip install --upgrade pip
     RUN pip install -r requirements.txt
     CMD ["python","run.py"]

    Finally, push the updated files to GitHub:

    $ git pull origin master
    $ git add docker-compose.yml flask.Dockerfile
    $ git commit -m "use Semaphore registry"
    $ git push origin master

    Continuous Delivery

    Once we have a working image, we’re ready to enter the continuous delivery stage. Here we’ll discuss how we can make deploy the app so our users can enjoy it.

    We’re going to add two new services to our scheme:

    Heroku

    Sign up for a Heroku account and get an authorization token:

    1. Sign up for a Heroku account.
    2. Click on your account profile, on the top right corner.
    3. Select Account Settings.
    4. Go to the Applications tab. Press the Create Authorization button:
    Screenshot of Manage Account section in Heroku

    5. Set the description as: “semaphore-demo-python-flask”. Leave Expires after blank:

    Screenshot of creating an authorization in Heroku

    6. Copy the authorization token:

    Screenshot of authorization token in Heroku
    The authorization token will populate after you enter a description.

    Finally, create an empty application. From your Heroku dashboard, click the New button and select Create New App:

    • Set your application name, or leave blank for a random one.
    • Select your preferred zone: US or Europe.

    In Semaphore, create a new secret called “heroku” to store the authorization token:

    Screenshot of creating a heroku secret in CI/CD platform Semaphore
    Creating the heroku secret

    MongoDB Atlas Account

    MongoDB Atlas offers a 500MB MongoDB database cluster for free. Not bad, not bad at all. However, the setup process is rather lengthy, so please bear with me:

    1. Sign up for a MongoDB Atlas Account
    2. Select AWS as a provider.
    3. Choose the region that matches your Heroku app:
      • For the US: pick us-east-1
      • For Europe: pick eu-west-1
    Screenshot of MongoDC Atlas account setup

    4. On Cluster Tier, select the M0 Sandbox:

    Screenshot of cluster tier in MongoDB Atlas

    5. You may set a name to describe your cluster. You can leave the rest of the settings alone.

    6. Click on Create cluster. Give it a few minutes to provision.

    Now, for the database user:

    1. On the left side navigation bar, open Security.
    2. Click on the +Add New User button.
    3. Add the user as follows:
      • Username: “semaphore-demo-python-flask”
      • Password: choose a secure password.
      • User privileges: Read and write to any database.
    Creating a new user in MongoDB Atlas

    4. Back in Security, select the IP Whitelist tab.

    5. Click on the +Add IP Address button.

    6. Choose Allow access from anywhere and Confirm:

    Creating a security whitelist entry in MongoDB Atlas

    Get the connection URI:

    1. Go back to Clusters from the left navigation bar.
    2. Click the Connect button for your cluster:
    Connecting your application in MongoDB Atlas

    3. Select the Connect Your Application option:

    4. Choose Python 3.6 or higher. Copy the entire connection string as shown.

    The connection string is incomplete; replace <password> with your actual password. If it has any special characters, you should first run it through URL Encode.

    Head back to Semaphore to create a “mongodb-atlas” secret:

    Creating a MongoDB Atlas secret in CI/CD platform Semaphore
    Creating mongodb-atlas secret

    The CD Pipeline

    In this section, we’re going to examine all the steps that the deployment pipeline goes through.

    To view the deployment pipeline:

    • In Semaphore, click on the demo project.
    • Click on Edit Workflow to open the Workflow Builder.
    • Scroll left, past the “Deploy to Heroku” promotion.
    • Click on the “Deploy to Heroku” block to see how it works
    Deploy to Heroku block

    Environment and Secrets

    In the Deploy block, we only need to explicitly set a single variable, $HEROKU_APP, which should point to your Heroku app name. Once you’ve done this, go ahead and replace the value with it.

    The deployment block needs access to all services, and the variables are imported from secrets: mongodb-atlas,pyflask-semaphoreheroku.

    Deploy Block

    The only new commands introduced in this block are related to Heroku Docker:

    1. Tag the image with Heroku Registry URL.
    2. Send the URI variable for the MongoDB connection.
    3. Set the stack mode to container.
    4. Release gets the app started.

    Deploy

    Only thing left to do is Run Workflow > Start.

    Start deployment

    After a few minutes, we should have the CI pipeline completed. Click on the Promote button to launch the CD pipeline:

    The completed CI/CD pipelines in Semaphore.

    Check your Heroku app dashboard. Then, the app should be online:

    Checking your app status in Heroku dashboard
    What you will see if your app is online in Heroku.

    Finally, click on the Open app button to access the live application. Happy task managing!

    Message in Heroku showing a successfuly created application

    Conclusion

    We’ve explored how to run a Python application using Docker. After that, we learned how to use Semaphore to create a pipeline that automates running the tests and the necessary build commands, as well as deploying the application to Heroku.

    Want to deliver continuously your applications made with Docker? Check out Semaphore’s Docker platform.

    Leave a Reply

    Your email address will not be published. Required fields are marked *

    Avatar
    Writen by:
    I picked up most of my skills during the years I worked at IBM. Was a DBA, developer, and cloud engineer for a time. After that, I went into freelancing, where I found the passion for writing. Now, I'm a full-time writer at Semaphore.