Docker is a container technology that enables developers to run entire applications as a unit. It offers all the benefits of virtual machines, without the high overhead:
- Consistency: Production and development environments are equal.
- Portability: fewer dependencies with the underlying OS; the same image can be deployed on any cloud provider.
- No overhead: better performance than virtual machines.
- Divide and conquer: distribute services among different containers.
However, Docker introduces a new variable to the equation: the app must be baked into the container image, then correctly deployed. In addition, setting up a test environment can prove more challenging.
Here is where a CI/CD platform can be of great value to us: by automating all tasks and giving us a fast and reliable environment to work in.
In this hands-on tutorial, we’ll learn how Semaphore can help us achieve all this in a few minutes. The tutorial is broken down into two sections:
- Continuous integration: build and test the app image.
- Continuous deployment: send the image to Heroku to run online.
Enter the Application
First, we will work with Semaphore’s ready-to-use Python Flask demo. The app consists of a simple task manager, written for the Flask web micro-framework, with a MongoDB acting as a database backend. To keep things nice and tidy, it has been split into two containers: one for the database and the other for the webserver.
The CI/CD pipelines will:
- Build a Docker image with our application.
- Push the image to Docker Hub.
- Test the application inside the container.
- Deploy it to Heroku.
While Semaphore will do most of the heavy lifting, you need to ensure to have the following things set up:
Then, grab a copy of the demo:
- Navigate to semaphore-demo-python-flask.
- Click on the Fork button.
- To get the git URL, click the Clone or download button.
- Use the git URL to copy the repository to your machine:
$ git clone http://github.com/...URL...
A well designed Continuous Integration setup will help us to:
- Spend less time testing and deploying.
- Get 100% automated testing.
- Avoid it-works-on-my-machine syndrome.
The objective of this section is to bake the app into a container. Ready to get started?
Sign up to Semaphore and Docker
Docker Hub provides free storage for images, so go ahead and get a Docker Hub account. You’ll also need a Semaphore account; by signing up with GitHub, you’ll get $10 worth of service time every month, which is enough for 1300 minutes of free build time.
In Semaphore, adding a project takes a couple of clicks:
- In the left-side navigation, click on + (plus sign) under Projects.
- Choose your GitHub repo and click Choose.
- Select the option I will use the existing configuration as the repository includes some ready-to-use CI/CD pipelines.
One last thing: Semaphore needs to be able to access your Docker Hub repository. The best way of storing sensitive data is by using Secrets, which are automatically encrypted and made available to jobs when called:
- On the left-side navigation menu, under Configuration, go to Secrets.
- Click the Create New Secret button.
- Name your secret “pyflask-semaphore” and type your Docker Hub credentials.
- Click on Save Changes.
Now, everything is in place to start integrating. How about a trial run? Semaphore will pick up the existing pipeline on the first push.
You can add an empty file from the command line with Git:
$ touch some_file $ git add some_file $ git commit -m "first run of the integration pipeline" $ git push
Or directly from GitHub:
- Click on Create new file.
- Type any file name. You can leave the contents of the file blank.
- Click on the Commit new file button.
Head back to Semaphore. in a few seconds, you’ll see the new workflow starting in your dashboard:
Inside the workflow, you’ll be able to see the pipeline:
Don’t mind the Promote button; we’ll get to it in a bit. The good news is that the integration pipeline is all green. If everything went according to plan you should have a new image in your Docker Repository:
We’re halfway there. But… what happened? How did Semaphore do all that?
Building an Image
Creating a custom Docker image is easily achieved with the right Dockerfile. The project already ships with a working
flask.Dockerfile. Take a look at the contents:
FROM python:3.8.7 ADD . ./opt/ WORKDIR /opt/ EXPOSE 5000 RUN pip install --upgrade pip RUN pip install -r requirements.txt CMD ["python","run.py"]
The first line defines the base image used as a starting point, in this case, a basic Debian image with Python 3.7. Next, we
ADDthe app files into the
RUN invokes pip, Python’s package manager, to install all the app dependencies. The final
CMDdefines how the app starts.
The container setup is completed with
version: '3.8' services: mongodb:
image: mongo:4.4.3-bioniccontainer_name: "mongodb" ports: - 27017:27017 command: mongod --logpath=/dev/null flasksemaphore: image: pyflasksemaphore container_name: semaphore-pyflask-docker_flasksemaphore_1 build: context: . dockerfile: ./flask.Dockerfile ports: - "5000:5000" volumes: - .:/opt/ environment: - DB=mongodb://mongodb:27017/tasks - PORT=5000 depends_on: - mongodb
With Compose, we define two services:
- flasksemaphore: the custom-built app container.
- mongodb: references a public MongoDB image.
Docker Compose can manage both containers as a unit:
- depends_on: defines the container starting order.
- environment variables:
DBpoints to the MongoDB service.
- build: references the
- ports and volumes: mappings between host and containers.
It only takes one command to build the image and start everything:
$ docker-compose up
The CI Pipeline
Next, let’s examine how is Semaphore implementing the integration pipeline. If you ever get lost, take a look at the Semaphore’s guided tour.
Click on the Edit Workflow button on the top-right corner of the pipeline to open the Workflow Builder:
Click on the CI pipeline to examine how it works:
Let’s break it down in more digestible bits.
Semaphore offers machines in several sizes. For our needs, the
e1-standard-2 with its 2 CPUs and 4GB of RAM is powerful enough to build a Docker image in a few seconds.
The super convenient Ubuntu 18.04 image includes everything you need to get started right away, including Docker, Docker Compose, Python, and Heroku. No additional components required.
Blocks and Jobs
Blocks define actions for the pipeline. Each block has a single task, and each task can have one or more jobs. Jobs within a block run concurrently, each one in its own fully isolated environment. Once all jobs in a block complete, the next block begins.
The “Build” block does exactly that; it builds the Docker image:
- Open the Secrets section to view the secrets imported in this block. pyflask-semaphore is the secret you created earlier with the Docker Hub credentials.
- Jobs have a name and a list of commands, one per line.
The Test Block
In this block, both containers are spun up to do integration tests. The prologue is executed before every job in a block. In this case, the prologue pulls the image and starts the app with docker-compose up.
We have two tests jobs:
- Run unit test: start a test script inside the container.
- Check running images: lists docker containers running.
We can create more pipelines and connect them with promotions. We can chain multiple pipelines with promotions to create complex workflows. Promotions can be manual or automatic. In this case, we have a manual connection to the deployment pipeline:
Optimizing the Build Job
Let’s take a few minutes to learn how we can make the build job a bit faster. The job spends a good chunk of time pulling the base images for MongoDB and Python from Docker Hub. To reduce this time, we can use the Semaphore Docker Registry, which offers faster downloads and pulling from it doesn’t count against the Docker pull limit.
To switch registries change:
docker-compose.yml should look like:
version: '3.8' services: mongodb: image: registry.semaphoreci.com/mongo:4.4 container_name: "mongodb" ports: - 27017:27017 command: mongod --logpath=/dev/null # --quiet flasksemaphore: image: pyflasksemaphore container_name: semaphore-pyflask-docker_flasksemaphore_1 build: context: . dockerfile: ./flask.Dockerfile ports: - "5000:5000" volumes: - .:/opt/ environment: - DB=mongodb://mongodb:27017/tasks - PORT=5000 depends_on: - mongodb
And Dockerfile should pull from
FROM registry.semaphoreci.com/python:3.8 ADD . ./opt/ WORKDIR /opt/ EXPOSE 5000 RUN pip install --upgrade pip RUN pip install -r requirements.txt CMD ["python","run.py"]
Finally, push the updated files to GitHub:
$ git pull origin master $ git add docker-compose.yml flask.Dockerfile $ git commit -m "use Semaphore registry" $ git push origin master
Once we have a working image, we’re ready to enter the continuous delivery stage. Here we’ll discuss how we can make deploy the app so our users can enjoy it.
We’re going to add two new services to our scheme:
Sign up for a Heroku account and get an authorization token:
- Sign up for a Heroku account.
- Click on your account profile, on the top right corner.
- Select Account Settings.
- Go to the Applications tab. Press the Create Authorization button:
5. Set the description as: “semaphore-demo-python-flask”. Leave Expires after blank:
6. Copy the authorization token:
Finally, create an empty application. From your Heroku dashboard, click the New button and select Create New App:
- Set your application name, or leave blank for a random one.
- Select your preferred zone: US or Europe.
In Semaphore, create a new secret called “heroku” to store the authorization token:
MongoDB Atlas Account
MongoDB Atlas offers a 500MB MongoDB database cluster for free. Not bad, not bad at all. However, the setup process is rather lengthy, so please bear with me:
- Sign up for a MongoDB Atlas Account
- Select AWS as a provider.
- Choose the region that matches your Heroku app:
- For the US: pick
- For Europe: pick
- For the US: pick
4. On Cluster Tier, select the M0 Sandbox:
5. You may set a name to describe your cluster. You can leave the rest of the settings alone.
6. Click on Create cluster. Give it a few minutes to provision.
Now, for the database user:
- On the left side navigation bar, open Security.
- Click on the +Add New User button.
- Add the user as follows:
- Username: “semaphore-demo-python-flask”
- Password: choose a secure password.
- User privileges: Read and write to any database.
4. Back in Security, select the IP Whitelist tab.
5. Click on the +Add IP Address button.
6. Choose Allow access from anywhere and Confirm:
Get the connection URI:
- Go back to Clusters from the left navigation bar.
- Click the Connect button for your cluster:
3. Select the Connect Your Application option:
4. Choose Python 3.6 or higher. Copy the entire connection string as shown.
The connection string is incomplete; replace
<password> with your actual password. If it has any special characters, you should first run it through URL Encode.
Head back to Semaphore to create a “mongodb-atlas” secret:
The CD Pipeline
In this section, we’re going to examine all the steps that the deployment pipeline goes through.
To view the deployment pipeline:
- In Semaphore, click on the demo project.
- Click on Edit Workflow to open the Workflow Builder.
- Scroll left, past the “Deploy to Heroku” promotion.
- Click on the “Deploy to Heroku” block to see how it works
Environment and Secrets
In the Deploy block, we only need to explicitly set a single variable,
$HEROKU_APP, which should point to your Heroku app name. Once you’ve done this, go ahead and replace the value with it.
The deployment block needs access to all services, and the variables are imported from secrets:
The only new commands introduced in this block are related to Heroku Docker:
- Tag the image with Heroku Registry URL.
- Send the URI variable for the MongoDB connection.
- Set the stack mode to container.
- Release gets the app started.
Only thing left to do is Run Workflow > Start.
After a few minutes, we should have the CI pipeline completed. Click on the Promote button to launch the CD pipeline:
Check your Heroku app dashboard. Then, the app should be online:
Finally, click on the Open app button to access the live application. Happy task managing!
We’ve explored how to run a Python application using Docker. After that, we learned how to use Semaphore to create a pipeline that automates running the tests and the necessary build commands, as well as deploying the application to Heroku.
Want to deliver continuously your applications made with Docker? Check out Semaphore’s Docker platform.