Introduction
This tutorial is the last in a series of tutorials concerning the container orchestration tool, Docker Swarm. In this series, we’ve already seen how to establish a Swarm cluster, schedule services to the cluster’s nodes, make those services consumable from within and outside the cluster, and update and rollback services in-flight. We’ve mainly focused on the theory associated with a Swarm cluster, so in this final tutorial, we’ll get to grips with deploying a multi-service application.
Prerequisites
In order to follow the tutorial, the following items are required:
- a four-node Swarm Mode cluster, as detailed in the first tutorial of this series,
- a single manager node (
node-01), with three worker nodes (node-02,node-03,node-04), and - direct, command-line access to each node or access to a local Docker client configured to communicate with the Docker Engine on each node.
The most straightforward configuration can be achieved by following the first tutorial. All commands are executed against the manager node, node-01, unless explicitly stated otherwise.
The Application
Our application was written by Jérôme Petazzoni, once a senior engineer at Docker, Inc., specifically for the purpose of demonstrating the deployment of a multi-service application to a cluster. It’s styled after Bitcoin mining, and is called Dockercoins.
The first of the services that make up the application is rng, which uses the urandom device to return a number of random bytes in response to an HTTP request to do so. The rng service is a small Python script.
The second service is a Ruby script, called hasher, which receives data in the form of an HTTP POST request, and returns a 256-bit hash of the data.
The worker service, another small Python script, uses a perpetual loop to retrieve 32 bytes of random data from the rng service which it uses to generate a corresponding hash, courtesy of the hasher service. If the hash starts with a ‘0’, a Dockercoin has been mined, and the hash is written to a Redis back-end. The worker service also stores in the back-end a measure of the units of work done, which is the number of hashes generated during the last second.
Finally, the webui service, written in Javascript, is used to present the hashing speed of the application by querying the Redis back-end, and displaying a graph in a web browser.
Building the Images
If we’re to deploy the Dockercoins application on the Swarm cluster that we’ve built, then we need the Docker images for each of the services. That means that we need to build each image using the source code, and an appropriate Dockerfile.
We can start by retrieving the relevant artefacts from the GitHub repository using the manager node. We don’t need the entire repository, just the sub-directories that contain the code for each of the services, so we must configure git to do a ‘sparse checkout‘. Additionally, we must specify the sub-directories that will constitute the sparse checkout by writing them to the file .git/info/sparse-checkout. If you provisioned your Docker hosts using Docker Machine, you can open a shell on any of the nodes in the cluster using docker-machine ssh (e.g. docker-machine ssh node-01). On node-01:
$ mkdir src && cd $_
$ git init
$ git remote add -f origin https://github.com/jpetazzo/container.training &> /dev/null
$ git config core.sparseCheckout true
$ for i in rng hasher worker webui; do echo "dockercoins/$i/" >> .git/info/sparse-checkout; done
$ git pull --depth=1 origin master
The artefacts for each service reside in a sub-directory of dockercoins:
$ cd dockercoins
$ ls -l
total 16
drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 24 11:12 hasher
drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 24 11:12 rng
drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 24 11:12 webui
drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 24 11:12 worker
Now, in order to build the images, we can set up a simple loop to build each image, and wait for them to complete:
$ for i in rng hasher worker webui; do docker image build -t $i:v1 $i; done
With the image builds completed, let’s check that we have all the components we need (note that we are customising the output of the command, using --format):
$ docker image ls --format 'table {{.Repository}}\t{{.Tag}}\t{{.ID}}\t{{.Size}}'
REPOSITORY TAG IMAGE ID SIZE
webui v1 daf916dc7fe3 217MB
worker v1 50641a6d3c11 97.3MB
hasher v1 d5c00774674d 225MB
rng v1 3bd59f088af9 98.9MB
ruby alpine 308418a1844f 60.7MB
python alpine 29b5ce58cfbc 89.2MB
node 4-slim f7b2d40c7ef3 211MB
Deploying a Registry
All of the images have been successfully built, but they reside in the cache on node-01. We need each of the nodes in the cluster to have access to the images, not just node-01. We could push the images to the Docker Hub or some other remote registry, which would make them available to each node in the cluster.
Instead, we’ll go for a local solution, and deploy a Docker registry on our cluster. You’ll remember from the third tutorial in this series – when we deploy a service with a published port on a Swarm cluster, the service can be consumed from any node in the cluster, irrespective of where the service’s tasks are deployed. If localhost is a resolvable name on each node, then it’s possible for each node to access the deployed registry service using the localhost name. Let’s see how to do this.
The official Docker registry image is not a ‘production ready’ image, but can be used for a simple use-case, such as this tutorial. The registry API expects to be consumed on port 5000, so we can deploy the service with the following (use eval $(docker-machine env node-01) to point a local Docker client at the daemon on node-01 (or issue the command directly on node-01):
$ docker service create --name registry --publish published=5000,target=5000 registry:2
8uye7u1b2njbyvygurvdl8cy7
overall progress: 1 out of 1 tasks
1/1: running
verify: Service converged
Now, let’s check to see which node the registry service has been scheduled on:
$ docker service ps --format 'table {{.ID}}\t{{.Name}}\t{{.Node}}\t{{.CurrentState}}' registry
ID NAME NODE CURRENT STATE
pkdg39j93dmi registry.1 node-02 Running 38 seconds ago
In this example, it’s been scheduled on node-02, so let’s check that we can access the registry from another node in the cluster. Using another node in the cluster other than the one that your registry is running on, query the _catalog endpoint (on node-03, in this instance):
$ curl localhost:5000/v2/_catalog
{"repositories":[]}
We get a response from the registry which indicates that we should be able to access the registry from any node in the cluster. The response shows that there are currently no repositories of images stored in the registry. Let’s fix this by pushing the images we built to the newly-created local registry.
Pushing Images to a Local Registry
To push the images to the registry service we need to tag the images with an appropriate name, so that it reflects the location of the registry. On node-01, where we built the images, they have the following names: rng:v1, hasher:v1, worker:v1, and webui:v1.
Each image must be provided with an additional name of the form localhost:5000/<image>. For example, the rng image must be given the additional name localhost:5000/rng:v1. On node-01, again, where our images currently reside in the local cache, we can run the following loop to tag the images:
$ for i in rng hasher worker webui; do docker image tag $i:v1 localhost:5000/$i:v1; done
$ docker image ls --format 'table {{.Repository}}\t{{.Tag}}\t{{.ID}}\t{{.Size}}'
REPOSITORY TAG IMAGE ID SIZE
webui v1 daf916dc7fe3 217MB
localhost:5000/webui v1 daf916dc7fe3 217MB
worker v1 50641a6d3c11 97.3MB
localhost:5000/worker v1 50641a6d3c11 97.3MB
hasher v1 d5c00774674d 225MB
localhost:5000/hasher v1 d5c00774674d 225MB
rng v1 3bd59f088af9 98.9MB
localhost:5000/rng v1 3bd59f088af9 98.9MB
ruby alpine 308418a1844f 60.7MB
python alpine 29b5ce58cfbc 89.2MB
node 4-slim f7b2d40c7ef3 211MB
As we can see from the image IDs, we still have one image for each of the Dockercoins services, but each has two different names to address it. On node-01 let’s push the images to the local registry service again:
$ for i in rng hasher worker webui; do docker image push localhost:5000/$i:v1; done
Now, if we return to the node where we issued the _catalog API request, node-03 in this instance, and repeat the request, we should get an altogether different response:
$ curl localhost:5000/v2/_catalog
{"repositories":["hasher","rng","webui","worker"]}
This time, the registry returns the list of repositories associated with the images that we’ve just pushed from node-01. The images for the Dockercoins application are now available to all of the cluster’s nodes.
Defining the Dockercoins Application
It’s time to deploy the application onto our cluster. We could do this service-by-service, but this would be a little tedious, and it would be far more convenient if we could handle the services as one logical entity. Thankfully, Swarm mode enables us to do this using the ‘stack’ abstraction.
If you’re familiar with Docker Compose, defining stacks is very similar to the definition of a multi-service composed application on a single Docker host. In fact, they use the same method for defining a multi-service application – a YAML-based configuration file called a ‘compose file’. The YAML-based syntax defining the composition of an application has evolved over time, but providing you use a compose syntax version greater than 3.0, then a compose file works with both the docker-compose utility and the Swarm command for deploying a stack, docker stack deploy. Some keys are ignored from the perspective of each command, when they apply to a single host or a Swarm, respectively. For example, the build key applies when using docker-compose, but not when using docker stack deploy, as Swarm only uses pre-built images.
So, what do we need to put in the compose file for our Dockercoins stack? You can retrieve a copy of the file, and refer to it as we walk through its contents. On node-01:
$ curl -sL https://git.io/vNodz > docker-compose.yml
The first key in the file defines the version of the compose syntax:
version: "3.5"
If this key/value pair is missing from the compose file, then version 1 is assumed, and parsing of the file will be based on version 1 syntax which does not support many of the structures for the stack abstraction. At the time of writing this tutorial, version 3.5 is the latest syntax version.
Next, we specify the networks that we want to use for the Dockercoins services:
networks:
front-end:
mining:
back-end:
We’ve specified three different networks: the front-end network is where the webui service will run, the mining network is where the worker, rng and hasher services run, and back-end is where the redis service will run. Defining networks to fit the architecture of our application helps to segregate the services and their traffic. We could specify the type of driver to use for our networks, but in the absence of doing this, the type of network that gets created depends on whether we use docker-compose or docker stack deploy to parse the file. If it’s the latter, the inference is that each network is an overlay network which can span multiple Docker hosts.
Networks are first-class objects in a Swarm environment, as are services. In order to define services we make use of the services key, placed at the beginning of a line to align with the version and networks keys:
services:
Under the services key, we define each of the stack’s services, indented by the same amount, as they are sibling ‘blocks’ in terms of YAML syntax. YAML syntax DOES NOT like tabs, so be sure to use spaces in the file when or if you edit a compose file.
The first service definition is for the rng service:
rng:
image: localhost:5000/rng:${RNG_TAG-v1}
networks:
- mining
deploy:
mode: global
The first key in the rng service definition is image, which defines the image to use when creating the service. The networks key defines the networks that the service will be attached to, whilst the deploy key, with its sub-key, mode, specifies the mode of deployment.
The value of the image key is the full name of the image with a tag that is either the value of the RNG_TAG environment variable (which we can set in our shell environment), or v1. The rng service is attached to the mining network, as it doesn’t need to communicate with the webui or redis services. Finally, it has a deploy key, which sets the service mode to global. We’ve omitted this config declaration in the other services, as the default service mode is replicated. Other than some resilience, running the rng service in replicated mode has no effect on the application. If you stretch your imagination a little, however, I might be able to convince you that running an rng service on each cluster node will increase the performance of mining Dockercoins!
Similar definitions exist for the other services that make up the application, with a couple of small differences. The webui service publishes a port (8000) on each cluster node, and the redis service uses the official Redis Docker image located on the Docker Hub rather than an image located on the local registry.
Deploying the Dockercoins Application
With the definition of the application stack complete, we can now deploy the service to the Swarm cluster using the docker stack deploy command. On the manager node, node-01:
$ docker stack deploy -c docker-compose.yml dockercoins
Creating network dockercoins_front-end
Creating network dockercoins_mining
Creating network dockercoins_back-end
Creating service dockercoins_hasher
Creating service dockercoins_worker
Creating service dockercoins_webui
Creating service dockercoins_redis
Creating service dockercoins_rng
What does this output show us? It shows the creation of the three networks we defined, as well as the five services that comprise the Dockercoins application. All objects are prepended with the stack name, dockercoins_. We can get some useful information about the services in the stack:
$ docker stack services --format 'table {{.Name}}\t{{.Mode}}\t{{.Replicas}}\t{{.Image}}\t{{.Ports}}' dockercoins
NAME MODE REPLICAS IMAGE PORTS
dockercoins_worker replicated 1/1 localhost:5000/worker:v1
dockercoins_hasher replicated 1/1 localhost:5000/hasher:v1
dockercoins_redis replicated 1/1 redis:latest
dockercoins_rng global 4/4 localhost:5000/rng:v1
dockercoins_webui replicated 1/1 localhost:5000/webui:v1 *:8000->80/tcp
We get a succinct description of each service, including it’s service mode, the number of tasks in the service, the image the service is based on, and the published ports. We can also get some information about the tasks associated with each service in the stack:
$ docker stack ps --format 'table {{.Name}}\t{{.Node}}\t{{.CurrentState}}' dockercoins
NAME NODE CURRENT STATE
dockercoins_rng.7jv5xe23t04xf7oill5fw48c7 node-04 Running about a minute ago
dockercoins_rng.qo2m2q97gukxfe9zdndrm0mwb node-03 Running about a minute ago
dockercoins_rng.ktjvlxcwrqn6oa8ynts81t040 node-02 Running about a minute ago
dockercoins_rng.tdfw0x1cg6l0e5kba97wcmyva node-01 Running about a minute ago
dockercoins_redis.1 node-01 Running about a minute ago
dockercoins_webui.1 node-03 Running about a minute ago
dockercoins_worker.1 node-01 Running about a minute ago
dockercoins_hasher.1 node-04 Running about a minute ago
All seems well, so it’s time to check the application itself. We can do this by pointing a web browser at the public IP address or name of one of the cluster nodes (i.e. 52.56.199.243:8000). This is where the webui service renders the performance of the mining operation. We should see a mining speed of approximately 4.0 hashes/second. If you don’t see the output of the webui service, it may be that you need to configure ingress for port 8000 for the AWS security group. The first tutorial showed how to achieve this.
In summary, we’ve been able to define a multi-service application and its deployment configuration within a compose file, and then use the docker stack deploy CLI command to deploy the application to a Swarm cluster. The deployment is not a ‘one-time’ restriction, and over time we might want to re-configure the deployment to reflect our changing needs. Let’s see how to do this.
Scaling a Stack Service
We could improve the mining speed by scaling the worker service. Scaling the service to 3 replicas will increase the mining speed to 12 hashes/second, but any further scaling has minimal impact. To do this, we can amend the definition of the service in the compose file, docker-compose.yml, to specify the number of replicas. If you’re re-creating this tutorial, when you edit the file, remember to use spaces to indent rather than tabs:
worker:
build: rng
image: localhost:5000/worker:${WORKER_TAG-latest}
networks:
- mining
- back-end
deploy:
replicas: 3
With this change written to our compose file, we can simply re-run the docker stack deploy command to implement the change:
$ docker stack deploy -c docker-compose.yml dockercoins
Updating service dockercoins_webui (id: xkeauoqu4exwfkwn79w3o0rkm)
Updating service dockercoins_redis (id: lqkx5abzpu2txt0h6du3ldz5o)
Updating service dockercoins_rng (id: rv6bvx6mtm0nahqwxb4v80np5)
Updating service dockercoins_hasher (id: jhcq1pumry7bj36bpuavzg58v)
Updating service dockercoins_worker (id: gavgv4gzo37luzamudgipjzbx)
Whilst the output shows us that each service has been updated, it’s only the worker service that has been affected. If we get a summary of the services, we can see the worker service now has 3 replicas:
$ docker stack services --format 'table {{.Name}}\t{{.Mode}}\t{{.Replicas}}\t{{.Image}}\t{{.Ports}}' dockercoins
NAME MODE REPLICAS IMAGE PORTS
dockercoins_worker replicated 3/3 localhost:5000/worker:v1
dockercoins_hasher replicated 1/1 localhost:5000/hasher:v1
dockercoins_redis replicated 1/1 redis:latest
dockercoins_rng global 4/4 localhost:5000/rng:v1
dockercoins_webui replicated 1/1 localhost:5000/webui:v1 *:8000->80/tcp
We can also see where these new tasks are running:
$ docker stack ps --format 'table {{.Name}}\t{{.Node}}\t{{.CurrentState}}' dockercoins
NAME NODE CURRENT STATE
dockercoins_rng.7jv5xe23t04xf7oill5fw48c7 node-04 Running 7 minutes ago
dockercoins_rng.qo2m2q97gukxfe9zdndrm0mwb node-03 Running 7 minutes ago
dockercoins_rng.ktjvlxcwrqn6oa8ynts81t040 node-02 Running 7 minutes ago
dockercoins_rng.tdfw0x1cg6l0e5kba97wcmyva node-01 Running 7 minutes ago
dockercoins_redis.1 node-01 Running 7 minutes ago
dockercoins_webui.1 node-03 Running 7 minutes ago
dockercoins_worker.1 node-01 Running 7 minutes ago
dockercoins_hasher.1 node-04 Running 7 minutes ago
dockercoins_worker.2 node-03 Running 2 minutes ago
dockercoins_worker.3 node-02 Running 2 minutes ago
If you refresh your browser, you should also see that the mining speed has increased to a peak of 12 hashes/second.
Configuring Service Update Policy
Before we finish up with our Dockercoins stack, let’s attempt to update the image associated with the hasher service. You’ll remember from the fourth tutorial in our series, Updating Services in a Docker Swarm Mode Cluster, we can fine-tune the way that services are updated, and even rollback in the event of a failed update. Let’s configure some update ‘policy’ for the hasher service:
hasher:
build: hasher
image: localhost:5000/hasher:${HASHER_TAG-latest}
networks:
- mining
deploy:
replicas: 8
update_config:
delay: 15s
parallelism: 1
monitor: 5s
failure_action: rollback
max_failure_ratio: 0.75
Firstly, we’ve increased the number of replicas of the hasher service to 8 in order to adequately demonstrate our update policy. Then, we’ve configured some attributes regarding updates – there will be a 15 second delay between each update, an update will be applied to one task at a time, the health of each updated task will be monitored for 5 seconds after update, and the failure action will be to rollback when the failure ratio reaches 75%.
Let’s go ahead and apply these changes:
$ docker stack deploy -c docker-compose.yml dockercoins
Updating service dockercoins_redis (id: lqkx5abzpu2txt0h6du3ldz5o)
Updating service dockercoins_rng (id: rv6bvx6mtm0nahqwxb4v80np5)
Updating service dockercoins_hasher (id: jhcq1pumry7bj36bpuavzg58v)
Updating service dockercoins_worker (id: gavgv4gzo37luzamudgipjzbx)
Updating service dockercoins_webui (id: xkeauoqu4exwfkwn79w3o0rkm)
Simulating Update Failure with Healthchecks
Our hasher service has a healthcheck defined in its Dockerfile (hasher/Dockerfile):
<snip>
HEALTHCHECK \
--interval=1s --timeout=2s --retries=3 --start-period=1s \
CMD curl http://localhost/ || exit 1
A hasher service task will attempt to curl port 80 on the loopback interface, and if it’s unable to do this within a set of parameters, the container will be deemed unhealthy. The Ruby script for the hasher service is coded to bind on all interfaces, and listen on port 80. If we change the port, and build it into a revised Docker image, the healthcheck will fail. If we use the revised image as the basis for an update to the hasher service in the Dockercoins stack, we’ll be able to observe the update failing, and the consequential rollback to the previous configuration.
Let’s go ahead and make the change to the source file:
$ sed -i "s/80/81/" hasher/hasher.rb
The new hasher script now needs to be encapsulated into a new Docker image for the hasher service, and then pushed to the registry deployed on the cluster:
$ docker image build -t localhost:5000/hasher:v2 hasher
<snip>
$ docker image push localhost:5000/hasher:v2
We can query the registry to retrieve the tags for the hasher repository:
$ curl localhost:5000/v2/hasher/tags/list
{"name":"hasher","tags":["v2","v1"]}
We now have two image versions for the hasher service.
In order to set our hasher service image update into motion, we need to set the HASHER_TAG environment variable so that when we execute the docker stack deploy command, the v2 image is applied instead of v1. We can follow this with the docker stack deploy command:
$ export HASHER_TAG=v2
$ docker stack deploy -c docker-compose.yml dockercoins
Updating service dockercoins_redis (id: lqkx5abzpu2txt0h6du3ldz5o)
Updating service dockercoins_rng (id: rv6bvx6mtm0nahqwxb4v80np5)
Updating service dockercoins_hasher (id: jhcq1pumry7bj36bpuavzg58v)
Updating service dockercoins_worker (id: gavgv4gzo37luzamudgipjzbx)
Updating service dockercoins_webui (id: xkeauoqu4exwfkwn79w3o0rkm)
We should check that the service update has been registered:
$ docker stack services --format 'table {{.Name}}\t{{.Mode}}\t{{.Replicas}}\t{{.Image}}\t{{.Ports}}' dockercoins
NAME MODE REPLICAS IMAGE PORTS
dockercoins_worker replicated 3/3 localhost:5000/worker:v1
dockercoins_hasher replicated 7/8 localhost:5000/hasher:v2
dockercoins_redis replicated 1/1 redis:latest
dockercoins_rng global 4/4 localhost:5000/rng:v1
dockercoins_webui replicated 1/1 localhost:5000/webui:v1 *:8000->80/tcp
The update to the hasher service has commenced – the image reported for the hasher service in the output to the docker stack services command reports its image as localhost:5000/hasher:v2.
We can also track the progress of the update with the following command:
$ watch -n 1 "docker service ps --filter "desired-state=running" --format 'table {{.ID}}\t{{.Image}}\t{{.Node}}' dockercoins_hasher"
The update is applied one task at a time, each of which eventually fails the healthcheck, and when 6 of the 8 tasks are in an unhealthy state, the rollback to the previous state commences, culminating in all 8 replicas running the original localhost:5000/hasher:v1 image. Running docker stack services once more also shows the service image as localhost:5000/hasher:v1.
$ docker stack services --format 'table {{.Name}}\t{{.Mode}}\t{{.Replicas}}\t{{.Image}}\t{{.Ports}}' dockercoins
NAME MODE REPLICAS IMAGE PORTS
dockercoins_worker replicated 3/3 localhost:5000/worker:v1
dockercoins_hasher replicated 8/8 localhost:5000/hasher:v1
dockercoins_redis replicated 1/1 redis:latest
dockercoins_rng global 4/4 localhost:5000/rng:v1
dockercoins_webui replicated 1/1 localhost:5000/webui:v1 *:8000->80/tcp
To clean up, let’s remove the stack:
$ docker stack rm dockercoins
Removing service dockercoins_hasher
Removing service dockercoins_worker
Removing service dockercoins_webui
Removing service dockercoins_redis
Removing service dockercoins_rng
Removing network dockercoins_front-end
Removing network dockercoins_mining
Removing network dockercoins_back-end
Conclusion
That concludes this series of tutorials on Docker Swarm. Over the course of the five tutorials, we’ve covered a lot of ground: the concepts associated with container orchestration, deployment of containerized services, consumption of those services, and their continued availability during updates. Of course, in tutorials, we can only paint a picture of an ideal world.
What is it really like, to deploy and maintain a multi-service, containerized application on a multi-node cluster? If you have any experiences to share, or would like to comment or pose a question, then please get in touch via the comments section below.
P.S. Want to continuously deliver your applications made with Docker? Check out Semaphore’s Docker platform with full layer caching for tagged Docker images.
Read next:
