No More Seat Costs: Semaphore Plans Just Got Better!

    17 Feb 2020 · Software Engineering

    Running Applications on a Docker Swarm Mode Cluster

    21 min read
    Contents

    Introduction

    This tutorial is the last in a series of tutorials concerning the container orchestration tool, Docker Swarm. In this series, we’ve already seen how to establish a Swarm cluster, schedule services to the cluster’s nodes, make those services consumable from within and outside the cluster, and update and rollback services in-flight. We’ve mainly focused on the theory associated with a Swarm cluster, so in this final tutorial, we’ll get to grips with deploying a multi-service application.

    Sign up for free CI/CD with Kubernetes ebook

    Prerequisites

    In order to follow the tutorial, the following items are required:

    • a four-node Swarm Mode cluster, as detailed in the first tutorial of this series,
    • a single manager node (node-01), with three worker nodes (node-02, node-03, node-04), and
    • direct, command-line access to each node or access to a local Docker client configured to communicate with the Docker Engine on each node.

    The most straightforward configuration can be achieved by following the first tutorial. All commands are executed against the manager node, node-01, unless explicitly stated otherwise.

    The Application

    Our application was written by Jérôme Petazzoni, once a senior engineer at Docker, Inc., specifically for the purpose of demonstrating the deployment of a multi-service application to a cluster. It’s styled after Bitcoin mining, and is called Dockercoins.

    The first of the services that make up the application is rng, which uses the urandom device to return a number of random bytes in response to an HTTP request to do so. The rng service is a small Python script.

    The second service is a Ruby script, called hasher, which receives data in the form of an HTTP POST request, and returns a 256-bit hash of the data.

    The worker service, another small Python script, uses a perpetual loop to retrieve 32 bytes of random data from the rng service which it uses to generate a corresponding hash, courtesy of the hasher service. If the hash starts with a ‘0’, a Dockercoin has been mined, and the hash is written to a Redis back-end. The worker service also stores in the back-end a measure of the units of work done, which is the number of hashes generated during the last second.

    Finally, the webui service, written in Javascript, is used to present the hashing speed of the application by querying the Redis back-end, and displaying a graph in a web browser.

    Building the Images

    If we’re to deploy the Dockercoins application on the Swarm cluster that we’ve built, then we need the Docker images for each of the services. That means that we need to build each image using the source code, and an appropriate Dockerfile.

    We can start by retrieving the relevant artefacts from the GitHub repository using the manager node. We don’t need the entire repository, just the sub-directories that contain the code for each of the services, so we must configure git to do a ‘sparse checkout‘. Additionally, we must specify the sub-directories that will constitute the sparse checkout by writing them to the file .git/info/sparse-checkout. If you provisioned your Docker hosts using Docker Machine, you can open a shell on any of the nodes in the cluster using docker-machine ssh (e.g. docker-machine ssh node-01). On node-01:

    $ mkdir src && cd $_
    $ git init
    $ git remote add -f origin https://github.com/jpetazzo/container.training &> /dev/null
    $ git config core.sparseCheckout true
    $ for i in rng hasher worker webui; do echo "dockercoins/$i/" >> .git/info/sparse-checkout; done
    $ git pull --depth=1 origin master
    

    The artefacts for each service reside in a sub-directory of dockercoins:

    $ cd dockercoins
    $ ls -l
    total 16
    drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 24 11:12 hasher
    drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 24 11:12 rng
    drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 24 11:12 webui
    drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 24 11:12 worker
    

    Now, in order to build the images, we can set up a simple loop to build each image, and wait for them to complete:

    $ for i in rng hasher worker webui; do docker image build -t $i:v1 $i; done
    

    With the image builds completed, let’s check that we have all the components we need (note that we are customising the output of the command, using --format):

    $ docker image ls --format 'table {{.Repository}}\t{{.Tag}}\t{{.ID}}\t{{.Size}}'
    REPOSITORY          TAG                 IMAGE ID            SIZE
    webui               v1                  daf916dc7fe3        217MB
    worker              v1                  50641a6d3c11        97.3MB
    hasher              v1                  d5c00774674d        225MB
    rng                 v1                  3bd59f088af9        98.9MB
    ruby                alpine              308418a1844f        60.7MB
    python              alpine              29b5ce58cfbc        89.2MB
    node                4-slim              f7b2d40c7ef3        211MB
    

    Deploying a Registry

    All of the images have been successfully built, but they reside in the cache on node-01. We need each of the nodes in the cluster to have access to the images, not just node-01. We could push the images to the Docker Hub or some other remote registry, which would make them available to each node in the cluster.

    Instead, we’ll go for a local solution, and deploy a Docker registry on our cluster. You’ll remember from the third tutorial in this series – when we deploy a service with a published port on a Swarm cluster, the service can be consumed from any node in the cluster, irrespective of where the service’s tasks are deployed. If localhost is a resolvable name on each node, then it’s possible for each node to access the deployed registry service using the localhost name. Let’s see how to do this.

    The official Docker registry image is not a ‘production ready’ image, but can be used for a simple use-case, such as this tutorial. The registry API expects to be consumed on port 5000, so we can deploy the service with the following (use eval $(docker-machine env node-01) to point a local Docker client at the daemon on node-01 (or issue the command directly on node-01):

    $ docker service create --name registry --publish published=5000,target=5000 registry:2
    8uye7u1b2njbyvygurvdl8cy7
    overall progress: 1 out of 1 tasks
    1/1: running   
    verify: Service converged
    

    Now, let’s check to see which node the registry service has been scheduled on:

    $ docker service ps --format 'table {{.ID}}\t{{.Name}}\t{{.Node}}\t{{.CurrentState}}' registry
    ID                  NAME                NODE                CURRENT STATE
    pkdg39j93dmi        registry.1          node-02             Running 38 seconds ago
    

    In this example, it’s been scheduled on node-02, so let’s check that we can access the registry from another node in the cluster. Using another node in the cluster other than the one that your registry is running on, query the _catalog endpoint (on node-03, in this instance):

    $ curl localhost:5000/v2/_catalog
    {"repositories":[]}
    

    We get a response from the registry which indicates that we should be able to access the registry from any node in the cluster. The response shows that there are currently no repositories of images stored in the registry. Let’s fix this by pushing the images we built to the newly-created local registry.

    Pushing Images to a Local Registry

    To push the images to the registry service we need to tag the images with an appropriate name, so that it reflects the location of the registry. On node-01, where we built the images, they have the following names: rng:v1, hasher:v1, worker:v1, and webui:v1.

    Each image must be provided with an additional name of the form localhost:5000/<image>. For example, the rng image must be given the additional name localhost:5000/rng:v1. On node-01, again, where our images currently reside in the local cache, we can run the following loop to tag the images:

    $ for i in rng hasher worker webui; do docker image tag $i:v1 localhost:5000/$i:v1; done
    $ docker image ls --format 'table {{.Repository}}\t{{.Tag}}\t{{.ID}}\t{{.Size}}'
    REPOSITORY              TAG                 IMAGE ID            SIZE
    webui                   v1                  daf916dc7fe3        217MB
    localhost:5000/webui    v1                  daf916dc7fe3        217MB
    worker                  v1                  50641a6d3c11        97.3MB
    localhost:5000/worker   v1                  50641a6d3c11        97.3MB
    hasher                  v1                  d5c00774674d        225MB
    localhost:5000/hasher   v1                  d5c00774674d        225MB
    rng                     v1                  3bd59f088af9        98.9MB
    localhost:5000/rng      v1                  3bd59f088af9        98.9MB
    ruby                    alpine              308418a1844f        60.7MB
    python                  alpine              29b5ce58cfbc        89.2MB
    node                    4-slim              f7b2d40c7ef3        211MB
    

    As we can see from the image IDs, we still have one image for each of the Dockercoins services, but each has two different names to address it. On node-01 let’s push the images to the local registry service again:

    $ for i in rng hasher worker webui; do docker image push localhost:5000/$i:v1; done
    

    Now, if we return to the node where we issued the _catalog API request, node-03 in this instance, and repeat the request, we should get an altogether different response:

    $ curl localhost:5000/v2/_catalog
    {"repositories":["hasher","rng","webui","worker"]}
    

    This time, the registry returns the list of repositories associated with the images that we’ve just pushed from node-01. The images for the Dockercoins application are now available to all of the cluster’s nodes.

    Defining the Dockercoins Application

    It’s time to deploy the application onto our cluster. We could do this service-by-service, but this would be a little tedious, and it would be far more convenient if we could handle the services as one logical entity. Thankfully, Swarm mode enables us to do this using the ‘stack’ abstraction.

    If you’re familiar with Docker Compose, defining stacks is very similar to the definition of a multi-service composed application on a single Docker host. In fact, they use the same method for defining a multi-service application – a YAML-based configuration file called a ‘compose file’. The YAML-based syntax defining the composition of an application has evolved over time, but providing you use a compose syntax version greater than 3.0, then a compose file works with both the docker-compose utility and the Swarm command for deploying a stack, docker stack deploy. Some keys are ignored from the perspective of each command, when they apply to a single host or a Swarm, respectively. For example, the build key applies when using docker-compose, but not when using docker stack deploy, as Swarm only uses pre-built images.

    So, what do we need to put in the compose file for our Dockercoins stack? You can retrieve a copy of the file, and refer to it as we walk through its contents. On node-01:

    $ curl -sL https://git.io/vNodz > docker-compose.yml
    

    The first key in the file defines the version of the compose syntax:

    version: "3.5"

    If this key/value pair is missing from the compose file, then version 1 is assumed, and parsing of the file will be based on version 1 syntax which does not support many of the structures for the stack abstraction. At the time of writing this tutorial, version 3.5 is the latest syntax version.

    Next, we specify the networks that we want to use for the Dockercoins services:

    networks:
        front-end:
        mining:
        back-end:

    We’ve specified three different networks: the front-end network is where the webui service will run, the mining network is where the worker, rng and hasher services run, and back-end is where the redis service will run. Defining networks to fit the architecture of our application helps to segregate the services and their traffic. We could specify the type of driver to use for our networks, but in the absence of doing this, the type of network that gets created depends on whether we use docker-compose or docker stack deploy to parse the file. If it’s the latter, the inference is that each network is an overlay network which can span multiple Docker hosts.

    Networks are first-class objects in a Swarm environment, as are services. In order to define services we make use of the services key, placed at the beginning of a line to align with the version and networks keys:

    services:

    Under the services key, we define each of the stack’s services, indented by the same amount, as they are sibling ‘blocks’ in terms of YAML syntax. YAML syntax DOES NOT like tabs, so be sure to use spaces in the file when or if you edit a compose file.

    The first service definition is for the rng service:

        rng:
            image: localhost:5000/rng:${RNG_TAG-v1}
            networks:
                - mining
            deploy:
                mode: global

    The first key in the rng service definition is image, which defines the image to use when creating the service. The networks key defines the networks that the service will be attached to, whilst the deploy key, with its sub-key, mode, specifies the mode of deployment.

    The value of the image key is the full name of the image with a tag that is either the value of the RNG_TAG environment variable (which we can set in our shell environment), or v1. The rng service is attached to the mining network, as it doesn’t need to communicate with the webui or redis services. Finally, it has a deploy key, which sets the service mode to global. We’ve omitted this config declaration in the other services, as the default service mode is replicated. Other than some resilience, running the rng service in replicated mode has no effect on the application. If you stretch your imagination a little, however, I might be able to convince you that running an rng service on each cluster node will increase the performance of mining Dockercoins!

    Similar definitions exist for the other services that make up the application, with a couple of small differences. The webui service publishes a port (8000) on each cluster node, and the redis service uses the official Redis Docker image located on the Docker Hub rather than an image located on the local registry.

    Deploying the Dockercoins Application

    With the definition of the application stack complete, we can now deploy the service to the Swarm cluster using the docker stack deploy command. On the manager node, node-01:

    $ docker stack deploy -c docker-compose.yml dockercoins
    Creating network dockercoins_front-end
    Creating network dockercoins_mining
    Creating network dockercoins_back-end
    Creating service dockercoins_hasher
    Creating service dockercoins_worker
    Creating service dockercoins_webui
    Creating service dockercoins_redis
    Creating service dockercoins_rng
    

    What does this output show us? It shows the creation of the three networks we defined, as well as the five services that comprise the Dockercoins application. All objects are prepended with the stack name, dockercoins_. We can get some useful information about the services in the stack:

    $ docker stack services --format 'table {{.Name}}\t{{.Mode}}\t{{.Replicas}}\t{{.Image}}\t{{.Ports}}' dockercoins
    NAME                 MODE                REPLICAS            IMAGE                      PORTS
    dockercoins_worker   replicated          1/1                 localhost:5000/worker:v1   
    dockercoins_hasher   replicated          1/1                 localhost:5000/hasher:v1   
    dockercoins_redis    replicated          1/1                 redis:latest               
    dockercoins_rng      global              4/4                 localhost:5000/rng:v1      
    dockercoins_webui    replicated          1/1                 localhost:5000/webui:v1    *:8000->80/tcp
    

    We get a succinct description of each service, including it’s service mode, the number of tasks in the service, the image the service is based on, and the published ports. We can also get some information about the tasks associated with each service in the stack:

    $ docker stack ps --format 'table {{.Name}}\t{{.Node}}\t{{.CurrentState}}' dockercoins
    NAME                                        NODE                CURRENT STATE
    dockercoins_rng.7jv5xe23t04xf7oill5fw48c7   node-04             Running about a minute ago
    dockercoins_rng.qo2m2q97gukxfe9zdndrm0mwb   node-03             Running about a minute ago
    dockercoins_rng.ktjvlxcwrqn6oa8ynts81t040   node-02             Running about a minute ago
    dockercoins_rng.tdfw0x1cg6l0e5kba97wcmyva   node-01             Running about a minute ago
    dockercoins_redis.1                         node-01             Running about a minute ago
    dockercoins_webui.1                         node-03             Running about a minute ago
    dockercoins_worker.1                        node-01             Running about a minute ago
    dockercoins_hasher.1                        node-04             Running about a minute ago
    

    All seems well, so it’s time to check the application itself. We can do this by pointing a web browser at the public IP address or name of one of the cluster nodes (i.e. 52.56.199.243:8000). This is where the webui service renders the performance of the mining operation. We should see a mining speed of approximately 4.0 hashes/second. If you don’t see the output of the webui service, it may be that you need to configure ingress for port 8000 for the AWS security group. The first tutorial showed how to achieve this.

    In summary, we’ve been able to define a multi-service application and its deployment configuration within a compose file, and then use the docker stack deploy CLI command to deploy the application to a Swarm cluster. The deployment is not a ‘one-time’ restriction, and over time we might want to re-configure the deployment to reflect our changing needs. Let’s see how to do this.

    Scaling a Stack Service

    We could improve the mining speed by scaling the worker service. Scaling the service to 3 replicas will increase the mining speed to 12 hashes/second, but any further scaling has minimal impact. To do this, we can amend the definition of the service in the compose file, docker-compose.yml, to specify the number of replicas. If you’re re-creating this tutorial, when you edit the file, remember to use spaces to indent rather than tabs:

        worker:
            build: rng
            image: localhost:5000/worker:${WORKER_TAG-latest}
            networks:
                - mining
                - back-end
            deploy:
                replicas: 3

    With this change written to our compose file, we can simply re-run the docker stack deploy command to implement the change:

    $ docker stack deploy -c docker-compose.yml dockercoins
    Updating service dockercoins_webui (id: xkeauoqu4exwfkwn79w3o0rkm)
    Updating service dockercoins_redis (id: lqkx5abzpu2txt0h6du3ldz5o)
    Updating service dockercoins_rng (id: rv6bvx6mtm0nahqwxb4v80np5)
    Updating service dockercoins_hasher (id: jhcq1pumry7bj36bpuavzg58v)
    Updating service dockercoins_worker (id: gavgv4gzo37luzamudgipjzbx)
    

    Whilst the output shows us that each service has been updated, it’s only the worker service that has been affected. If we get a summary of the services, we can see the worker service now has 3 replicas:

    $ docker stack services --format 'table {{.Name}}\t{{.Mode}}\t{{.Replicas}}\t{{.Image}}\t{{.Ports}}' dockercoins
    NAME                 MODE                REPLICAS            IMAGE                      PORTS
    dockercoins_worker   replicated          3/3                 localhost:5000/worker:v1   
    dockercoins_hasher   replicated          1/1                 localhost:5000/hasher:v1   
    dockercoins_redis    replicated          1/1                 redis:latest               
    dockercoins_rng      global              4/4                 localhost:5000/rng:v1      
    dockercoins_webui    replicated          1/1                 localhost:5000/webui:v1    *:8000->80/tcp
    

    We can also see where these new tasks are running:

    $ docker stack ps --format 'table {{.Name}}\t{{.Node}}\t{{.CurrentState}}' dockercoins
    NAME                                        NODE                CURRENT STATE
    dockercoins_rng.7jv5xe23t04xf7oill5fw48c7   node-04             Running 7 minutes ago
    dockercoins_rng.qo2m2q97gukxfe9zdndrm0mwb   node-03             Running 7 minutes ago
    dockercoins_rng.ktjvlxcwrqn6oa8ynts81t040   node-02             Running 7 minutes ago
    dockercoins_rng.tdfw0x1cg6l0e5kba97wcmyva   node-01             Running 7 minutes ago
    dockercoins_redis.1                         node-01             Running 7 minutes ago
    dockercoins_webui.1                         node-03             Running 7 minutes ago
    dockercoins_worker.1                        node-01             Running 7 minutes ago
    dockercoins_hasher.1                        node-04             Running 7 minutes ago
    dockercoins_worker.2                        node-03             Running 2 minutes ago
    dockercoins_worker.3                        node-02             Running 2 minutes ago
    

    If you refresh your browser, you should also see that the mining speed has increased to a peak of 12 hashes/second.

    Configuring Service Update Policy

    Before we finish up with our Dockercoins stack, let’s attempt to update the image associated with the hasher service. You’ll remember from the fourth tutorial in our series, Updating Services in a Docker Swarm Mode Cluster, we can fine-tune the way that services are updated, and even rollback in the event of a failed update. Let’s configure some update ‘policy’ for the hasher service:

        hasher:         
            build: hasher
            image: localhost:5000/hasher:${HASHER_TAG-latest}
            networks:
                - mining
            deploy:
                replicas: 8
                update_config:                               
                    delay: 15s
                    parallelism: 1
                    monitor: 5s
                    failure_action: rollback
                    max_failure_ratio: 0.75

    Firstly, we’ve increased the number of replicas of the hasher service to 8 in order to adequately demonstrate our update policy. Then, we’ve configured some attributes regarding updates – there will be a 15 second delay between each update, an update will be applied to one task at a time, the health of each updated task will be monitored for 5 seconds after update, and the failure action will be to rollback when the failure ratio reaches 75%.

    Let’s go ahead and apply these changes:

    $ docker stack deploy -c docker-compose.yml dockercoins
    Updating service dockercoins_redis (id: lqkx5abzpu2txt0h6du3ldz5o)
    Updating service dockercoins_rng (id: rv6bvx6mtm0nahqwxb4v80np5)
    Updating service dockercoins_hasher (id: jhcq1pumry7bj36bpuavzg58v)
    Updating service dockercoins_worker (id: gavgv4gzo37luzamudgipjzbx)
    Updating service dockercoins_webui (id: xkeauoqu4exwfkwn79w3o0rkm)
    

    Simulating Update Failure with Healthchecks

    Our hasher service has a healthcheck defined in its Dockerfile (hasher/Dockerfile):

    <snip>
    HEALTHCHECK \
      --interval=1s --timeout=2s --retries=3 --start-period=1s \
      CMD curl http://localhost/ || exit 1
    

    A hasher service task will attempt to curl port 80 on the loopback interface, and if it’s unable to do this within a set of parameters, the container will be deemed unhealthy. The Ruby script for the hasher service is coded to bind on all interfaces, and listen on port 80. If we change the port, and build it into a revised Docker image, the healthcheck will fail. If we use the revised image as the basis for an update to the hasher service in the Dockercoins stack, we’ll be able to observe the update failing, and the consequential rollback to the previous configuration.

    Let’s go ahead and make the change to the source file:

    $ sed -i "s/80/81/" hasher/hasher.rb
    

    The new hasher script now needs to be encapsulated into a new Docker image for the hasher service, and then pushed to the registry deployed on the cluster:

    $ docker image build -t localhost:5000/hasher:v2 hasher
    <snip>
    $ docker image push localhost:5000/hasher:v2
    

    We can query the registry to retrieve the tags for the hasher repository:

    $ curl localhost:5000/v2/hasher/tags/list
    {"name":"hasher","tags":["v2","v1"]}
    

    We now have two image versions for the hasher service.

    In order to set our hasher service image update into motion, we need to set the HASHER_TAG environment variable so that when we execute the docker stack deploy command, the v2 image is applied instead of v1. We can follow this with the docker stack deploy command:

    $ export HASHER_TAG=v2
    $ docker stack deploy -c docker-compose.yml dockercoins
    Updating service dockercoins_redis (id: lqkx5abzpu2txt0h6du3ldz5o)
    Updating service dockercoins_rng (id: rv6bvx6mtm0nahqwxb4v80np5)
    Updating service dockercoins_hasher (id: jhcq1pumry7bj36bpuavzg58v)
    Updating service dockercoins_worker (id: gavgv4gzo37luzamudgipjzbx)
    Updating service dockercoins_webui (id: xkeauoqu4exwfkwn79w3o0rkm)
    

    We should check that the service update has been registered:

    $ docker stack services --format 'table {{.Name}}\t{{.Mode}}\t{{.Replicas}}\t{{.Image}}\t{{.Ports}}' dockercoins
    NAME                 MODE                REPLICAS            IMAGE                      PORTS
    dockercoins_worker   replicated          3/3                 localhost:5000/worker:v1   
    dockercoins_hasher   replicated          7/8                 localhost:5000/hasher:v2   
    dockercoins_redis    replicated          1/1                 redis:latest               
    dockercoins_rng      global              4/4                 localhost:5000/rng:v1      
    dockercoins_webui    replicated          1/1                 localhost:5000/webui:v1    *:8000->80/tcp
    

    The update to the hasher service has commenced – the image reported for the hasher service in the output to the docker stack services command reports its image as localhost:5000/hasher:v2.

    We can also track the progress of the update with the following command:

    $ watch -n 1 "docker service ps --filter "desired-state=running" --format 'table {{.ID}}\t{{.Image}}\t{{.Node}}' dockercoins_hasher"
    

    The update is applied one task at a time, each of which eventually fails the healthcheck, and when 6 of the 8 tasks are in an unhealthy state, the rollback to the previous state commences, culminating in all 8 replicas running the original localhost:5000/hasher:v1 image. Running docker stack services once more also shows the service image as localhost:5000/hasher:v1.

    $ docker stack services --format 'table {{.Name}}\t{{.Mode}}\t{{.Replicas}}\t{{.Image}}\t{{.Ports}}' dockercoins
    NAME                 MODE                REPLICAS            IMAGE                      PORTS
    dockercoins_worker   replicated          3/3                 localhost:5000/worker:v1   
    dockercoins_hasher   replicated          8/8                 localhost:5000/hasher:v1   
    dockercoins_redis    replicated          1/1                 redis:latest               
    dockercoins_rng      global              4/4                 localhost:5000/rng:v1      
    dockercoins_webui    replicated          1/1                 localhost:5000/webui:v1    *:8000->80/tcp
    

    To clean up, let’s remove the stack:

    $ docker stack rm dockercoins
    Removing service dockercoins_hasher
    Removing service dockercoins_worker
    Removing service dockercoins_webui
    Removing service dockercoins_redis
    Removing service dockercoins_rng
    Removing network dockercoins_front-end
    Removing network dockercoins_mining
    Removing network dockercoins_back-end
    

    Conclusion

    That concludes this series of tutorials on Docker Swarm. Over the course of the five tutorials, we’ve covered a lot of ground: the concepts associated with container orchestration, deployment of containerized services, consumption of those services, and their continued availability during updates. Of course, in tutorials, we can only paint a picture of an ideal world.

    What is it really like, to deploy and maintain a multi-service, containerized application on a multi-node cluster? If you have any experiences to share, or would like to comment or pose a question, then please get in touch via the comments section below.

    P.S. Want to continuously deliver your applications made with Docker? Check out Semaphore’s Docker platform with full layer caching for tagged Docker images.

    Read next:

    Leave a Reply

    Your email address will not be published. Required fields are marked *

    Avatar
    Writen by:
    Nigel is an independent Docker specialist who writes, teaches, and consults all things Docker-related. Based in the UK, he travels regularly, and can be found at windsock.io, and on GitHub.