31 May 2017 ยท Software Engineering

    Managing Secrets in Docker Swarm

    10 min read


    Docker changed how development, testing and running software are done, and that’s why software engineers and companies should develop new habits and follow best practices. On the other hand, orchestration technologies like Docker Swarm, K8S or Mesos are also changing depending on the industry demands. Security management, including secret management in Docker Swarm, is one of the things that have evolved as a reaction to these changes.

    When working with Docker and orchestration tools, following security best practices is important. Software is becoming more distributed and with the rise of new development models, e.g. Microservices Architecture, communication between software components is adopting new patterns like message-based communication.

    In a production environment, containers running different services may exchange some critical information. When deploying a software to different environments, it is evident that storing critical information like production passwords in the development environment is not a good practice. There are multiple scenarios where we can notice the need of a standardized interface for accessing secrets.

    Datacenter Operating System, Kubernetes and other orchestration technologies have their own integrated secrets management solutions. In this tutorial, we re going to discover Docker native secret management.

    Continuous Delivery with Kubernetes

    Managing Secrets in Swarm Mode

    In this part of this tutorial, we are going to see how to create, attach and rotate sensitive data using Docker Secrets.

    How Docker Secrets Work

    Starting from the version 1.13, Docker users can use Docker Secrets in a Swarm cluster. Managers in Docker Swarm act as an authoritative delegation to coordinate secrets management.

    Secrets could be passwords, keys, certificates, sensitive environment data or any other custom data that a developer wants to protect e.g a database name, the administrator username, etc.

    Docker Secrets is only available in the Swarm mode, so standalone containers can not use this feature. Swarm Mode allows you to centrally manage sensitive data and messages by encrypting and transmitting them in a secure way and only to containers that need to access to it ( the principle of least-privilege).

    When a user adds a new secret to a Swarm cluster, this secret is sent to a manager using a TLS connection.

    Note: TLS is a cryptographic protocol that provides communications security over a network by providing communication encryption, privacy and data integrity.

    Docker Swarm Secrets Management With Raft

    In order to make all managers up to date with a new secret, when one manager node receives the secret, it saves it to a Raft store with a 256-bit key.

    Creating a Secret

    In order to see how everything works, let’s create a new Docker machine:

    docker-machine create --driver virtualbox secrets-testing
    > Running pre-create checks...
    > (secrets-testing) Default Boot2Docker ISO is out-of-date, downloading the latest release...
    > (secrets-testing) Latest release for github.com/boot2docker/boot2docker is v17.05.0-ce
    > (secrets-testing) Downloading /home/eon01/.docker/machine/cache/boot2docker.iso from https://github.com/boot2docker/boot2docker/releases/download/v17.05.0-ce/boot2docker.iso...
    > (secrets-testing) 0%....10%....20%....30%....40%....50%....60%....70%....80%....90%....100%
    > Creating machine...
    > (secrets-testing) Copying /home/eon01/.docker/machine/cache/boot2docker.iso to /home/eon01/.docker/machine/machines/secrets-testing/boot2docker.iso...
    > (secrets-testing) Creating VirtualBox VM...
    > (secrets-testing) Creating SSH key...
    > (secrets-testing) Starting the VM...
    > (secrets-testing) Check network to re-create if needed...
    > (secrets-testing) Waiting for an IP...
    > Waiting for machine to be running, this may take a few minutes...
    > Detecting operating system of created instance...
    > Waiting for SSH to be available...
    > Detecting the provisioner...
    > Provisioning with boot2docker...
    > Copying certs to the local machine directory...
    > Copying certs to the remote machine...
    > Setting Docker configuration on the remote daemon...
    > Checking connection to Docker...
    > Docker is up and running!
    > To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env secrets-testing

    You should see the created machine using :

    docker-machine ls
    > NAME              ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER        ERRORS
    > secrets-testing   -        virtualbox   Running   tcp://           v17.05.0-ce

    Let’s see the machine variables:

    docker-machine env secrets-testing
    > export DOCKER_TLS_VERIFY="1"
    > export DOCKER_HOST="tcp://"
    > export DOCKER_CERT_PATH="/home/eon01/.docker/machine/machines/secrets-testing"
    > export DOCKER_MACHINE_NAME="secrets-testing"

    If everything is ok, we can connect the current shell session to the new machine:

    eval "$(docker-machine env secrets-testing)"

    We can now start a Swarm cluster to run our tests. Using the IP address of the created machine, initialize the Swarm:

    docker swarm init --advertise-addr

    You should see something similar to:

    > Swarm initialized: current node (sqd5o5vj5c2r9gn53jd1uc4ig) is now a manager.
    > To add a worker to this swarm, run the following command:
    >     docker swarm join \
    >     --token SWMTKN-1-0atbw3c3geeyievi62dkgrw03pauh93rhfz41mdveo21ei1fb6-abf6azeaezaea \
    > To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

    Our manager is created, and you can create new machines and add new managers or workers of your choice.

    Now, let’s create a new random Docker secret. While the secret will be created using date |md5sum|awk '{print $1}', a Docker secret is created using this command:

    docker secret create [OPTIONS] SECRET file|-

    We can run a single command to create a Docker secret:

    date |md5sum|awk '{print $1}' | docker secret create my_secret -
    > qk4k9adgri6b3ubjua9nca5fp

    The created secret is called my_secret.

    When you type, docker secret ls you can get a view of the used secrets:

    > ID                          NAME                CREATED              UPDATED
    > qk4k9adgri6b3ubjua9nca5fp   my_secret           About a minute ago   About a minute ago

    qk4k9adgri6b3ubjua9nca5fp is a random and unique identifier of the recently created secret.

    If you want to have more information about my_secret, you can use:

    docker secret inspect  my_secret

    and you will see a JSON output with different other information such as the creation or the update date.

    > [
    >     {
    >         "ID": "qk4k9adgri6b3ubjua9nca5fp",
    >         "Version": {
    >             "Index": 10
    >         },
    >         "CreatedAt": "2017-05-22T13:25:23.815462834Z",
    >         "UpdatedAt": "2017-05-22T13:25:23.815462834Z",
    >         "Spec": {
    >             "Name": "my_secret",
    >             "Labels": {}
    >         }
    >     }
    > ]

    Our secret is now stored in the encrypted Raft logs.

    Consuming Docker Secrets

    Let’s create a new service that will be using the created secret ( my_secret ).

    docker service create --name my_test_app --secret my_secret eon01/infinite

    eon01/infinite is a Docker image that I created for this kind of tests that uses Alpine to run an infinite loop.

    Now, we have a running service named my_test_app using the secret my_secret. We can view it using docker service ls or even docker ps.

    docker ps
    > CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS              PORTS               NAMES
    > db38ae5f19fa        eon01/infinite:latest   "/bin/sh -c 'tail ..."   10 minutes ago      Up 10 minutes                           my_test_app.1.cc71i2rm0cl6pghn7z5ht7b98

    Only the created service is using the password, and since our service is running one container, only this container can see the secret. In other words, when a secret is attached to a service, only this service and its tasks have access to this sensitive data. The secret is now stored to a temporary file system (tmpfs) under /run/secrets/my_secret

    Tmpfs is the temporary file storage facility on many Unix-like operating systems stored in volatile memory instead of a persistent storage device.

    Since our container id is db38ae5f19fa, we can check the tmpfs content using:

    docker exec -it db38ae5f19fa cat /run/secrets/my_secret
    > 21ac8406497dded57ceafb7767e53633

    If you create other secrets for the same services, you can find them in the same directory under /run/secrets/<secret_name>

    Sharing Docker Secrets

    In order to see how sensitive data could be shared between two applications, we can create a new service called my_other_test_app and send it the created secret. In our case, my_other_test_app will use the same password but with a different name (my_other_secret):

    docker service create --name my_other_test_app --secret source=my_secret,target=my_other_secret,mode=0400 eon01/infinite

    We have set the mode to share the secret (0400) but it is possible also to add other options like UID or GID. Let’s get inside the container to verify the content of the create secret:

    docker exec -it 51fe104e416b cat /run/secrets/my_other_secret

    And we will find the same output as the previous command docker exec -it db38ae5f19fa cat /run/secrets/my_secret which is the first secret 21ac8406497dded57ceafb7767e53633.

    Rotating Docker Secrets

    Rotating sensitive data like passwords is a security best practice and it is possible using the Docker Swarm Secrets feature.

    Start by creating a new secret:

    date |md5sum|awk '{print $1}' | docker secret create my_new_secret -

    After that, the old secret should be deleted and updated by the new one and this is done in one command:

    docker service update --secret-rm my_secret --secret-add  source=my_new_secret,target=my_secret  my_test_app

    If you want to verify, you will find that the old secret (21ac8406497dded57ceafb7767e53633 in my case) has now a new value:

    docker exec -it 5152ba89ea7f cat /run/secrets/my_secret

    You can also inspect the new secret:

    docker secret inspect my_new_secret
    > [
    >     {
    >         "ID": "loame64czkbto3z87s37iy2ds",
    >         "Version": {
    >             "Index": 34
    >         },
    >         "CreatedAt": "2017-05-22T14:03:46.627206229Z",
    >         "UpdatedAt": "2017-05-22T14:03:46.627206229Z",
    >         "Spec": {
    >             "Name": "my_new_secret",
    >             "Labels": {}
    >         }
    >     }
    > ]

    Deploying Dockerized Applications with Semaphore

    To continuously deploy your project to Docker Swarm, you can use Semaphore’s hosted continuous integration and deployment service. Semaphore’s native Docker platform comes with the complete Docker CLI preinstalled and has full layer caching for tagged Docker images. Take a look at our other Docker tutorials for more information on deploying with Semaphore.


    We have seen a native feature in Swarm, it was introduced in Docker 1.13 after realizing how much important secret management is.

    All recent releases of Docker CLI and API support this feature but only in the Swarm mode where secrets are stored in the Raft log of the associated Swarm cluster and stored in a temporary filesystem inside a container.

    Secret management with Docker is useful: sensitive data is immutable, never written to a disk and never sent in clear text to the network. Other orchestration technologies like Kubernetes have different models of security and secrets management. If security is important in your project, you should compare these different models and choose the best fit for you.

    If you have any questions or comments, feel free to leave them in the section below.

    Read next:

    Leave a Reply

    Your email address will not be published. Required fields are marked *

    Writen by:
    Aymen is a Cloud/Software Architect. He wrote Painless Docker and is the creator of DevOpsLinks, a global community of thousands of IT experts and DevOps enthusiasts.