Docs Navigation

Docker layer caching

How to use Docker layer caching

Once you've configured Semaphore's Docker integration, you can use the following snippet to build an image using --cache-from. A pre-existing image will be used as the cache source. The following commands should be part of your project's build settings:

export CACHE_IMAGE=aws_account_id.dkr.ecr.us-east-1.amazonaws.com/my-app

docker pull $CACHE_IMAGE:$BRANCH_NAME || true
docker build --cache-from $CACHE_IMAGE:$BRANCH_NAME --tag $CACHE_IMAGE:$BRANCH_NAME .
docker push $CACHE_IMAGE:$BRANCH_NAME

To ensure a good cache hit, you should choose Semaphore's environment variable BRANCH_NAME as a tag. This way, each branch will have its own cache to avoid cache collision.

The CACHE_IMAGE environment variable specifies the name of the Docker image that is used as a cache source. This is usually your main application image.

The first command attempts to pull the image from the registry. If this is the first time building this branch's image, this step will be skipped.

The second command builds a Docker image, and tags it with the branch name. If the tagged image is available, it is used as a cache source.

The last command pushes the resulting tagged Docker image along with its dependent layers to your container registry. Once pushed, it becomes available for reuse as a cache source in consequent builds. To automate the authentication process for your chosen registry, configure your project to integrate with one of the supported methods.

After completing these steps, you can tag and push images as you normally would.

Layer caching in Docker Compose

You can define one or more images to be used as cache sources using the following clause:

version: '3.2'

services:
  web:
    build:
      context: .
      cache_from:
        - ${CACHE_IMAGE}
    volumes:
     - .:/code

You can include cache_from in the build section of target services if your Docker Compose version is 1.12.0 or newer.

How Docker caching works

When building an image from a Dockerfile, each line generates its own layer. These layers are cached, and they can be reused if no changes are detected. If a layer changes, all the following layers have to be created from scratch.

Starting with Docker 1.13, the docker build command introduced a new switch --cache-from, which allows us to specify a tagged image as a cache source. More than one image can be specified with several --cache-from switches. Each gets a tagged image. The image resulting from the build can also be used as a cache source in another Docker build.

As a result of changes in Docker's caching mechanism, our docker-cache tool is deprecated. We recommend using --cache-from combined with a Docker container registry instead.

Best practices

When working with Docker images, it's convenient to export the name of the image as an environmental variable, or define it in your project's environment variables. This will allow you to write compact Docker commands.

To maximize layer re-use between builds, it's important to structure the Dockerfile so that frequently changing steps (eg. ADDing code) are towards the end, and less frequently changing steps (eg. RUNing apt-get install) are on top. This will ensure that the steps concerned with doing the same action are not unnecessarily rebuilt. See our guide for Lightweight Docker Images in 5 Steps for more details.

In order to improve the speed of pushing and pulling images in your Semaphore builds, the Docker container registry should be geographically as close as possible to the Semaphore build servers in Germany. To compare approximately how much time is needed to push or pull a Docker image, depending on the registry's physical location and the image's size, see the results of our benchmarks visualized in the following charts:

Time needed to push Docker image to registry Time needed to pull Docker image from registry

Note that local refers to a local Docker container registry, running inside the build environment. This functionality isn't shipped with the environment, and it serves only as a base line. To avoid any compression that might occur during the push and pull operations, image contents are generated using /dev/urandom. As such, these transfer speeds are not likely to occur, and standard images will have considerably better performance.

Semaphore Docs are open source — Edit on GitHub

Newsletter

Occasional lightweight product and blog updates. Unsubscribe at any time.

2009-2017 © Rendered Text. All rights reserved. Terms of Service, Privacy policy, Security.