Benchmark Report: Docker Builds with Semaphore vs. Docker Hub

How many things feel as unproductive as waiting for a build? Not a lot of them I think. Docker build is a hungry beast; not only for computing resources but also for precious minutes of our lives.

What should we do about it? To find the answer, I benchmarked Semaphore and Docker Hub with a real use case scenario. I found that I can build my image at least 7 times faster on Semaphore. Read on to find out how I did it.

Continuous Integration: 10 Minutes or Bust

We’re not doing proper continuous integration unless our commit takes less than 10 minutes to build. Any longer than that and the integration feedback loop breaks down. Teams with slow pipelines merge fewer times a day, increasing the danger of conflicts. Besides, waiting is hard so we are tempted to multitask, which hurts productivity on the final account.

Most teams will do their builds as part of the CI/CD workflow; this is the correct way of doing things. Docker Hub includes for free the convenient autobuild feature, which takes a Dockerfile from your git repo and builds the image right into the registry. Convenience is nice, but we should also take into account how much of the 10 minutes budget it eats up.

The Benchmark Setup

The question I want to answer is simple: how long does it take from the moment I do a git push up to the moment the image is ready to use?

Here’s is my benchmark plan:

1. I set up a GitHub repository with the official Couchbase Dockerfile—this is the source for both build paths.
2. I created two repositories on Docker Hub:
   • docker-build: I connected it with the GitHub repo and enabled autobuild.
   • semaphore-build: I kept autobuild disabled in this one and used Semaphore to build and push to it.
3. Each GitHub push triggers both builds at the same time.

The couchbase image is around 500MB—quite above average. Its Dockerfile, however, is pretty standard, so the benchmark is likely to be representative of most workloads.

The source repository only needs to have a Dockerfile and the related scripts. This is how I set it up:

$ wget https://raw.githubusercontent.com/couchbase/docker/201af2d1fd4988d23d980cef5b91763ee5fdc9b7/enterprise/couchbase-server/6.0.3/Dockerfile
$ mkdir scripts
$ cd scripts
$ wget https://raw.githubusercontent.com/couchbase/docker/201af2d1fd4988d23d980cef5b91763ee5fdc9b7/enterprise/couchbase-server/6.0.3/scripts/run
$ wget https://raw.githubusercontent.com/couchbase/docker/201af2d1fd4988d23d980cef5b91763ee5fdc9b7/enterprise/couchbase-server/6.0.3/scripts/entrypoint.sh
$ wget https://raw.githubusercontent.com/couchbase/docker/201af2d1fd4988d23d980cef5b91763ee5fdc9b7/enterprise/couchbase-server/6.0.3/scripts/dummy.sh

For the course of this post, I’ll call build time to the total time from git push to image ready; this includes push/pull times, queue time, and docker build time.

I resorted to the git reflog to measure the push time:

$ git reflog --date=local master

To measure the image last modification time, I queried the Docker Hub API:

$ curl https://hub.docker.com/v2/repositories/$DOCKER_NAME/$REPO_NAME/ | jq .last_updated

Building With Cache

Build times can be reduced by using the cache feature, which reuses image layers from previous builds. In Docker Hub this feature is enabled by default when autobuild is configured. In Semaphore we must use docker pull and the cache-from options:

version: v1.0
name: Docker benchmark 
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804

blocks:
  - name: "Docker build"
    task:
      secrets:
        - name: dockerhub 
      jobs:
      - name: Pull, Build & Push
        commands:
          - echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
          - checkout
          - docker pull "$DOCKER_USERNAME"/semaphore-docker-benchmark-semaphore:latest || true
          - docker build --cache-from "$DOCKER_USERNAME"/semaphore-docker-benchmark-semaphore:latest -t "$DOCKER_USERNAME"/semaphore-docker-benchmark-semaphore:latest .
          - docker push "$DOCKER_USERNAME"/semaphore-docker-benchmark-semaphore:latest

I ran both builds simultaneously 11 times and discarded the first run. All the tests were done during US business hours. Here are the results:

Docker Hub builds consume almost half of the 10-minute budget. On average, Docker Hub jobs spent one or two minutes queued before starting.

Semaphore, on the other hand, starts the build immediately. Most of the time in the pipeline is spent doing the push and pull.

Median build time:

• Docker Hub: 265s
• Semaphore: 35s (x7.5 faster)

On average, the 1,300 monthly minutes included in Semaphore free plan is enough to build 1,900 couchbase images. For the pro plan, 100 builds should cost around 50 cents and take about an hour from your day. For comparison, a 100 builds would take 7:30 hours in Docker Hub—we can gain back a whole workday for the price of lemonade.

Without Cache

Sometimes Docker’s cache can be too aggressive. Docker will reuse the cache unless the Dockerfile has changed, which isn’t always what we want it to do. The surefire solution in those cases is to build from scratch.

I repeated the benchmark with the cache setting disabled in Docker Hub. I also modified the Semaphore pipeline to force a full build.

version: v1.0
name: Docker benchmark
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804

blocks:
  - name: "Docker build"
    task:
      secrets:
        - name: dockerhub
      jobs:
      - name: Pull, Build & Push
        commands:
          - echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
          - checkout
          - docker build -t "$DOCKER_USERNAME"/semaphore-docker-benchmark-semaphore:latest .
          - docker push "$DOCKER_USERNAME"/semaphore-docker-benchmark-semaphore:latest

These are the full build times:

Docker Hub has taken us well over the 10-minute limit. Semaphore keeps winning the benchmark and is 3.4 times faster.

• Docker hub: 665s per image – 100 builds take 21 hours.
• Semaphore: 194s per image – 100 builds take 5:30 hours.

The free plan lets us build at least 380 of these images per month. For those on the pro plan, 100 builds will only set you back $2.50. By switching from Docker Hub to Semaphore, we gain back 15 hours for the price of a cup of coffee.

I found that the sweet spot for this image can be found when I switch to the more the middle-tier e1-standard-4 machine. The cost rises to $3.26 per 100 builds, but the total time spent cuts back to 3:30 hours. These results, however, will vary depending on the Dockerfile.

Building Faster With Semaphore

At Semaphore our motto is “build great products at high velocity.” Each second you gain back by optimizing your CI/CD is multiplied hundreds or thousands of times over the project’s lifecycle.

The Docker Hub upgraded plan doesn’t offer faster builds; upgrading only allows concurrent builds and additional private repositories.

The Semaphore pro plan also enables concurrent builds and offers more powerful machine types. So, if you feel your Semaphore builds are still taking too long, you can try a faster machine.

What to learn more about Docker and Semaphore? Learn how to do powerful CI/CD for Docker and Kubernetes.