Have a look at our new Handbook: "Transitioning from Monolith to Microservices"!  Discover →

    15 Sep 2021 · Software Engineering

    Beyond Docker with Earthly

    11 min read
    Contents

    We’ve been using containers for software packaging for a long time. But let’s face it, Docker’s user experience isn’t really the greatest. Building and testing usually entail having a bunch of Makefiles, fiddly Dockerfiles, a bit of black magic, and a bundle of Bash scripts. It’s a path that can lead into insanity.

    And it’s a shame because Docker could be much friendlier. Earthly developers certainly believe in this: they have created a tool that fills the space between Docker and Make.

    At the end of this tutorial, you’ll know how Earthly works. As a bonus, you’ll have a pipeline that builds, tests, and pushes a Docker image into a remote repository.

    Final pipeline

    How Earthly works

    Earthly is a Docker-based build tool. It does not, however, replace language-specific tools like Maven, Gradle, or Webpack. Instead, it leverages and integrates with them — acting as a glue.

    Using Docker containers as the core mechanic to achieve repeatability, every Earthly command runs in an isolated environment and is effortlessly parallelized whenever possible.

    Earthly consists of the CLI binary and a Docker image called earthly/buildkit, based on Docker’s BuildKitd.

    Earthly components
    Earthly components

    Earthly devs want you to get the same experience everywhere. BuildKit’s role is to yield portable Docker-compatible layers. Thus, we get the same result whether on our local machine and in our CI systems.

    Does Earthly deliver on its promises?

    Earthly promises better, repeatable builds. But, does it deliver? Here are some of the best features and some of the problems found while doing a test drive.

    Pros:

    • Earthly, like Docker, is language-agnostic. Its syntax feels familiar since it mixes Make and Dockerfiles.
    • Their site has good documentation.
    • Configuration is low maintenance. There are only a few settings to tweak.
    • Useful even in the cases you don’t need Docker. Container images are only one of the possible outputs. For instance, you can use Earthly to test and compile binaries in a clean environment.
    • The import system works well in multirepo and monorepos.

    Cons:

    • It doesn’t actually replace Makefile or Docker Compose. And you will still need some shell scripts.
    • It’s slower than native Docker because it relies on copying files instead of mounts. Of course, this is by design to ensure repeatability.
    • Multiplatform builds sometimes use emulation, so it may be too slow for some use cases.
    • Error messages can get really confusing.
    Error message

    Setting up Earthly

    Earthly configuration files are called, as typical in software engineering, Earthfiles. Opening one reminds us of Dockerfiles with a twist of Makefiles.

    Earthly takes the syntax from Docker and expands it for a general use case. While Dockerfiles are only meant to produce container images, Earthly can generate artifacts.

    The following Earthfile was taken from Earthly’s official guide. It describes two targets: build and docker.

    FROM golang:1.15-alpine3.13
    WORKDIR /go-example
    
    build:
        COPY main.go .
        RUN go build -o build/go-example main.go
        SAVE ARTIFACT build/go-example /go-example AS LOCAL build/go-example
    
    docker:
        COPY +build/go-example .
        ENTRYPOINT ["/go-example/go-example"]
        SAVE IMAGE go-example:latest

    The build target compiles a Go program inside a container, while docker copies it into a production-ready image that can be pushed into any container registry and run anywhere.

    The FROM keyword works as you would expect if familiar with Dockerfiles. Likewise, COPYENTRYPOINTWORKDIR, and RUN, behave similarly. Two concepts deserve a pause to explain further:

    • targets: the thing we want to build. Earthly uses a target-based system inspired by Make.
    • references: targets can reference other targets. As in Make, Earthly will sort dependencies and do the right thing.
    • extended commands: Earthly features specialized commands such as SAVE IMAGE and SAVE ARTIFACT.

    To run this Earthfile we need to run: earthly +TARGET. The first time, Earthly will pull the image and create a cache volume. The cache will persist the build files between runs.

    $ earthly +build
    
               buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)
    golang:1.15-alpine3.13 | --> Load metadata linux/arm64
                 context | --> local context .
                   +base | --> FROM golang:1.15-alpine3.13
                 context | transferred 1 file(s) for context . (2.1 MB, 4 file/dir stats)
                   +base | *cached* --> WORKDIR /go-example
                  +build | *cached* --> COPY main.go .
                  +build | *cached* --> RUN go build -o build/go-example main.go
                  output | --> exporting outputs
    ================================ SUCCESS [main] ================================
                  +build | Artifact +build/go-example as local build/go-example
    

    Or we can directly go to the Docker generation target with:

    $ earthly +docker
    
                   +base | --> FROM golang:1.15-alpine3.13
                   +base | *cached* --> WORKDIR /go-example
                 context | transferred 3 file(s) for context . (2.1 MB, 4 file/dir stats)
                  +build | *cached* --> COPY main.go .
                  +build | *cached* --> RUN go build -o build/go-example main.go
                  +build | *cached* --> SAVE ARTIFACT build/go-example +build/go-example AS LOCAL build/go-example
                 +docker | --> COPY +build/go-example ./
                  output | [██████████] exporting layers ... 100%
                  output | [          ] exporting manifest
    ================================ SUCCESS [main] ================================
                 +docker | Image +docker as go-example:latest
                  +build | Artifact +build/go-example as local build/go-example
    

    Since the docker target depends on build, Earthly first runs the build step and then proceeds to build and export the image into the local Docker directory.

    Extending Earthfiles

    Let’s try Earthly in a more involved example. Please go ahead and fork this demo repository.

    The provided Dockerfile builds a Node Alpine container image.

    FROM node:14.17-alpine3.12
    WORKDIR /app
    COPY package.json package-lock.json ./
    RUN npm install
    COPY src src
    EXPOSE 3000
    ENTRYPOINT ["npm", "start"]

    You can build it with: docker build . -t mydemo and then run it as: docker run -it -p 3000:3000 mydemo. Browsing the localhost on port 3000 should print a “hello, world” message.

    How would we go about turning this into a working Earthfile? Let’s start one from scratch.

    The first two lines remain the same and will be used for every target.

    FROM node:14.17-alpine3.12
    WORKDIR /app

    Now we add the build target, which downloads dependencies and copies the source.

    build:
        COPY package.json package-lock.json ./
        COPY --if-exists node_modules node_modules
        RUN npm install
        COPY src src

    We have added a COPY statement to copy the node_modules folder into the build environment.

    Since npm runs in the container, we need to copy its output files back outside. We do this with Earthly’s special command SAVE ARTIFACT ... AS LOCAL.

        SAVE ARTIFACT node_modules AS LOCAL ./node_modules
        SAVE ARTIFACT package.json AS LOCAL ./package.json
        SAVE ARTIFACT package-lock.json AS LOCAL ./package-lock.json

    Running earthly +build should install the Node dependencies. The end result of this process is similar to what we would get if we run npm install directly, with the added benefit that we get the same outcome everywhere.

    Next, we should create the Docker image. We can achieve this with:

    docker:
        FROM +build
        EXPOSE 3000
        ENTRYPOINT ["npm", "start"]
        SAVE IMAGE semaphore-demo-earthly:latest

    The new target picks up from where the build stopped. It adds the start command and exports the image into the host’s local Docker registry.

    Testing with Earthly

    One of the most powerful features Earthly brings is the ability to extend Dockerfiles with tests. Let’s add unit tests by extending build with a target that runs: npm test.

    tests:
        FROM +build
        COPY spec spec
        RUN npm test

    Now we can run earthly +tests to get the results.

    Adding a linting target is also trivial:

    lint:
        FROM +build
        COPY .jshintrc ./
        RUN npm run lint

    To run more involved tests, such as integration tests or end-to-end tests, we can reuse existing Docker Compose manifests in order to bring up other containers during the integration stage.

    integration-tests:
        FROM +build
        COPY docker-compose.yml ./
        COPY integration-tests integration-tests
        WITH DOCKER --compose docker-compose.yml --service db
            RUN sleep 10 && npm run integration-tests
        END

    Earthly uses a Docker-in-Docker approach to start helper containers:

    1. Copies docker-compose.yml and the integration tests in the main container (the application container).
    2. Starts a PostgreSQL container inside the main container. The special WITH DOCKER command accepts precisely one RUN statement.
    3. Runs the integration tests.

    Our Earthfile is good enough for now. Let’s upload it into the repository so we can configure a CI/CD pipeline in the next section.

    $ git add Earthfile
    $ git commit -m "push Earthfile"
    $ git push origin master

    Earthly Continuous Integration Pipeline

    We are now all set to test the application with Earthly and continuous integration. Our pipeline will pull to and push from Docker Hub, something that will require authentication with the service.

    Authentication in Semaphore happens via secrets. To create an encrypted secret, go to your organization menu and click on Settings > Secrets. Then, press Create secret and type the variables DOCKER_USERNAME and DOCKER_PASSWORD as shown below:

    Docker Hub secret
    Docker Hub Secret

    Next, initialize the forked demo repository. The procedure for adding a project is detailed in the getting started guide. Go check it out if this is the first time using Semaphore.

    Since Earthly doesn’t come preinstalled in the Ubuntu CI image, we need to install it for every job. The simplest way of achieving this is with a pipeline-level prologue. The commands in the global prologue are executed before every job in the pipeline.

    The Earthly installation command for Linux is:

    sudo /bin/sh -c 'wget https://github.com/earthly/earthly/releases/latest/download/earthly-linux-amd64 -O /usr/local/bin/earthly && chmod +x /usr/local/bin/earthly && /usr/local/bin/earthly bootstrap --with-autocomplete'

    Click on the pipeline and type the line in the prologue section.

    Pipeline config

    Build block

    The first block will start a build stage. In it, we will download and cache Node dependencies.

    Type the following commands in the job:

    echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
    checkout
    cache restore
    earthly --ci +build
    cache store

    We’ll run these commands or a variation of them in every successive job in the pipeline:

    • docker login: logs into Docker Hub with the credentials defined in the secret.
    • checkout: clones the repository into the CI machine.
    • cache: With the cache command, we can store node_modules in the project-level storage provided by Semaphore.
    • earthly ci: runs Earthly with some CI-optimized settings.
    Build block

    To finish the block, enable the “dockerhub” secret, so the variable is decrypted and exported in the job.

    Tests block

    The next block will run fast tests such as unit tests and linting. Add the following commands in the block’s prologue:

    echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
    checkout
    cache restore
    New block

    Next, create two jobs:

    earthly --ci +tests

    And:

    earthly --ci +lint
    Test block

    Integration tests block

    The last job will run the integration test. Type the following commands. The -P switch is needed to enable privileged mode inside Docker (needed for Docker-in-Docker).

    echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
    checkout
    cache restore
    earthly --ci -P +integration-tests
    Integration test block

    Don’t forget to check that the secret is enabled in all blocks. When ready, run the workflow to try the CI pipeline.

    Try CI pipeline
    CI pipeline
    CI pipeline

    Continuous Delivery with Earthly

    The final result of a CI/CD pipeline is to deliver something that can be deployed. Here’s where the continuous delivery part comes into play. We’ll push the image to a remote repository so it can be later deployed in production.

    A Semaphore workflow can span multiple pipelines, as long as they’re connected by promotions. Go back to the workflow editor and click on Add Promotion. Click the new pipeline and copy the global Earthly install commands, again:

    sudo /bin/sh -c 'wget https://github.com/earthly/earthly/releases/latest/download/earthly-linux-amd64 -O /usr/local/bin/earthly && chmod +x /usr/local/bin/earthly && /usr/local/bin/earthly bootstrap --with-autocomplete'

    The block in the pipeline will contain a single job to generate and push the Docker image into Docker Hub. The commands are as follows:

    echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
    checkout
    cache restore
    earthly +docker
    docker tag semaphore-demo-earthly $DOCKER_USERNAME/semaphore-demo-earthly
    docker push $DOCKER_USERNAME/semaphore-demo-earthly
    

    We use earthly +docker to export the image into the CI machine, then a combination of docker tag and docker push to send it to the remote registry.

    Push block

    The only thing left is to enable the Docker Hub secret … and we’re done!

    Rerun the workflow and click on promote to try pushing the image.

    Final build
    Final build

    What do you think?

    The Earthly project is young and under heavy development, with many features still in the experimental stage (we stuck to stable features during the course of the post). It’s not perfect, but it feels like a massive step in the right direction.

    Skill up your Docker knowledge with these tutorials:

    Have you tried Earthly? Tell us about your experience.

    Leave a Reply

    Your email address will not be published. Required fields are marked *

    mm
    Writen by:
    I picked up most of my skills during the years I worked at IBM. Was a DBA, developer, and cloud engineer for a time. After that, I went into freelancing, where I found the passion for writing. Now, I'm a full-time writer at Semaphore.