10 Feb 2020 · Software Engineering

    In Continuous Integration, Run Fast and Fundamental Tests First

    3 min read

    While it’s great to keep your entire CI/CD pipeline fast, on many occasions you don’t even need to run all tests to get the feedback you need.

    Unit tests run the fastest, because they test a very small piece of code in complete isolation from the rest of the system. For example, they usually don’t touch the database.

    Unit tests deal with fundamental business logic, and are the most numerous, as is commonly depicted in the “test pyramid” diagram:

    test pyramid
    Test pyramid

    A failure in unit tests is a sign of a fundamental problem, which makes running the remaining high-level and long-running tests irrelevant.

    For these reasons, projects with test suites that run for anything longer than a minute should prioritize unit tests in the continuous integration (CI) process.

    How to Use Pipelines to Run Fast Tests First

    A classic CI setup is to split all tests in parallel jobs:

    ci build with parallel jobs
    CI build with parallel jobs

    In this approach, the total CI build time is determined by the duration of the slowest parallel job. For example:

    • If some unit tests have failed in less than a minute,
    • But one of the jobs with UI tests runs for 6 minutes,
    • Then you need to wait for 6 minutes to receive a report about unit test failures.

    The unit tests that fail typically make some higher-level tests fail as well. However, in this case the results of higher-level tests are irrelevant and running them is a waste of time and money.

    A better approach is to use customizable CI/CD pipelines. This way you can configure sequential blocks of single or parallel jobs. As soon as one block fails, the pipeline stops.

    ci pipeline
    CI build as a pipeline with sequential blocks and parallel jobs

    This strategy allows developers to get feedback on trivial errors in seconds. It also encourages all team members to understand the performance impact of individual tests as the code base grows.

    More Tips for Fast Feedback

    There are additional tactics that you can use with your CI system to get fast feedback:

    • Conditional stage execution lets you defer running certain parts of your build for the right moment. For example, you can configure your CI to run a subset of end-to-end tests only if one of the related components was changed. (Here’s how it works on Semaphore.)
    • A fail-fast strategy gives you instant feedback when a job fails. CI stops all currently running jobs in the pipeline as soon as one of the jobs has failed. This approach is particularly useful when running parallel jobs with variable duration. (Here’s how you can configure this on Semaphore.)
    • Automatic cancelation of queued builds can help in situations when you push some changes, only to realize that you have missed something small, so you push a new revision immediately, but then need to wait for twice as long for feedback. With this approach you get feedback on revisions that matter while skipping all the intermediate ones. (Here’s how to do it with Semaphore.)

    The same principle applies to stages in the delivery pipeline. For example, if you’re running smoke tests after deployment, you want to detect issues as quickly as possible.

    What is your approach to getting fast feedback? Let me know in the comments. Happy building!

    Leave a Reply

    Your email address will not be published. Required fields are marked *

    Writen by:
    Marko Anastasov is a software engineer, author, and co-founder of Semaphore. He worked on building and scaling Semaphore from an idea to a cloud-based platform used by some of the world’s engineering teams.