20 Mar 2024 · Software Engineering

    8 Ways To Retry: Finding Flaky Tests

    8 min read

    Handling flaky tests in software development can be a tricky business, specially for tests that fail a small percentage of the times. The most reliable way we have to detect flaky tests is to retry the test suite several times.

    Finding Flaky Tests: To Retry or Not To Retry?

    The decision to retry these tests depends on the environment you’re working in.

    Decision tree to decide to retry flaky tests. On local environments we should try to retry always (preferably with the IDE).

On CI environments, we don't want to retry as it would hide the flakiness of the tests. The only exception is when we can log the test results in a file or database to analyze.

    Local Development

    In local environments, to retry flaky tests can be beneficial as it allows developers to identify and address transient errors. Most Integrated Development Environments (IDEs) support running tests directly, providing immediate feedback. Alternatively, most test frameworks offer configuration options to automate retries, helping to smooth over these intermittent issues.

    Continuous Integration Environments

    For CI environments, the approach is more nuanced. If your CI platform has specific support to retry flaky tests, such as a flaky test dashboard, it’s better to let tests fail and use these tools to track and fix them. This ensures that flaky tests are not hidden but rather highlighted for further investigation. However, if you can log test failures for later analysis without retrying, this could also be a viable approach. Generally, if you lack the tools to properly track and analyze flaky tests, avoiding retries in CI environments is advisable to ensure that every test accurately reflects the state of the code.

    How to Configure Retry For Flaky Test Detection

    JavaScript and TypeScript with Jest

    For JavaScript and Types testing using Jest, you can configure retries directly in your Jest configuration. First, we create a small initialization file at the root of our project:

    // retry-tests.js
    jest.retryTimes(5, {logErrorsBeforeRetry: true});

    Then, we load it in jest.config.js:

    // jest-config.js
    module.exports = {
      setupFilesAfterEnv: ['<rootDir>/retry-tests.js'],
      reporters: [ "default" ]

    This is useful for automatically rerunning failed tests a specific number of times, with options to log errors before each retry.

    We can also retry specific files and test by adding jest.retryTimes to the test file, for example:

    jest.retryTimes(5, {logErrorsBeforeRetry: true});
    test('Flaky Test', () => {
        const value = Math.random()

    The option logErrorsBeforeRetry will make Jest show the error on the console when the test begins to flake.

    Ruby with RSpec-Retry

    In Ruby, using the RSpec framework, flaky tests can be managed by installing the rspec-retry gem.

    $ gem install rspec rspec-retry
    $ rspec --init

    Next, we need enable the gem in the spec/spec_helper.rb file by adding:

    require 'rspec/retry'

    Finally, in the RSpec.configure do |config| section add the following lines:

    RSpec.configure do |config|
      config.verbose_retry = true
      config.display_try_failure_messages = true
      config.default_retry_count = 20
    # rest of the config ...

    This will make RSpec retry up to 20 times failed tests.

    Alternatively, you can specify the number of retries directly in your tests, giving flaky tests several chances to pass before being marked as failures.

    describe "Flaky Test" do
        it 'should randomly succeed', :retry => 10 do
        expect(rand(2)).to eq(1)

    You can also override the default number of retries by changing the RSPEC_RETRY_RETRY_COUNT environment variable:


    Python with Pytest-Retry and FlakeFinder

    The PyTest framework provides two switched to re-run failed tests:

    • pytest --lf: re-run last failed tests only
    • pytest --ff: re-run all test, failed tests first

    So out of the box we get decent retry features. But in order to have PyTest automatically retry failed tests without running additional commands, we can install the pytest-retry plugin:

    $ pip install pytest-retry

    Once installed, we can use the @pytest.mark.flaky decorator to our tests to automatically retry tests. For example, this test will run up to 20 times:

    import pytest
    import random
    def test_flaky():
        if random.randrange(1,10) < 6:
            pytest.fail("bad luck")

    In the spirit of “fail fast”, we can combine this with pytest --ff to rerun failed tests first.

    Pytest-retry takes care of retries, but we Python developers have a tool specifically intended to find flaky tests with retry: pytest-flakefinder.

    We can install the tool with:

    $ pip install pytest-xdist

    Then, we can run test multiple times in parallel with:

    $ pytest --flake-finder --flake-runs=20

    This will run each test 20 times and show a report at the end. A great way for quickly identifing flaky tests.

    Java with Surefire

    For Java projects using Maven, integrating retries requires adding specific plugins like the Maven JUnit and Surefire plugins.

    First, we should add the most current JUnit version to our pom.xml:


    Next, we add a plugin into the build/pluginManagement/plugin section of the pom.xml. This enables the Maven Surefire Plugin:


    Now we can ask Maven to re-run failed plugins by adding surefire.rerunFailingTestsCount to the test command:

    $ mvn -Dsurefire.rerunFailingTestsCount=20 test

    This allows for configuring test retries directly in the Maven configuration, providing a systematic way to rerun failing tests a certain number of times.

    Rust with NexTest

    In Rust, while Cargo does not natively support test retries, extensions like cargo-nextest can be installed to add this functionality.

    $ curl -LsSf https://get.nexte.st/latest/mac | tar zxf - -C ${CARGO_HOME:-~/.cargo}/bin

    Running cargo init should create the nextest config file .config/nextest.toml. We can customize the test behavior here:

    retries = { backoff = "fixed", count = 20, delay = "1s" }

    Now, in order to run test with retries we need to run cargo nextest instead of cargo test:

    $ cargo nextest run

    We can also specify the number of retries in the command line:

    $ cargo nextest run --retries 10

    Configuring retries either via command-line arguments or configuration files allows developers to automatically rerun failed tests.

    PHP with PHPUnit

    [PHPUnit] has retry support out of the box. We only need to install the plugin with composer:

    $ composer require --dev phpunit/phpunit

    Then, configure PHPUnit to look for tests in our test folder:

    <phpunit bootstrap="vendor/autoload.php"
            <testsuite name="Application Test Suite">

    We can now use the repeat option to re-run failed tests:

    $ ./vendor/bin/phpunit --repeat 10 

    We can re-run failed test first by adding --cache-result --order-by=depends,defects to the invocation:

    $ ./vendor/bin/phpunit --repeat 10 --cache-result --order-by=depends,defects

    This will re-run test up to 10 times, cache the results and run failed tests first on the next test run.

    Elixir with ExUnit

    Elixir projects use [ExUnit] by default as the test runner. This framework does not provide a re-run functionality, however, it does a --failed switch:

    $ mix test --failed

    This option will re-run failed tests in the last execution. We can leverage it in a shell script to automatically re-run failed tests until they succeed or reach the maximum number of retries:

    # rerunner.sh: re-reun failed tests in Elixir
    mix test
    # retry up to 20 times
    for i in {1..20}; do
        echo "=> Re-running failed tests"
        mix test --failed && break

    With this simple script we can emulate the retry behavior of other frameworks.

    Go with GoTestSum

    Go has a built-in test runner in the framework, which, unfortunately, does not support automatic retries. For that, we need to install [GoTestSum]():

    $ go install gotest.tools/gotestsum@latest

    Now we can use gotestsum --rerun-fails like this:

    $ gotestsum --rerun-fails --packages="./..."

    In the packages section we can list the packages to test or ./... to recursively search for test files in the project.

    One problem you may encounter is that Go caches the build results, which can hide flakiness in the tests. In order to bypass the cache you can add -- -count to the command invocation. For example, to re-run 20 times the tests:

    $ gotestsum --rerun-fails --packages="./..." -- -count=20

    This will efectively rebuild the binary each time the test runs.


    Whether working in local or CI environments, the key is to strike a balance between identifying and fixing flaky tests with retry and not letting them undermine the overall confidence in your test suite.

    Learn more about flaky tests:

    Leave a Reply

    Your email address will not be published. Required fields are marked *

    Writen by:
    I picked up most of my skills during the years I worked at IBM. Was a DBA, developer, and cloud engineer for a time. After that, I went into freelancing, where I found the passion for writing. Now, I'm a full-time writer at Semaphore.