Semaphore is using the Test Boosters gem to parallelize and run your test suite. This document describes what is the Test Booster gem, its assumptions about your project, and tips for troubleshooting issues that can occur.
The Test Boosters gem is an open source test runner that runs your tests in multiple test jobs. It is used on Semaphore to support the execution of boosted builds. The Test Booster gem comes preinstalled in Semaphore boxes.
If you notice any issues with the test booster gem, please report it as an issue on the GitHub repository.
We are also glad to receive contributions via pull requests.
Semaphore collects test file durations from each of your builds. Based on these reports, test files are distributed accross your boosted jobs. When a build starts, Semaphore calculates the estimated duration for each test file in your suite.
To calculate the estimated test file duration, Semaphore uses the last 50 build reports on your project. For this calculation the avg(test_files.durations) + 2 * standard_deviation formula is used.
If you notice that your test files are not distributed properly, one of the following could have happened:
There is a big variation in the duration of your test files. For example, if a file takes 10 seconds to complete in one build, and 5 minutes to complete in the next build, Semaphore will have a hard time to properly allocate a job for your file. Try to adjust the test file, and limit duration changes up to 10%.
If you have added or removed a new test file on one of your branches, it will have an effect on all of your builds. Boosters work under the assumption that new branches are short lived and frequently merged into the master branch.
Boosters run under the assumption that you have no additional filters on your test suite. For example, the :focus filter that runs only a portion of your RSpec test suite will prevent proper booster execution.
If neither of the above applies to your project, observing the test distribution of your test files might help. Split Configurations are available in each boosted build you run. SSH sessions into the build environment is the best way to explore them. Read more about split configurations on the Test Boosters README page.
Boosters are designed to work out of box for most Rails applications. However, edge cases are always possible when it comes to project configuration. Here are some tips that might help you to track down the issues:
Make sure that you are using a supported Ruby version. Versions from 1.9.x to the latest 2.4.x are supported out of box. Older or newer Ruby versions might work, but are not officially supported.
Make sure that your Rails version is supported. Rails 3, 4, 5 are officially supported. Rails 2 might work, but is not officially supported.
Verify that your tests are runnable with bundle exec rspec <file-list> and bundle exec cucumber <file-list>. Boosters are running your test suite via those commands.
If you have changed Ruby version in your builds with rbenv use <version> you will need to manually install the test booster gem for that ruby version. This can be achieved by running rbenv rehash or by manually installing the test booster gem in your builds gem install semaphore_test_boosters.
Semaphore Docs are open source — Edit on GitHub