If you have turned Test file performance Insights on for a project on Semaphore, but you don't see any test files on the Test file performance tab, or some stats are missing, this guide will help you find the source of the problem.
Semaphore supports Test file performance Insights for RSpec and Cucumber testing frameworks. If you are using some other framework, but you would still like to use Test file performance Insights, please let us know by answering few short questions.
At the moment, Semaphore supports only RSpec 3.x. If you are using earlier versions of RSpec, it's likely that you will see test files in the Test file performance tab, but you won't see all stats. Percentage of passed builds will be visible, but stats for mean time, max time and standard deviation will be missing. The solution to this problem is to upgrade to a newer version of RSpec.
If you use RSpec 3.x, but you don't see any test files in the Test file performance tab, there are a few possible causes described below.
Semaphore Test filer performance Insights are displayed only for test files that take more than 10 seconds to finish.
There are a few ways to pass configuration options to RSpec runner:
When you turn RSpec Test file performance Insights on on a project, Semaphore will set the value of SPEC_OPTS environment variable:
export SPEC_OPTS="--format documentation --format json --out rspec_report.json"
However, if your project is using the SPEC_OPTS environment variable to pass options to the RSpec runner, they will override the options that Semaphore previously set and Semaphore will not be able to gather stats for the Test file performance Insights.
A typical user RSpec command with the SPEC_OPTS environment variable might look like this:
bundle exec rake spec SPEC_OPTS="--order random"
To use Test file performance Insights, replace the SPEC_OPTS environment variable with an .rspec file. You can commit the file to the repository. In that case, these options will be used on all development machines, unless a developer overrides them. If you don't want to commit the .rspec file to the project repository, you can create the file using configuration files. That way, the .rspec file will be available only on Semaphore.
An alternative is to construct the SPEC_OPTS environment variable yourself. The variable should include --format json --out rspec_report.json and all the options you need.
If you run your RSpec tests using parallel_test gem, the report with stats for each test example will not be in valid format, and you will not be able to use Test file performance Insights. The issue is described in detail at the official parallel_tests issues page.
rspec_rerun gem is using a special formatter to save the report about failed tests. When you turn on Test file performance Insights for a project that's using rspec_rerun, Semaphore overrides the formatter used by rspec_rerun and your tests will fail with an error such as:
Errno::ENOENT: No such file or directory @ rb_sysopen - rspec.failures
To use RSpec Test file performance Insights in combination with rspec_rerun, you need to set options for both Semaphore Test file performance Insights and rspec_rerun in the SPEC_OPTS environment variable. Change your RSpec command to look like this:
SPEC_OPTS="--format documentation --format json --out rspec_report.json --require rspec-rerun/formatter --format RSpec::Rerun::Formatter" bundle exec rake rspec-rerun:spec
After that, your tests should work as expected and Semaphore should be able to collect Test file performance Insights data for the project.
Semaphore Docs are open source — Edit on GitHub