If you have turned Insights on for a project on Semaphore, but you don’t see any test files on the Insights page, or some stats are missing, this guide will help you find the source of the problem.
Semaphore supports Insights for RSpec and Cucumber testing frameworks. If you are using some other framework, but you would still like to use Insights, please let us know by answering few short questions.
At the moment, Semaphore supports only RSpec 3.x. If you are using earlier versions of RSpec, it’s likely that you will see test files on the Insights page, but you won’t see all stats. Percentage of passed builds will be visible, but stats for mean time, max time and standard deviation will be missing. The solution to this problem is to upgrade to a newer version of RSpec.
If you use RSpec 3.x, but you don’t see any test files on the Insights page, there are a few possible causes described below.
All test files finish faster than 10 seconds
Semaphore Insights are displayed only for test files that take more than 10 seconds to finish.
Your options are defined with the SPEC_OPTS environment variable
There are a few ways to pass configuration options to RSpec runner:
- command line options
When you turn RSpec Insights on on a project, Semaphore will set the value of
SPEC_OPTS environment variable:
export SPEC_OPTS="--format documentation --format json --out rspec_report.json"
However, if your project is using the
SPEC_OPTS environment variable to pass
options to the RSpec runner, they will override the options that Semaphore
previously set and Semaphore will not be able to gather stats for the Insights.
A typical user RSpec command with the
SPEC_OPTS environment variable might
look like this:
bundle exec rake spec SPEC_OPTS="--order random"
To use Insights, replace the
SPEC_OPTS environment variable with an
file. You can commit the file to the repository. In that case, these options
will be used on all development machines, unless a developer overrides them. If
you don’t want to commit the
.rspec file to the project repository, you can
create the file using custom configuration
files. That way, the
file will be available only on Semaphore.
An alternative is to construct the
SPEC_OPTS environment variable yourself.
The variable should include
--format json --out rspec_report.json and all the
options you need.
Your tests are executed using parallel_test gem
If you run your RSpec tests using parallel_test gem, the report with stats for each test example will not be in valid format, and you will not be able to use Insights. The issue is described in detail at the official parallel_tests issues page.
Your tests are executed using rspec_rerun gem
rspec_rerun gem is using a special formatter to save the report about failed
tests. When you turn on Insights for a project that’s using
Semaphore overrides the formatter used by
rspec_rerun and your tests will fail
with an error such as:
Errno::ENOENT: No such file or directory @ rb_sysopen - rspec.failures
To use RSpec Insights in combination with
rspec_rerun, you need to set options
for both Semaphore Insights and
rspec_rerun in the
variable. Change your RSpec command to look like this:
SPEC_OPTS="--format documentation --format json --out rspec_report.json --require rspec-rerun/formatter --format RSpec::Rerun::Formatter" bundle exec rake rspec-rerun:spec
After that, your tests should work as expected and Semaphore should be able to collect Insights data for the project.