3 Oct 2016 Β· Software Engineering

    Using RSpec Metadata

    10 min read
    Contents

    Post originally published on https://rossta.net. Republished with author’s permission.

    A useful feature of RSpec is the ability to pass metadata to tests and suites.

    You may already be familiar with how Capybara uses the :js option to enable the JavaScript driver.

    describe "a javascript feature", :js do
      # tests run against the Capyabara.javascript_driver
    end

    Capybara provides an RSpec configuration hook that changes the web driver for any example where :js metadata is present. Here it is, oversimplified:

    # capybara/rspec.rb
    RSpec.configure do |config|
      config.before do
        Capybara.current_driver = Capybara.javascript_driver if example.metadata[:js]
      end
    end

    We may reach a point in the maturity of our test suite when it makes sense to add our own configuration options.

    The examples in the post are based on RSpec version ~> 3.

    Changing Test Runner Behavior

    Testing libraries like RSpec and Capybara do some heavy lifting to set up the Rails environment and make it suitable for running in test mode. For performance reasons, it may be beneficial to run each of our specs in a database transaction so test data can be easily rolled back at the start of each spec.

    Here’s a common base configuration for using the popular DatabaseCleaner gem to set up transactional database behavior for RSpec:

    RSpec.configure do |config|
      config.use_transactional_fixtures = false
    
      config.before(:suite) do
        DatabaseCleaner.clean_with(:truncation)
        DatabaseCleaner.strategy = :transaction
      end
    
      config.before(:each) do
        DatabaseCleaner.start
      end
    
      config.after(:each) do
        DatabaseCleaner.clean
      end
    end

    Not all specs can be run this way β€” once we’ve added a JavaScript acceptance spec, for example, the JavaScript driver will likely need its own connection to the database, so it won’t have access to data setup in the tests. We need to run JavaScript acceptance specs in truncation mode to ensure database changes are committed to the database, so multiple database connections will have access to the same data.

    Let’s use RSpec metadata to toggle database behavior automatically when using the JavaScript driver (i.e. not the default :rack_test driver). We’ll add the following hooks, borrowed from the DatabaseCleaner README:

    # spec/spec_helper.rb
    config.before(:each, type: :feature) do
      # :rack_test driver's Rack app under test shares database connection
      # with the specs, so continue to use transaction strategy for speed.
      driver_shares_db_connection_with_specs = Capybara.current_driver == :rack_test
    
      if !driver_shares_db_connection_with_specs
        # Driver is probably for an external browser with an app
        # under test that does *not* share a database connection with the
        # specs, so use truncation strategy.
        DatabaseCleaner.strategy = :truncation
      end
    end

    We also run into problems with ActiveRecord after_commit callbacks β€” when running tests in transaction mode, these callbacks will never fire. We can also add an option for enabling truncation mode outside of acceptance specs when isolated specs are needed for these callbacks:

    # spec/model/user_spec.rb
    it "triggers background job after creating new user", :truncation_mode do
      # test after_commit callback
    end
    
    # spec/spec_helper.rb
    config.before(:each, :truncation_mode) do
      DatabaseCleaner.strategy = :truncation
    end

    Here’s a consolidated configuration for providing hooks for the issues related to database truncation mentioned above:

    # spec/spec_helper.rb
    RSpec.configure do |config|
      config.use_transactional_fixtures = false
    
      config.before(:suite) do
        DatabaseCleaner.clean_with(:truncation)
      end
    
      config.before(:each) do
        DatabaseCleaner.strategy = :transaction
      end
    
      config.before(:each, type: :feature) do
        driver_shares_db_connection_with_specs = Capybara.current_driver == :rack_test
    
        if !driver_shares_db_connection_with_specs
          DatabaseCleaner.strategy = :truncation
        end
      end
    
      config.before(:each, :truncation_mode) do
        DatabaseCleaner.strategy = :truncation
      end
    
      config.before(:each) do
        DatabaseCleaner.start
      end
    
      config.after(:each) do
        DatabaseCleaner.clean
      end
    end

    Changing Application Settings

    Rails provides a number of settings that can be easily configured based on the environment, so we avoid undesired work in development or test environments, such as sending emails. For any mature Rails application, we’ll likely have our own custom settings layered on top of the Rails defaults.

    There are many cases where we’ll still want to test the production settings in our test environments. For example, controller caching is disabled in the tests by default:

    # config/initializers/test.rb
    Rails.application.configure do
      # ...
      config.action_controller.perform_caching = false
    
    end

    For selected acceptance specs, we may still want to test behavior of caching at the view layer, say that users can see new info when a model attribute changes. We don’t need this caching behavior, so it may be useful to toggle specs on/off during the test run.

    First Attempt

    We could try to stub the setting in the context of a single spec run with the enabled state.

    # spec/spec_helper.rb
    RSpec.configure do |config|
      config.before(:each, :caching) do
        allow_any_instance_of(ActionController::Base).to receive(:perform_caching).and_return true
      end
    
      config.after(:each, :caching) do
        Rails.cache.clear
      end
    end

    This may require changing behavior of instances which is typically discouraged. We may also need to clean up other global states, like clearing the Rails cache after the test run.

    A Better Attempt

    Alternatively, we can set the actual values on while settings are derived. Here’s how it might look for enabling controller caching with an around block:

    # spec/spec_helper.rb
    RSpec.configure do |config|
      config.around(:each, :caching) do |example|
        caching = ActionController::Base.perform_caching
        ActionController::Base.perform_caching = example.metadata[:caching]
    
        example.run
    
        Rails.cache.clear
        ActionController::Base.perform_caching = caching
      end
    end

    The around block takes the RSpec example object as an argument. When running specs, the given block is triggered when :caching is detected as a key in an example’s metadata. The example object provides a number of methods for test introspection, allowing us to make changes before and after calling run to execute the spec.

    As a result, we now have a simple and explicit mechanism for introducing caching to individual specs and suites:

    # spec/features/homepage_spec.rb
    describe "visit the homepage", :caching do
      it "expires cache" do
        # test cached stuff
      end
    end

    The main concern with this approach is that modifying a global state can affect other tests unintentionally β€” a big no-no.

    To avoid this, we need to reset the original value when the example completes.

    Here, we are storing the previously set value of ActionContoller::Base.perform_caching, setting it for the local suite, and resetting it back to the original value after it completes.

    This technique may come into play when integrating with certain gems like PaperTrail, which may generate expensive logic or queries that aren’t needed in most cases. PaperTrail even provides a helper to take advantage of RSpec. It may be worth considering whether to provide an interface to toggle behavior and RSpec helpers next time we write gem metadata to toggle behavior in specs.

    Filtering Specs

    One useful technique while developing is running a selected set of specs. We may be editing acceptance specs, model validations, and other disparate tests while test driving a feature from outside in.

    Manual tagging

    Adding arbitrary metadata like :focus to a set of specs is one way of approaching this.

    # spec/models/user_spec.rb
    it "validates a user", :focus do
      # unit test
    end
    
    # spec/features/sign_up_spec.rb
    it "displays error message", :focus do
      # acceptance spec
    end
    

    We can now filter our test run to a subset at the command line as follows:

    $ rspec --tag focus

    We can also add some global configuration so this will be the default behavior when using :focus specs, as long as we don’t make the mistake of unintentionally filtering on the build server.

    RSpec.configure do |config|
      # enable auto-focus only when running locally
      config.filter_run_including focus: ENV['CI_SERVER_SETTING'].blank?
    
      config.run_all_when_everything_filtered = true
    end
    

    Alternatively, avoid running broken or flaky specs when tagged accordingly:

    it "test that fails intermittently", :flaky do
      # probably a JavaScript test
    end

    Using either a command line option

    $ rspec --tag ~flaky

    or a configuration option, we can filter out specs we wish to ignore.

    RSpec.configure do |c|
      c.filter_run_excluding flaky: true
    end

    Auto Tagging

    A less-known feature of RSpec 3 is an API for telling RSpec to derive additional metadata automatically based on the other metadata.

    For example, each spec example has metadata that includes its file path. This, along with the RSpec::Core::Configuration#define_derived_metadata method, allows us to alter spec behavior based on the spec directories, for example.

    Why is this useful and how do we use it? Glad you asked.

    Let’s say we want to isolate model specs that require database truncation since they are more like functional specs than unit specs. We may set up our spec directory as follows:

    spec/
      truncation/
        example1_spec.rb
        example2_spec.rb
        ...
      transaction/
        example1_spec.rb
        example2_spec.rb
        ...
    

    Instead of manually tagging each file with our :truncation_mode metadata we used earlier to toggle DatabaseCleaner’s truncation strategy, we can configure all the specs in spec/truncation as follows:

    # spec/spec_helper.rb
    RSpec.configure do |config|
      config.define_derived_metadata(file_path: %r{spec/truncation}) do |metadata|
        metadata[:truncation_mode] = true
      end
    
      # rest of DatabaseCleaner config below
    end
    

    Now, all specs in the directory will run with the :truncation_mode metadata, and the database strategy will be set to :truncation as long as it is declared ahead of the additional DatabaseCleaner configuration we referenced earlier.

    Note that this is the same method used in rspec-rails to add custom behavior to specs in specific directories, e.g. spec/controllers', spec/requests, etc.

    Using and Abusing

    While using RSpec metadata can be a powerful technique for altering test behavior and application settings in specs, it can also be taken too far.

    As @avdgaag notes in his blog post on the topic, make sure to distinguish between how a spec is run from what the spec should test. For example, we probably shouldn’t use metadata to create records specific to certain tests, or authenticate users for a given context.

    One rule of thumb for adding metadata is to decide whether it would be generally useful in any Rails app (good), or if it’s specific to the business logic of your current application (bad). The latter is best set up more explicitly within or alongside your tests.

    Before considering a new metadata tag, I ask the rubber duck “Could I extract this configuration into a gem?” To answer yes, the behavior would have to be non-specific to my application. If so, the behavior might be useful as metadata.

    While metadata can nicely separate the boilerplate required to set up and tear down test behavior, it also adds a layer of indirection that can cause readability issues when stretched too far. Understand that there is a big increase in mental overhead to permuting test behavior with each new tag option, and consider the tradeoffs with the rest of the team.

    Use it wisely!

    If you have any questions or comments about this article, feel free to leave them in the comment section below.

    P.S. Would you like to learn how to build sustainable Rails apps and ship more often? We’ve recently published an ebook covering just that β€” “Rails Testing Handbook”. Learn more and download a free copy.

    Leave a Reply

    Your email address will not be published. Required fields are marked *

    Avatar
    Writen by:
    Ross Kaffenberger is passionate about teaching, learning, and contributing to open-source. He writes about Ruby, JavaScript, and Elixir on his site, speaks at local meetups, and enjoys being a dad.