This is the second blog post in a series inspired by a Stack Overflow thread. You can read the first post here.
A local hero is defined as:
A test case that is dependent on something specific to the development environment it was written on in order to run. The result is the test passes on development boxes, but fails when someone attempts to run it elsewhere.
Leaving database state aside, this typically arises due to differences in configuration between a developer’s machine and somebody else’s or CI system.
Machines used for CI for web apps usually run in headless mode, which means that they lack a hardware display. Instead, the OS renders the screen as a bitmap in memory. On Linux the most frequently used implementation is Xfvb, which is also used on Semaphore.
In earlier versions of Semaphore’s build platform, the virtual resolution was set to be 1280×800. Eventually we learned that this resolution is not large enough for all users. Some developers that use Selenium to run tests in a real browser were getting mysterious failures due to an expected element not being present on the screen. For example, the navigation bar was implemented as a responsive element, hiding certain links in smaller browser windows. Nowadays the default resolution on Semaphore is 1900×1200.
In case above, the tests were implicitly depending on screen resolution, a rather unexpected component. But sometimes a local hero arises when the application simply was not configured completely. For example, Semaphore’s front-end application tests depend on the existence of a file config/config.yml, which is not stored in version control (because it contains sensitive credentials) but should be copied or created manually during project setup. To make it work, it was necessary to create an encrypted custom configuration file on Semaphore with just enough content to make the tests pass.
A related case is…
The hidden dependency
> a unit test that requires some existing data to have been populated somewhere
> before the test runs. If that data wasn’t populated, the test will fail and
> leave little indication to the developer what it wanted, or why… forcing
> them to dig through acres of code to find out where the data it was using
> was supposed to come from.
Management of database state in tests is best to be to completely automated by your tools. If your tests require you to manually clear the database before each run, look for a tool to do that for you. For example, Ruby developers often use database_cleaner, whose purpose is to maintain a clean state during tests. There’s a similar package for Node.js.
The problem occurs because test runners — or your own code — often wrap tests in a transaction, so other processes, such as the web server, may not be able to access test data which was previously set.
Having established this practice, you also insulate your test suite from the bad practice of requiring some special data to be manually inserted before the tests run.
See also Incidental database state in Rails Testing Antipatterns: Models.