The Benefits of Acceptance Testing

At some point, most of us have worked on a project where tests were an afterthought — a grueling experience that shows the value of testing discipline (consider yourself one of the lucky few if you haven’t been in this situation). In this article, I’d like to talk about one of the most complex forms of testing, one that will tell when we have met our software design goals: Acceptance Testing.

What is acceptance testing?

Acceptance testing is the practice of running high-level, end-to-end tests to ensure that a system follows spec. Acceptance tests are derived from acceptance criteria, which define how an application responds to user actions or events.

Acceptance tests shift attention towards the end goal: shipping software that fulfills a business need. They cross the gap between developers and end-users, ensuring that the application works in the real world.

What do acceptance tests test?

Acceptance tests are different from other types of tests. Why? Because they are primarily about business goals. While technology-facing unit tests ask: “is this function returning the correct value?” and integration tests ask: “are the application components interacting well?”, acceptance tests focus on what matters most when all is said and done: “is my application providing valuable functionality to end users?”

The importance of these questions cannot be overstated. If your customers don’t get what they came for, either because you didn’t meet the goals or because you overengineered the solution, you won’t be in business for long.

Acceptance tests come in two variants:

• Functional acceptance tests deal with application behavior. They ask: “does the application work as users expect?”
• Non-functional acceptance tests cover things like security, capacity, and performance. These are questions such as: “is my system secure and fast enough?”

Testing business goals

An application that passes all acceptance tests is, by definition, complete and functional according to its specification. Acceptance testing is an iterative process whereby:

1. We define criteria in cooperation with product managers, who collaborate with end-users.
2. We write tests to meet acceptance criteria. These tests should initially fail.
3. We code until we pass the tests.
4. Once acceptance tests pass, progress is evaluated. A new cycle may then begin.

At the end of every cycle the specification is reviewed and refined. The process continues when all acceptance criteria are met.

How to write acceptance tests

Writing and maintaining acceptance tests is not a trivial thing, but it’s an investment that will repay itself many times over throughout the project. Unlike unit tests, which can run piecemeal, acceptance tests must test the system as a whole. You have to start the application in a production-like environment and interact with it in the same way a user would.

Organizing acceptance tests

Once the acceptance criteria are defined, you can start writing the tests. The most high-value targets for acceptance tests are the happy paths: the default scenarios where there are no exceptions or error conditions.

We organize acceptance tests into two layers:

1. Acceptance Criteria Layer: this is a conceptual description of the case being tested.
2. Test Implementation Layer: encodes acceptance criteria layer cases using a testing framework. This layer interacts with the application, simulates user actions, and deals with the UI.

Acceptance criteria layer

The top layer describes the case under test in plain English. It states what the application does without saying anything about how it’s doing it. Here we ask questions such as: “if I buy this product, will the order be accepted?”, or “if I don’t have funds, will the order be rejected and the user notified?”

We use the Behavior-Driven Development (BDD) pattern of Given-When-Then to formalize the case under test.

1. Given: gives context and pre-conditions. This is the initial state of the application.
2. When: describes the events or actions the user takes.
3. Then: lists the expected outcome(s).

For instance, a criteria for testing the login feature in an application might look like this:

Feature: Sign into the system
  Scenario: User logs in and sees the welcome page
    Given I have an account
    When I sign in with my valid credentials
    Then I see the welcome page

BDD libraries like Cucumber, Ginkgo, Behat, Behave, or Lettuce allow you to use plain text to describe acceptance criteria and keep them synchronized with test execution. If these tools are not an option, we can use the Given-When-Then pattern in the test’s description text.

Test implementation layer

If the acceptance criteria layer focuses on building the right things, then the implementation layer is about building them right. Here is where we find Test-Driven Development (TDD) frameworks like JUnit, Mocha, or RSpec, of which many employ a Domain Specific Language (DSL) to map the conditions and actions in the acceptance layer into executable code. The test layer’s function is to evaluate the pre-conditions, execute the actions, and compare the outputs.

For instance, the acceptance test above requires a log in routine. Here’s where the expressive power of a DSL like Capybara manifests itself:

When /I sign in/ do
  within("#session") do
    fill_in 'Email', with: 'user@example.com'
    fill_in 'Password', with: 'password'
  end
  click_button 'Sign in'
end

Dealing with the UI

If the application has a UI, acceptance tests should cover it; otherwise, we’re not really testing the end users’ experience. The test layer must then include a window driver that knows how to operate the UI, i.e. clicking buttons, filling fields, and parsing the results. In this category, we have libraries such as Puppeteer, Cypress, and Selenium to help us.

UI testing has some downsides though. The tests are slower, harder to scale up, and need more maintenance. Acceptance tests must include the UI, but that doesn’t mean that every test should go through it. It’s perfectly acceptable to channel some of the tests via alternative paths like API endpoints.

Automating acceptance tests

Being in business means fulfilling business objectives. When an acceptance test breaks, we have to drop everything else we’re doing and triage the problem right away.

While manual tests are possible, the results are unreliable. It’s slow and painstaking work that makes testers miserable. Automated acceptance tests, on the other hand, give immediate feedback about business objectives.

Acceptance tests should be automated because they:

• Alleviate tester’s workloads.
• Reduce test runtime.
• Allow for regression testing.
• Allow testers to focus on exploratory testing, increasing test coverage and improving the performance of the test suite.
• Allow testers to find the exact point at which the error was introduced.

Continuous acceptance testing with CI/CD

Once we have written our automated acceptance tests, we must run them continuously. Acceptance tests take more resources to run than any other type of test. So, use caution when setting up the continuous integration process.

If the CI/CD pipeline is going to fail, it’s better that it happens sooner, rather than later. The shorter the feedback cycle, the earlier we know that there is a problem. Therefore, you should place the fastest tests first during the build stage.

Next, we add integration, security, and any other medium-level tests. Acceptance tests, which take longer, should go at the very end of the pipeline using parallelism to reduce total test time.

When should acceptance tests run?

At this point, we have a choice: do we want to run the acceptance tests on every commit? Ideally, yes. But in practice, if the tests take too long, or are too costly due to infrastructure concerns, you should run them only on specific branches or before doing a deployment with promotions.

To manage more complex situations, there is the change_in function, which allows us to run blocks on specific conditions, such as when certain files change. This function is a key component of monorepo workflows.

Debugging acceptance tests in CI

Bugs happen. To make matters worse, debugging can feel like punishment on most CI/CD platforms. This is not the case on Semaphore — which has two features that let you debug in a snap:

• SSH access: you can log in directly to the CI machine responsible for the failed job. In there, you can rerun commands, inspect logs, and try quick fixes. Being able to try something without having to restart the pipeline and wait for the job is game-changing.
• Port forwarding: while testing the UI, you may find that SSH access is not enough. Semaphore lets you forward ports in the remote machine so you can see what’s actually happening while the jobs are running.

Better insight with test reports

Last, but not least, you can configure your test framework to generate reports in a format compatible with Semaphore test reports. This way, Semaphore can collect the output from several runs into one convenient, easy-to-read report that shows a wider perspective of the state of your acceptance tests over time.

Streamlining acceptance tests

Optimizing the application for testability dramatically reduces the effort of maintaining tests. Stick to these points in order to keep them streamlined:

• Avoid production environments: acceptance tests should not run in a real-life production system. At the same time, the test environment must closely resemble the production environment. When integration with external systems takes place, you should mock services and use test doubles.
• Avoid production data: avoid the temptation of using a production database dump in the testing environment. Instead, each test should populate the database with the needed values and clean up afterward. Keep the test dataset as minimal as possible.
• Respect encapsulation: don’t break code encapsulation. Test using the same public functions or APIs offered by your code. Avoid the temptation of adding a privileged backdoor to run a test.
• Keep them loose: coupling tests to code too tightly leads to false positives and extra maintenance work.
• Expose programmatic access: this only happens in UI-only applications. When the UI is the only way of interacting with the code, testing it is a lot more complicated.

Conclusion

Acceptance tests are an integral part of behavior-driven development and the primary tool we have to ensure that we fulfill our business goals.

Further reading:

• Continuous Integration & Delivery Explained
• 20 Types of Tests Every Developer Should Know
• Revving up Continuous Integration with Parallel Testing
• Continuous Integration for Monorepos
• Design an Effective Build Stage for Continuous Integration

Thank you for reading!