In the Italian language, there is a single word for computer science: Informatica. It is a portmanteau of “informazione automatica” and, as you may have already spotted, it means something like information automation.
Automation is a pillar of information technology. Among experienced software professionals, it becomes instinctual. Every time you find yourself doing the same job over and over again, the desire to write a script and automate a repetitive task asserts itself.
This attitude is key for software developers, but sometimes things get overcomplicated. Then, automation becomes difficult, and manual work becomes necessary. This article describes what automated testing is, the path to a fully automated test suite, and the challenges that lie along the way.
What is automated testing?
Like any other material good, software needs to be tested before reaching the customer. The obvious way to test something is to use it for a while to make sure it behaves as expected. This is the way most developers in the past used to check their work before shipping it.
After all, you have written it, you have it on your computer, you know how to run it, and you know how to quickly test it as you make changes.
The flip side is that you as the creator are biased. You might forget to test some parts or might not realize the ramifications of a change. Also, you can’t be 100% sure that your code will work in an environment that isn’t your development machine.
Even if you do everything right, testing is a boring task — a time-consuming activity where skilled, creative professionals are forced to do mindless drudgery. Historically, the countermeasure was hiring people to test, making developers happy and removing biases.
This seemingly win-win solution makes things worse more often than not. Developers lose the overall picture and awareness of all the moving parts when they aren’t involved in testing. Testers, on the other hand, spend an enormous amount of time just making things work and have to bother developers whenever they find something they don’t understand.
The role of a QA team when testing is automated
Testers are part of what is known as the Quality Assurance (QA) team. They are paramount for shipping high-quality products, so it’s important to let them do their job efficiently. They do not have to test everything every time, nor catch unhandled exceptions or 404 errors. They are there to help developers and companies to deliver good quality software. Otherwise, the “QA is doing the testing anyway” mindset settles in, giving rise to dysfunctional behaviors.
The inflection point for automated testing
In 1999, the book Extreme Programming Explained laid the groundwork for a revolution, which later resulted in the Manifesto for Agile Software Development. In this text, we find the cornerstones of modern software development:
- Rapid feedback loops.
- Simple and safe ways to modify code. As Kent Beck said, “make the change easy, then make the easy change.”
- A happy and satisfied team when the work is done.
In this field, automated testing plays a key role. Let’s have a look at some kinds of tests we can put into use.
From unit tests to end-to-end tests
The topic of testing is very broad and there are dozens of different types of tests, depending on the needs of the developer. Let’s suppose, for example, that we want to create software that can manage prices in a supermarket. The code below is an example of a unit test:
@Test
public void When_I_add_an_apple_the_system_charges_50_cents() {
CashRegister register = new CashRegister();
register.add("apple");
int expectedCost = 50;
int actualCost = register.getAmount();
assertEquals(expectedCost, actualCost);
}In this test, we are verifying that our cash register charges 50 cents for an apple. Although simplified, the example gives a good idea of what it means to test a “unit of code” (the cash register), in a very focused way. If the price of an apple changes or the system stops pricing apples as expected, this test will alert us. A unit test validates an assertion on a specific portion of code.
There are other types of tests, aimed at ensuring the expected behavior of the application as a whole, from the graphical interface to the database. These kinds of tests are called acceptance and end-to-end (E2E) tests.
Our range of tests, therefore, goes from unit tests, which are focused on the technical details, to acceptance tests, which instead show that the required business objectives are being met.
Feature: Supermarket Bundles
To increase the revenues as a Supermarket Manager,
I want to apply discounts and offers to my customers
@Bundles
Scenario: if customer buys 3 apples, he pays $1.00
Given the apples costs 50 cents each
Given the “buy 3 apples, pay 2” promotion
When the cashier scans 3 apples
Then the cash register charges $1.00In the excerpt above, we see an example of a specification written in Gherkin, a language made famous by Cucumber and used in Behavior Driven Development (BDD). Despite appearing as natural language, there are automated tests hidden below the statements in this scenario.
The characteristics of a good test
All good tests have some features in common:
- Good tests improve quality. Tests drive quality and can act as a gate to prevent bad commits from reaching production. A low quality commit can be anything ranging from code does not pass static tests to code that does not meet basic security checks.
- A good test reduces the risk of introducing failures. Tests find errors and regressions. They should be robust in order to not introduce false positives or flakiness.
- Good tests helps to understand the code. We should always be able to use the test as a reference for code behavior.
- Good tests are easy to write. Tests are code too and can have errors. Because it’s impractical to test tests, they should be easy to write to reduce the chance of introducing false positives.
- Good tests are easy to run. Automated tests should start without any human intervention. A CI/CD platform like Semaphore can run your tests on every change. You can also configure tests to run automatically using Git hooks or by configuring your IDE.
- A good suite needs minimal maintenance. Refactoring should not break tests unnecessarily. This can be achieved by not tying the code to tightly to the implementation.
The heart of Continuous Integration (CI)
At regular intervals, developers integrate changes with those of the other team members, using a versioning tool such as Git. A well-established team has a strong Continuous Integration policy and a CI pipeline that, upon noticing the changes, will take the new code from the repository and run tests on it — from quick unit tests to more rigorous acceptance tests. More costly tests can be run less frequently, such as once or twice a day. This lets the developers know immediately, with a good degree of confidence, if something stops working.
As in many aspects of life, there are trade-offs in testing strategy. It’s up to the developers and testers to negotiate the best compromise, ensuring quality and efficiency.
Simplicity and automation are prerequisites for safety
As mentioned earlier, software professionals must ensure the quality of their own work. Quality means low defectiveness, and low defectiveness means a vast amount of testing. Thus, writing good tests and building a system that can execute them quickly and accurately isn’t optional. Teams must commit to building a safety net with CI/CD to ensure smooth development.
The Myths of Test Automation
Despite the growing awareness achieved by companies and software developers in recent years, test automation is still plagued by some myths.
Testing slows down development
Those who claim that testing slows down development are both wrong and right. They’re right because in some cases, especially in TDD, writing tests makes developers take the time to think ahead about the result they want to achieve, setting aside the frenzy of writing code for a moment. In this case, slowing down is an act of will, a precise means of focusing on the problem ahead.
In the long run, however, those who think that writing tests are a waste of time are wrong. We must think of the time spent writing tests as an investment because a failure-safe environment makes it possible to make changes and experiments at a very high speed. Developers can come to a solution in a fraction of the time it would take if they constantly had to be careful about where they put their feet.
It’s like walking home on the sidewalk versus walking on a tightrope stretched between two buildings: tightrope walkers are not famous for their speed.
Abraham Lincoln used to say “Give me six hours to chop down a tree and I will spend the first four sharpening the axe“. Testing automation is like having a little helper that constantly sharpens your axe, so you can “hack” down any problems that pop up without spending hours sharpening it yourself first.
Testing is only for finding bugs
Another myth is that testing is only for finding bugs. Going after bugs is one possible purpose, but certainly not the only one. The real advantage lies in making bugs easily detectable and fixable after every single change. That is what test automation is all about: making it easier for a developer to alter code because they know they have a safety net.
You must achieve 100% coverage
Modern development environments allow you to calculate a metric called code coverage. It represents the amount of code covered by tests. It is quite natural for the non-test-savvy to think that 100% coverage guarantees the total absence of bugs in their codebase. This belief, apart from being incorrect, is a harbinger of bad practices.
Coverage only guarantees that there is at least one test that puts a certain portion of code under scrutiny. It does not guarantee that said test makes sense, that it verifies all the possible combinations of inputs, or that it will always behave as expected.
Writing tests is a profession somewhere between art and science, and coverage is certainly not the tool that guarantees the quality of a test suite. Furthermore, the drive for 100% coverage leads developers to test trivial portions of code, wasting time and money, and feeding an unhealthy perception of procedural infallibility. Setting a more reasonable threshold, such as 80%, might be a good compromise.
How to define a good testing strategy?
At this point, the question arises: what are the steps to creating a good test suite? Let’s try to define a roadmap for a company or a team that wants to change course and start testing its codebase.
Start testing
No, this is not a joke: the first thing you need to do is start testing. Usually a simple query on a search engine like “<your language> testing tools” provides plenty of hints and tutorials to get started.
But even if you’re using outdated or abstruse languages, you can always find a way to test your product. If there are no frameworks or ready-made tools on the market, you can always write tests using pure programming languages or with bare shell scripts.
To paraphrase Walt Disney, “If you can run it, you can test it.“
Identify a meaningful part, and write an automated test
In codebases that have developed with no inclination to testability, the road is uphill at first.
The first tests written should offer a good balance between value and affordability, i.e. code that is sufficiently easy to approach, but that at the same time can provide you with some value. Testing a part that never breaks, a trivial feature, or one that almost nobody uses is not helpful. Moreover, going straight to the main feature of your product usually ends with brittle tests and frustrated developers.
Most of the time, we can start by writing some end-to-end tests that try to emulate the behavior of the user. If we have a web application, for example, we can begin by training a browser with tools like Selenium or Cypress. Let’s not, however, limit ourselves to tools; we also have to be ready to manually write our test routines, shell scripts, and whatever else is needed to make the test automatic.
In the journey we take, we always discover useful details for increasing our understanding of the codebase, details that will be valuable when we go to modify or refactor bits of code. Once we understand the basic principles of testing, we can start learning about testing frameworks, mocking libraries, and all the tools that make a developer’s job easier.
NB: Semaphore has a neat Test Reports feature that allows you to see which tests have failed, find the slowest tests in your test suite, and find skipped tests in your Test Reports dashboard. Read more about Test Reports.
Create your safety net
A good test suite is a safety net. In the beginning, when you don’t have tests, every change is an unknown risk. Over time, and with the addition of tests, the risk decreases, confidence increases, and development accelerates.
It’s a long process. You can’t hope to solve all the accumulated problems in a few weeks. The important thing is to keep adding tests, and reinforcing testing practices in the team.
Make automation an integral part from the beginning. If a new project is about to take off, don’t waste the opportunity to start off with the right foot with a well-rounded testing strategy and a solid set of CI tools. Once the right procedure for conducting a test has been identified, the test must be automated, and then it must be run in a secure, isolated environment: a CI/CD pipeline.
This is about feedback: the longer you wait, the later you will know if your automated tests are as isolated and idempotent as you want them to be, or if they “just work on your computer”.