Software development is a discipline that includes multiple means to ensure quality. Testing, with its many practices and tools, is one of the most prominent. Every developer should have a clear picture of the possibilities available in this field. This article gathers common knowledge on the subject and offers an up-to-date picture to choose the most suitable approach and tools for each and every circumstance.
How to categorize types of tests
One of the earliest and most famous attempts to shed light on the testing subject was the Agile Testing Matrix by Brian Marick. In this set of articles, the author offers a reading key to connote different approaches to testing:
- The Y Axis spans the length between technology-facing and business-facing tests.
- The X Axis goes from supporting development to critiquing the product.
X-axis: development support and product critique
Consider the division from left to right in the matrix. On the left, there are the tools and practices that support the act of programming. Their purpose is to help developers build software components that meet quality and functional requirements. Some examples are static tests, unit tests, and integration tests.
On the right side, there are tests that put the entire software solution under the lens in search of defects. The aim is to critique the product and discover potential issues or omissions. In this category, there are various non-functional tests, which may evaluate performance and verify security concerns.
Y-axis: technology and business
On the Y-axis, we can arrange tests as business-facing or technology-facing. The first represent tests that demonstrate the suitability of a solution to a concrete business case. It describes features a business expert is interested in without technical jargon, just plain language and domain concepts.
Let’s say that you are developing software for online banking. In this scenario, a business-facing test would check that: “Given the account balance is $1000. When the Account Holder makes a wire transfer of $200. Then the account balance should be $800.”
On the other side, you can spot technology-facing tests because they speak the language of programmers. The concern is less interesting for business experts, but equally important to realize a robust solution. Continuing with the previous example: “Different browsers implement JavaScript in different ways, so we test whether our wire transfer feature works with the most important ones.”
An updated testing matrix
Over time, new testing practices and tools have proven effective. In the following paragraphs, we’ll examine an updated view of the four quadrants, describing the traits that make each testing approach attractive.
Support development, technology-facing quadrant
Starting from the lower-left quadrant, we have tests that support the act of code writing. These types of tests are used by developers as a guide for building working software, with an eye on quality and technical details.
Static testing
In this portion of the map, we find static testing, which is an umbrella term for all tests performed without actually executing code. Here are some of the most relevant ones.
- Linting: this term identifies the process of checking the source code for programmatic and stylistic errors. A lint (or linter) is a program that enables linting. Linters are available for most languages. Some renowned linters are JSLint, CSSLint, Pylint.
- Type checking: in the world of programming languages, we have strong and weak type systems. When the type system is strong, the compiler warns you in the event of typos and errors. But when the type system is weak, like in JavaScript, some mistakes are hard to detect. In this case, a good type-checking library like Flow can come in handy.
- Static code analysis: finally, there are other tools that help to ensure code quality by analyzing code and giving useful insights. Suites like SonarQube, PhpMetrics, or SpotBugs can provide metrics such as cyclomatic complexity, vulnerability reports, and feedback about technical debt.
Unit testing
Unit tests are typically automated tests written and run by developers. They ensure that a part of the application (called a “unit”) conforms to its scope and behaves as expected. In object-oriented programming, a unit is usually a complete class or a single method. Using a car as a metaphor, a disk brake is a unit. A unit can be a complete module, but more commonly it is a separate function or process.
The Test-Driven Development (TDD) approach invites the developer to write unit tests before the actual implementation, using them as a guide for coding. Developers are used to accomplishing this job with the support of testing frameworks like JUnit for Java or Jest for JavaScript.
Integration testing
One level above unit testing, there are integration tests. While unit testing is about writing a good implementation of the desired feature, integration testing is about verifying the cooperation of a set of units (components). Recalling the previous metaphor, an integration test checks the entire braking system of the car.
Integration tests can be written by using the same testing frameworks available for unit testing. There are some specialized tools that help in applying test doubles techniques, like Mockito or Mocha for example, or some other solution to test the integration of units with databases, like DBUnit, or with REST APIS, like REST-Assured.
System testing
In system testing we move even higher, testing the interaction of the components. Finally, getting back to our metaphor, it’s time to put the whole car on the test bench and check to see if the important things are working, like the engine and transmission.
System testing evaluates the system’s compliance with the product’s requirements; furthermore, it looks for defects both in the assemblages of components and also within the system as a whole. A well-known tool for this purpose is the Robot Framework.
Honorary mention: mutation testing
Mutation testing is a fascinating technique. A “mutation” is a portion of code purposely modified to verify if there are tests in place able to catch the generated errors. If a mutation survives the tests, you know there’s a leak. In the car example, it is like disconnecting a headlight cable to see if the corresponding warning light shows up on the dashboard. In this area, PITest is a preeminent tool, along with StrykerJS.
Support development, business-facing quadrant
Moving upwards in our matrix. For these types of tests, we shift to the business-facing region, holding all the tests that allow stakeholders and developers to check the current understanding of the business rule. While developing a car, this is where test drivers start pushing it to its limits on the track.
Acceptance testing
Acceptance tests are a sort of contract within business and development teams. For example, when considering a feature done, it’s useful to have at least an acceptance test that demonstrates that it respects business rules and behaves correctly. Acceptance tests are usually part of user stories.
This could be an example for our car metaphor: “Taking a car that has just left the factory, and a road with asphalt in good condition, the braking distance to go from 100km/h to zero must be less than 100 meters.”
Acceptance tests are different in meaning, but not in form. We can use some of the aforementioned testing frameworks to compose them, or maybe take advantage of the Behavior Driven Development (BDD) approach and tools like Cucumber to better express the intent and foster a common language between developers and business experts.
End-to-end testing
End-to-end testing (E2E) happens when we try out our software solution to see if the main features work as expected, i.e. to see if “all the cables are connected”. With this method, we can see if all the functional requirements are met. Continuing with the metaphor, we take the new car on a short test drive to ensure everything is working as expected before delivering it to the car dealer.
Many of the aforementioned tools are suitable for putting these kinds of tests in place; while building a web app, Selenium, Cypress or Postman can come in handy.
Smoke testing
Smoke tests are a subset of E2E tests, looking for possible showstoppers. The objective is not to perform exhaustive testing, but to verify that the essential functionalities of the system are working. These kinds of tests are usually automated, but in some complex solutions, human intervention is required.
Honorary mention: contract testing
Contract testing is a methodology for ensuring that two independent systems (such as two microservices) are compatible and able to communicate with each other. It captures the interactions that are exchanged between each service, storing them in a contract, which can then be used to verify that both parties adhere to it. The difference between this form of testing and other methods with the same objective is that each system can be tested independently of other systems. The contract is generated by the code itself, which means that it is always up-to-date. The main tool for this is Pact.
Critiquing the product, business-facing quadrant
Moving to the right, we have a family of tests aimed to verify things we cannot easily automate, like the accessibility and usability of a website or the presence of undesired behaviors.
Tech demos
If you know about Scrum and its events, you already know about the sprint reviews. Or maybe you are familiar with User Acceptance Tests (UAT). Demonstrating software at work (and collecting feedback, of course) is one of the good things you can do to minimize the risk of errors. Tech demos are mostly to confirm expected behavior, by using the software solution as users do. This is of course a manual activity. It is like inviting the potential buyer to take a test drive and see if yours is the car they’re looking for.
Exploratory testing
While demos are about confirming existing paths, exploratory testing is about finding new ones. This is where QA engineers shine in their ability to find “unintended functionalities”.
Exploratory testing is also often performed by project or demand managers. To make a comparison, one of the most famous exploratory tests in the automobile industry is the moose test, which determines how well a certain vehicle can evade an unexpected obstacle.
Usability testing
Usability testing consists of observing a person handling with the product. Generally, there is a facilitator that asks the person to perform open tasks (such as: “you’re looking for a gift on this e-commerce site: what would you do?“) or closed (such as: “use the navigation menu to see the degree courses of the University“).
Another distinctive element is the presence of a researcher in an adjoining observation room, analyzing the subject’s behavior. The purpose of these types of tests is to collect feedback to improve the solution.
Alpha and beta testing
Big-bang delivery can be risky. A mitigation strategy is to first offer the new or revamped features to a strict circle of people who are willing to accept glitches and happy to provide feedback.
This is basically the purpose of the alpha and beta testing, even if there are some slight differences between the two: in the alpha stage, you enlist employees working for your company. While the beta testing is open to people outside the company, who comprise a circle of trustworthy customers.
The other important thing is that in the alpha stage you generally use testing environments, focusing more on the features than other technical aspects like performance or reliability; whereas in the beta stage you test the final solution, in the actual environment. When the beta stage ends, the solution is ready to be opened to all customers.
Honorary mention: Accessibility, Internationalization, Localization
A global solution cannot ignore themes like internationalization, localization, and accessibility. Accessibility (a11y) is the practice of making the product usable by as many people as possible. We traditionally think of this as being about people with disabilities, but the practice of making sites more accessible also benefits other groups such as those using mobile devices or those with slow network connections.
In the web field, the standard is the Web Content Accessibility Guidelines (WCAG). While there are some tools (e.g. to verify the correct use of available HMTL tags for screenreaders), these tests are mostly performed by humans. For example, in web applications, basic tests include browsing in keyboard-only mode or blindly with the help of a speech synthesis system.
Internationalization (i18n) is the process of designing a software application so that it can be adapted to various languages and regions without any code changes. This is the home of themes like encoding, date formats, or text orientation.
Localization (l10n) is the process of making the application adaptable to meet the cultural, lingual, and other requirements of a specific region or a locale, by adding locale-specific components. It’s not only about translating text, but also about satisfying local regulations and habits.
Critiquing the product, technology-facing quadrant
The last quadrant groups all the tests that explore non-functional aspects of software solutions, like resilience and security. The focus is to ensure the readiness of the entire system.
Non-functional testing
The purpose of this group of tests is to verify non-functional aspects (performance, usability, reliability, among others) of a software application. Functional aspects, like business rules and features, are already tested elsewhere: the focus is on its availability in all possible situations.
Below is a categorization of the most relevant test types:
- Resilience testing: is a way to refer to testing techniques that observe how applications act under stress. It’s meant to ensure the product’s ability to perform in chaotic conditions without loss of core functions or data. This ensures quick recovery after unforeseen, uncontrollable events. Today, there are software solutions so vast that you can only test them in production. In this situation, techniques like chaos engineering show what happens when things go wrong. Netflix is a pioneer in this field, with its well-known chaos monkey tool.
- Performance testing: aimed to test the sustainability of our application and its components. People and tools push software to the limit to discover and define acceptable loads. Stress testing or load testing are other common ways to classify these activities. JMeter is a well-established tool in this field, but new tools like Taurus are gaining momentum.
- Security testing: under this term, we have all the tests you can do to ensure your software solution addresses known security pitfalls. The experts in the software development field know there is no “100% secure” software. However, this testing technique defines what “secure enough” means to our solution. The biggest proponent in this field is the OWASP Foundation. On the foundation’s website, there is a curated list of tools for application security and vulnerability scanning.
NB: for any type of tests, Semaphore has a neat feature that allows you to see failed tests via the Test Reports dashboard. You can also see the slowest tests in your test suite and find skipped tests. Read more about Test Reports.
Conclusion
This article has shed some light on the important testing categories that characterize the world of software development. There is no precise recipe regarding which to adopt, much depends on the context of your software. The proposed diagram design helps to understand which testing areas need to be covered, leaving it to the individual developer to draw on their experience and find the right balance in a given situation.
Deepen your testing kung-fu with these recent posts: