Software testing has undergone a significant transformation in recent years. It has evolved from being an isolated, time-consuming activity performed at the end of the development cycle, to an integrated and ongoing process tied to customer success. Various factors have driven this shift, including the rise of agile development methodologies and the need for faster and more efficient testing processes. In this article, we’ll explore, alongside Unity VP of Engineering Services Alan Page, the evolution of software testing and its impact on the software development industry, and discuss how testing practices have changed the way software is built and delivered.
Edited transcription
Alan Page started his career in tech in the early nineties. Upon working on diverse jobs, he didn’t take long to specialize in testing, API testing, and test tool writing. Moreover, his long career also led him to pursue infrastructure, as well as software build processes and tools even before continuous integration (CI) practices became popular. Nonetheless, Alan’s experience in testing would lead him to work at Microsoft doing network testing and eventually teaching testers, designing courses, and guiding clients on how to use Microsoft’s products.
In 2008, as a way of writing down his knowledge from educating teams on testing subjects, Alan published How We Test Software at Microsoft. With a few chapters written by his colleagues Ken Johnston and BJ Rollison, Alan’s book became a milestone on the subject and crowned him as a testing authority. While Alan admits that part of the book has become dated, he is confident that another part of it stands up to this day as a valuable source of knowledge in testing.
Around five years ago, Alan joined Unity, a 3D engine popular for creating video games, to work doing both testing and infrastructure. After decades of working in the industry, Alan recognizes that, as time went by, he now cares “more and more about quality every day”, as his work now focuses on “improving quality and customer experience.” To this end, he now has “become less concerned about testing,” and believes that “testing is one way in which we achieve quality, but it’s not the only way.”
Dedicated testers or developers testing?
Does defining testing as one of many ways of achieving quality challenge the role of testers and testing? While Alan points out that, in the future, dedicated testers will not be as needed, this doesn’t mean software will stop demand testing.
“Functional correctness, no matter what you are working on, is, today, the job of the developer,” says Alan. As functionality depends on the developers, dealing with functional bugs, those related to product functionality, should be in their domain. Therefore, developers are responsible for “monitoring to understand how it’s being used [the code] and to find the potential issues that need to be fixed.”
Withal, Alan concedes that complex software will still require dedicated testers. However, their role will not be separated from the rest of the team and the product’s success. Dedicated testers should be committed “to find bugs that really could only be found by someone using the product end-to-end and thinking about customer outcomes”. Testing should not be involved externally (and even less bypassed) but integrated into production and shape how teams think and work.
From that perspective, Alan considers that “everything needs to be collaborative from day one”. Testers should pair with developers and help them understand testing techniques and ideas to the point “everybody should do some aspect of testing”, following a philosophy in which every team member“ should know a lot about a little and a little about a lot.” Hence, even though some individuals are going to be more proficient in certain tools than their colleagues, it is crucial not to rely on their knowledge to avoid growing dependent on them.
Testing and release cycles
The transformation of testing has also had an impact on software delivery. Long shipping cycles have become a thing of the past in most industries, so customers can (and have become used to) get the latest software faster than ever before.
Not so long ago, Alan recalls, testing was a sort of releasing bottleneck that halted release on the premise of testing and finding bugs so they could release a rather perfect product; However, Alan believes that “these days, testing is truly about accelerating the business and making things move faster.”
What’s more, aside from delivery frequency, testing plays a central role in finding the right balance between delivering earlier and creating a superb product. CI testing toolings —such as code quality checkers and static analysis tools such as linters— have transformed the developer’s role to be more testing oriented and make them responsible for balancing quality and release time,
Delivering and updating on-premise software
While testing and CI are meant to feed shorter feedback loops. This is also the case for on-premise software, whose feedback loops are slower than, for example, web services.
On-premise software can be tracked with DORA metrics, a popular set of metrics that serve to measure product success. “Just because it’s an on-prem product doesn’t mean you can’t use the traditional DevOps metrics,” Alan affirms. These metrics are:
- Deployment frequency
- Mean leader time for changes
- Mean time to recover
- Change failure rate
Another aspect to consider when developing a product to be delivered on-premise is the customer’s preference for update cycles. In his experience working at Unity, Alan says that many customers working with on-premise versions choose to stick with it for as long as their project goes to prevent having to deal with update changes. Despite this being mostly a risk assessment IT choice, customers should be willing to adopt the latest iteration of your product and not be forced to.
Of course, delaying updates is not the case for every user; many others are willing to update to get the latest features, solve bugs, and security updates. Hence, what’s important is making the latest builds available for those who want them and allowing those who prefer past versions to go with them. In this regard, Alan advises companies to implement a versioning model: “You can have one version scheming, Semantic Version or whatever, that ships yearly; another one that ships monthly; another one that ships weekly, and customers can choose which train to get on depending on, well, initially their aversion to risk.”
Fixing legacy code through testing
Alan recognizes that dealing with legacy code can frustrate testers. The organization they work for might be unwilling to fix a bug to prevent breaking the code, even if affects the end customer experience. Besides, legacy code documentation might be incomplete properly, or the people who worked on it and knew about it might as well have left the organization long ago.
While fixing legacy code might be difficult —and even advised against—, Alan encourages developers to push over the line. Working with legacy code, he says is a matter of learning to deal with the unknown: “a lot of times the legacy code that no one wants to touch is also because no one understands it, and there may be parts of it that you understand why nobody wants to touch, but there may be parts of that code that may be okay to touch.”
As a way of learning more about legacy code, Alan recommends writing unit tests. Unit tests are automated tests that test individual units or components of the software, such as functions, methods, or classes, in isolation from the rest of the system. The goal of unit testing is to validate that each unit of the software application is working as intended.
When reading and working with legacy code, unit tests can serve as a valuable source of information about the code’s design and functionality, allowing developers to understand how it works and make changes with more confidence. In this way, it is possible to determine the morphology of the bug and obtain “a better idea of where to look and what the risks may be in making a change.”
The bottom line
Alan has a podcast, ABTesting, on testing and other software development topics. Visit Alan’s blog, angryweasel.com, to learn more about his views and read his published works. You can also follow him on Twitter.
