There’s so much more to testing than just writing automated tests that run in CI. Testers on high-performing teams don’t just write tests. They work closely with site reliability engineers, ensuring that the infrastructure is tested as well. They get involved in production.
But how does one introduce this holistic approach to testing to one’s team? Even more so, how does one introduce continuous integration to an organization, if it hasn’t been adopted yet?
In this podcast episode, we welcome Lisa Crispin, Author, Agile Testing Coach, “tester by trade”, in her own words. Among other things, we talk about a holistic approach to testing, how to shift from shipping many times a day to once a month, and how to help organizations adopt continuous integration. Listen to the full episode or read the edited transcript.
Table of contents:
- What’s more to testing than just writing tests?
- Exploratory testing
- Holistic approach to testing
- Modern trends in testing
- Implementing CI in organizations
- Testing as a practice: patterns and antipatterns
You can also get Semaphore Uncut on Apple Podcasts, Spotify, Google Podcasts, Stitcher, and more.
Like this episode? Be sure to leave a ⭐️⭐️⭐️⭐️⭐️ review on the podcast player of your choice and share it with your friends.
Introduction
Darko Fabijan (00:02):
Hello and welcome to Semaphore Uncut, a podcast for developers about building great products. Today, I’m excited to welcome Lisa Crispin. Lisa, thank you so much for joining us.
Lisa Crispin (00:11):
Well, thank you very much, Darko. It’s an honor to be here and I’m very interested in learning more about your community of developers.
Darko Fabijan (00:20):
Great, happy to have you here. Can you just go ahead and introduce yourself?
Lisa Crispin (00:26):
Sure. I’m a tester by trade. I’ve been in the software business for 40 years now, which is really amazing. For more than 20 years, I’ve been lucky enough to work on agile teams, mostly what would be termed high performing agile teams.
We didn’t start out that way. In the due course, I learned all the good modern software development practices, and saw the ways we can build quality into our products. Through sharing my experiences along with a lot of other people’s experiences, we could learn from each other and work together to solve whatever new problems that we come up with.
My goal is to help people learn to collaborate, work together so they can learn more easily and faster. Because it’s all about continuous improvement.
I live in Vermont on a little farm with donkeys and dogs and cats and a husband. It’s very nice that just when we moved to Vermont, more people started working remotely. Even though it wasn’t for a great reason, it’s a silver lining.
Holistic approach to testing
Darko Fabijan (01:36):
Thanks for that introduction. So as I was thinking about this episode and the majority of our listeners, it’s that definition of testing and it could mean different things for different people.
Probably for the majority of our listeners, it’s that element of writing automated tests to ensure that your small or bigger piece of code feature is working as intended and then passing it off to a continuous integration. Everything is green pushing into production.
But systems are complex. They’re also public clouds, different architecture types of microservices and mobile labs and so on. It’s always just getting more complicated. There are more components in place.
So I think it would be great if you could shed some light on the aspects of testing in a more holistic approach. What’s more out there and how does it connect together, apart from writing those continuous or integration tests?
Lisa Crispin (02:49):
Of course, continuous integration is a core practice that I believe no team will succeed long term without. And of course, we want automated regression tests to run in our continuous integration and make sure that we’ve met the requirements for our features.
We’ve provided value that customers want in terms of functionality, we’ve guided our development using these tests with test driven development, behavior driven development. That’s super important and again, that’s a foundational practice.
Lisa Crispin (03:25):
Over time, we realized there were a lot of other aspects of quality that were as important as meeting the requirements that the customers gave us.
So testing activities encompass all of those. At the same time, our systems have gotten way more complex. We’re running in the cloud, we have microservices, the architectures have changed.
Even in the old days, we could not reproduce a production environment in tests, it’s generally impossible. So we can test as much as we want in our test environment, but we’re going to miss some things. Customers are unpredictable, everything is unpredictable.
Exploratory testing
Lisa Crispin (04:15):
We started expanding into other types of testing activities that are equally important, things like exploratory testing. Once we’ve checked that we’ve met the requirements, what else might we have missed? What might our customer have missed in defining their requirements?
So we explore our features. We put on our customer persona hats and it’s like, what would a customer do here? We might do chaos engineering. What would happen if a server went down or a database went down and we start having to explore various scenarios that we couldn’t have thought of upfront?
And then we have things like accessibility, security, reliability, sustainability. They’re all these aspects of quality that are really important.
Lisa Crispin (05:02):
With the advent of the DevOps model with those two loops, we’re building a feature on the left and we’re deploying it and taking care of it in production and learning on the rate. Dan Ashby was the first person I know of that took that DevOps loop model and said, “You know what? We do testing activities all the way around this loop.” He called it the continuous testing loop.
But over time we discovered that when people say continuous testing, they’re only referring to the automated tests that run in CI. Those tests are important, but there’s so much more to testing all the way around that DevOps loop.
There’s so much more to testing all the way around that DevOps loop than just the automated tests that run in CI.
-Lisa Crispin, Author, Agile Testing Coach
What is holistic testing?
Lisa Crispin (05:42):
So, my co-author and business partner, Janet Gregory came up with this model for holistic testing. Let’s look at the activities in the software development continuous loop and what testing activities are involved along with those activities.
We really need the tester mindset, the testing skills, things like we’re great question askers. When people use the term QA, I think question asker. We’re great at identifying risks, we’re great at noticing patterns.
When people use the term QA, I think – question asker. We’re great at identifying risks and noticing patterns.
-Lisa Crispin, Author, Agile Testing Coach
Lisa Crispin (06:17):
I’ve been trying to get testers to see that they have to be involved in testing and production. People get scared by that term. Observability, learning about our systems in production, identifying problems quickly because we couldn’t anticipate all those problems.
At the same time, working with site reliability engineers to see that infrastructure needs to be tested as well. Testing perspective is useful when we’re looking at all of our log data and all of our metrics and all of our alerts.
Lisa Crispin (07:08):
My mission for the last few years is integrating those things and taking that holistic testing. I’m really glad that Janet came up with that term holistic testing, because when I’ve shared that with people I know who do the same kind of work I do, they’re like, “Yes.” They look at that model and they say, “That is exactly what we do.”
So that’s what testers on high performing teams are doing and we want to spread that idea to more people.
Shift from shipping many times a day vs once a week
Darko Fabijan (07:58):
And in terms of changes, you mentioned at some point extreme programming and the practices that came out of that. It was also a time where a lot of things started moving to being always available and offer as services through the internet.
But although there is still a significant portion of software which is shipped many times a day, it has a different release cycle.
As you, through your career, experienced that shift from being always on deploying many times per day, versus we are going to have a monthly, biweekly, whatever recycle. I’m curious to hear, how do you see those differences and shifts and how that changes things for QA people?
Lisa Crispin (08:55):
That’s a good question. I’ve been on teams that for business reasons, we only released every two weeks because our customers didn’t want even small changes more frequently than that. And because our customer support had to be notified, there was just some coordination that had to happen.
And yet we considered ourselves a team practicing continuous delivery because we always had a deployable bill that we could put in production.
And I think that’s really key. I think, no matter how often you deploy or release a production, I think continuous delivery is a benefit. It’s hard to imagine where it wouldn’t be a benefit, if somebody knows of a place, I guess I’d like to know.
But even back in my days on waterfall teams, waterfall has a bad name now, but I was on a waterfall team in the early to mid nineties, we had all the practices.
Lisa Crispin (09:53):
We did continuous integration, we had automated deployments, we had practically a hundred percent regression test coverage at the unit level and the UI level. We worked together through all those different phases, we were a database software company.
Back in the early to mid nineties, nobody needed a new version more than every six months or once a year but yet, we pretty much had a deployable build at any given time because we were running our continuous integration every day. And these are not new practices, it’s just that extreme programming popularized them.
So I think it’s a separate thing and I think it’s a thing where the business and the technology sides of the company have to work together. The business has to decide what cadence works for them and they have to adapt how they’re requesting new capabilities from the technology side, make that congruent with their cadence so that on the technology side, we understand we still want to deliver in small increments.
But when do they want to flip the switch and actually show things to customers, and I think it’s important to coordinate those things.
The business has to decide what cadence works for them and they have to adapt how they’re requesting new capabilities from the technology side, make that congruent with their cadence so that on the technology side.
-Lisa Crispin, Author, Agile Testing Coach
Modern trends in testing
Darko Fabijan (11:03):
Clear. And over time, as you are entering organizations and helping them in various ways, what are some things that are still around perhaps and maybe shouldn’t be? And what’s the rate of change that you see?
Lisa Crispin (11:20):
I’ve been lucky to be in magical unicorn land for a lot of the last 22 years. Not that teams necessarily started out that way, but that we had enlightened executives or co-founders who were like, “Well, our business model depends on everything being automated. So let’s hire some really good people who know how to develop software and let’s let them develop the software in the way they know the best and let’s support them and collaborate with them.”
The majority of teams are a long way from high performing
Lisa Crispin (11:50):
So unfortunately I find that’s extremely rare. So yes, I still see the majority of teams are a long way from high performing. And we see that in surveys like the DORA’s State of DevOps surveys and things like that. There are more and more teams achieving those levels of being able to deliver frequently at a sustainable pace so that their teams are happy, their customers are happy.
Things changing for women in the software industry
Lisa Crispin (12:18):
We see that’s happening a little more. But when I started in the software business in the early 1980s and was asked by the person interviewing me if I baked because they really liked having baked goods in the office, I thought things changed a lot for women in software too.
But I see that things have changed, but they’re slow. And in the testing side of things, I have seen a pretty dramatic shift in the last 20 years.
When I started in the software business in the early 1980s, I was asked by the person interviewing me if I baked because they really liked having baked goods in the office. I see things have changed a lot for women in software too. But the changes are slow.
-Lisa Crispin, Author, Agile Testing Coach
Lisa Crispin (12:44):
In the early 2000s, Janet Gregory and I were among the lone voices in the wilderness saying, “Well first of all, you XP teams need testers. We can add a lot of value. And second of all, you testers, this is a good way to work. It’s focused on quality, it’s focused on people. We’re building quality in, we’re preventing bugs, not finding them after the fact. It’s a lot more fun, we’re doing it together”.
And so these concepts, there’s a wide testing community now that has embraced these. And those are normal ideas to them, whereas when Janet and I were first going to conferences or writing articles about them, people were like, “What?”
Many organizations still don’t have continuous integration
Lisa Crispin (13:27):
So I’ve seen some positive changes. And yet I still run into sizable, significant, big companies and government organizations that still don’t have continuous integration, which to me is just a basic core practice you can’t do without.
And if I’m doing a training course and they don’t have continuous integration, I want to say, “Well, save your money. Stop doing this, go get your continuous integration running and come back, because the things I’m teaching you will not help you without that.”
Implementing continuous integration in organizations
Darko Fabijan (13:58):
You almost asked the question that I wanted to ask. And we do also from time to time face organizations who are reasonably successful, not small, but are very far behind.
What you mentioned you were doing in the early nineties, with all practices being in place, although it didn’t deploy and release five times per day, that’s fine, there are organizations who are still very behind on that train.
So yeah, you answered the question, maybe partially. Go set up your continuous integration and don’t come back then we are going to level that up.
Can you give us a glimpse of what that looks like? Let’s say when you’re going to an organization to help them reach their next level. What are a couple of high level bullet points that you go through with them? I assume it also depends on the organization.
Lisa Crispin (14:52):
It’s not my area of expertise, implementing continuous integration. But all I can say is that the teams I’ve been on were committed to delivering a high quality product. That’s the first thing that we did. It just wasn’t that hard.
Now these were smaller teams that I was on back then, but a day or two, we had continuous integration running. I don’t know why it’s so hard.
Now with larger organizations I’ve worked with, I joined them and they already had continuous integration. So I have not had to help a larger organization implement it, but I have helped with infrastructure changes.
So, maybe we’re going to change our deployment. Our continuous integration maybe isn’t getting many interactions, but we don’t like the deployment side of that. So we’re going to replace the deployment side with Harness, because we think that’s going to be a better tool.
Lisa Crispin (15:41):
As a tester, I can be involved with that, help come up with a test strategy for moving over to that, reducing the risk of things going wrong when we switch. And working together with the SREs and developers to make that a smooth process.
And they’re appreciative of having somebody with a testing background because they don’t know how to come up with a test strategy. I didn’t start out knowing anything about Harness either but when we collaborate together, we can figure it out together.
And so that’s the model, I think that DevOps culture of people working together and bringing their specialized skills to the table and being able to help each other. I see that work really well.
Testing as a practice: patterns and anti-patterns
Darko Fabijan (16:56):
We already identified that a lack of continuous integration can be one of the major anti-patterns and then it’s hard to move things forward without that component. And after that, what would you say are some of the key anti-patterns that you are seeing most frequently in organizations in terms of testing as a practice?
Lisa Crispin (17:17):
Well, it’s not a testing practice really. But back to why it’s changed so slowly, test driven development, I’m told by people like David Farley, who co-wrote Continuous Delivery with Jez Humble. They’re my heroes.
He says that teams that practice TDD pretty much avoid 80% of the bugs that most teams have and I can say, that’s my experience. When I’ve been on teams, practicing TDD, I’m not wasting my time finding unit level bugs.
Teams that practice TDD avoid 80% of the bugs that most teams have, and I can say, that’s my experience.
-Lisa Crispin, Author, Agile Testing Coach
And if I’m spending all my time finding unit level bugs, I’m never going to have time to find the more important system-wide bugs that are going to really affect the customers.
Why teams don’t adopt TDD
But yet there were a couple of panel discussions in 2020, that I think Dave and Jez were involved in, of why haven’t people adopted TDD. I have my theory as to why that is, because it’s really hard to learn. The teams I’ve been on took months even to just get traction on it, because they didn’t know how to write unit tests.
Lisa Crispin (18:24):
They didn’t know how to practice TDD and Brian Merrick calls it the hump pain. When you’re first doing it, you’re spending more and more time than you used to spend developing because you’ve got to write these tests and it’s hard.
But over time you start to get a library of test components and it becomes easier and you start to know how to do it and all of a sudden you’re saving effort. And now you can spend that effort on other things.
But it takes so much time and unfortunately, most businesses do not have the vision to realize that if they let their teams make that learning investment, over time, it would just pay off hugely. Preventing 80% of the bugs, how much time do we spend fixing those bugs after the fact? If we prevent them, we have all that saved effort.
Why companies oppose slow changes and go for quick wins instead
Lisa Crispin (19:13):
That’s just an example of one of the things I’ve seen that’s especially slow to change. I think part of it is, at least in the United States, our economy is driven by public companies on the stock market, it’s all based on your quarterly results. So who’s going to make a long term investment.
And it’s really frustrating. But teams do that. Where I work right now, they embrace tester and development from the start and other good practices like pairing and ensemble programming and continuous integration. And they’re a startup but they’ve done really well with their quality, where a lot of startups suffer from very poor quality and a lot of times that causes them to fail.
And I guess another area of testing, on the other side, I think a lot of people don’t invest enough. They don’t even understand what observability is and they don’t invest enough in good telemetry and being able to understand what their product is doing in production and being able to diagnose problems frequently.
People don’t invest enough. They don’t invest in good telemetry and being able to understand what their product is doing in production.
-Lisa Crispin, Author, Agile Testing Coach
Lisa Crispin (20:14):
Now I’ve also worked for companies that did a really horrible job of testing, but they were so good at observability that they could fix their problems for their customers in a couple minutes. And that works as long as you are a startup with a few customers, it doesn’t scale because your beginner price customers are not going to tolerate that.
Why it’s important to learn how your customers will use your product
So you need both of those things and you need things like exploratory testing. Learning about the product after you think you’re done with your initial functional testing and your test automation of what we didn’t think of, how will our customers use this product? What environment may they use it in? What devices may they use? What’s their network speed? Where will they be?
And you start trying all these real world things and learn a lot of, “Oh, we’re missing a feature.” Or, “We didn’t think of that and we have to go back and add some capability to deal with this situation.”
Lisa Crispin (21:10):
You need time to do that. But if your testers are spending all their time, either manually regression testing, because I also see many organizations with no automated testing. If they’re doing it manually, they don’t have to time. If they’re dealing with flaky automated tests, they don’t have time.
So there are a lot of things that have to come together. All these testing activities are important and we need to master the basic ones to have time to learn to do the others, because they’re all important.
Darko Fabijan (21:44):
One of the conclusions for me is to automate absolutely everything that is possible to automate and what the human doesn’t have to do, because humans should do those things that are very hard and very creative and exploratory and complex that are almost impossible to automate.
Automate everything that is possible to automate and what the human doesn’t have to do.
-Darko Fabijan, Co-Founder and CTO of Semaphore
Lisa Crispin (22:05):
Well said, I like that.
How to progress a career in the QA domain
Darko Fabijan (22:08):
Yeah, thanks. And your career is long, as you said, and you have seen a lot of war stories in the domain. What would you give, some of the key tips for the people who are maybe for even some time in the QA domain, want to progress their careers further, what would be your roadmap that you would suggest?
Lisa Crispin (22:32):
Another anti-pattern that I see is our testers who are often a silo team. Or maybe there’s a tester embedded on the team, but the test automation is done either by the testers or by some separate test automation team that has software developers, engineers and automated tests.
I think all that’s an anti-pattern because as you know, our tests need to guide our development. And as we’ve seen in the data from the DORA’s State of DevOps survey, the correlation for high performing teams is that their developers are on those automated tests, even at the UI level.
And they collaborate with the testers who know how to specify the test and design the test and do good coverage. It has to be a collaborative effort. So I think what I tell everybody, regardless of where they are in their career, is number one, build relationships.
What I tell everybody, regardless of where they are in their career, is number one: build relationships.
-Lisa Crispin, Author, Agile Testing Coach
Lisa Crispin (23:29):
I’m a shy person, so it’s not easy for me to start a new job. And everybody around me is probably a shy person, we’re an industry of introverts. I contact somebody on the platform team and set up a, “Can we have a one-on-one every couple weeks?”
Then I might learn about something within Harness, which I end up having to test people on the database team, if they’re separate teams. And then try to encourage your organization to take that more DevOps approach.
Where I work now, I really love the approach they take because they call their SREs dev advocates and they embed them on the feature teams and they take developers in the feature team and embed them on the platform teams. And their mission as a platform team is to provide the infrastructure, but turn it over to the future teams to own and manage their own infrastructure.
I think that’s a great model, but we have to be able to work together to do that.
Get to know people from other specialties
Lisa Crispin (24:28):
And many people, whether they’re a tester or developer or platform engineer, have not worked with people in other specialties. So get to know those people, build those bridges and it starts a foundation for working together.
When we have a bunch of people with diverse experiences and diverse skills, we can solve problems a lot faster. That’s why Janet and I have always promoted this whole team approach to testing your quality. It takes everybody.
Even back in the nineties, I used to hear testing experts say, “You can’t test quality into a product.” And that’s absolutely true in terms of if you’re testing after the fact, but you can use things like test driven development and behavior driven development and other practices to build quality into that product. And then do a lot of other things to help with all the other mini quality attributes that are so important today.
Lisa Crispin (25:26):
We could have the best product in the world, but if our security is bad, we’re going to be out of business for lacking in accessibility. Depending on where we are, where our customers are, we could be in legal trouble.
There’s just so many ways that you go wrong and it takes a village, it takes people. There are studies that show that companies who employ a diverse workforce or people from diverse backgrounds and experiences, they make more money, they’re more innovative.
Lisa Crispin (25:55):
And this applies to software teams too. So my mission again, is just trying to get people to work together and as individuals, it takes that first step. It can be very hard to influence an organization if the culture is really dysfunctional and toxic and unfortunately, those are out there.
But as an individual, I can start building relationships, I can start talking to somebody on the monitoring and observability team and get them interested in testing. And they can show me, “Here’s what we have. Here’s what we’re logging, here’s what’s on our dashboard.”
And I could say, “Well what about this information? Could I find this information? It would be really helpful.” And so we can work together in those ways. I hope that makes sense.
Darko Fabijan (26:40):
Makes a lot of sense, yeah. It was a fantastic answer and advice. And for people who want to learn more about your work, about the domain, you mentioned a number of people who are obviously leaders in the domain. What would you list for us as a couple of key references where we could go and learn more?
Lisa Crispin (27:02):
My website is just lisacrispin.com and I have a page there for testers who are interested in learning more about the “DevOps side of things,” how they can learn about continuous delivery, observability, monitoring. I’ve just got a huge list of books, podcasts, videos, webinars and various blog posts.
Agiletester.ca is the website for the books that Janet Gregory and I have written and also, we have a video course, so you can go to that site and there’s a lot of free content there.
We have book chapters that are downloadable and lots of free testing and resources, cheat sheets and things that can help people learning how to test or wanting to expand their skills. And then we have a training course, agiletestingfellow.com.
Janet and I have a training course, which we’re going to rename to Holistic Testing, but right now it’s called Agile Testing for the whole team.
Lisa Crispin (28:03):
And we’re adding a couple courses, including a more advanced course, the working title is Testing in DevOps. That’s so vague and hand wavy, but how testers can get more involved in helping their team succeed with continuous delivery. So we’re really excited about working on that.
So those are just a few areas and the book “Continuous Delivery” that I mentioned, that’s one of my Bibles. “Accelerate” by Dr. Nicole Forsgren and Jez Humble. Anyway, that is a great book because it also goes into how you can change your culture. Culture is so important.
The modernagile.org from Joshua Kerievsky, I think that’s a great site to go to with his prerequisites for success.
So I could go on all day with helpful resources, but also anybody that wants to contact me, my email is lisa@lisacrispin.com or my Twitter, I’m @lisacrispin pretty much everywhere. So feel free to reach out to me on Twitter or LinkedIn although I don’t always look at LinkedIn, but I should.
Darko Fabijan (29:06):
Thank you. Yeah, that’s a list I checked prior to the episode, a couple of them and there is a huge library of amazing resources. So yeah, we are for sure, going to release those out in episode notes.
It was super interesting to see and hear your approach to the craft. Thank you so much and good luck with future books, courses and everything. Thank you.
Lisa Crispin (29:33):
Well, thank you. And I’m excited to learn more about Semaphore. Anything I can do to help companies embrace continuous integration. I am happy to have those tools in my toolbox.
Darko Fabijan (29:43):
Great, thanks a lot.
Lisa Crispin (29:46):
Thanks.
