In this episode of Semaphore Uncut, Justin Cormack, Senior Security Engineer at Docker and member of the Technical Oversight Committee at CNCF, shares insights from the security industry. We talk about why it’s important to think about what could go wrong when building software, how hackers are now exploiting vulnerabilities before shipping your code to production, and what companies can really do and use to secure their products.
- Security – a matter of software quality
- The threat modeling practice – understanding the potential security threats
- Using the experience of experts
- Supply-chain security
- Security integration into CI/CD pipelines
- Important vs. overhyped practices in the security industry
Listen to our entire conversation above, and check out my favorite parts in the episode highlights!
Like this episode? Be sure to leave a ⭐️⭐️⭐️⭐️⭐️ review on the podcast player of your choice and share it with your friends.
Justin (00:12): Thanks for having me. I’m Justin Cormack, I’ve been working at Docker for almost exactly five years now, and I’ve been involved in everything happening with containers. I’ve done a lot of work with CNCF; I started with SIX Security – many people interested in the security parts of cloud-native. Since the beginning of this year, I’ve been on the technical oversight committee at the CNCF; I’m the maintainer of the Notary project, a CNCF project around supply chain security for containers.
Security: a matter of software quality
Darko: For our listeners who are maybe not on a day-to-day basis involved in security and the whole thinking framework around security, can you maybe give us a brief introduction about general security?
These things are not the things you notice if you just try and do the things you’re supposed to do; it’s like, what happens if you do the things you’re not supposed to do? And software should work when things go wrong as well as when things go right.
I think of security as just being that collection of everything about things going wrong. Most of the things that go wrong in that area of quality, and often it’s hard to tell which things are security issues until they’re exploited. It seems to be the case that many bugs can be security bugs if people try hard enough to exploit them.
The threat modeling practice
Darko: Docker is the infrastructural layer of our software stack for many developers. And I guess the culture at Docker is a bit different from a typical, maybe a company making web apps or web app shops. What’s your experience in terms of educating developers and people working on the software in the lines of security? You mentioned that, from the perspective of quality, as a more general thing, do you have experience of maybe educating developers who are not that security aware?
Justin: One of the exciting things about working at Docker, unlike a company that just builds web apps, we build a wide variety of very different systems. We do build web apps. I mean, Docker Hub has a web app in front of it basically, but we also build software that we ship to other people, like the Docker Engine. And then we also have Docker Desktop, which is a different thing because it’s a desktop application with all the different security surface that has that does automatic updates. We have a really wide variety of development environments and CI environment and test environments. With web apps, there’s just a lot of security surface of things that are really important. And generally, relatively well-documented because the web app space is quite good at doing this.
Threat modeling is not widely taught, and is about understanding how someone might attack your piece of software; what they might be trying to achieve by doing that, and what kinds of things they might try and do. Many people are not exposed to that at all, it seems, to be a sort of specialist thing, but it’s the sort of thing that’s valuable to learn about and to sit down and think about it. If you’re a security person, part of your job is to help people think about these things. There’s a good threat modeling book called Threat Modeling. Just sit down and think, what might someone do?
A production system is very different from a system sitting on a developer’s laptop or a system where people might try and misuse resources. For example, in the last few years, one of the most common things is that people try and use any kind of thing available on the internet to do Monero mining, which is usually coin mining. So any sort of CI system, any computer that’s exposed to the internet, people will run this because they can make money out of it. And they will use all available resources they can get by any means whatsoever, bypassing security systems or even just signing up to your service and running CI jobs. One of the positive things about that is these people are not hostile; they’re just trying to steal resources. For many people, finding out where the weaknesses in their systems are has been educational.
Using the experience of experts
Justin: And then there’s really specialist areas, I think, generally difficult and not many people are going to understand them in depth. And so things like the details of authentication protocols and cryptography and those kinds of things, they’re difficult, and specialist, and there are experts in those fields. Security people’s role is to package these up so they’re safe to use, and they’re hard to make them go wrong. And I think, historically, security people have been quite bad at that. They provide interfaces that are very easy to use incorrectly. There’s been a history over the last, maybe, 15 years of trying to fix that, but it’s a very slow process. So building things like that is a very specialist thing because you’ve got to understand how the protocols work and how you should and shouldn’t use them and provide interfaces. That’s important for security specialists to work on to make sure people do it well.
You often know that pieces you think might have a problem, and doing that internally is helpful because you know those areas, but someone external knows what mistakes other people who’ve done the same kind of thing do as well. So if you read security audits, you can understand the kinds of things they’re looking for. One of the things about security audits that’s changed recently is that historically most security audits were never published, but more open-sourced projects are doing open security audits. So, the CNCF has started funding security audits for its projects, which has been valuable because they’re all public, and you can read them and find out what kinds of problems people have found and what they haven’t found.
Darko: During our prep call, we talked about the area of security, which is connected to CD pipelines. Can you maybe share some of the concrete examples of issues you see in those and how generally people can integrate more of those security tests and those pipelines?
Justin: There’s this whole field now that’s new, called supply-chain security. There are attacks on software before it gets to production, as well as in production. And they’ve become much more common recently. One of the reasons is that people who spent more time hardening the software while it’s running, like these cross-site scripting attacks, have worked out how to protect against those. But the attackers have found that why attack the software in production if you can hack it before it gets to production? And then ship something into production that’s not what you meant or has got a backdoor in it.
We’ve seen a lot more of these in the last few years, especially. There’s been some sort of famous ones. There was a great write-up in Wired about the NotPetya attack. The attackers had hacked into a software company in the Ukraine that made accountancy software commonly used across Europe. They compromised all the build machines in this accountancy company to ship out their own hacked versions of the accountancy software. And when people updated, they attacked their internal networks and caused enormous damage, like shutting down hold companies for weeks.
Then there are other cases where open-source projects had been attacked. There was the event stream thing with NPM recently where someone neglected an open-source project that lots of people use, and someone offered to maintain it, very helpful. Then, so they go in, do a new release, but it turns out that this new release has encased a targeted attack on a particular company’s Bitcoin wallets and this is a very, very carefully targeted attack.
Security in CI/CD pipelines
Justin: Jenkins, for example, because it’s so popular, has been attacked a lot. To be honest, it’s never been seen as a high-security piece of software because it’s just been assumed that you’re running it behind a firewall or something. But actually, of course, people are running it on the internet because it’s more convenient to use it like that, and it has a lot of plugins as well.
There’s a lot of work going on in this field to improve security. CI providers probably have more resources to harden than often individual companies do. People realize that cloud and services often have a considerable amount more security resources than their company does, especially for smaller companies where security is a scarce resource. And so outsourcing is probably a good thing from the security point of view for many people.
There’s a whole bunch of tooling that’s starting to happen. We’re working on a v2 of Notary for container signing, but basically, there’s a whole set of technology being built around signing and authentication and transparency logs and things like that for making sure that the stuff you ship are the things you meant to ship. And then, there’s a whole set of things about your dependencies; whether you’re pulling in dependencies that have security issues that are known, so security scanning, there’s an entire industry around that. Then there’s static analysis tooling and things like that we’ll look and see if there’s suspicious code.
Static analysis tooling is getting much better, and I believe that there are many good tools that you can use. Then there are things like hardening developer machines, making sure that people are updating the OS, and that your people are getting software from where it’s supposed to come from.
It is essential to keep building software and testing it because all sorts of software break over time because it talks to an external API and the API changes. You need to upgrade the base operating system, other dependencies; there’s a whole lot of things that can break software. We have this term bit rot for a reason. You think software doesn’t have to change every time, but it does. You can’t control everything in the environment. And certificates expire and things as well, these things just happen because they’re built into processes.
Important vs. overhyped practices in the security industry
Darko: Can you give us an overview of how you see that area of what security vendors are offering and what is maybe less talked about, but also matters a lot?
Justin: The security industry is a slightly weird space in a way. Traditionally, they’re trying to sell fear and trying to sell it to executives because of how they do things. You don’t really know much about the products they are actually doing; there’s not much open source or possibility to try it before you buy it.
Understanding where your risk is and what you want from the security industry is important, what you’re trying to achieve, and your actual risk profile. The thing that’s gradually changing is that we’re seeing more open source and cooperation in the security space, which is a refreshing change.
One of the most exciting things about CNCF SIX security is talking to big companies who are saying, “We don’t want to do all this security work on our own because we know it’s the same as what the other companies in our space are doing. It’s not a competitive advantage. Why don’t we do this collaboratively?” So that’s a real change in attitude that’s happening slowly.
Security is often seen as a cost. There’s a bunch of people telling you things could go wrong, and they need money to fix them. You’re like, “But it’s not gone wrong yet.” And you do talk to people who don’t invest in security for that reason. Understanding how much you should spend, what you should buy, what you should build, and how much work you should do is very difficult because it happens so rarely for many people. Many of the large security incidents you hear about don’t end up costing the people that much money. People are surprised by that, they seem that it will be a disaster, but it’s something you can live with.
One of the other interesting trends in security now and something that we’ve been working on, for example, with Notary and TUF, is this whole idea of survivable compromise. It’s like, what happens when things go wrong? If this key gets stolen, how do you rotate it? How do you get back to a secure position from an insecure position? Some of that kind of survivability and compromise resistance stuff is another way of approaching this, it’s like, “Can we carry on even if there’s a minor compromise? Can we recover without having to rebuild everything from scratch?”
You can rebuild things more easily in the cloud world, and I think compromise resistance is important. If you don’t collect unnecessary personal data, it can’t get stolen. Personal data is a liability; if you don’t have that liability, you’re in a much better position.
We have lots of flexibility now. Computers are cheap, and you can run a separate cluster or separate machines to run code that has a different security level from other code, just to give you that kind of isolation. So, anything to do with payment processing can not be on the same infrastructure as the stuff that handles personal data, and that can be on a separate infrastructure from the code that you’re running builds on and things like that.
Separation of the data so that if one thing goes wrong, it’s spilling over into another area is difficult. But the separation of different kinds of risk levels and different bits of data is fundamental.
Darko (35:20): Thank you, Justin, for sharing all these insights in the area, which is maybe less talked about, and I would dare to say, less loved among the developers, although very, very important.
Justin: I do recommend people get into security. Ideally, in advance of it being a real problem.