Continuous Container Vulnerability Testing with Trivy

💛 Huge thanks to Teppei Fukuda for reviewing this tutorial.

You’ve adopted continuous integration (CI), followed TDD and BDD principles — having mastered testing, you deploy anytime, at a moment’s notice. You are a CI guru. You can live happily ever after.

Like all fairy tales, the story ends abruptly; before real life intrudes. Before we realize that a vital component has been left out, security; more specifically vulnerability testing.

In this tutorial, I’ll propose ways of integrating security at every stage of your CI/CD workflow. I can’t promise a happy life, but I guarantee you’ll sleep better at night.

Reactive vs. proactive security

When we do vulnerability or penetration testing in a live production system, we say we’re doing reactive security. We’re in a race — trying to find the issues before anyone else can exploit them.

Even if the security scan is very comprehensive, when all is said and done, we’re just patching holes. Better it would be if holes wouldn’t be there in the first place.

On the other end of the spectrum, we have proactive security. Proactive security is all you do before release or deployment. By being proactive we prevent the system from ever being exposed to a vulnerability.

There’s nothing wrong with being reactive, as long as it’s not the only solution you’re using.

Proactive security with Trivy

Trivy is an open-source security and misconfiguration scanner. It works at every level: it can check the code in a Git repository, examine container images, advise regarding configuration files, look into Kubernetes deployments, and verify Infrastructure as Code (IaC).

Trivy is maintained by Aqua, and feeds from their vulnerability database and many other data sources. It runs on Linux, macOS, Docker, as Helm Chart, and a VS Code Extension.

Automated security in your CI/CD

There are at least four security checkpoints in every CI/CD. We’ll learn how to use Trivy at each one of these stages:

1. The source code dependencies.
2. Artifacts such as Docker images. Attackers exploit vulnerabilities deep down in the application or the supporting libraries to break out from the container.
3. Configuration files.
4. Infrastructure code describing cloud services that power the application.

Vulnerability testing for dependencies

Before we start, we need something to test. In this tutorial, we’ll be using a “Hello, World” Kubernetes demo. So go ahead and fork it here:

For maximum effect, you can combine these steps with the ones laid down in past tutorials:

• With Structure Tests for Docker Containers we run custom tests inside the container image.
• In Secure Kubernetes deployments we confirm that the Kubernetes manifests are using sane deployment practices.

Next, install Aqua Trivy. The first time Trivy runs, it downloads the vulnerability database and creates a cache folder for results. You can clean it up with trivy --reset.

To run a dependency scan use trivy fs. Trivy detects the Gemfile in our project and searches for vulnerabilities.

$ trivy fs .
​
Need to update DB
Downloading DB...
Number of language-specific files: 1
Detecting bundler vulnerabilities...
​
Gemfile.lock (bundler)
======================
Total: 0 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 0)

You can even scan a repository without cloning it with trivy repo.

$ trivy repo https://github.com/knqyf263/trivy-ci-test

You may simultaneously run a scan for vulnerabilities and misconfigurations by adding --security-checks vuln,config. The scan will include any Dockerfiles, Kubernetes, and Terraform files in the repo.

$ trivy fs --security-checks vuln,config .

💡You can see all the languages and package managers supported by Trivy here.

Scanning dependencies with CI/CD

The dependencies on our demo are clean since the repository receives frequent dependabot pull requests from GitHub. But even dependabot is reactive so it’s not enough to prevent us from shipping unsafe code. We can switch to proactive mode by integrating Trivy into our CI/CD pipeline.

Let’s see how it works. After adding your repository to Semaphore, click on Add Block and type the following commands in the job:

wget https://github.com/aquasecurity/trivy/releases/download/v0.20.1/trivy_0.20.1_Linux-64bit.deb
sudo dpkg -i trivy_0.20.1_Linux-64bit.deb
checkout
trivy fs --exit-code 1 .

The first two lines install Trivy in the CI machine. The third, checkout, clones the repository. The last one runs Trivy with --exit-code 1 to force the pipeline to stop when some problem is detected.

The CI pipeline looks like this after adding the Trivy scan:

For extra security, we can verify the checksum of the Trivy package. The checksum file is included in releases. The following line will make the pipeline stop if checksums don’t match. Adjust the filename and checksum as needed.

echo "46119ad9571f740201461b7529059afacb01ff74549de60bca657deba2f556cd trivy_0.20.1_Linux-64bit.deb" | shasum -c -a 256

Scanning Docker images

Now we take the next step in the vulnerability testing journey. However, before we can deploy the application, we have to package it with Docker. Since container images may have misconfigurations that leave the system open, we can’t go on without scanning them. We’ll break down the task into three steps:

1. The base image we’re building from.
2. The Dockerfile that packages the application.
3. The final container image.

You can learn more about Docker and Kubernetes with our free ebook: CI/CD with Docker and Kubernetes.

Vulnerability testing for base images

Our base image is Debian 11.1 “Bullseye”. We use trivy image to scan it directly from the image registry. You’ll need a Docker installation for this to work.

$ trivy image ruby:2.7

TIP 💡: You can get other output formats with -f FORMAT (for example -f json).

In all likelihood, there are dozens of vulnerabilities in the image ranging from low to critical. At this point we need to start applying risk management strategies and think what level of security is “good enough” for our needs. We’ll worry for now about high and critical issues. By adding --severity HIGH,CRITICAL to the command we can filter results.

The next step is to go through the list of issues and assess what actions, if any, are needed. Some of the vulnerable packages are not actually needed and can be removed. Let’s do that by adding these lines into Dockerfile and Dockerfile.ci:

# remove unneeded packages with vulnerabilities
RUN apt-get purge -y curl "libcurl*" libaom0 python3.9
RUN apt-get autoremove -y

And build a new “base image” to see if the vulnerabilities are gone.

$ docker build -t my-test-image .
$ trivy image --severity HIGH,CRITICAL my-test-image

Other vulnerabilities may be patched or ignored. We have two ways of skipping vulnerabilities with Aqua Trivy:

• Adding --ignore-unfixed to the command hides vulnerabilities that do not have a fix or patch.
• In .trivignore we list the CVEs we want to skip.

For instance, if after reading about a particular issue and deciding I can safely ignore it, I add the following line into .trivyignore:

# a libc vulnerability in the base image, currently unfixed
CVE-2021-33574

Commit the changed files once satisfied with the results.

Vulnerability testing container images

With the base image and the dependencies scanned, the resulting container should be pretty safe. Let’s confirm this by testing the final Docker image in the pipeline

The check runs after the “Docker build” job in the continuous delivery pipeline. As you can see in the screenshot below, I added an image scan in parallel with Container Structure Tests.

The job pulls the recently-built Docker image and scans it with trivy image. You may need to add docker login and a secret if your image is private.

wget https://github.com/aquasecurity/trivy/releases/download/v0.20.1/trivy_0.20.1_Linux-64bit.deb
sudo dpkg -i trivy_0.20.1_Linux-64bit.deb
checkout
docker pull "${DOCKER_USERNAME}"/semaphore-demo-ruby-kubernetes:$SEMAPHORE_WORKFLOW_ID
trivy image --severity HIGH,CRITICAL "${DOCKER_USERNAME}"/semaphore-demo-ruby-kubernetes:$SEMAPHORE_WORKFLOW_ID

Scan the Dockerfiles

Scanning the image artifact is not the end of the story; there’s one more thing we can do to improve its security: check if we’re using good practices in our Dockerfiles. Trivy understands Dockerfiles and can offer suggestions on improving them.

Dockerfile-scanning is done with trivy config. We have to copy the files into a temporary folder to avoid scanning all config files in the repository:

$ mkdir -p audit-dockerfiles
$ cp Dockerfile* audit-dockerfiles
$ cd audit-dockerfiles
$ trivy config .
$ cd -

The analogous job in the pipeline looks like this:

wget https://github.com/aquasecurity/trivy/releases/download/v0.20.1/trivy_0.20.1_Linux-64bit.deb
sudo dpkg -i trivy_0.20.1_Linux-64bit.deb
checkout
dockerdir=$(mktemp -d)
cp Dockerfile* $dockerdir
(cd $dockerdir; trivy config --exit-code 1 .)

The new job should run before “Docker build” in the pipeline.

Scan Kubernetes

Keeping infrastructure safe is as important as securing the application. We’re running the demo project in a Kubernetes cluster. This means we have at least two levels of security checks ahead of us: the cluster infrastructure and the deployment.

Test the Infrastructure

Being proactive in this area means using IaC tools such as Terraform, so Trivy can enforce a set of rules that encode good security practices.

Our demo project doesn’t include any IaC files, but this is where the Trivy job would go if it did.

As with Dockerfiles, we use trivy config to scan the Terraform files:

wget https://github.com/aquasecurity/trivy/releases/download/v0.20.1/trivy_0.20.1_Linux-64bit.deb
sudo dpkg -i trivy_0.20.1_Linux-64bit.deb
checkout
cd terraform
terraform init
trivy config --exit-code 1 --severity MEDIUM,HIGH,CRITICAL .

Check Kubernetes manifests

You may not manage or have access to the cluster, so infrastructure tests may not apply to you. But if you’re using Kubernetes, for sure you’ll have to worry about manifests.

Manifest are configuration files that completely describe how Kubernetes runs the application. trivy config serves as an excellent way of rounding out deployment checks.

We’ll add this test after the infrastructure scanning and next to other manifest tests done in a previous post: Secure Your Kubernetes Deployments.

The job runs trivy config on the deployment files in a temporary folder:

wget https://github.com/aquasecurity/trivy/releases/download/v0.20.1/trivy_0.20.1_Linux-64bit.deb
sudo dpkg -i trivy_0.20.1_Linux-64bit.deb
checkout
k8sdir=$(mktemp -d)
cp deployment*.yml $k8sdir
(cd $k8sdir; trivy config --exit-code 1 --severity HIGH,CRITICAL .)

Extending Trivy

Let me close up this post by mentioning that Trivy can be extended with plugins and custom policies. For example, Aqua provides the kubectl plugin to better integrate Trivy with Kubectl. The plugin lets us scan images running in a Kubernetes pod or deployment:

$ trivy plugin install github.com/aquasecurity/trivy-plugin-kubectl
​
# Scan a pod
$ trivy kubectl pod mypod
​
# Scan a deployment
$ trivy kubectl trivy deployment mydeployment

Custom policies are written in the rego language and may be used to enforce specific rules in your organization. To learn about how to write policies, check out the custom policies doc page.

Conclusion

A 100% secure system is a chimera. But that doesn’t mean we can’t use every tool at hand to reduce the odds of having a security breach.

You have added a total of five new security checkpoints into your pipeline. You can tweak the severity levels until you reach the right balance between functionality and security.

Thanks for reading!

Related reads:

• [Podcast] Justin Cormack on Integrating Security into Software Building.
• Design an Effective Build Stage for Continuous Integration.
• Kubernetes vs Docker: Understanding Containers in 2021.
• Beyond Docker with Earthly.