Continuous Integration (CI) pipelines enable software development teams to build and test code quickly and efficiently. However, the need for robust security measures becomes increasingly important. In this article, we interview Solutions Engineer Adib Saikali about his experience with CI pipeline security. From protecting sensitive data to detecting and preventing malicious code, Adib provides valuable guidance for ensuring the security of your CI pipeline.
Edited transcription
Adib Saikali is a principal solutions engineer at VMware Tanzu, a platform for integrating cybersecurity in software development. As part of the services offered by his company, Adib has experience with role-based access control (RBAC) security, including securing CI pipelines. CI (Continuous Integration) pipelines automate the processes of writing code and are designed to catch errors or issues before they get to the production environment (CD or Continuous Delivery). CI implementation grants rapid feedback and easy identification and resolution of problems that can occur during development. Withal, CI pipelines are not free of security concerns, such as unauthorized access to sensitive data, the introduction of malicious code, and the potential for attackers to gain access to production systems.
Challenges when securing CI pipelines
Over the years consulting companies, Adib has found two patterns regarding security in CI pipelines. First, how teams confirm if the security system code they wrote works with the real-world security system. When developers check out code locally and disconnected from the real security system, they might have difficulties running the application as intended. Hence, developer teams implement workarounds to try out the code without compromising the real system.
However, Adib has found that each team might develop a unique, not intuitive workaround to try out their code. As Adib puts it, “whenever you look at the code bases in these teams, it’s like ‘How do we do this? What are the unique ways of configuring security?’ ” Establishing a unique, centralized, and documented way of doing things will help organizations understand how to work with their code bases.
Secondly, Adib found that when writing tests, it can be difficult to know the identity of the user and how to handle complex permissions and workflow scenarios tied to security. Implementing this kind of test demands a complex permissions workflow.
To solve this last challenge, Adib recommends OpenID Connect (OIDC), an open standard for identity authentication and authorization. Built on top of the authorization framework OAuth 2.0, OIDC is an extra identity layer to verify the identity of a service end user using their existing credentials from a different service, without having to create new credentials. According to Adib, the great thing about OIDC is that “it allows us to define backend APIs, which are stateless,” meaning that “any state is stored in an external service like a database.”
With OIDC it is possible to create resource servers that demand whoever calls to have a certain token that will help them identify who is making the call. You can use OIDC to create a resource server that doesn’t have any information about how to log users but that merely reads whether or not they have a valid OIDC token and provides access depending on it.
According to Adib, the login should be handled by an OIDC server. The user experience could be, for example, in the case of a single page application, a user landing on the page and clicking on the login link, which would redirect them to the login server. In this regard, Adib advises against choosing client-side login. “The better way to do it is to in fact delegate that authentication to some sort of API gateway that’s in front of your stateless backend resource servers,” he affirms.
An API gateway is a server that acts as an intermediary between an application and a set of microservices. The API gateway is responsible for request routing, composition, and protocol translation. Moreover, it can also be used for authentication. As such, the API gateway can act as the single entry point for external consumers of the microservices and hide their underlying structure from the consumer, providing a consistent and simplified interface. The client (a native or single page app, or server-side rendered web app) will just HTTP redirect to the login server. The login server will display a login screen for users to log in. Upon successful login, they will establish an HTTP session with the API gateway. With the HTTP session, it is possible to track identities: on the client side, with the HTTP session ID, and on the server side, with the API gateway forwarding the backend services.
For implementing an API gateway, Adib recommends Spring Cloud Gateway, part of the Spring ecosystem. With Spring Cloud Gateway, “you can now actually do development the same way that it would be in production, except that you can program the OIDC server to have your standardized set of users to test different scenarios.” In this regard, for testing, Adib recommends creating fake users for the many plausible characters who would use that system and testing if their access limitations are working as intended.
Optimizing CI pipelines
When asked about any practical tips or information that can be shared about maintaining CI pipelines, Adib recommends keeping an eye open for new emerging standards, such as SLSA, a comprehensive security framework that includes a checklist of standards and controls to protect against tampering and maintain integrity in your projects. However, he also admits that the industry is five to ten years away from a holistic solution being widely available.
From the time being, Adib encourages us to try out everything, especially cutting-edge solutions, and see what fits best. As an example of this kind of technology, if you are you’re running your CI pipeline with coding containers, he encourages you to look at buildpacks.io, which offers both framework and runtime support for applications, and analyze your apps to identify necessary dependencies and properly set them up for execution on any cloud platform.
On the other hand, Adib recommends associating application projects with test fixture projects. Test fixtures are pre-configured environments to test software that allow developers to fix states. Test fixtures “make all the tests simpler to write by having them all leverage common helper classes,” Adib explains. In this way, test fixtures become “common code libraries’ ‘ that can be as handy as any internal utility library.
The challenge of writing test fixtures code, says Adib, consists of being “able to create things where you can produce these test errors that are hard to reproduce.” For this reason, Adib recommends maintaining three types of source code for each project: the project code, the test fixture code, and the test code. The test code you produce will depend on the other two types of code. Withal, Adib reminds us that creating this environment can be as challenging, if not more, than writing the actual application code: setting up the right security contacts, getting dependencies like the database or the application, and simulating error conditions.
Lastly, to simulate error conditions, Adib also suggests implementing a way of testing databases. “A lot of the time our state is stored in some sort of database and, depending on what’s in the database, your test may pass or it may fail,” he points out. He recommends using Java library Testcontainers and running JUnit tests through it. You can spin up containers for the specific version of the database before running the test. The containers are then thrown away after the test has finished ensuring that each test starts with a clean environment and eliminates the chance of interference between tests.
Leveling up the security culture of your team
On a skill level, Adib encourages developers to get into cybersecurity. He argues that while it might seem intimidating at first, they should start by learning to separate “what you need to know from what you don’t need to know”. In this regard, he advises learning three kinds of technologies:
- Cryptography. Specifically, the basics, including hash functions, public key cryptography, and symmetric key cryptography.
- Digital certificates. Mainly, X.509 certificates. “X.509 certificates are super important to understand because they’re used everywhere, and they’ve been used everywhere for decades, and they’re going to continue to be used everywhere for decades more,” Adib affirms.
- Transport Layer Security (TLS). He understands that “TLS is everywhere and we should be doing mutual TLS everywhere, whichever way we do it.”
The next step after learning about these technologies, says Adib, is learning how to deal with two security concerns. First, on how users log in and what technologies come into place. Adib advises getting used to the OIDC protocol, which works with all programming languages. He also considers it essential to pick up WebAuth for logging in without a password.
The second security concern is how to secure API call chains. An API call chain is a series of calls made to different APIs or services in a specific order, with the first call at the top of the chain and the last call at the bottom. These calls are made to perform tasks or access data, and the chain of calls together forms the overall program functionality. According to Adib, “there isn’t an industry-standard way to do it”, so you are forced to judge by yourself among the employed methods, such as API gateways and service mesh, an architecture that can be implemented to control and monitor the communication between different services.
The bottom line
Currently, Adib is writing a book on cybersecurity named Securing Cloud Applications. His approach is writing a book about cybersecurity not for cybersecurity experts but developers. “A lot of the books I went through were very much written by security people for security people, so this book represents the thing I wish I had when I started learning about security years ago,” he confesses. The book follows the teaching of the cybersecurity fundamentals he mentioned earlier, including cryptography, digital certificates, TLS, user login, and securing the call chain, as well as Kubernetes. You can check his book progress here.
Adib understands that one of the things that differentiate his book is that it does not demand doing math. Instead, it features sample applications. “The idea is to help developers understand security by running a sample app that lets them see the security protocols and actions,” he says.
Follow Adib on Twitter and LinkedIn and check his website for finding his posts, talks, and workshops. You can also find Adib’s past talks and presentations on Youtube.
