AppSignal is a leading European real-time APM provider founded in 2012.
Working remotely, the team develops an application monitoring tool that merges error catching, performance tracking, server monitoring, metric dashboards and anomaly detection into a combined UI. Bringing these features together allows AppSignal users to generate clear and focused insights.
AppSignal integrates with a diverse range of languages used to build the web: Ruby, Elixir, Node.js and Javascript and is currently used by over 1,000 teams.
The Challenge
The AppSignal team found themselves at the mercy of their old CI tools. Builds were unstable. The team decided it was time to change when they couldn’t get the help they needed from their old CI provider. “CircleCI made a change somewhere in their stack and that broke our build consistently,” explains Tom de Bruijn, an AppSignal developer. What’s more, it wasn’t just one CI provider. Tom continues, “We had a similar problem with TravisCI, builds were flaky and these issues only occurred on TravisCI.”
Because of the spurious build and test failures, the value of the team’s CI setup started to deteriorate. First, a lot of time was lost retrying builds to see if a failure was real. Second, Tom sank a lot of effort into debugging the brittle builds. Third, and most importantly, the random failures started to mask real issues. As Tom describes, “Failing tests snuck in. If the build always or randomly fails you’re not going to look at the build result anymore.”
To top off the problems with their previous CI setup, the team found the UI difficult to use. “People got lost in the CircleCI workflow UI”, relates Tom. This meant that the team avoided building the kind of advanced workflows that could improve build times. Moreover, the array of disparate jobs generated lots of notifications in Slack. Therefore, it was hard to tell when a full set of jobs for one pull request was complete. “The team thought builds were done before they actually were,” Tom recalls.
Overall, AppSignal needed their new CI/CD solution to
- Give them full control of the execution environment
- Use low maintenance, SaaS build agents
- Be supported by prompt customer service
- Simply model the whole workflow
The Solution
Tom’s number one piece of advice to anyone considering switching CI providers is to move the test environment to containers. Semaphore’s seamless support for Docker containers allowed the AppSignal workflow to be ported quickly. The team’s server test container, from AppSignal’s own, private Docker Hub collection, gave them total control of the runtime environment and insulated them from external changes.
All jobs in the workflow are collected in one model. “The workflow UI is better,” says Tom, “I no longer have to show people where they can see the entire workflow for example.” As a result of the simpler workflow modelling, Slack notifications arrive only for complete workflows – not for each individual job as before.
Through the improved user-experience of Semaphore, the team has been able to exploit the parallelism their workflow permits. “We’ve split up more builds so separate parts can run in parallel.” Semaphore’s high-quality documentation has been a boon, “I’m able to find most of the answers to my questions in the documentation,” Tom concludes.
Another important aspect of the solution is its low maintenance requirements. Semaphore provisions build-agents as and when needed. As a result, AppSignal engineers don’t have to expend time and people on setup and maintenance of the agents.
The Results
Tom’s assessment of the results? “Running the Ruby gem builds on Semaphore runs perfectly, no random failures, and it’s done a lot faster.” The eradication of flaky builds directly produces a time-saving. Indirectly, the increased reliability boosts the team’s productivity even further.
The elapsed- and CPU-time of each build cycle has radically improved too. For example, AppSignal’s Ruby gem used to take about 14 wall-clock minutes (and 85 minutes of CPU time). Now it takes just 6.5 minutes (40 minutes of CPU). JRuby builds also sped up: 7 minutes on the old solution, and just 1 minute 30 seconds with Semaphore.
“The build is faster out of the box with the default machine type,” observes Tom. “We’ve switched all of our projects from CircleCI to Semaphore, and have also started migrating the TravisCI builds for our integrations to Semaphore.”
