This is a guest post by Brendon Murphy, CTO at Kajabi. Read it to find out how Kajabi got to a 10-minute CI build for 100k lines of Rails code and got rid of tedious manual test suite balancing with Semaphore Boosters, fully automated parallel CI for Rails.
Kajabi is a SaaS company positioned squarely in the area of Knowledge Commerce. Our customers typically use our platform to sell video based courses of their knowledge, which their members purchase and consume on the platform. Our core stack is Ruby, Rails, with RSpec for testing. Our application is primarily run on Heroku, with assets spread across S3, FileStack, Wistia, and Fastly.
Kajabi’s Workflow
Kajabi is a fairly large and relatively complex for a Rails app. We’re over 100k lines of code at this point (including tests). We have a fair amount of complexity in the areas of billing, as we directly integrate with both PayPal and Stripe for single time purchases, installment sales, and subscription billing. A file count of our app/ and lib/ directories, excluding views, is about 2600 files.
We store and work on our code using GitHub. We track progress of features across Productboard, and then use GitHub organizational projects for tracking at the code level. Our deployment strategy is to develop in feature branches off of master, merge into master, and then deploy to staging followed by product from a deployment specific branch.
The Challenge: A Slow Rails Test Suite
At some point, our test suite was too slow to complete for our liking. We were using a parallel runner configuration we constantly had to try and balance ourselves, which was very challenging, so we felt that the parallel runner was constantly imbalanced and slow for us to fix.
Changing the number of threads, even though it was easy, still felt tedious. We’d have to go in, copy the config for a runner thread, create a new thread, and paste the config in for every new thread we wanted to create. It was nowhere near as nice as having a Heroku-style slider, I didn’t like the cut-and-paste.
We also tried using Knapsack. It was a great first experience and served us very well, but having to balance it constantly apart from our build felt like it got in the way of automation.
We’d have to manually balance the test suite every so often, and it always felt tedious. Technically we should have done the balancing on Semaphore via SSH so we balance on the proper hardware, but our suite wouldn’t finish inside the 60 minute session timeout. This meant a developer would have to run the balancing locally, which created 2 problems. First, we always wondered if the hardware performance mismatch could cause any issues, and more importantly, it was annoying for a developer to have their fans spun up and CPU pegged for over an hour when they’d run the entire suite.
The Solution: Semaphore Boosters – Automatic Parallel Testing for Rails
Our interest in automatically parallelizing our test suite was first piqued by blog posts from Semaphore. On a conference call when were were struggling with long running tests and timeouts, Semaphore recommended trying Semaphore Boosters, automatic test parallelization for Rails, to see if it helped improve our build time.
When we started researching various ways of speeding up our test suite, we were having builds (some even timing out) at the 45 minute mark. Prior to that, we’d consider a good parallel build to be about 23 minutes. With enhancements we made, pulling Boosters in for very reliable parallel builds, and spreading the work across even more threads, we are almost at the 10 minute mark some days.
Why Boosters
The fact that Boosters would self-balance after initial setup felt like a huge win to us. Additionally, that it is directly supported by Semaphore, and we have had good, warm experiences with the same team, was compelling. Lastly, the fact it provides a Heroku-like slider to scale the Boosters made it feel like home. It’s a small detail, but having to create threads manually before and make sure the count and config for each was right, is just not as nice as a slider. Great UX makes for a great product!
Our build had gotten to the point where we had weeks off and on of timeouts and failed builds we couldn’t confidently deploy. After settling in, making changes to the suite, and switching to Boosters, we could start doing pretty continual deploys once again.
Setting Up Boosters
The process of setting up Boosters was pretty smooth. The biggest challenge we had was that, a lot of feature specs using javascript drivers in our suite can cause intermittent failures. The initial Boosters build needed to go green for us to start running it fully, so it was a lot of trial and error pending problematic specs to get a green build. That said, those challenges were pretty self contained to our test suite; the setup on Semaphore itself was intuitive and had that first class feeling that made us confident with continuing forward with it.
Auto-balancing the test suite for us is a huge win, it’s just one less thing to think about. The fact that it’s baked in and feels like a “first class citizen” on the Semaphore platform gives us a lot of confidence it will be well maintained and integrate well with the platform on an ongoing basis.
Getting up and running easily with Boosters was the kind of experience we’ve come to expect from Semaphore (i.e. easy and user friendly). We saw tremendous benefits without us having to go tweak a bunch of dials and get back to thinking about our code and tests.
If you’re using RSpec or Cucumber for your tests, spend 15 minutes on setting up Boosters, speed up your tests, and get rid of the overhead of manually balancing your test suite.