Watch the replay of our CI/CD Express Conference 2023 →

    11 Apr 2018 · Software Engineering

    Lies, Damn Lies, and the State of Testing in Rails

    4 min read

    Ruby on Rails was the first major web framework to include automated tests as part of every new project. The effects have been enormously positive. Many developers have learned to write tests well, which in turn helped them get better at design. Experienced developers were drawn by Rails’ focus on productivity and beautiful code. Their community activities have helped push the craft of web development forward. Over the years positive effects have rippled to other communities too.

    In practice though, how much do Rails developers really write tests? An interesting outcome of building Rails Testing Grader – a website which lets you compare your test suite to the industry – was that we learned what an average Rails app looks like.

    The average Rails app has 15,443 lines of code, a 1.03 test-to-code ratio and a continuous integration build that runs for 6 minutes and 36 seconds.

    The problem with averages is that they’re, well, unsatisfying. According to National Geographic, the average human is a 28-year old Han Chinese man. A lot of people are not a 28-year old Han Chinese man, and your Rails app probably doesn’t fit the description above.

    To get a better picture, let’s take a look at the statistical distribution.

    The following graph shows the size, in lines of code, of randomly sampled 1,000 commercial Rails apps running CI on Semaphore. Commercial Ruby on Rails applications by size Many Rails apps are relatively small, with 5,000 lines of code or less. But the power of Rails is that you can build an entire business on it, and the right-hand side of the chart shows some examples.

    Test to code ratio and continuous integration

    Can you have too many tests? Better questions to ask are:

    • Are you testing the right things?
    • Are you adding unnecessary tests while leaving important parts of your code untested?
    • Are you keeping your test code clean?

    The test-to-code ratio can’t answer all those questions, but it’s a simple metric that gives a sense of the codebase. A ratio lower than 1 usually means that your test coverage has room for improvement.

    It turns out that half of all Rails apps have a test to code ratio of 0.67 or lower.

    Private Ruby on Rails projects test to code ratio

    And half of all apps have a fast CI build of around 3 minutes. However it’s obvious that the spectrum of projects is very wide.

    Commercial Rails apps continuous integration build time distribution

    Segmenting data by code size

    We know from experience that as the app and team grow, it gets more challenging to maintain the quality of code. Here’s how the distributions change if we focus on applications of a specific size. Test to code ratio: Commercial Rails apps test to code ratio by code size distribution Continuous integration build time: Commercial Rails apps CI build time by code size distribution

    Why do bigger projects struggle with test coverage?

    20,000 lines of code seems to be the threshold when the proportion of tests sharply drops. We don’t know whether these projects never had a higher test to code ratio, or it has dropped over time. A large code base has complexity which certainly brings both technical and social challenges:

    • Without optimizations or specialized solutions, CI runtime increases. This slows down the feedback loop and introduces more interruptions to the flow state.
    • The discipline of writing tests requires constant dedication from all team members.
    • Code size usually correlates with having reached some business success. The engineering team needs to face a new set of challenges which didn’t exist when they were coding for a small number of users.

    Overall, we’d say that the practice of writing automated tests remains very strong in the Rails community.

    What are your experiences and thoughts on the data presented? Let us know in the comments.

    If you’d like to compare your app’s stats more closely with similar Rails apps, check out Rails Testing Grader.

    Leave a Reply

    Your email address will not be published. Required fields are marked *

    Writen by:
    Marko Anastasov is a software engineer, author, and co-founder of Semaphore. He worked on building and scaling Semaphore from an idea to a cloud-based platform used by some of the world’s engineering teams.