In this episode of Semaphore Uncut, I talk to Benjy Weinberger, co-founder of Toolchain. We discuss the open-source build tool, ‘Pants‘, and hear Benjy’s views on the monorepo strategy for managing your codebase.
Key takeaways:
- Pants: a fast, scalable build system
- Explicit modelling of dependencies is key to Pants performance
- Monorepo gives visibility and ownership of the effects of your changes
- Monorepo helps avoid dependency hell
- How Pants works: a concrete example
- Tools to make adopting Pants easy
- How to contribute to Pants V2
Listen to our entire conversation above, and check out my favorite parts in the episode highlights!
You can also get Semaphore Uncut on Apple Podcasts, Spotify, Google Podcasts, Stitcher, and more.
Like this episode? Be sure to leave a ⭐️⭐️⭐️⭐️⭐️ review on the podcast player of your choice and share it with your friends.
Edited transcript
Darko (00:02): Hello, and welcome to Semaphore Uncut, a podcast for developers about building great products. Today, I’m excited to welcome Benjy Weinberger. Benjy, please introduce yourself.
Benjy: I’m the co-founder of a startup called Toolchain. We’re working on a distributed software build system based around an open-source core. I’ve been a software engineer for over 25 years now. I’ve been really focused on the software build space for probably the last 10 years or so.
Pants: a fast, scalable build system
Darko (00:38): Great. And during the prep call, we talked about a build system that some of our listeners might know about, called Pants. Can you give us a history of how it came about and where it’s going?
Benjy: Pants was a system that started out as an internal system at Twitter for building Scala code at scale. And that project was initiated by John Sirois, who is now my co-founder at Toolchain. And I joined Twitter and encountered this system, and was pretty excited to use it. I got connected to John that way. Fast forward a bit, after I left Twitter, I went to Foursquare. And Foursquare had a similar set of problems with scaling its Scala builds. So, I reached out to John and suggested that he get approval to open source Pants, and we could collaborate on it in the open.
Fast forward again, and the code in Pants was getting quite old. We had learned a lot from building it. We started working on the next generation of Pants, which we unimaginatively refer to as V2, launched that last October.
One of the big differences is that the initial use case that we use to drive this development was Python. Because, in those intervening 8, 10 years, Python has really exploded in popularity. The idea of having large, scalable Python code bases that span multiple libraries and multiple projects is now quite common. So that’s where the Pants project is right now, relaunched with brand new technology built in a combination of Rust and Python, and with a focus on Python as the target language.
Explicit modelling of dependencies is key to Pants performance
If you think about older systems like Make, or Maven, or SBT, the state of the system is essentially the state of your file system. Nothing is explicitly modeled.
Now with Pants V2, everything is modeled as workflow. Your build is broken down into a very large number of small steps, work units, and small tasks. And each of those is fully specified by its inputs and outputs. Everything is explicitly modeled.
And what that allows you to do is it gives you very fine-grained invalidation. You can cache because you can know all the inputs and you can fingerprint them. You can run concurrently because, again, every piece of work is fully encapsulated and so you know all the dependencies. And finally you can run build work remotely. I can run a hundred tests at the same time because I have this cluster that can accept and perform that kind of work.
Monorepo gives visibility and ownership of the effects of your changes
Darko (05:42): In the industry, there is a heated debate about monorepos. With your experience in build systems, what’s your position on that?
Benjy: Obviously every case is different, but I would say this: scaling your code base, the problem comes with analyzing changes and how they propagate through your dependencies. You will have this problem in a monorepo and you will have this problem with multiple repos. Why I come down on the side of monorepos is that at least monorepos make that problem explicit. So, your first party dependencies all live in the repo with you. You have visibility into everything that depends on you and when you make a change, you have the opportunity to propagate the ripple effects of that change correctly in your repo.
If you’re going with a multiple repo architecture, to some degree you are kicking the can down the road to other people. You’re creating a situation where there’s no incentive and no tooling to even enable correctly propagating changes through your code base. So generally I found that monorepos have advantages as long as you have the right tooling, and a lot of the work that we put into Pants is to provide that tooling.
Monorepo helps avoid dependency hell
Darko (08:12): And is Pants going that far that the person which is making the change, introducing new version of a library, should propagate that change across the projects?
Benjy: Well, generally in a monorepo, if you make a change then, with the right tooling, with something like Pants, you can run all the tests that depend on your change, and you can see what you broke. Generally, I prefer the mode where it is your responsibility to fix that. So essentially the versioning and the whole point of a monorepo is that you avoid dependency hell, where you have transitive dependencies at conflicting versions. The flip side of that is your repo has to be internally consistent. The right tooling allows you to achieve that internal consistency fairly easily, because you can reason about what are the possible effects of my change? Where could they ripple out to?
How Pants works: a concrete example
Darko (11:43): You mentioned using high-level concepts where each unit of work during the build process can be fingerprinted, and then later reused, which could save a lot of time. Can you give us maybe a better concrete example?
Benjy: The classic example is running tests, because all languages have that, and ideally you have a good amount of tests in your code base. And if not, then go away, write a bunch of tests, and then come back and finish listening to this podcast.
Basically, in most of today’s build setups, I have to run all the tests and I have to run them one at a time because the system can’t reason about what tests could potentially be impacted by my change, so it has to rerun them all. It doesn’t know how to cache the results because it has no coherent sense of a key to cache against. As I mentioned, the state of your build in a system like Make, say, is the state of the entire file system or the relevant part of the file system.
Pants takes the approach of no, this needs to be modeled as an explicit workflow. So basically, it knows about all the static code dependencies, it knows that this test imports this file, and this file imports this other file, and so on, transitively. So it knows which changes to which files affect which tests. It also knows all the other inputs needed to run a test. The system knows about all of these things, so it looks at those and it says okay, all of these inputs have been resolved.
To make the example more complicated, let’s say that you have tests that depend on generated codes. So you have a protocol buffer IDL, and you need to run the protocol compiler to turn that into Python code. The system knows that, so it can say okay, I cannot run this test until I generate code, because it depends on the outcome of generating this code. So it knows that it can’t run the code generator and the test concurrently.
Furthermore, let’s say now someone makes a change to that protocol buffer IDL that I mentioned, but it’s just a white space change. The file has changed. Its fingerprint has changed. The system knows that it has to rerun the generator, but when it does, the resulting Python file is identical. So its fingerprint is identical as some previous result that we have. So we don’t need to run those tests again. That’s what I mean by treating a build like a workflow.
Tools to make adopting Pants easy
Darko (17:46): How about the initial implementation and maintenance of that workflow in Pants, is it something which is reasonably easy?
Benjy: We put a lot of effort into making it reasonably easy. So the thing about the build tools like Pants, you have to teach it the structure of your code. The graph of your dependencies has to live in these build files. We know from our own experience that maintaining those is a giant hassle, especially when you start trying to adopt a new tool. So we have several mechanisms for making this much, much easier.
The first and probably most important one is dependency inference. So unlike other tools, we do not require you to explicitly enumerate your dependencies in those build metadata files if we can infer them from your import statements. Occasionally, there’s a dependency you can’t infer. Sometimes you load code by name at runtime or you depend on assets that are not imported. There are cases like that, and in those cases you have to potentially manually add those dependencies. But 99% of the time you don’t.
And also, that logic is itself pluggable, the dependency inference logic.
The second is we have sensible defaults for things. So when you declare this directory contains a Python library, or a bunch of Python tests, or whatever, it knows that tests are typically called test_something, or something_test, or test.py, or conftest.py for configuration. So it knows that, and it knows that everything else that ends in .py or .pyi is a library or so on.
And then the third thing is called the Tailor feature because it helps you make your ‘pants’. Tailor basically generates build files by looking at what files you have. And it’s idempotent, so you can just keep running it as you go and it will add new build metadata as it goes.
So with these features, we’ve put a lot of energy into making it much, much, much easier to adopt this kind of system than it’s ever been, and that continues to be an active area of work for us because we will not rest until it is really, really easy.
How to contribute to Pants V2
Darko (21:36): So you mentioned Python and Scala being in Pants V2. And you also talked about extensibility and supporting other technologies. Can you give us a roadmap?
Benjy: Pants is fully extensible and has this plug-in architecture. Anyone can write their own rules using the exact same APIs that the official rules use. Many organizations have custom build rules, or people have custom code generators, or custom linters, or custom checks. So you can just write code to do that.
If there’s a language, or a library, or a framework that’s not supported and you want to add support, we’re an open source project and we obviously welcome contribution. Anyone who wants to add functionality that’s useful to them and may be useful to others is welcome.
Darko (24:09): Yeah, great. Okay, so folks can head to …?
Benjy: It’s pantsbuild.org. And you can also find me on Twitter. I’m @Benjy or benjyw@gmail.com. The best place to ask questions about Pants is Slack, and you can find the link at pantsbuild.org.
Darko (24:49): Great. Okay, thank you, Benjy, for sharing all this and good luck with the project.
