This episode of Semaphore Uncut features Jonathan Creamer, Senior Software Engineer at Microsoft. We hear his monorepo experiences and about his work in the field of ‘DivOps’ – the term he coined to describe the engineering of front-end tooling.
Key takeaways:
- DivOps is DevOps for the front-end
- Monorepo’s power is having all dependencies at your fingertips
- Make and test sweeping changes across the whole monorepo in seconds
- Embrace monorepo early to keep options open
- Monorepo tooling enables micro-frontend architectures
- Workspaces and Webpack module federation are ones to watch in future
Listen to our entire conversation above, and check out my favorite parts in the episode highlights!
You can also get Semaphore Uncut on Apple Podcasts, Spotify, Google Podcasts, Stitcher, and more.
Like this episode? Be sure to leave a ⭐️⭐️⭐️⭐️⭐️ review on the podcast player of your choice and share it with your friends.
Edited transcript
Darko (00:02): Jonathan, thank you so much for joining us. Can you introduce yourself?
Jonathan: I’ve been programming now for going on 11 years. I started in computer engineering technology with a degree from Middle Tennessee State, then started following folks in the .NET space like Scott Hanselman. .NET MVC lent itself towards the V a lot and put an emphasis on doing more client-side development, which I decided I liked. Eventually, I moved on to a company where we did all JavaScript and I realized that JavaScript is all I want to do all day every day.
At Lonely Planet, a travel company, and did a lot of React development and architecture and started getting into more of the tooling side of JavaScript. I realized how much I liked building tools that you may not see on the browser, but giving my engineering team the ability to do their job. Then I got an opportunity to go work at Eventbrite, a big ticketing company, and they have a team dedicated to doing front end infrastructure, FE Infra. And we were basically taking code out of a big monolith that they had and putting it into a monorepo, which is a really cool opportunity to learn the ins and outs of building a monorepo from scratch, and the cultural side of building and working inside of a monorepo.
And now I’m at Microsoft. There’s a lot of different repos all over, and Microsoft has a goal of improving JavaScript developers’ lives. And so that’s what I’m going to be doing here is building a really nice monorepo and just tools in general for Microsoft Office, Microsoft, and hopefully beyond.
DivOps is DevOps for the front-end
Darko (06:37): There are two very interesting things. One is monorepos and the journey that you mentioned. And the term that you coined, DivOps.
Jonathan: Yeah. DivOps is this funny thing. I am a front end developer. I write in JavaScript, I use front end tools and technologies, like Node and React and all this stuff, but I don’t actually build front ends. So, what do we actually call ourselves? I put this poll out on Twitter and eventually somebody said, “You guys should call yourself DivOps, like <div> ops.”
And we were all just laughing and said, “That is so perfect. We’re kind of like DevOps, but we’re working specifically on this front end bent.” It causes a discussion, which is what I love about it more than anything. It causes people to say “DivOps? What is that? Explain that.” My goal with the whole thing was to just bring people who do kind of what I do into a place where we can talk, because we’re solving same problems, just slightly differently. I know that people inside my new company, Microsoft, are doing things, literally the exact same types of things that Atlassian was doing, Eventbrite were doing things, and how Shopify is doing some things. And so if you get all these people together, maybe we can come up with some really cool solutions. So yeah, it’s been really fun to see that take off.
Monorepo’s power is having all dependencies at your fingertips
Darko (10:03): Going back to the Eventbrite and the challenges that you saw there, you mentioned it was about separating a front-end part from a monolith. Can you give us an overview how it started?
Jonathan: When I first got to Eventbrite, all of the front end code, everything, the Python, the JavaScript, was all in one ginormous monolith. The biggest issue and most annoying problem was if you wanted to change a line of CSS, you had to wait for the monolith to ship, which only could happen periodically. The deployment processes were so complicated.
Jamie Kyle joined our team for a little while and he obviously was big in helping build Yarn workspaces and Lerna. He’d actually been working on a tool called Bolt at the time, that we were able to prototype and adopt. Prototyping this monorepo concept at first just sort of blew my mind. I hadn’t really done much monorepo development, but he showed us the power or how, if you have that dependency tree in your code base and all your tools are right there next to the code, how so much is just easier.
You can run scripts against your whole code base in seconds and parallelize it. And it had some built-in CI smarts to go to CI to help you get to production. We were able to work really hard on this idea of the dependency tree and shipping only certain parts of the monorepo, using independent deployments. I think the monorepo, what I’ve learned is it’s as much of a cultural thing as it is anything. And once we were able to show everyone a lot of the benefits of it, people were like, “Okay, let’s go for this thing.”
Make and test sweeping changes across the whole monorepo in seconds
Darko (15:12): Monorepo is still something of a new term. What would be your pitch?
Jonathan: I think a monorepo is going to be the most successful when you have cultural buy-in and then a team who is dedicated to that monorepo.
When you start separating your app out into lots of small packages, this tooling just makes everything easier, especially if you have shared components, shared utilities, or a component library. Then with that monorepo dependency tree, because everything is just right there next to your code, you’re able to say definitively, “If I change this grid component, here is the blast radius downstream of all the changes.” And then you can run your unit tests and run your integration tests against that list of things, that’s powerful.
Part of the DivOps thing is writing code that writes code using something called ASTs. If you’ve never heard of that, it’s basically how Babel, Jest and ESLint all work. Using that AST, you can make broad sweeping changes across your entire code base, in a matter of literally seconds. Upgrading React was one really good example of this. We were on React 15.4 for a long time, and there were some changes we needed to make to the entire code base, hundreds of thousands of lines of code, to upgrade to React 16. And so our team, using that technology, Babel, codemod, and all this stuff, was able to write a few transformers and convert thousands of lines of code in a few seconds inside the monorepo. And we could just see it all happen and then run all those tests and just make sure everything was still working.
Embrace monorepo early to keep options open
Darko (21:37): When is the right time to embrace? Have you seen or heard maybe some anti-patterns in this area where people run into something which is maybe not for them at this stage?
Jonathan: I would say it’s a lot harder to take code from all over, from multiple repositories and put it back into one, than it is to start with one and spread it out. If you just are a new startup and you just use vanilla Lerna out of the box or vanilla Yarn out of the box and don’t do anything crazy, there’s hundreds of good posts on how to do that, you’re going to find it to be easier to keep that in one place and then decide later. That’s why I would say it’s probably okay, even from the get-go.
The main other thing with a monorepo, you start dividing up your packages into the monorepo, then you sort of start building and thinking about your units of code, your packages in a similar way you’d think about writing a unit test and trying to test small things. You don’t want to have too many large swaths of code and big packages. It forces you into this mindset where you’re already thinking in smaller units of code and these small packages. So that’s why I say, even if you’re a new startup, it might be a good approach. It’s that culture thing. And that’s why I would just recommend it. I think it’s a really good practice.
Monorepo tooling enables micro-frontend architectures
Darko (24:48): Something that you refer to also, is can I ship independently? It seems that there is some tooling that needs to be present?
Jonathan: A lot of people are moving towards this idea of micro-frontends: small, independently deployable apps, just like microservices. If you are going to move into the micro-frontend space, having those micro-frontends in a monorepo is really nice because if some package that touches both the dashboard and homepage changed, you can see that effect happen and run those tests and verify that your dashboard and your homepage are still good to go.
Once you’ve got that dependency tree it comes down to having that tooling team understand that dependency model and being able to then extrapolate that into what needs to go out to your staging server first and then into production.
And it’s even more powerful than that because a micro-frontend could even be just a component on a page. It doesn’t necessarily have to be the entire page. If that little component on your dashboard is a micro-frontend, same principle applies. That team changes that micro-frontend, then all the tests run and you can have some cool integration tests where this micro-frontend runs inside of this other micro-frontend. And because of that tooling that you have built with your monorepo, you can effectively determine that blast radius. This changed, so now I need to verify that it all works over here, and that’s just a really powerful thing to be able to do.
Workspaces and Webpack module federation are ones to watch in future
Darko (29:39): Bolt, you’re mentioning, you mentioned Lerna and Rush is also referenced in your blog. And to this point in time, is there something which you feel that it’s still itching and needs to be solved?
Jonathan: I’ve seen lots of Lerna repos and I’ve seen lots of Yarn workspace repos. I’m curious to see how well this works once we are dealing with a thousand packages versus we have 500 right now. Is it going to continue to not go exponential in its scale? Is it going to stay kind of this linear thing? I think the answer’s yes. And I hope it is. I’m very curious to see how NPM workspaces end up working.
There’s a Federation idea which is you could build your packages and put them somewhere and then different Webpack runs could remotely pull your bundles in. So that way, talk about not building things didn’t need to build, you could just literally fetch bundles from somewhere else and pull them down at runtime. It’s going to be really, really powerful for front ends in the future, especially as this micro-frontends architecture is taking off even more.
Darko (33:28): Great, great. So thank you for sharing all this. I think especially for people who maybe have not ventured in these waters, this would be probably a word of encouragement. So at least try it out in some limited scope and see what this brings. Okay. Thank you again so much. Good luck with your adventures as part of Microsoft.
