In this episode of Semaphore Uncut, José Valim, the creator of the Elixir programming language shares with us insights on developing Elixir, from its inception to the road ahead.
José is the Chief Adoption Officer at Dashbit, speaker, and the author of three technical books.
We talked about:
- How Elixir was born
- Elixir’s functional roots
- The role of Erlang VM
- Modeling concurrency in Elixir
- The milestones of creating a new programming language
- Extensibility as a must
- Elixir macros enable the ecosystem
- Tips to stay focused
Listen to our entire conversation above, and check out my favorite parts in the episode highlights!
You can also get Semaphore Uncut on Apple Podcasts, Spotify, Google Podcasts, Stitcher, and more.
Like this episode? Be sure to leave a ⭐️⭐️⭐️⭐️⭐️ review on the podcast player of your choice and share it with your friends.
Edited transcript
Darko (00:02): Hello, and welcome to Semaphore Uncut, a podcast for developers about building great products. Today, I am excited to welcome José Valim. José, thank you so much for joining us.
José: Thanks for having me. I am José Valim, and I believe most people know me as the creator of Elixir. I am also the chief adoption officer at Dashbit. We are a company where we are helping startups and enterprises run and adopt Elixir, in different stages, different industries, and so on.
How Elixir was born
Darko (00:45): Can you give us a bit of history, how you ended up doing what you’re doing and how that progression came to the point where you felt that the world needs a language that will address a certain group of problems?
José: I was a Ruby/Rails developer and at the time, back in 2008, we were starting to hear a lot about concurrency and that machines, they’re going to have multiple cores.
I knew from the experience of maintaining Rails and other root projects that writing multi-core software, which is software that runs on all cores with Ruby, was not really straightforward. So for example, we are trying to make Rails thread-safe. Sometimes we would get a bug report like, “When are we having a spike in production, and we deploy a new server, we have this error, but we don’t have this error any other time.
And that means when there’s a spike in production in the part of the server, there was a bunch of requests coming at the same time and that is going to exercise some code paths that it’s very hard to exercise otherwise that would make bad things happen. And those things, they were hard to debug, hard to understand, hard to fix. So, I started looking. Originally I started to say, “Okay, if concurrency is going to be important, I want to see how people are solving those problems.”
Elixir’s functional roots
Throughout this journey, I’d like to say, there were two particular moments of like no return where I said, “Okay.” That effectively changed the way I thought about software. The first one was functional programming. And functional programming can mean a lot of different things to a bunch of people. But to me, what it meant at the time was immutability.
Because a lot of the time, when we are working with most languages that are imperative languages, Ruby, Java, JavaScript, Python, C, we have a lot of issues with concurrency because we are changing things in memory. So it happens that if you have two threads trying to change the same place in memory, you have two cores in your machine trying to change the same place in memory they can corrupt that memory, or they can put in a state that is not understandable.
In functional languages, generally, we avoid doing that. So everything’s immutable. If you have a list of three elements, and you delete one element, now you have a whole new list with two elements. That was when things clicked for me. I understood like, “Wait, if I’m not mutating my data structure, if everything’s immutable, there’s a whole category of problems that I have been debugging and trying to fix in the past that they no longer exist.”
And to me, that’s like my favorite way of solving problems. I can fix problems, but if I can disappear with the root cause of the problem altogether, yeah. I’ll take that. So I was like, “Okay, this is great. If I’m following those principles, I think my software is going to be better. It is going to be ready for this multi-core future.” And that was the first point of no return.
The role of Erlang VM
And the second point of no return was when I found out about the Erlang virtual machine, and the thing was the focus of the Erlang virtual machine was not concurrency. It was distribution and it has been distribution for three decades. is that back when I was exploring those things, everybody was talking about concurrency.
Imagine that you’re writing your software and you wanted to use a single machine as efficiently as possible. So if the machine has eight cores, you’re going to leverage concurrency. But what happens when that machine is not enough? You cannot run everything your system has to do on a single machine. Or if you want to have two machines, because you want some resilience. They want to have some robustness there not to have one single machine in production, you need to have multiple machines.
So how can you make those multiple machines now communicate and collaborate together? That’s exactly when distribution comes in. Everybody was trying to solve the issues with concurrency, but we have this virtual machine, the Erlang virtual machine, that has already solved those issues, and they are now talking about distribution. So they are ahead of everybody. And I still believe the virtual machine in this sense is still ahead of everybody today.
And then I said, “Okay, this is the platform that I want to build software with.” So I started using more and more the Erlang virtual machine and the Erlang language. I found out about some places that I would like to improve the language or there were some particular kinds of software, some particular kinds of constructs that I felt were important and were not there. And that’s when I decided to start working on Elixir.
Modeling concurrency in Elixir
Darko (12:01): What are your experiences with people generally wrapping their minds around concurrency and using it? Can you share how that part of the language is being adopted in use? What are your experiences?
José: One thing that’s worth talking about is that Elixir doesn’t solve all concurrency issues. You don’t just write the software and they are no concurrency issues whatsoever. It does remove the majority of data issues, but you still have concurrency issues. But I don’t know if there’s a better word for this, but I like to say those are natural concurrency issues.
It’s like, imagine there is a cup of glass on the table. Two people are going to get it at the same time. You can’t split the glass. The water’s going to be everywhere. Those are natural concurrency issues, and they exist everywhere.
But the way that you model concurrency issues in Elixir is by having this idea of actors. In the community, we call them processes. Those are lightweight processes. They’re not operating system processes, but I’m going to go into the less confusing term in general: actors. We have actors and actors, they communicate with messages. It’s such a natural way to think about concurrency that once you get familiar with how Elixir models concurrency and approaches concurrency, you’re actually better prepared to think about those natural issues that happen. It’s going to give you a mental framework to approach those problems in general.
I think the mental framework is going to help you as a developer, as an engineer in general, throughout your career and how you’re going to approach those problems when you have clients and servers or systems in general. It’s a way to give that foundation.
The milestones of creating a new programming language
Darko (16:26): Something that might also be interesting for folks listening is, how the journey of creating the language looks like. If you look back and connect with that, how do you plan future iterations? Because this will obviously be a decade long project with a huge impact.
José: In the beginning, I was mostly driven by curiosity and trying things out. The prototypes kind of failed. You expect the prototype to fail in a way, but the important thing about the prototype is the lessons you learn from it. So I did a couple of those, and some point I was actually able to come up with a vision of what I thought the Elixir could be.
When it was clear what the goals for the language were, then everything was much easier because I had a direction. Because if you think about creating a programming language, there are like just so many options that you can take, just so many different approaches.
And then I was able to go to my company, do a presentation to them, Plataformatec at the time, and told them, “This is what I want the language to be. This is why I think the language is going to be important.” So, they accepted the proposal for me to work part-time on it.
And then, after one year and a half of working on this vision, more people starting using it. More people were actually betting on writing books, putting their human hours into it. And that was when I started feeling, “Oh, this may actually work after all.” We see that there are new people coming all the time and learning language, exploring and building new things, like solving completely new problems that I personally wouldn’t have thought about.
Extensibility as a must
One of the things that I had in mind for Elixir was that I wanted Elixir to be an extensible language. And what it means specifically is that I want the Elixir to be small… Not like a super small language, but the language itself should be small, and the developers should be able to get the language and extend it to the domains that they want.
So if you want to get Elixir for data processing, you can do that. You can extend it to web development. You can use it for distributed systems, doing embedded devices and all of that. So, that was the design of the language. I didn’t want the language to be able to support all those things natively. You should be able to extend and grow the language.
If you have to be constantly changing the language and constantly improving the language, this goal kind of failed. If the language is extensible for the whole ecosystem to grow and for the community to grow, the language doesn’t need to change. Everybody can build on top of that. And we have seen exactly this happening. If you look at the last Elixir releases, we are working on the infrastructure; we’re working on making the compiler faster. We are working on finding errors earlier and improving the developer experience.
For Elixir itself, I feel like the language is in stable mode. We are continuously improving it, but it’s not like big flash news. And the bulk of the work is happening rather in the ecosystem, with new ideas, new frameworks, new projects.
Elixir macros enable the ecosystem
Darko (22:16): When you are talking about the extensibility of language, is in practice the tooling to achieve that journey, the macros and the macros system that’s available, or there is something more?
José: Macros play an important role, not necessarily the macros themselves, but what we had to do to have the macros. The Elixir syntax is quite regular in terms of constructs and things that there are in the language. There are also aspects of how we make software extensible as a whole.
And for example, one of the features in Elixir that plays an important part in that is protocols. Protocols allow library authors, for example, to define contracts. And users, they can fill in those contracts. To me, as a library author, I can say, “Hey, I can create this library that is going to work with anything as long as this thing implements this contract.” And then you can plug this hole, and then you can have a bunch of libraries that are just depending on those contracts and allowing the ecosystem to grow without a lot of interference, “Oh, I can’t do this because the thing’s expecting this from this other thing. And we’ve got a lot of conflicts.”
Tips to stay focused
Darko (25:53): Since let’s say you’re holding this title of uber-programmer, who created the language, can you share some tips with developers on how you work today and how it changed since you started working on the language now, primarily spending time in the language that you have created?
José: The main tip I like to give is to be very mindful of distractions. I know this is something that you hear from other people, but it makes such a big difference for me. So I don’t have anything that can interrupt me when I’m working. That has been extremely helpful. And I usually tell people, just turn off.
The other thing that I do as well is that I do a pre-planning. I think of what I’m going to do tomorrow. When you have to start doing something new, the hardest part is getting that energy for starting that new thing. If you take a little bit of planning before, it actually turns out you are kind of curious, “Wait, I think I can solve it this way. Or I think that’s how I’m going to approach this problem.”
It helps remove blockers, it reduces your potential downtime. So those are things that I do each day, and they kind of give me going. There is one downside, which works well for me because sometimes I am working on something, and then I start planning the next thing. But the next thing that I want to do is super exciting that I drop the thing I’m doing right now, so I do the other thing. I tried to not do that a lot, but sometimes it does happen!
Darko (30:04): José, thank you so much for sharing all this with us.
José: Thank you very much. See you around.
