We invite software industry professionals to Semaphore Uncut YouTube show to discuss what problems they’re currently solving and what excites them about the emerging technologies.
In the first episode with a guest appearance, we were chatting with Eddie Zaneski, Developer Relations Manager at DigitalOcean.
This chat was recorded on April 3rd, 2019 and is also available in the form of a podcast.
Full transcription below.
Darko: [00:00:06] Hello everyone. Welcome to Semaphore Uncut. This is the first episode where we have a guest. We plan to talk about developer tools, people who are making them and about people who are helping developers to embrace those tools. My name is Darko Fabijan and today I have Eddie Zaneski from DigitalOcean with us. Thanks so much for joining us!
Eddie: [00:00:31] Thanks for having me Darko.
Darko: [00:00:34] Before I give you the chance to introduce yourself I would just like to invite people to let us know in the comments which are somewhere on the side. Let us know if you have any trouble seeing or hearing us and feel free to jump in with questions at any point and we will take care of them as we go into the show. Eddie, feel free to go live and introduce yourself.
Eddie: [00:00:59] My name’s Eddie Zaneski and I serve the developer community at a company called DigitalOcean. If you’re not familiar we are a cloud provider, we’ve got everything you need to deploy, scale and build the cool things that you’re working on. You might be very familiar with us from our tutorials that might have saved your ass on Google search at some point. We reach a lot of developers and get a lot of lovely feedback from the community. We’ve got a good team working on that. I’m based out of Denver Colorado in the United States right now.
Darko: [00:01:35] Yeah. So when you touched upon that topic I saw a tweet on your Twitter stream with climbing gear on your back and hanging from your vest. Maybe you can tell us a bit about your hobby and how you got into that.
Eddie: [00:01:58] I used to live in Brooklyn, New York and one of my friends invited me to go to a climbing gym one time (I think it was 2014 or 2015) and I fell in love with it. It was pretty awesome. I started climbing a lot more in the gym eventually worked my way outdoors – started climbing some mountains and I didn’t realize till was you know pretty deep into this climbing hobby that it is a very stereotypical thing for a developer to be into because it’s very technical. I’ve got a lot of hardware here…
Darko: [00:02:33] You’re always ready?
Eddie: [00:02:36] These things go into the rock the save my ass from falling and it’s very hands-on, it’s very technical and it’s a lot of problem-solving. It turns out developers gravitate towards climbing as I played myself into another stereotype which is fine. I recently moved out here to Colorado back in November because the mountains are so much bigger. So my wife and I hopped in the car we drove the three days with our dog and we’ve been climbing big mountains and since we got here so l Iove doing it.
Darko: [00:03:07] Yeah great. It’s one of those categories – it’s obviously high on your priority list – you’re moving because of that. Yeah. That’s great to hear that.
Eddie: [00:03:17] Yeah. I was never into sports before. So this was the first time I was doing anything physical.
Darko: [00:03:26] To be honest I’m in the same category – not many sports but during college, I got into hiking. So yeah that probably in a stereotypical developer thing.
Darko: [00:03:45] Let’s move to the business area. In the developer relationship area – I was looking around preparing for the interview and I saw that on one of DigitalOcean YouTube channels you gave a fantastic sum up in a couple of minutes about developer relationships. To be honest, that’s one of the best essentials that I heard about that, so maybe you can just quickly repeat it for us.
Eddie: [00:04:20] I like to describe developer relations (people hate when I say this) but at the root of it – it’s marketing for developers. Developers have a strong shield and a wall that goes up whenever they’re talking to someone sales-y or marketing or just even non-technical in general. It makes a lot of sense right: as a developer you want to talk to other developers and so that this grassroots movement of like having developers who enjoy talking to other developers is is a great way to build trust and relationships with your brand, with your company. I always say that technical credibility is everything right. It’s very important that you know what you’re talking about – you can back it up. Because as soon as you shed any type of chink in your armor of being a technical developer, folks will write you off as a marketer and they don’t really wanna talk to you anymore. It’s pretty great to be able to you know engage communities and help folks out with their first journey to the cloud. I’ve been doing that well since the early days and I used to work at a company called SendGrid and I’ve loved it ever since.
Darko: [00:05:36] It’s like a specific set of skills that you need to have. So it’ll be kind of very hands-on and you have to get that great technical knowledge but on the other hand, you will you have to be into talking with people you know to engage with the community in a broader sense. How did you get attracted to that? Was it natural, that from your developer job you ended up doing that or how it came about?
Eddie: [00:06:02] That’s a great question. I went to a company called Twilio and so that’s where I learned a lot from a great man named Rob Spectre. We would say that the qualities that make a good developer advocate or evangelist are the head of a hacker and the heart of a teacher. Being able to empathize with other developers. I guess my story starts when I was in high school you know figuring out what the heck I wanted to do with my life. I was always good at math and so OK you know maybe I’ll be a math teacher because I guess that’s what people who are good at math do. So I went to school for math education I was going to an accelerated master’s program and then I started student teaching and I just hated it. It was just not fun. I was working with some freshmen I think teaching Algebra 1 and you know the common phrase came up “When am I ever going to use this when am I ever going to need this?”.You know you want to say like “All the smart kids are going to use it”, but you can’t say that obviously right? I just couldn’t see myself doing and teaching something that I enjoyed so much to folks who really didn’t care about it. At the same time, I had to take an intro to programming class for my major and my dad is an old-school COBOL DB2 Fortran developer back in the IBM mainframe days. I never thought I’d be doing nerdy computer stuff like my dad. My first intro to programming class – I fell in love with it. I was the coolest thing I’d ever done. I was never creative, so I hated art class and that kind of stuff, but you know being able to print out “Hello world!” for the first time on the screen just blew my mind. It’s the first time that I could do something that you know didn’t really it didn’t look like crap and I was proud of it. Fast forward I switched computer science I wind up landing an internship at SendGrid doing DevOps because I had always been interested in Linux systems. Summer ends and I stay on the team while I finish school and go into a lot of hackathons and go into a lot of meetups and go into conferences. I keep seeing that the SendGrid DevRel team around and they’re like “Hey you really enjoy doing this kind of stuff and working with the community. Why don’t you just join our team and you get paid to do this kind of stuff?” I’m like “Oh that sounds great – sign me up”. That’s really how I wound up in DevRel and I think it’s a mixture – I wanted to be a teacher so I still get to work with people who want to learn and they actually want to learn, that’s the difference there. What I realized at the end of it was I didn’t care too much about math. I like problem-solving and that’s what programming lets you do on a daily basis. It’s kind of like the perfect storm – all my pieces came together and I’m pretty stoked with how things worked out so far.
Darko: [00:09:02] This is a much better story that I expected to hear. I imagined that there were some life changes because there is no school for developer relationships and so on.
Eddie: [00:09:27] The missing piece of the puzzle for me was this. Right after I left doing math education I went to a community college and I was doing my associates in computer science, had to repeat a lot of courses. The ironic thing about community college is there’s a giant lack of community: everyone shows up, they hate each other and they go home. When I finished my associates I transferred to a four-year school to finish my bachelor’s and that was my missing piece – the community at Rutgers where I went. The Rutgers hackers community is very well known for getting together. We had a computer lab called The Cave where we would hang out and build cool things. That is like what really sealed the deal – having a supportive community that taught me about the broader world and then being able to give back. That that is my school for DevRel.
Darko: [00:10:24] It’s interesting though this story. From math to development and teaching together. In that DevRel world, there are more mediums than ever from Twitter, blogging, live streaming that you’re doing right now is huge but podcasting also. From your experience and maybe even biases where would you dedicate more of your time and energy and what is more successful in these terms. Going somewhere live, spending time with people, engaging with them online?
Eddie: [00:11:16] People describe DevRel different ways. One of the things that’s always stuck out to me is these three pillars that we call the three C’s: code, content, and community. The code is working with sample apps, GitHub projects, Open Source projects, quickstarts that kind of stuff. Content is a blog post online, preparing in-person talks and workshops. The community is going to places and actually evangelizing – building relationships with people, maybe working with students at a code school that kind of different stuff. Those are the three things that most people focus on. Generally, people gravitate towards two of those. For myself, I generally like doing code and community. I like speaking, so I’ll give talks when I can find time there, but I just hate writing. I always hated writing even in school. I’d wait till the last minute to start that 10-page essay I had to do. But that’s when I got my best focus. I’m not a writer.
Darko: [00:12:21] I feel you completely on writing. We have a question here. It says… I don’t know if you can see it. Probably not.
Darko: [00:12:40] It says: “Is DevRel the holy grail of marketing of developers or is it about technical credibility and being the devil’s advocate inside the company?” You partially answered that but if you want to extend a bit on that…
Eddie: [00:12:56] I’m going to share a link real quick – I always reach for this. There is a difference between developer evangelism and developer advocacy and I’m not one for titles at all, but I always reach for this tool whenever that discussion comes up. You can click through some of the boxes there, it lists some different responsibilities and the real TLDR is that evangelism is generally a lot more in person – a lot more in person working with developers. Actually going where developers live and gather. Then advocacy generally is more about enabling developers. It’s about enabling them through quickstarts and online documentation and those kinds of things. Engagement versus enablement – that’s one of the big distinctions that I’ve always seen. When you’re approaching DevRel for a software product you need to think how would you best benefit. Both sides can contribute feedback back to the product team and the engineering teams. It’s actually one of the most important parts about DevRel is being out in the field working with folks who give you direct product feedback. People pay for that kind of research by having people go out and gather product feedback. Having people out there doubling as community and relationship builders while getting valuable product feedback is great. It’s a lot about surfacing that up to the rest of the org, helping to influence the product and really being a voice for developers inside the organization. That’s where a lot of this whole thing comes together. I guess the answer to the question is: it’s not a Holy Grail because I’ve definitely seen it done and implemented not the best. I’ve seen some pretty good teams and I still think that the Twilio team is definitely the best DevRel team out there right now. I learned a lot from them. You have to have the right attitude. Creating a DevRel team isn’t magically going to bring you more developers. It’s not going to turn your relationship around if you have a south one. It’s going to take a lot of work by a lot of people. I like to use PayPal’s example there. PayPal used to be a pretty despised brand amongst developers, they used to just not enjoy working with PayPal, they didn’t have the best documentation, they didn’t have the best reputation. So PayPal built a pretty big developer relations team and they started… Have you ever heard of the PayPal battle hacks?
Darko: [00:15:41] No. No. But I heard pain stories about using PayPal’s integration – I’ve heard the bad side of the story.
Eddie: [00:15:49] They’ve built a big DevRel team and they started putting on these battle hacks which I think they had 12 different hackathons throughout the world. They had some out in the UK, just everywhere right, and out in Tel Aviv. But they were just like the most glorious implementation of a hackathon ever. They had like ice sculptures and they had massages throughout the whole thing that you could get a massage when you’re tired. It’s not the grand gestures, it’s all the work that was put in to really turn the reputation around and when the PayPal DevRel team was at its peak people loved PayPal – they were excited to go to PayPal events because they were so glorious. That took a lot of work and a lot of time by folks to turn that reputation around. It’s not like I said it’s not a magic bullet, it’s not a holy grail. You have to actually like dedicate the time and resources and commit to doing it the right way.
Darko: [00:16:57] In terms of those investments – it also depends on the size of a company and the team. As you said for PayPal they’ve put together quite a big team. There is that tipping point – how much do you have to invest in something. If you don’t invest enough it’s almost tipped over and it’ll not be successful. You only touched upon a question that I wanted to ask. It’s about someone from the DevRel team actually bringing the information back to the product team. Can you give us an example when there’s some piece of information you brought back to the product team and they embraced it was a feedback loop that was closed – what would be an example of that?
Eddie: [00:17:51] I’ll give you two. When I was at SendGrid we had a monthly DevRel product meeting where we would gather and compile all the feedback and work with the person in charge of product to prioritize the community requests and kind of surface stuff like what the most important easy wins were for the community. A lot of changes came directly out of that. SendGrid at one point implemented API keys because that was something that people really wanted – before they had just had usernames and passwords. That was a very strongly requested feature from the community and eventually got implemented. At Twilio over the years the DevRel team built a very strong relationship with the product team. Whenever a product was ready to launch or open up or even give as early customers access to it, they would come to the DevRel team and be like “Hey here’s what we built. It’s very rough right now. Could you guys like take a day and just do like an internal hackathon on the product?” and that was the best thing you can imagine. The product team coming to you with something as soon as they can get someone’s hands on it, they want your hands on it and they are listening to you all day while you build and implement something cool with this thing. We did that for almost every product that came out of Twilio. The thing with that is that’s earned and you have to earn that respect and build that relationship. So that’s what I definitely aspire to do here still at DigitalOcean.
Darko: [00:19:26] Yeah those connections are powerful. In our example, we have a collaboration with the customer success team. In our case, most of the talking that we do is with existing customers. There’s also that kind of a magical moment when you’re coming from some of those meetings with customers and speaking with developers and bringing (in most cases) just the good news like telling that people love the feature that a specific person in the product team built – that feedback is pretty magical.
Eddie: [00:20:06] It’s hard right because you have a roadmap already laid out so how do you prioritize the stuff you want to build as a product team versus the stuff your customers want. You probably know that better than I do.
Darko: [00:20:18] Yeah. What is good for us that we have these quarterly plans. We end up putting those in a Gantt chart: this will take this long and we will do this and this. At the end of the quarter we are happy if we do 60-70% of that, but we’re even happier if some things that people requested that we haven’t even thought that were important were implemented along the way. Being flexible there is very important and being able to react quickly to what customers ask for, then they appreciate that a lot. Unfortunately, it’s one of the hardest things to do, not to blow up your timeline, but it’s a great feeling when you manage to do that.
Darko: [00:21:18] So moving on maybe closer to technology – the idea for this call and everything came at the beginning of the year. I was doing some hands-on demos – deploying something from Semaphore to Kubernetes clusters and I think I started with DigitalOcean actually and then I also deployed to Google Cloud and AWS. I have to give a shout out to you guys, because the experience of setting up Kubernetes cluster on DigitalOcean, the speed and also the amount of brain power that I had to put in was very small so that that was an amazing experience and then we connected somehow on Twitter (I don’t remember exactly how) It was a huge difference comparing AWS to DigitalOcean to get that Kubernetes cluster up and running. It was like night and day for me so I’m sure that AWS is going to improve that, especially the time I had to wait to get this cluster up and running. I wanted to ask about core principles, values and the culture in DigitalOcean. Probably I discovered the company from tutorials, as you already mentioned, but whenever I touched any of the products it was so easy to use. How is the culture shaped and what are the principles that you stand for and you end up coming out with those products that are so easy to use. It’s always like you’re three clicks away to get what you want and you’re never lost or confused. So if you can maybe share with us a bit about that.
Darko: [00:27:44] Coming back to the ease of use that you mentioned and your team that used Kubernetes for a while and made exactly what they wanted – when I use some product and when I have to spend a lot of time figuring things out, I feel that it’s not empowering me, it’s making me ask myself “Am I smart enough to do this?”. When you use a product where you feel in a completely different way – a few clicks and “Wow I have now this superpower that I can just handle deploying now” that’s a completely different story. When you touched upon Kubernetes – it’s very interesting how pieces fall into place. For us – I have to say Semaphore was never more stable. There are many components to it, why it happened – we’ve learned a lot over the years, we rewrote some things a couple of times and so on. Not having that Server 01 and Server 02 and knowing that I have five instances, but now I just don’t know how many instances we have in our Kubernetes cluster and it’s more stable than ever. It really shows that there are many many engineering hours and years and years of experience that Google built into that and shared with the world. It sounds great how you benefited from that, but other public cloud providers also because competing now with AWS in that area is a completely different game. Whenever I hear something coming from CNCF and the brands that I see there I see a kind of alliance on one side and AWS on the other side. We are all going to benefit from that.
Eddie: [00:29:55] We want to evolve that experience too, so I think that the worst part about Kubernetes for a lot of people is writing large YAML files, right? The elephant in the room is the YAML manifest and we’ve built a tool and abstraction layer on top of Kubernetes internally that we call DOCC and this has been talked about a few times out in the community. It basically boils down to very simple JSON files with very same defaults for the organization and it talks directly to the Kubernetes API server – it translates and makes all the requests it needs to. But you can basically take his tiny JSON manifest, put your image and all that stuff. You can opt-in to having Prometheus metrics right out of the box. You can opt-in to let’s encrypt or whatever we use for TLS, load balancing and it’s all done through this tiny little JSON manifest. There have always been rumblings of open sourcing that, but maybe one day we’ll actually build that into the product, some form of that where we can help provide same defaults so folks don’t have to use those manifests with JSON all the time. It could be more of like a “Choose Your Own Adventure” where you pull in the right pieces for the components.
Darko: [00:31:14] I’m sure that would make a lot of people very happy. Our experience with Kubernetes is that it’s super stable, we never used anything better, but it comes at the cost. We have years of building a distributed system so we know that the network will fail you many times and know all about retries and time outs and all the things that you have to worry about. When we came to Kubernetes world at some point we just had maybe a month, month and a half setback in our timeline because we had to figure out all those retries but it scales. So now it’s not a couple of services, but it’s maybe 20+ services that are talking together and there are so many connections. We just had to bring Istio in. Learn that the hard way from scratch to just get those retries in place and time outs and all that. If we weren’t in this distributed-system-building world for a couple of years I think that I would be very unhappy about all these things that we had to deal with. Just one tiny abstraction layer more which would help developers is is this still missing. If you guys can do something about it that would be great.
Eddie: [00:32:52] We always say Kubernetes solves a lot of problems but it also introduces a whole another set of problems right?
Darko: [00:33:00] Yeah luckily some people are a few steps in front of us, so we just import Istio and use it.
Eddie: [00:33:08] The power of open source.
Darko: [00:33:10] Yeah it really is. Okay, so before moving on we have another question “I agree. Setting up a Kubernetes cluster on DigitalOcean is very easy – the only thing I have issues with it expires after seven days”. Okay, I guess that one is for you.
Eddie: [00:33:36] I’m pretty sure for GA we have a token so you can issue actual security and API tokens inside Kubernetes, so we’re building first-class support for that. I don’t think that has to be refreshed every seven days. I’m honestly not too plugged into that feature. As a stopgap, you could pull down the kubeconfig from the API, so there’s an endpoint where you can actually download the config and that could be a step in your build. I know it’s not the best but right now a step in your build pipe could be to use our DOCTL command line tool or even just curl it with your DigitalOcean API key and pull down the most current kubeconfig from the endpoint, so that that should hopefully get you past that trouble until we can actually ship the proper token support.
Darko: [00:34:32] Being able to connect very quickly is also something. I sometimes have a laptop one iMac, other iMac and I just need to jump into some cluster from time to time, so just going in copying a single line pasting it terminal to connect is a great thing.
Darko: [00:34:55] You mentioned about the computing part that you focused on for years and always had fun discovering those names that you have like droplets and spaces and all other ones. I think that sometime in the beginning of this year there was a blog post about announcing a couple of areas that you guys are working on and I think one of those were databases. Can you maybe share a bit of what’s coming up in 2019?
Eddie: [00:35:31] We’re pretty open with our roadmap. I think every January our VP of product Shiv will write up a blog post about what our roadmap looks like for this year. We’re very transparent, we want community feedback there. Managed databases we shipped either February or March – it was very recently and that is our take on a database as a service and we heard from developers out in the community that they don’t want to manage a database, they don’t want to have to deal with updates and backups. We abstract all that away. Right now we give you a PostgreSQL cluster, the pricing is pretty straightforward and pretty cheap. You can add up to highly available nodes so you have full HA and then you can add read-only nodes in different regions that can all talk to each other. You can fork your database so it will give you a production copy of your data. Some folks use that for running migrations and you can do blue-green deployments on that if you want with just forking a database. We handle backups, we handle upgrades for you. It’s our first shot at it. We’re definitely looking for community feedback. On the roadmap there we have Redis and MySQL coming up and then we’ve talked about doing Apache Kafka. The question in the chat is “Do you have a plan to provide a new sequel database?” Redis is probably the closest we’ll get there along with Kafka if we make it that far. I think we wanted to do MongoDB but the licensing changes there made us reconsider and really look into how to handle all of that the right way. Those are the ones that we’ve said we’re coming out with so far.
Darko: [00:37:26] Managing database is probably something that’s very few people want to do, so I think it’s a huge thing when you don’t have to worry about it. It was always for us the part that we were most scared of. I remember early days when there was a single Postgres instance running on a single physical machine and you were kind of just waiting every week for something bad to happen that you will not know how to get out of that. Someone managing that for you is a huge thing definitely.
Eddie: [00:38:05] I have a quick story I’ll share. We’ve sponsored a Postgres conference in New York. Like two or three weeks ago you know kind of like supports Postgres community and share and market our product. It was kind of our breakthrough into the database community as a whole. My hypothesis was it wasn’t really going to be a fit audience-wise turned out to be pretty correct. The main reasoning is that if you have like a Venn diagram of people who would use managed Postgres and then people who would go to Postgres conf, that overlap is so tiny because people that are going to Postgres conf want to tune every knob. They want to get the most performance. They don’t want a managed product. They want to be able to optimize the heck out of their database. In terms of thinking about what conferences and things to support in DevRel, it was a great experiment for us. That’s something you have to take into account – what is the audience, who is going to be there, what do those folks look like.
Darko: [00:39:06] It’s more interesting when you need to expand your team – that’s the place to go. Yeah. OK so you answered the question but NoSQL – so Redis and potentially Kafka. Those were the questions that I wanted to ask. Yeah. Is there anything that I should have asked or that you maybe wanted to share.
Eddie: [00:39:36] No nothing particularly. We’re hiring some pretty cool roles right now. We do a lot of Go internally. We’re hiring someone to run our Hatch which is our startup program, so if anyone out there is interested in taking on the leadership of our startup program feel free to hit me up – I’m just at Eddie Zane on Twitter. We’re also hopefully opening a role very soon for our developer advocate focused on Hatch. So we’ll have someone on the DevRel team working with startups. Any of those things sound interesting to folks?
Darko: [00:40:13] Yeah. You know where to find Eddie So one additional question: “I built a Go service that updates config for now in the DigitalOcean API with Sem CLI”. Okay so here is a user which is using both Semaphore and DigitalOcean. Good to know that you are working on a solution.
Eddie: [00:40:36] Feel your pain – will pass the feedback along. Hopefully, we ship token support very soon.
Darko: [00:40:44] Well it was great talking to you. I think that I learned quite a bit about the journeys of DevRel people and what are some best practices and so on. Thank you very much for joining us and for the people watching: feel free to leave any questions in the comments below and both Eddie and me can answer those at any point. Thank you, Eddie, see you!
Eddie: [00:41:17] Thank you so much for having me. Man, this is great.
Interested in future chats streamed live? Subscribe to Semaphore YouTube channel!