Semaphore Blog: Community posts Community articles from the Semaphore Blog: news, analysis and developer interviews https://semaphoreci.com/blog/tags/community.html 2019-02-14T00:00:00+00:00 Semaphore A Reality Check About Cloud Native DevOps https://semaphoreci.com/blog/2019/02/14/a-reality-check-about-cloud-native-devops-john-arundel-interview.html 2019-02-14T00:00:00+00:00 2019-02-14T00:00:00+00:00 Wojtek Cichoń <p class="figure"> <img alt="Interview with John Arundel, consultant and author of 'Cloud Native DevOps with Kubernetes'" src="https://semaphoreci.com/blog/assets/images/2019-02-14/semaphore-interview-john-arundel-5b3f0102.png" /> </p> <p><a href="https://twitter.com/bitfield">John Arundel</a> is a consultant and author of &quot;<a href="https://learning.oreilly.com/library/view/cloud-native-devops/9781492040750/">Cloud Native DevOps with Kubernetes</a>.&quot; We spoke with John on how adopting DevOps destroys walls, builds bridges, and influences company cultures for the better.</p> <p> <p class="figure"> <img alt="Interview with John Arundel, consultant and author of 'Cloud Native DevOps with Kubernetes'" src="https://semaphoreci.com/blog/assets/images/2019-02-14/semaphore-interview-john-arundel-5b3f0102.png" /> </p> <p><a href="https://twitter.com/bitfield">John Arundel</a> is a consultant and author of &quot;<a href="https://learning.oreilly.com/library/view/cloud-native-devops/9781492040750/">Cloud Native DevOps with Kubernetes</a>.&quot; We spoke with John on how adopting DevOps destroys walls, builds bridges, and influences company cultures for the better.</p> <p></p> <p><strong>You&#39;ve just co-authored (with Justin Domingus) a book entitled &quot;Cloud Native DevOps with Kubernetes: Building, Deploying, and Scaling Modern Applications in the Cloud.&quot; From the description of the book, it looks like it can take you from zero to hero in building and scaling apps the Kubernetes way. What do you think makes this book stand out from other books on the topic available on the market?</strong></p> <p>It&#39;s the book that Justin and I wish we&#39;d had when we started trying to deploy real apps on Kubernetes! The first and most important introductory book on Kubernetes that anybody should read is &quot;<a href="http://shop.oreilly.com/product/0636920043874.do">Kubernetes Up And Running</a>,&quot; by Kelsey Hightower, Brendan Burns and Joe Beda. But what should come after that?</p> <p>We wanted a book that answers the question &quot;OK, I have Kubernetes; now what do I do with it?&quot; We found lots of great tutorials and introductions and blog posts explaining how to set up a simple Kubernetes cluster. But it seemed like as soon as we started asking hard questions, the blog posts dried up.</p> <p>We were trying to deploy real apps for clients; complicated, messy things which don&#39;t look like a conference slide, and we had a ton of questions like:</p> <ul> <li>Should we build our own cluster or use a managed Kubernetes service?</li> <li>How big should the cluster be?</li> <li>How should we deploy app updates to the cluster?</li> <li>How do we configure replicas, shared volumes, load balancers, databases, IP addresses, DNS records, and all those fun things, and update them along with the app using infrastructure as code?</li> <li>How do we set up the cluster and the app to be secure?</li> <li>How do we integrate all this with our CI systems?</li> <li>Which of the zillion Kubernetes add-on tools are actually helpful? How should we use them?</li> </ul> <p>You get the idea. It&#39;s funny that when you try and do something real with a technology, you realize how little you actually know about it. “Cloud Native DevOps with Kubernetes” is not a book about Kubernetes, in fact; it&#39;s a book about doing DevOps, in a cloud-native context, where the de facto standard platform is Kubernetes. A few years from now that might not be the case, but we can envisage a future edition of the book which teaches the same principles and practices, only using some other software instead of Kubernetes.</p> <p><strong>Looking at the table of contents one can notice that you&#39;re going through a lot of Kubernetes-related topics that range from practical knowledge through monitoring, continuous delivery all the way to monitoring and metrics.</strong></p> <p><strong>How would you define the ideal reader of the book? A cloud native beginner? A senior developer that wants to learn about Kubernetes? A CTO who wants to get a bit more than the general overview of Kubernetes and its purpose? Or is it targeted at people with hands-on experience that will put the theory into practice?</strong></p> <p>We discussed this a lot in the planning stages. It&#39;s important to know your audience, or you may finish writing the book and find no one wants it.</p> <p>We&#39;ve aimed the book at three main groups of people:</p> <ul> <li><p>DevOps and infrastructure engineers who are responsible for operating applications in production, at scale, and who want to know where Kubernetes fits into this, what it can do for them and how to get the best out of it;</p></li> <li><p>Developers who don&#39;t have a dedicated ops team, but who need to deploy their apps to the cloud and run them reliably, scalably, without having to be an infrastructure genius. Kubernetes is great for this;</p></li> <li><p>CTOs, engineering managers and tech leads who want to know the business angles: how is Kubernetes going to benefit my company? How is it going to help us ship working software faster, to more customers, with fewer defects, with lower fixed costs and better flexibility? What arguments will help sell the idea of adopting Kubernetes to my management?</p></li> </ul> <p>We didn&#39;t want to leave out anyone who hasn&#39;t yet encountered containers, or even cloud infrastructure, so there are a couple of introductory chapters which explain the whole thing from scratch. We like practical books, so we included lots of hands-on tasks, like setting up a Kubernetes cluster on your laptop, building a containerized application in Go and deploying it to the cluster.</p> <p>Justin and I make a good team in this regard. While I like to explore concepts and principles, Justin is a very pragmatic, get-things-done kind of guy. So I think that creative tension has helped us avoid getting bogged down in abstruse technical detail, and keep the book focused on real, practical and everyday problems and solutions without hand-waving.</p> <p>We&#39;re both &quot;show me the code&quot; types, too; it&#39;s all very well saying airily that you can do X, Y and Z with Kubernetes, but how, exactly? Like, what do I type to make this work? To that end, we&#39;ve included usable code examples in just about every chapter, and they&#39;re all available in a GitHub repo that accompanies the book.</p> <p><strong>In <a href="https://www.activestate.com/blog/john-arundel-devops/">one of the interviews</a> you define DevOps very broadly: &quot;If you wake up in the morning feeling depressed about another day as an unvalued cog in a pointless machine, your company doesn’t do DevOps.&quot;</strong></p> <p><strong>This almost sounds as if DevOps is a solution to a lot of issues connected with company culture. Could you tell us a bit more about this approach and why aligning your company&#39;s culture with DevOps best practices can increase happiness inside the company and create business value?</strong></p> <p>Let&#39;s leave aside the term “DevOps” for a moment and think about specifics. DevOps is often associated with infrastructure as code. We have lots of great tools and practices in software engineering that we can use to collaborate on developing code, to learn and share knowledge across teams and to ship reliable, maintainable code. Now that more and more infrastructure is defined, provisioned and managed by code; what&#39;s the real difference between that code and the application software that the infrastructure is there to support? I don&#39;t see one.</p> <p>Another example: when some feature blows up at 3:00 AM, some poor ops person with a pager, who never saw this code before, has to get out of bed and triage it. What sense does that make? Isn&#39;t the person who wrote the code the most likely person to be able to fix it? Shouldn&#39;t we close the loop between writing features and understanding how they perform in production? I think we should.</p> <p>We talk a little in the book about the future of operations teams. Kubernetes can automate many of the processes that kept ops people busy in the past, but that doesn&#39;t put them out of a job. Far from it. Instead, they can now do more interesting work, rotating around development teams bringing their knowledge and experience to bear on high-level problems, teaching and coaching, designing and maintaining internal tools and platforms. These are some very smart and skilled people. If parts of their work can be automated, that means they&#39;re free to do things which bring more value to the business. They&#39;ll also be happier and more fulfilled, which is what every manager wants for her team, isn&#39;t it?</p> <p>If your company doesn&#39;t look like what I&#39;ve described, it&#39;s worth taking some time to think about why not. Talk to your people and find out what they want. Ask them what would need to change to make them excited about coming to work. If they sense that you&#39;re genuinely interested in fixing a broken culture, you may find you&#39;ve got all the help you need, right there in your own team.</p> <p><strong>You&#39;ve worked as a consultant for various brands ranging from medium-sized companies to enterprises. I know that every scenario is different, but what would you identify as a common obstacle that your clients meet in adopting DevOps as you define it?</strong></p> <p>First, any criticism of the status quo is implicitly a criticism of the people who created it. But it&#39;s possible for good people to do good things with good intentions, and still end up in a bad place. As the late, great Jerry Weinberg liked to say, &quot;Things are the way they are because they got that way.&quot; If you don&#39;t change the way you make decisions, you can&#39;t expect different results.</p> <p>Second, it would be nice to think that you could just wave a wand and everybody in the organization would instantly understand your priorities, share your values and radically transform things to deliver them. But that doesn&#39;t always happen.</p> <p>While it&#39;s always desirable to bring about change by explaining, motivating, inspiring and guiding people towards new attitudes, there will be people who don&#39;t agree with you and aren&#39;t willing to change.</p> <p>If all other options have been exhausted, you may have to gently explain to them: &quot;This is what the future of this organization looks like. If you don&#39;t see yourself as part of that future, it may be time to start looking for another one.&quot;</p> <p>In other words, if you can&#39;t change your people, you may eventually have to change your people.</p> <p>A third obstacle is what I call “consultant fatigue.” Failing companies often start trying different things more or less at random, flailing around and half-implementing half-understood ideas. Is it any wonder that people in that situation become cynical and pessimistic? They start to develop antibodies to management medicines. As one group of consultants parachutes in, changes everything, creates chaos, fails to deliver real results and is replaced by another group, the organization stumbles on from one missed earnings forecast to the next.</p> <p>Sometimes all it takes to put the light back in people&#39;s eyes is a gleam of hope. Hearing about companies where things do work can make a big difference. When I talk to my clients about teams I&#39;ve known that are happy, productive, and high-functioning, it can help dissolve years of encrusted cynicism. “It doesn&#39;t have to be this way” is a powerful sentence. It can open minds, and doors.</p> <p><strong>In &quot;<a href="http://bitfieldconsulting.com/bridges-not-walls">Build bridges not walls: DevOps is about empathy and collaboration</a>&quot; you write: &quot;Developers, you might think your job ends with a ‘git push’. But software that doesn’t work in the real world is a waste of bits. (...) The truth is there was never a neat line between dev and ops. The overlap is precisely where things get interesting.&quot;</strong></p> <p><strong>You&#39;ve been in the industry for some years now and how would you describe the current state of developers sharing responsibilities with their infrastructure colleagues? Does it work the other way round — with infrastructure teams doing pair programming with their dev teams?</strong></p> <p>The future is already here, as William Gibson noted; it&#39;s just not evenly distributed. I know, and have worked with, many teams that are really flying high. They have wide-ranging and complementary skill sets, covering all aspects of development and operations. They&#39;re constantly teaching and learning from each other, by pair programming, code reviews and other means. They collaborate in powerful ways, making sure everybody sees the big picture as well as the tiny piece they&#39;re working on, and making everybody feel involved in decisions. Instead of little code fiefdoms, stoutly defended by local warlords, and which no one else dare touch, there&#39;s a sense of collective ownership. When you feel valued, and you&#39;re thoroughly invested in the success of what you&#39;re all building together, there&#39;s no need for consultants anymore.</p> <p>It&#39;s at this point that I like to quietly slip away. There are always other teams who need my help.</p> <p><strong>With <a href="https://semaphoreci.com/blog/2018/12/20/kubernetes-tim-hockin-on-whats-real.html">Kubernetes being complex and giving birth to many &quot;follow-up-ecosystems&quot;</a> plus the entire cloud native technology landscape growing at an incredible speed is it even realistic to demand from development teams to keep up with the upcoming novelties in how the software they build can be deployed and maintained? It&#39;s all about dividing responsibilities between teams, but this again might lead to working in silos. Is it possible to close the loop between people who create software and deliver it?</strong></p> <p>Can everybody know everything about everything? No, indeed, and they shouldn&#39;t try. I have a general idea of how engines work, and I know enough to be able to drive my car to the store and back. But I couldn&#39;t build an engine, nor could I discourse insightfully on the relative merits of gasoline, hybrid, or even Wankel rotary engines.</p> <p>The same applies to Kubernetes. Not only do you not have to know how it all works under the hood, it would actually be a waste of your time to learn (unless you&#39;re a car mechanic). You should do only the things that only you can do. &quot;Undifferentiated heavy lifting&quot; — time-consuming work which doesn&#39;t relate to your core business or unique strengths — should be outsourced. We argue strongly in the book that unless you&#39;re in the business of running Kubernetes, you shouldn&#39;t be in the business of running Kubernetes.</p> <p>To overload that metaphor slightly, the Cloud Native DevOps” book is a road map, not an engine manual. It&#39;s about where Kubernetes can take you, not how to change the spark plugs. Yes, you&#39;ll need people on your teams who are experienced and skilful at operating Kubernetes clusters and getting the best out of them, in the same way that you need people who understand programming languages and compilers or databases and load balancers. But these are all just tools. They&#39;re not the thing; they&#39;re the thing that gets us to the thing.</p> <p><strong>I noticed that the chapter on choosing a Continuous Delivery tool is missing a solution called Semaphore. What can we do to be included in the next edition of the book?</strong></p> <p>As a matter of fact, we&#39;re big fans of <a href="https://semaphoreci.com">Semaphore</a>, but we faced a dilemma when writing this book. If we just listed all the tools available, that would be a ridiculously long and unhelpful book, not to mention instantly out of date. Similarly, if we just picked our favourite tools, we might justly be accused of leaving out some really important alternatives. And if we didn&#39;t mention specific tools at all, we would leave a lot of questions unanswered.</p> <p>What we&#39;ve tried to do, therefore, is sample. In each particular category (managed services, Kubernetes add-ons, deployment tools, CI systems, metrics servers and so on) we&#39;ve chosen to highlight a few representative examples, to give readers a flavour of what&#39;s out there, while making it clear that there are many other options. I apologize humbly to anyone whose favorite tool or product we had to leave out for reasons of space.</p> <p>It&#39;s worth saying that the GitHub repo, which contains our code samples, is available as a free download, whether or not you buy the book, and it&#39;s open source — we welcome issues and pull requests to add support for tools and options that we didn&#39;t have space to cover in the book. For example, you could add a CI configuration example for Semaphore, a demo application in Python or Terraform examples for Microsoft Azure. We&#39;ll then have a really comprehensive set of code samples which everybody can dip into and use for inspiration, or adapt to their own circumstances (<em>Editors note: challenge accepted, watch this space</em> :).</p> <p>Please check out the repo and let us know what you think. We&#39;d love for you to help us make it better!</p> <p><em><strong>Looking to deliver your next project with Kubernetes? <a href="http://eepurl.com/dLkTuw">Sign up for Semaphore&#39;s free ebook on CI/CD with Kubernetes</a>.</strong></em></p> <p><em>Article originally published on <a href="https://thenewstack.io/a-reality-check-about-cloud-native-devops/">The New Stack</a></em>.</p> Travis CI Alternative for Private Projects https://semaphoreci.com/blog/2019/01/29/travis-ci-alternative-for-private-projects.html 2019-01-29T00:00:00+00:00 2019-01-29T00:00:00+00:00 Marko Anastasov <p>Travis CI is one of the most popular hosted <a href="/blog/2017/03/02/what-is-proper-continuous-integration.html">Continuous Integration solutions</a>. Most notably, it has made a huge contribution to the developer community by serving the biggest share of open source projects.</p> <p>However in 2019 most <a href="/blog/2018/11/22/high-velocity-cicd.html">new software projects</a> are looking for a solution that can drive the entire <a href="/blog/2017/07/27/what-is-the-difference-between-continuous-integration-continuous-deployment-and-continuous-delivery.html">Continuous Delivery lifecycle</a>. There&#39;s also little evidence in the developer tools space of companies making customer-centric innovations after being acquired by a private equity fund. So after the last week&#39;s announcement of Travis CI&#39;s acquisition by Idera Inc., many developers have started exploring alternatives.</p> <p> <p>Travis CI is one of the most popular hosted <a href="/blog/2017/03/02/what-is-proper-continuous-integration.html">Continuous Integration solutions</a>. Most notably, it has made a huge contribution to the developer community by serving the biggest share of open source projects.</p> <p>However in 2019 most <a href="/blog/2018/11/22/high-velocity-cicd.html">new software projects</a> are looking for a solution that can drive the entire <a href="/blog/2017/07/27/what-is-the-difference-between-continuous-integration-continuous-deployment-and-continuous-delivery.html">Continuous Delivery lifecycle</a>. There&#39;s also little evidence in the developer tools space of companies making customer-centric innovations after being acquired by a private equity fund. So after the last week&#39;s announcement of Travis CI&#39;s acquisition by Idera Inc., many developers have started exploring alternatives.</p> <p></p> <p>Over the years the most common benefit that we&#39;ve heard from customers who migrated from Travis CI to <a href="https://semaphoreci.com">Semaphore</a> has been higher productivity. Developers work better when continuous integration is faster and more reliable: with consistent job runtime and no flaky builds.</p> <p>Here&#39;s a summary of the key Semaphore features that would be relevant to people looking for an alternative to Travis CI for their private projects:</p> <div class="overflow-x-scroll"><table class="f5"><tr> <th style="text-align: left">Semaphore Feature</th> <th style="text-align: left">Summary</th> </tr> <tr> <td style="text-align: left">Lowest tier</td> <td style="text-align: left">Free for up to 1,300 minutes</td> </tr> <tr> <td style="text-align: left">Pricing model</td> <td style="text-align: left">Pay-as-you-go (billed per second)</td> </tr> <tr> <td style="text-align: left">Concurrency</td> <td style="text-align: left">Autoscaling, no limit</td> </tr> <tr> <td style="text-align: left">Time to start a new job</td> <td style="text-align: left">1s</td> </tr> <tr> <td style="text-align: left">Time to SSH session</td> <td style="text-align: left">2s</td> </tr> <tr> <td style="text-align: left">Configuration</td> <td style="text-align: left">YML, CLI, Encrypted secrets</td> </tr> <tr> <td style="text-align: left">Machine types</td> <td style="text-align: left">3, can vary per job</td> </tr> <tr> <td style="text-align: left">Continuous Integration Pipelines</td> <td style="text-align: left">Single stage, Multi stage sequence, Parallel jobs, Fan-in, Fan-out, Branch-based workflows, Conditional continuation, Build matrix</td> </tr> <tr> <td style="text-align: left">Continuous Delivery Pipelines</td> <td style="text-align: left">Deploy to multiple environments, Automatic or manual deployment promotions</td> </tr> <tr> <td style="text-align: left">Environment</td> <td style="text-align: left">Ubuntu Linux-based, Docker natively available</td> </tr> </table></div><h2 class="f3 lh-title">Next steps</h2> <p>In addition to capacity provided by the free tier, all new Semaphore customers receive $200 of free credit, and enjoy support which has been rated as <a href="https://www.g2crowd.com/products/semaphore/reviews">#1 in the CI/CD space on G2 Crowd</a>.</p> <p><a href="https://semaphoreci.com">Sign up for a free Semaphore CI/CD account</a> and start using it on your projects today.</p> What’s Real and What’s Not: Interview with Kubernetes Pioneer Tim Hockin https://semaphoreci.com/blog/2018/12/20/kubernetes-tim-hockin-on-whats-real.html 2018-12-20T00:00:00+00:00 2018-12-20T00:00:00+00:00 Wojtek Cichoń <p class="figure"> <img alt="Interview with Tim Hockin on Kubernetes, Principal Software Engineer at Google" src="https://semaphoreci.com/blog/assets/images/2018-12-20/semaphore-interview-Tim-Hockin-0993641d.png" /> </p> <p>In the Developer Interview series, we talk to engineers and pick their brains about how they work, what wisdom they would like to pass on, and the most challenging problems they’ve faced during developing.</p> <p>This time we had a chat with Tim Hockin, Principal Software Engineer at Google.</p> <p> <p class="figure"> <img alt="Interview with Tim Hockin on Kubernetes, Principal Software Engineer at Google" src="https://semaphoreci.com/blog/assets/images/2018-12-20/semaphore-interview-Tim-Hockin-0993641d.png" /> </p> <p>In the Developer Interview series, we talk to engineers and pick their brains about how they work, what wisdom they would like to pass on, and the most challenging problems they’ve faced during developing.</p> <p>This time we had a chat with Tim Hockin, Principal Software Engineer at Google.</p> <p></p> <p><strong>You describe yourself as <em>&quot;a systems software engineer [who writes] software most people don&#39;t see&quot;</em> on <a href="http://www.hockin.org/%7Ethockin/">your homepage</a>. That&#39;s hard to believe when looking at Kubernetes success and the exposure it gets. What went wrong? How come so many people see &quot;your project&quot;? What’s the key to its success?</strong></p> <p>I think the key to its success is that it solves real problems that real people have. There&#39;s a lot of projects that try to change the way people work or try to change the way people think about things. Kubernetes came directly from a real experience at Google where we built Borg. We&#39;ve been using it for 14 years and it worked very well. I suddenly realized that &quot;one day I&#39;ll leave Google and when I do, what will I do? How will I exist without all these tools?&quot; </p> <p>The reason that Kubernetes is successful is because people look at it and they don’t understand why they need it until they see it do stuff. Then they say &quot;Oh my God, I need that!&quot;. I can&#39;t say how many talks and presentations I&#39;ve done in front of skeptical audiences where they don&#39;t understand what it&#39;s for. Just by showing short and simple features like “let&#39;s do a rolling update” I watch what happens. I stop it halfway through and then roll it back and they&#39;re like ”that&#39;s what I do by hand. It takes me a ton more energy than what you just did.” So I think it captures things that people really wanted.</p> <p>I would also add that in the bigger picture, Kubernetes is really not seen by a lot of people. It is not and will never be as visible as Android, or Chrome, or Google Maps. It is still very much &quot;software that most people don&#39;t see&quot;.</p> <p><strong>You mentioned Borg and I was wondering what’s the correlation between Borg and Kubernetes. I doubt that it’s 1 to 1.</strong></p> <p>It&#39;s derived from the same ideas and the same patterns, but Borg is millions of lines of code that are very Google-specific. It has thousands of features that nobody would use except Google.</p> <p><strong>There was a movement called &quot;Google infrastructure for everyone&quot; way before Kubernetes got so popular. Do you think that there are still some tools you use inside of Google infrastructure that can be open sourced and added to the stack of modern DevOps tools?</strong></p> <p>I hope so. We have made a ton of investment in tools increasing developer productivity, compiler tools, distributed builds and automatic testing. There&#39;s been some blog posts and some papers written about how Forge works which is our distributed build system. We have little silly things that you might not think about, like test logs collection. Every time I run a test through our test system the logs are captured and stored somewhere. Then I go back, look at all them and realize that they are giant. </p> <p>We&#39;ve got some really great code review tools that are integrated with static analysis. Googlers can add plugins that say &quot;I want to do an analysis of this sort of thing on my code base&quot;. You put these things in and when somebody sends you a code change it will run your analyzer against their change. Neat stuff. </p> <p>We&#39;ve got these live debugging tools which will capture traces across test runs. If a test fails you can actually go back and single step through the test and figure out what happened and why. All this happens automatically. It&#39;s not like you had to say &quot;all my tests fail. Let me go run it in the debugger.&quot; You just captured it automatically because the cost of saving the data is just so much less. </p> <p>So this is one area where I think Google infrastructure can continue to help influence the world. We&#39;ve got a lot of other things - internal storage systems and other things that are sort of trickling out a little. Bigtable and Spanner are out as Google Cloud products. These are fantastic tools and they have really transformed the way we do things internally. I think we&#39;re getting better at putting out more of what we do.</p> <p><strong>You mentioned that Kubernetes is actually helping people in what they do but it&#39;s no mystery that it has a steep learning curve. It&#39;s complicated, spans across a variety of topics in the software development and maintenance niche. In one of the interviews, you mentioned that the initial mission behind the project was to &quot;provide the hub of ecosystems, plural.&quot; K8s was initiated to minimize human effort in running and scaling but, in the meantime, the project itself adds layers of complexity to itself. Could you elaborate a bit on that? Is there an end to the new ecosystems it brings to life?</strong></p> <p>I hope that each of these &quot;follow-on-ecosystems&quot; is optional and that nobody should be forced to adopt if it doesn&#39;t actually bring any benefits. Istio is just a great example - Kubernetes is an application management platform and it does networking out of necessity because apps are networked. But it doesn&#39;t make sense for Kubernetes to try to be the ultimate network abstraction. It can&#39;t be the perfect service discovery because there is no such thing as perfect. There are a hundred service discovery applications out there. It can&#39;t be the perfect load balancer and it can&#39;t be the perfect traffic router, etc. </p> <p>These things are complicated services and we&#39;ve been trying to keep Kubernetes relatively simple at its core. Early on in the process we created this document &quot;What is Kubernetes.&quot; We defined there what we thought it was, what our swim lanes are, what we’re doing, and what we&#39;re not going to do. We’ve defined how big the Kubernetes box was and how everything else was outside of it.</p> <p>Networking is in the box but only a little bit. We need to keep just enough in order to keep the stuff working. Then you look at what people really need out of the box. For things like Layer 7 HTTP load balancer, go read the spec for Nginx or Envoy or HAProxy. They have hundreds and hundreds of features that are specific to their implementations and they&#39;re all different from each other. </p> <p>There&#39;s no way that Kubernetes could implement all of those things - we would be in a race forever. That’s not where we want to be and we don&#39;t want to be competing with these companies and those projects. We don’t want to provide an abstraction over these things because abstractions loose information, details, access to the underlying systems. We decided that our swim lane is the minimum that we can get away with that helps basic users do basic things. The only thing we’ve got in is the Ingress API. It&#39;s a really simple HTTP API, simple to the point of being not satisfying. Almost everybody who uses Ingress ends up doing something with their specific implementation of Ingress because they need the extra features. </p> <p>Here Istio comes along and says &quot;We’re going to tackle the problem of networking!” They are not a container management system. They work with container management systems. They work with Kubernetes. They are a networking app and like Kubernetes has 100 features for container management, Istio has 100 features for traffic management. You don&#39;t have to use Istio if you don&#39;t want to, if it doesn&#39;t bring you any value. But for a lot of people it does bring incremental value. You can start with a simple API and move into a more detailed, robust and advanced API. It brings complexity which comes with the value. Complexity needs to justify itself because complexity for its own sake is cancer. </p> <p>I&#39;m a little bit sad how complex Kubernetes is right now. In some sense Kubernetes is not for end users, it’s for people who set up clusters and clusters are for end users. At Google, we&#39;ve always separated the roles that were involved in cluster management into two very specific ones: the cluster operators and the application operators. </p> <p>The cluster operators know everything about Borg and they they know a little bit about each application that runs on Borg in a profile they understand, but not the details. They&#39;re able to keep the Borg clusters up and running and they have SLAs around that and so forth. The application teams (like Google search or Gmail) come in and they say &quot;I know how to use Borg but I don&#39;t know how Borg runs and I don&#39;t need to.&quot; They come in and they run their applications on top of Borg and that split gives people a really nice ability to focus. Kubernetes is really aimed at those cluster operators. </p> <p>The truth is that a lot of companies and organizations today are set up with those two roles in one. It&#39;s tough getting resources out there right, so they get this complexity. It&#39;s unfortunate that networking and security are hard, and they are a big source of complexity for Kubernetes. These are the hardest parts of setting up in Kubernetes because they&#39;re still the hardest parts of our industry in general.</p> <p><strong>As one of the initiators, you have a big picture of where Kubernetes is heading and what are the biggest issues ahead. Could you give us a &quot;Founding Father summary&quot; on the state of Kubernetes in 2018?</strong></p> <p>So 2018 being almost over I think we&#39;ve actually done a decent job at responding to some of the complaints about it being hard to set up certain things in Kubernetes. It has gone a long way towards making cluster setup easier (not easy but easier). There&#39;s still more that we can do there. </p> <p>I think Kubernetes is starting to get to a place where it&#39;s a little bit more stable. We&#39;re not seeing as many major features come in anymore. The development of this system has moved more and more to plugins and things that are outside of the core. You can see this even if you look at the graphs of number of commits against the GitHub kubernetes/kubernetes repo that&#39;s trending down. It’s a good thing because more and more projects are moving out of the core repo. Overall, that&#39;s good. </p> <p>Meanwhile the community has grown, Slack has grown, the number of engagements and customers have grown enormously. These are pretty healthy signals to me. </p> <p>We still have too many bugs and too many PRs open. The developer experience process is still being worked on. The code review process and the design process are still being iterated on. This year was really successful. Looking back at the beginning of a year, version 1.8 feels very primitive. We&#39;re closing out more and more of our wishlist and getting further down the list of important features.</p> <p><strong>Do you think that the ideal state (if there is such a thing in the software world) is that the project remains stable? i.e.no big changes occur, it just works and the only thing remained to do is to add plugins to it?</strong></p> <p>Yes, that would be wonderful. It&#39;s software so nothing is really timeless. We&#39;re constantly updating for things like security and there&#39;ll always be a trickle of new features and things coming in. But the rate of change of things (especially things that have user facing impacts like API changes or functionality changes) is definitely slowing down and they have to, for the sake of the project.</p> <p>More and more things have to be in the follow-up ecosystems. I don&#39;t want to do a ton of new HTTP oriented work. I want that to land in other systems like Istio and other service meshes like Conduit. They&#39;re more experts in networking than I am. We’re putting various extension mechanisms together which allow people to do more and more things that are not second class. </p> <p><strong>Kubernetes mission is to ease off developers and devops professionals’ pains in scaling and maintaining software - it&#39;s all about reliability and uptime. Continuous Integration and Deployment are also touching the same topics but on a level closer to source code and testing. Is there a common space between K8s and CI/CD that needs to be addressed in the future?</strong></p> <p>So this is one of those places where I don&#39;t think Kubernetes itself should go. I think CI/CD is the most important thing that people want to use a system like Kubernetes for, but I don&#39;t think that Kubernetes should be trying to be a CI/CD system. There are commercial offerings, there are Open Source offerings, there are cloud centric offerings and I think those experts should be focusing on how to do CI/CD. </p> <p>The interesting question is more what can Kubernetes do to make it easier to consume from CI/CD. Whether that’s more triggers, more push-based solutions and the GitOps workflow process - those sorts of questions are the interesting ones. If anybody submits a patch to add CI/CD to Kubernetes API, I will do my best to shut it down. I don&#39;t think that makes sense. But certainly I want Kubernetes be useful for that. The number one thing we hear from people is &quot;this looks really cool, can I run my build against it?&quot;. We can do a better job at making that easy and possible. I think we&#39;ve done a lot of evolution there. A good example is the ability to do Docker builds without Docker. This sort of thing is important.</p> <p><strong>In one of the interviews you’ve said that &quot;people don&#39;t wanna bet on a new horse&quot; and I guess this was the way with Kubernetes in its early days. A lot of effort has been put to persuade decision-makers and senior devs to use the project and make it one of the busiest communities on GitHub. What do you think really contributed to making people adopt K8s despite its complexity and singularity?</strong></p> <p>Kubernetes got the most attention from us delivering real demonstrations that showed people how to do the old things. We tried not to be academic. We tried not to be theoretical about what to do and think. For example, we were showing how to make a rolling update of an application and rolling it back once there&#39;s something bad with it during the update. This is something that DevOps people do every day and it’s not always automated. With Kubernetes you can make it easier. You can have the automation around a consistent API and consistent tooling, so that you can do this more often with more confidence. This is exactly the issue we&#39;ve heard about from a lot of people and that’s why they now talk about Kubernetes adoption. They say &quot;I went from one push a day to 20 pushes&quot; and I think those are the things that got the most attention. </p> <p>The first KubeCon was less than a thousand people and now we&#39;re looking at sixty five hundred. But it was important that we would always put in the demos. A lot of my presentations early on were just demos like &quot;Let me show you 10 things that I can do with Kubernetes&quot;. Those are the things that captured people&#39;s attention. I remember I did one at USENIX LISA for system administrators and after the talk people came up and were like ”This is my job - you just automated my job”. I feel a little bit bad about that, but at the same time I hope this frees them up to do other things and jobs. </p> <p>What Kubernetes does is self-evident. It has allowed people to take code and deploy it quickly and whether you&#39;re a CEO or a developer you see great value in this. </p> <p><strong>Kubernetes is now a bleeding edge technology keyword. It dominates the so-called &quot;DevOps industry&quot;. The project is in fact so cool that when I saw the <a href="https://kubernetes.io/blog/2018/10/04/introducing-the-non-code-contributors-guide/">Non-Code Contributor’s Guide</a> I was excited that I might be of some help (even before properly educating myself on the topic). I even saw an article that made some resonance in the community entitled <a href="https://blog.jessfraz.com/post/you-might-not-need-k8s/">“You might not need Kubernetes”</a>. Is the hype around the project any good or does it create too much noise and shifts the original focus of the project?</strong></p> <p>There&#39;s a lot of noise and anytime there&#39;s a successful thing and money&#39;s involved there&#39;s a lot of people who are trying to shift the message back to themselves, their product or how to make money on this thing. One of the things that we were instructed when we were reviewing talks for a KubeCon was to make sure that the talks are not vendor pitches. We don&#39;t want people to pitch their product. One of the main reasons that I end up rejecting proposals was that &quot;this is just a pitch, a company is showing me their product&quot; and that&#39;s just not what people want to see at a conference like this. I don&#39;t think the noise itself is inherently bad. The more awareness we have the better off the project will be. </p> <p>Obviously I care a lot about the success and the longevity of Kubernetes. I see that when people come in now, they come with higher expectations. As the bar gets higher and people start to accept that you can do certain things automatically, they’re looking for what&#39;s next. Now that Kubernetes is out there, there&#39;s also a half dozen other systems that can do things in the same way. They make different tradeoffs, of course. People come in with different levels of expectations but I don&#39;t think that diminishes the value of the community.</p> <p>The non-code contributors guide was there because we have a lot of people who are working at companies using Kubernetes or who are interested in the technology and say &quot;I&#39;m not really a developer but this is interesting and compelling stuff. How can I help you?&quot; There are hundred things that non-coders can do to contribute.</p> <p><strong>I&#39;m a careful reader and I noticed on your homepage that you&#39;re a Fine arts minor! Are you an after-hour artist and paint large-scale steering wheels?</strong></p> <p>I got my first computer when I graduated from high school. Before that I was a painter and all through high school I studied painting and sculpture. I went to university to become a painter and my parents tried to talk me out of it. My father was an engineer, my brother is a mechanical-electrical engineer and my mom kept telling me &quot;why don&#39;t you go be an engineer and you can paint on the side&quot;. I said “No, I want to be an artist and a teacher (I had a really influential art teacher in high school).” So I was majoring in Fine Arts and did a double major in Fine Arts and Education. Then I got a computer as a graduation gift. I started to play with it and realized how much fun it was. One of my friends took a C programming class, so I took it as well and it was easy. It just spoke to me. I understood it all. I realized that I actually really enjoy doing this and I wanted to do more of this. I took more classes and eventually changed majors from art to computer science. </p> <p>I don&#39;t paint much after work as it requires more time and space commitment which is hard to meet. Sculpture even more so. But I do draw a lot and I&#39;m known around our office now for doodling on sticky notes during meetings especially. It’s like my brain&#39;s screensaver. There’s even a running joke around that if Tim is drawing, that means he is paying attention. </p> <p>Doodling is not exactly the same as painting, but it&#39;s a creative outlet. The truth is that writing code satisfies the same urge that painting does — the need for creative expression. I don’t write as much code today as I wrote a couple years ago but when I do, I feel that same sense of pride from doing something creative on my own.</p> <p><br></p> <p>Big thanks to Tim for taking the time to answer our questions. Read our related articles below, and subscribe to our newsletter to <a href="http://eepurl.com/drCQf1">keep up to speed with our future posts, updates and interviews</a>.</p> <p><em>Looking for fast CI/CD that works well with Kubernetes? <a href="/">Try Semaphore for free</a>.</em></p> <p><em>Article <a href="https://thenewstack.io/kubernetes-pioneer-tim-hockin-on-whats-real-and-whats-not/">originally published on The New Stack</a>.</em></p> <h3 class="f4 lh-title">Next up in cloud-native development:</h3> <ul> <li><a href="/blog/2018/11/22/high-velocity-cicd.html">Why Cloud Native Success Depends on High-Velocity CI/CD</a></li> <li><a href="/blog/2018/11/06/semaphore-2-0-launched.html">Semaphore 2.0 launched with customizable CI/CD pipelines, autoscaling and more</a></li> <li><a href="/blog/2018/01/10/interview-traefik.html">Building Open Source Solutions for Microservices: Meet Emile Vauge of Traefik</a></li> </ul> Why Cloud Native Success Depends on High-Velocity CI/CD https://semaphoreci.com/blog/2018/11/22/high-velocity-cicd.html 2018-11-22T00:00:00+00:00 2018-11-22T00:00:00+00:00 Marko Anastasov <p><em>Article <a href="https://thenewstack.io/why-cloud-native-success-depends-on-high-velocity-ci-cd/">originally published on The New Stack</a>.</em></p> <p class="figure"> <img alt="High Velocity CI/CD" src="https://semaphoreci.com/blog/assets/images/2018-11-22/high-velocity-cicd-5b507d39.png" /> </p> <p>The goal of every tech leader is to deliver bug-free products to customers at high velocity. Today’s cloud-native technology can empower engineering teams to iterate, at scale, faster than ever. But teams that don’t also change how they deliver software will struggle to benefit from the agility and speed to deployment that cloud native can offer.</p> <p> <p><em>Article <a href="https://thenewstack.io/why-cloud-native-success-depends-on-high-velocity-ci-cd/">originally published on The New Stack</a>.</em></p> <p class="figure"> <img alt="High Velocity CI/CD" src="https://semaphoreci.com/blog/assets/images/2018-11-22/high-velocity-cicd-5b507d39.png" /> </p> <p>The goal of every tech leader is to deliver bug-free products to customers at high velocity. Today’s cloud-native technology can empower engineering teams to iterate, at scale, faster than ever. But teams that don’t also change how they deliver software will struggle to benefit from the agility and speed to deployment that cloud native can offer.</p> <p></p> <p>Hosted CI/CD solutions can do a very good job of solving the “integration” part, such as compiling code and with varying degrees of performance and usability. <a href="/">Semaphore</a>’s platform, for example, is based on bare metal hardware to provide the best possible performance. This makes a big difference in productivity in scenarios of parallel testing.</p> <p>But in a cloud-native approach, managing many microservices in parallel makes delivery more challenging than integration. We need a quick, standardized way of delivering new services to production. And we need to make sure that ops work doesn’t explode in complexity, as we no longer deploy and run a single application, but dozens or even hundreds of services.</p> <p>The first step is to embrace the programmability of the cloud — you describe your infrastructure as code and manage it in a version control system. Once you treat all infrastructure as you do application code, you can apply continuous delivery to address the challenges of operational complexity. But for that to happen, your CI/CD tool needs to be programmable, too.</p> <p>Traditional hosted CI/CD services offer deployment features optimized for monolithic Web application deployment, with limited flexibility. When teams move to containers and cloud-native computing, they tend to rely on a CI vendor as a “workhorse” — but often invest engineering time in operating their own instances of Jenkins, which is sometimes combined with Spinnaker, for CD. This is a forced move due to lack of a well-rounded CI/CD product. Teams then face the time-sinking task of maintaining and scaling internal infrastructure, while suffering from a poor user experience and stability issues. It’s a lot of work that brings no value to the company’s end customers.</p> <p>At Semaphore, we subscribe to the idea that while problems can be complex, solutions must be simple. Since its inception in 2012, we guided product development by two principles: speed and simplicity. About a year ago, we asked ourselves: what kind of a CI/CD product would these principles translate to today?</p> <p>The answers we found have led us to create <a href="/product/">a whole new CI/CD product — Semaphore 2.0</a>, which launched this month. The new Semaphore is based on a number of principles and patterns that aim to improve developer productivity for cloud native environments:</p> <p class="figure"> <img alt="CI/CD Autoscaling" src="https://semaphoreci.com/blog/assets/images/2018-11-22/semaphore-2-0-ci-cd-autoscaling-graph-4e29f55f.png" /> </p> <h3 class="f4 lh-title">1. Speed always matters.</h3> <p>Nobody wants their builds to run slower. If developers need to wait for more than about 10 minutes for CI results, they lose focus and work less effectively. A CI/CD product should provide tools to measure and speed up build time.</p> <p>Take the engineering team at Par8o, an enterprise healthcare company. They use a development workflow that is similar to GitHub flow, merging to master once a story has been verified in the QA environment.</p> <p>The problem was this was happening too rarely, as running all tests on development laptops was taking more than two hours. So in one workday, the entire engineering team is unable to merge more than three times per day.</p> <p>Once they started using Semaphore’s automatic test parallelization feature called <a href="/landers/boosters-rails">Boosters</a>, average build time dropped to just 13 minutes, improving team morale and productivity. Such challenges will continue to matter in the future.</p> <h3 class="f4 lh-title">2. Every developer should be able to define and run a custom CD pipeline in minutes.</h3> <p>Most teams developing apps with Docker need to use multi-stage pipelines. For example, in the CI phase we usually need to build container images once, “fan out” to multiple parallel jobs for testing, then “fan in” to collect test coverage results.</p> <p>Some teams use different strategies across stages. For example, teams at companies such as Toyota have tests that run against different environments, such as QA, staging and production. The code base is the same, but is combined with different configuration or parameters. These tests don’t need to run at the same time, but manually on demand with a system of record. Changing one step in a build should seamlessly reflect in all environments.</p> <p>Another common use case is to “gate” deployments on the success of another build. For example, we may have multiple applications that all use one library. In that case, we want those applications to not deploy if the tests for that library fail.</p> <h3 class="f4 lh-title">3. Infrastructure is moving from fixed to ephemeral resources.</h3> <p>When CI/CD resources are limited, developers are often blocked in busy times of the week. This is in contrast with the cloud-native “pay only what you use” model, in which the resources we use scale automatically to support the team’s actual needs.</p> <h3 class="f4 lh-title">4. A CI/CD feedback loop is more than a pull request status.</h3> <p>The tool of choice should be programmable and act as a handy extension of developers’ natural workflow.</p> <p>Bootstrapping a new project, including a working CI/CD pipeline, should ideally take minutes. As always automation helps move faster while removing the risk associated with human error.</p> <p>By describing our CI/CD pipelines in code, as we already do for our cloud infrastructure, we also increase visibility, enable quick onboarding and enable fast transfer of knowledge. If all functions of the CI/CD tool are exposed through a command-line interface, developers can interact and build upon it as easily as they work with code libraries.</p> <h3 class="f4 lh-title">5. CD throughput needs to be measured.</h3> <p>Instead of prescribing the same key numbers for every project, CI/CD tool should let users choose their own metrics and define their own dashboards. Some projects are having an issue with build time. On others it’s most critical to know how often we deploy to users.</p> <p>Most metrics, however, are no longer relevant on a per-repository level. When an application is composed of many microservices, what matters are aggregate metrics. So CI/CD dashboards can include summaries of the latest work across a family of services, insights about deploys to production or average build duration.</p> <p>Cloud-native computing was once reserved only for the biggest and most advanced engineering organizations like Netflix or Spotify. The rise of Docker, Kubernetes and related tools and services is making the model accessible in the mainstream. Likewise, doing Continuous Delivery at scale is in a transition from requiring a specialized in-house team to becoming a turn-key service. At Semaphore, we’re thrilled to be part of this journey, and remain committed to making CI/CD easy for everyone.</p> <p><em>Accelerate the way your team builds, tests and deploys projects with powerful and fully customizable CI/CD pipelines. <a href="/product">Give Semaphore 2.0 a spin today, and get $20 of free monthly credit</a>.</em></p> <p><strong>Read next:</strong></p> <ul> <li><a href="/product">Set up CI/CD that just works</a></li> <li><a href="/blog/2018/11/06/semaphore-2-0-launched.html">Semaphore 2.0 launched with customizable CI/CD pipelines, autoscaling and more</a></li> <li><a href="/blog/2018/09/13/semaphore-uncut-first-look-at-2-0.html">Semaphore Uncut: A First Look at Semaphore 2.0</a></li> <li><a href="/blog/2018/09/20/semaphore-uncut-auto-promotion-continuous-delivery-pipelines.html">What is Continuous Integration?</a></li> </ul> Semaphore Uncut: CI/CD Slack Notifications https://semaphoreci.com/blog/2018/11/21/semaphore-uncut-cicd-slack-notifications.html 2018-11-21T00:00:00+00:00 2018-11-21T00:00:00+00:00 Dunja Radulov <p class="figure"> <img alt="Semaphore Uncut: Slack Notifications" src="https://semaphoreci.com/blog/assets/images/2018-11-21/semaphore-uncut-slack-notifications-e10c39ff.jpg" /> </p> <p>Semaphore Uncut is our weekly live show where we talk about continuous integration, continuous delivery and <a href="/product">Semaphore 2.0, our new product with customizable CI/CD pipelines</a>. In our latest episode of Semaphore Uncut, we showed you how to set up Slack notifications in Semaphore 2.0. Read on to learn more and watch the video. </p> <p> <p class="figure"> <img alt="Semaphore Uncut: Slack Notifications" src="https://semaphoreci.com/blog/assets/images/2018-11-21/semaphore-uncut-slack-notifications-e10c39ff.jpg" /> </p> <p>Semaphore Uncut is our weekly live show where we talk about continuous integration, continuous delivery and <a href="/product">Semaphore 2.0, our new product with customizable CI/CD pipelines</a>. In our latest episode of Semaphore Uncut, we showed you how to set up Slack notifications in Semaphore 2.0. Read on to learn more and watch the video. </p> <p></p> <h3 class="f4 lh-title">How to set up Slack notifications in Semaphore 2.0</h3> <p>In the previous episode, <a href="/blog/2018/11/14/semaphore-uncut-semaphore-classic-vs-2-0.html">we talked about the differences between Semaphore Classic and Semaphore 2.0</a>. This time around, we showed you Slack Notifications in Semaphore 2.0. See how they work below:</p> <div itemprop="video" itemscope itemtype="http://schema.org/VideoObject"> <meta itemprop="duration" content="00:36:46" /> <meta itemprop="thumbnailUrl" content="https://i.ytimg.com/vi/dG7ilfyL4Mw/2.jpg?time=1438331231591" /> <meta itemprop="URL" content="https://www.youtube.com/embed/rF6IvDDGtUo" /> <meta itemprop="embedURL" content="https://www.youtube.com/embed/rF6IvDDGtUo" /> <meta itemprop="uploadDate" content="2018-11-14T14:19:00+01:00" /> <meta itemprop="height" content="315" /> <meta itemprop="width" content="560" /> <span itemprop="description"> <p><span itemprop="name">Semaphore Uncut: Slack Notifications</span></p> </span> <iframe width="560" height="315" src="https://www.youtube.com/embed/rF6IvDDGtUo" frameborder="0" allowfullscreen></iframe> </div> <h3 class="f4 lh-title">What&#39;s next</h3> <p>In today&#39;s episode of Uncut, we&#39;re <a href="https://www.youtube.com/watch?v=0WIwILAq4O0">setting up a CI/CD pipeline for a Rails application</a>.</p> <p>To give Semaphore 2.0 a try, <a href="https://docs.semaphoreci.com/article/88-migration-guide-for-semaphore-classic-users">read our migration guide</a>, or <a href="/product">jump straight in and get $20 of free monthly credit</a>. </p> <p>Happy building, and see you in the next episode of Semaphore Uncut!</p> <p><strong>Read/watch next:</strong></p> <ul> <li><a href="/blog/2018/11/06/semaphore-2-0-launched.html">Semaphore 2.0 launched with customizable CI/CD pipelines, autoscaling and more</a></li> <li><a href="/product">Set up CI/CD that just works</a></li> <li><a href="/blog/2018/09/13/semaphore-uncut-first-look-at-2-0.html">Semaphore Uncut: A First Look at Semaphore 2.0</a></li> <li><a href="/blog/2018/09/20/semaphore-uncut-auto-promotion-continuous-delivery-pipelines.html">Semaphore Uncut: Auto-promotion of Continuous Delivery Pipelines</a></li> </ul> Maker to Manager: Interview with Director of Engineering at Dribbble https://semaphoreci.com/blog/2018/11/15/dribbble-developer-interview.html 2018-11-15T00:00:00+00:00 2018-11-15T00:00:00+00:00 Wojtek Cichoń <p class="figure"> <img alt="Developer interview with Jeffrey Chupp, Director of Engineering at Dribbble" src="https://semaphoreci.com/blog/assets/images/2018-11-15/dribbble-developer-interview-jeffrey-chupp-d36aab8e.png" /> </p> <p>In the Developer Interview series, we talk to engineers who use <a href="/">Semaphore</a> and pick their brains about how they work, what wisdom they would like to pass on, and the most challenging problems they’ve faced during developing.</p> <p>This month, we had a chat with Jeffrey Chupp, the Director of Engineering at <a href="https://dribbble.com">Dribbble</a>.</p> <p> <p class="figure"> <img alt="Developer interview with Jeffrey Chupp, Director of Engineering at Dribbble" src="https://semaphoreci.com/blog/assets/images/2018-11-15/dribbble-developer-interview-jeffrey-chupp-d36aab8e.png" /> </p> <p>In the Developer Interview series, we talk to engineers who use <a href="/">Semaphore</a> and pick their brains about how they work, what wisdom they would like to pass on, and the most challenging problems they’ve faced during developing.</p> <p>This month, we had a chat with Jeffrey Chupp, the Director of Engineering at <a href="https://dribbble.com">Dribbble</a>.</p> <p></p> <p><strong>You joined Dribbble in 2015 as a part of the development team, and currently your job title is Director of Engineering. What was the most challenging part of this transition from a “hands-on experience” developer to a more business-related/strategic role?</strong></p> <p>I think my experience has been fairly typical of people moving from maker to manager roles: the hardest part for me has been getting comfortable with the change in how my value is determined. When I was purely a developer, my value was easy to measure in terms of my speed and the quality of the features I worked on. Now my value is measured in how effective my teammates are.</p> <p>That’s a lot fuzzier and harder to quantify – you have to take a much more zoomed-out view. And you aren’t great at being a manager at first because the skill set is so different from the developer skill set.</p> <p>Because of how difficult it is to quantify your value as a manager and because you aren’t great at it yet, it is tempting to fall back into the maker role for that dopamine hit that comes from the visible progress of shipping code.</p> <p>That temptation hasn’t gone away entirely but I’m making good progress rewiring my brain to get that same feeling by amplifying my teammates. It helps that I have great teammates.</p> <p><strong>I understand that this change has a lot to do with Dribbble scaling its engineering team. Dribbble is a 100% remote team, and one of your co-founders, Dan Cederholm, <a href="https://www.webdesignerdepot.com/2011/05/interview-with-dribbbles-founder-dan-cederholm/">admits in an Interview</a> that for <em>... a tiny, bootstrapped operation, the scaling has been a challenge.</em> Can you tell us more about what are the biggest challenges in terms of scaling a remote engineering team?</strong> </p> <p>You’re exactly right, I was promoted internally because of growth. We’re still growing (shameless plug: <a href="https://dribbble.com/careers">come work with us!</a>).</p> <p>I think remote jobs have a huge advantage in hiring: When location isn’t a factor, you can hire the best people from a much larger pool of candidates.</p> <p>Asynchronous communication has been a challenge as we’ve grown our remote team. When you’re 8 people, interrupts are naturally capped by the small team size. Once you grow to 40-ish people and stretch across time zones (from the UK to California) things start to get trickier. </p> <p>For example, when you’re co-located in an office, there are visual cues to help you see that someone is deeply focusing and shouldn’t be interrupted (e.g. Patrick has his headphones on and is typing furiously while squinting at his text editor). Working remotely, you have to be more deliberate about choosing the best method for communicating without breaking someone’s flow.</p> <p>To minimize interrupts, we try to have as few scheduled meetings as possible to optimize for heads-down time. Monday and Friday are 100% meeting-free. Engineering has stand-up on Tuesday through Thursday to ensure face-to-face time with other engineers and do some knowledge sharing. </p> <p>Our product teams have weekly sprint check-ins. Finally there’s a weekly company meeting. Beyond that we’ve landed on communicating heavily through our tools.</p> <p><strong>When it comes to scaling up a dev team, how much do you rely on management techniques, how much on company culture and how much on tooling?</strong></p> <p>This is a good question. I think our company culture drives all the other things you mention. Put simply, we try to hire excellent people then trust and empower them to do their best work.</p> <p>If our company Slack is constant @channel and @here notifications, that’s not helping people do their best work. If our tools aren’t reliable and async-friendly, that’s not helping people do their best work. You see where I’m going with this. I try to get people to talk about the frictions they’re experiencing in 1:1s and then I own figuring out what steps we can take to reduce those frictions.</p> <p>Our tools are mostly about facilitating async communication to get things done. Broadly speaking: Slack is where ideas are discussed before they become tasks in <a href="https://www.getflow.com/">Flow</a>. More discussion happens in Flow as-needed, and then a developer submits a pull-request on GitHub proposing a solution. More discussion happens in the PR. If the tests pass and the code is approved, things go to production.</p> <p>Sometimes that process takes minutes and sometimes it happens across days. Being spread across time zones means it is common for a developer to submit a PR as they leave for the day, and find it already reviewed when they start work again the next day. That’s async at its best.</p> <p><strong>Speaking of tooling - Dribbble has been using <a href="/">Semaphore</a> since 2013 - if you could point out a couple of specific Semaphore features that enable you to deliver your software and scale up without obstacles, what would those be?</strong></p> <p>Our Semaphore usage has been really boring – in a good way! We have a pretty vanilla Rails app and Semaphore <em>“just works.”</em> Everyone on the team feels comfortable using it, from hardcore backend developers to markup and style experts.</p> <p>There are a couple features that have helped us scale: we make good use of Semaphore’s caching between builds and have our test suite split across multiple parallel jobs to help keep it speedy.</p> <p>We’re curious to investigate Semaphore’s Docker support in the near future. We’ll keep ramping up our parallel jobs pool as we grow as well.</p> <p><strong>To what extent is a CI/CD tool such as Semaphore also a collaboration-empowering tool?</strong> </p> <p>I appreciate how reliable Semaphore has been. This means that we get to focus on making our users happy, and don’t have to spend time trying to divine why our CI isn’t working.</p> <p>With <a href="/docs/adding-github-bitbucket-project-to-semaphore.html">its GitHub integration</a>, Semaphore adds to the PR conversation. We have a thorough test suite, so a green build gives us a high degree of confidence that the code changes aren’t going to have unexpected side effects.</p> <p><strong>What is your current focus when it comes to advancing the product and its dev infrastructure and why?</strong></p> <p>We have a lot of cool features on the roadmap for our users. My current focus is doubling-down on reducing frictions for my teammates and enabling them to be more productive so they can deliver on those features. Some practical applications of this include being hyper-aware of fixing missing technical documentation, flaky tests, and clarifying confusing parts of our ever-expanding codebase.</p> <p>Things on my list to investigate in the short term are how we might be able to leverage continuous deployment, further speed up our test suite, and improve collaboration across product teams.</p> <p><strong>I took a peek at your side-project <a href="http://www.hallofstats.com/">“Hall of stats”</a>. I would love to see its equivalent when it comes to the NBA Hall of Fame. What was the main motivation behind it - was it just for fun, or were you on a mission to demonstrate that according to maths the National Baseball Hall of Fame should look differently?</strong> </p> <p>Haha. You’re not the first one to request an NBA version of the Hall of Stats. I love that side-project. My BFF Adam Darowski is the brains behind all the goodness there.</p> <p>Players get into the Hall of Fame via a voting process and (like all voting) the results are political and somewhat arbitrary. Adam felt it was worth exploring what a merit-based Hall might look like and he’s worked hard to refine a powerful (but simple) algorithm for determining qualification. You can read more about this on the <a href="http://www.hallofstats.com/about">Hall of Stats about page</a>. The project has been hugely successful and I’ve been happy to help Adam work on it.</p> <p>Fun fact: When I first saw a rendering of <a href="http://www.hallofstats.com/player/ruthba01">Babe Ruth’s stats chart</a> for the Hall of Stats, I was convinced that something was broken in the code. I told Adam and he assured me that the code worked fine, Ruth was just that good.</p> <p><br></p> <p>Big thanks to Jeffrey for taking the time to answer our questions. Read our related articles below, and subscribe to our newsletter to <a href="http://eepurl.com/drCQf1">keep up to speed with our future posts, updates and interviews</a>.</p> <p>Happy building! <br></p> <p><em>Want to let your team focus on the code and have <a href="/">the fastest CI/CD service</a> take care of CI/CD in the background? <a href="/product">Try Semaphore for free</a>.</em></p> <p><br></p> <h3 class="f4 lh-title">Related reads:</h3> <ul> <li><a href="/blog/2018/09/25/slimjs-interview.html">Performance and Usability as Top Priorities: Interview with Slim.js Creator</a></li> <li><a href="/blog/2017/07/27/what-is-the-difference-between-continuous-integration-continuous-deployment-and-continuous-delivery.html">What&#39;s the Difference Between Continuous Integration, Continuous Deployment and Continuous Delivery?</a></li> <li><a href="/blog/2017/03/02/what-is-proper-continuous-integration.html">What is Proper Continuous Integration?</a></li> </ul>