NEW Run your Ruby tests in a few minutes and ship faster with Boosters, one-click auto parallelization · Learn more…
×

Semaphore Blog

News and updates from your friendly continuous integration and deployment service.

Introducing Easy Continuous Deployment to Amazon S3

Get future posts like this one in your inbox.

Follow us on

Amazon Simple Storage Service (S3) is a part of AWS that provides an API and a simple web interface to store and retrieve any type of data from anywhere on the web. An S3 bucket is conceptually similar to an API driven file folder.

One of the features of AWS S3 is that you can also use it to host static websites. The Semaphore blog, for example, is hosted on S3. Today, we’re happy to announce an integration which makes it easy to automatically build and deploy static websites as you push new versions to your Git repository. It works great for all other kinds of content too.

Continuous deployment from GitHub via Semaphore CI to Amazon S3

Similar to our continuous deployment integration with AWS Elastic Beanstalk, which we launched last week, our goal with S3 has also been to simplify the process as much as possible. Semaphore can detect your existing S3 buckets and also lets you create new ones, all without visiting the S3 web interface. There are also options to enter your custom site building commands and set the index page of your website. The brief screencast below shows this quick setup process.

You can tell Semaphore to automatically deploy your application from a selected branch whenever all tests pass, but you can also choose to do it manually yourself. At any time, your team will have a shared deployment history and uniform way to publish your website online.

If you don’t have any experience with static website engines, staticgen.com provides a good overview of the most popular options in various programming languages.

For more detailed help in getting started using our new feature, please see the related documentation page which explains the basics behind deploying to S3 and how to retrieve your AWS credentials.

We hope that the new Semaphore integration with Amazon S3 will help improve your continuous deployment workflow. Happy building!

Continuous Deployment with Phoenix 1.0 via Semaphore

By Josh Adams of ElixirSips

I have an application up at http://elixirfriends.com. It’s a Phoenix-based Elixir webapp. Since Phoenix hit 1.0 this week, I thought it would be a good time to outline how you can set up Continuous Integration and Deployment of a Phoenix-based application using Semaphore. Let’s get started.

Project Details

So this project lives at on GitHub at knewter/elixir_friends. It’s a pretty standard Phoenix application, and it uses a React-based frontend to render the views after talking to the API to get the data. One particularly fun piece of it is the ImageTweetStreamer, which gets a stream of tweets matching a given search term, filters out those that don’t have images, and stores them in the database if they haven’t already been stored. Using one of Phoenix’s more interesting features, Channels, we broadcast any newly-stored tweets to the frontend for real-time addition to the page. We then run this as a supervised Task when our application starts.

Alright, so we have our application and you have a good feel for how it works now. Where does Semaphore come into play?

Continuous Integration

Well, I’ve been writing software for long enough that I’m now of the opinion that if you don’t have tests you haven’t really written anything. Consequently, I have a few tests to verify that my API works as expected. Here’s the controller test, which very basically confirms that my API returns posts in our system.

But what good are tests if you aren’t running them? I run my tests as I’m building my software, but I also know that if I don’t automate things I’m not likely to be as rigorous as I’d like. To avoid merging Pull Requests that break the tests, I’d like to automatically run the tests on any Pull Request. In the past I’ve used Jenkins for this sort of thing, but honestly running my own infrastructure for CI is a bit time-consuming, and I’m especially unlikely to set it up for open source projects. Semaphore provides a very simple means to set up CI for anything ranging from small open source projects to large private projects.

Setting up Semaphore for Continuous Integration

Adding a Project

Once you’ve logged into semaphore, there’s a link in your dashboard to add a new project.

Add a new project

Selecting Your Repository Host

Semaphore provides nice integration with various potential repository hosts. Our app is on GitHub, so we’ll pick that:

Select Repository Host

Choose Repository

Semaphore now connects to GitHub via OAuth to list your repositories, and you can just click on the repository you want to set up CI for:

Choose Repository

Select Branch

Your branches will be listed, and you can choose the branch you care about:

Select Branch

Select Account

Next, you choose your Semaphore Account. I only have one:

Select Account

Analysis

This is one of the coolest features of Semaphore. It will analyze the project to determine which kind of project it is.

Analysis

Review the Build Plan

Since Semaphore knows it’s an Elixir project, it has a very good default plan in place. You’re provided with a screen to make any changes to the build in case your project requires additional setup.

Review the Build Plan

First Build

Once you’ve OKed the Build Plan, your first build is kicked off immediately:

First Build

Our First Build Fails

Our first build fails…you can see in the browser why it failed:

Build Fails Due to DB Settings

Set Up Database Configuration

The problem was due to our database connection failing. Semaphore provides you with a username and password to interact with your database in two environment variables:

  • DATABASE_POSTGRESQL_USERNAME
  • DATABASE_POSTGRESQL_PASSWORD

We modify our test environment for the Phoenix application to use these environment variables if they’re present:

Setup DB in Phoenix

Successful Build

When we push that fix up, Semaphore immediately kicks off the new build. This time it passes, because we’re able to talk to the database successfully:

Build Success

Setting up Semaphore for Continuous Deployment

So now we’ll know when our builds fail, which is very good. As soon as I’ve got Continuous Integration set up on a project, though, I prefer to immediately add Continuous Deployment. I find that automating my deployment from the master branch is a great way to avoid getting into the situation where all of my deployments require manual intervention. If you always have to support automated deployment, then you’re going to write better software and be more careful about what makes it into the master branch. Semaphore provides extremely easy-to-use support for adding Continuous Deployment.

First off, I set up my app so that I could deploy it with Heroku. For most of my applications I don’t use Heroku, but I definitely prefer to deploy open source and community-focused apps to Heroku because it makes collaboration a lot easier since I don’t have to manually manage permissions to my servers. Once I set that up, I set out to use Semaphore for Continuous Deployment for the first time.

Set Up Deployment

If you visit the projects page in your dashboard, there’s a button to that says “Set Up Deployment.” Seems like what we want, so let’s click that.

Set Up Deployment

Choose Deployment Type

Next, we’re presented with a choice of deployment types that are supported out of the box. We have a Heroku application, so we’ll choose that.

Choose Deployment Type

Select Strategy

Now we’re prompted to choose our strategy. We can pick either Automatic or Manual, but our whole point is to have Continuous Deployment so we choose Automatic.

Select Strategy

Choose Branch

Now we can pick the branch we want to deploy on success. We pick master, which happens to be our only branch at the moment anyway.

Choose Branch

Paste API Key

Since we’re deploying to Heroku, we’re not prompted for our Heroku API Key. There’s a handy link that takes you right where you need to go to get it from your Heroku Dashboard.

Paste API Key

Pick Heroku Application

We’re now presented with a list of our applications on Heroku. We’ll pick elixir-friends.

Pick Heroku Application

Name Your Server

The last step allows you to name your server.

Name Your Server

Awaiting First Deploy

Now that our deploy is set up, we’re prompted with a button that will allow us to deploy immediately. We’re also told that the next successful build of master will trigger this deployment.

Awaiting First Deploy

Deploying After Build Passes

Here we can see our next commit triggering a deployment after the CI build succeeded:

Deploying After Build Passes

Deploy In Progress

We can click the deployment that got started and see that it’s in progress. Here it’s actually queued up to start:

Deploy In Progress

Deploy Success

Finally, we can see that our automated deployment to Heroku was successful and our latest commit is now live on the site!

Deploy Success

Summary

So with that we went from an application that we had manually deployed to Heroku, to a rock solid CI + CD setup that we can use from now on. Semaphore made it very simple to set up, so someone without any experience with Continuous Integration or Continuous Deployment can just click through very straightforward wizards and get the same setup that I historically had to run fairly heavy infrastructure to support. On one hand, that seems so unfair! On the other hand, it means that we can avoid running this sort of infrastructure on our own for a lot of projects going forward. I hope you enjoyed it!

Bio

I’m Josh Adams, CTO of Isotope11 and the voice and keyboard behind ElixirSips. ElixirSips is a screencast series that can take you from Elixir newbie to experienced practitioner, as it walks through my entire experience with Elixir from my first bit of code through to building production applications. It contains over 22 hours of videos covering a wide variety of projects. If you’re interested in learning Elixir, give it a shot - I think it’s great! :)

Introducing Easy Continuous Deployment to AWS Elastic Beanstalk

Amazon Web Services (AWS) is a global leader in cloud computing services. It comprises dozens of services and the default approach, of course, is to learn how to optimally use, provision and monitor each one that best fits your application’s needs.

Continuous deployment from GitHub via Semaphore CI to AWS Elastic Beanstalk

But what if you would rather use AWS as a platform and not worry about what’s happenning in the lower layers, such as EC2 or Elastic Load Balancing? In that case something like git push aws would be great.

That’s what AWS Elastic Beanstalk is for. With Elastic Beanstalk, you can quickly deploy and manage applications in the AWS cloud without worrying about the infrastructure that runs those applications.

Without much knowledge about the AWS and Elastic Beanstalk platform, the road from working source code to a continuously deployed application that’s running in production can still be a bit bumpy. So today we’re happy to announce an integration that lets you easily set up continuous deployment from Semaphore to AWS Elastic Beanstalk. Check out the brief screencast that shows the quick setup process below.

Our goal has been to simplify the process as much as possible. For this reason, Semaphore can detect your currently available Elastic Beanstalk applications and guide you through the rest of the process with little input required.

You can tell Semaphore to automatically deploy your application from a selected branch whenever all tests pass, but you can also choose to do it manually yourself. At any time, your team will have a shared deployment history and an easy and uniform way to deliver new versions of your applications.

To help you get started from scratch, we have also published a detailed tutorial. It uses a Ruby application as an example but the same process applies to any programming language:

We hope that this integration will help you in your continuous integration workflow. Happy building!

Platform Update on October 20th

The upcoming platform update is scheduled for October 20th, 2015.

Cassandra is now on version 2.2.2.

Git has been updated to version 2.6.1.

Go gets an update with version 1.4.3, which includes security fixes for the net/http package and other bugfixes.

io.js gets an update with version 3.3.1.

MySQL has been updated to version 5.6.27, including a lot of InnoDB and Replication bugfixes.

NodeJS gets one version update with 4.1.2. To use this version in the release candidate platform, add nvm use 4.1.2 to your build commands.

PHP receives two updates with versions 5.5.30 and 5.6.14.

RethinkDB gets an update with version 2.1.5.

Trying the new platform

You can evaluate the new platform right away by selecting Ubuntu 14.04 LTS v1510 (release candidate) from the Platform section of Project Settings. We encourage you to try it out, so that we can fix the issues you may encounter until the final release on the next Tuesday.

Changes in the final release

Cassandra is now on version 2.2.3.

Git has been updated to version 2.6.2.

NodeJS 4.2.1 LTS has been added to the platform.

Redis gets and update with version 3.0.5.

Qt WebKit gets an update with version 5.4.2, including important security fixes. This version maintains backwards compatibility with Qt 5.4.1. Keep in mind that switching between Qt versions requires expiring your project’s cache.

A full list of changes is available in the platform changelog.

Build Commands Editor 2.0

Here at Rendered Text, we’re always thinking about ease of use. And with continuous integration, that ease — or pain — shows itself at the very beginning: the moment you start setting up your project. While Semaphore tries to infer or recommend you the right build commands, many projects need to customize their build commands sooner or later.

The build commands editor has remained mostly unchanged since we’ve launched parallel CI execution back in 2012, which let you speed up your test suite by splitting it in parallel build threads.

The editor definitely has some limitations. For example, adding a new command to a project that is utilizing multiple parallel threads, aside from keyboard input, has required about five mouse clicks.

We’ve been thinking deeply about how we can make this process faster and easier. So today we’re happy to announce availability of a completely new version of CI build commands editor on Semaphore.

Watch a quick screencast below for a taste.

As you can see, we’ve unbundled what was previously one list of commands into distinct sections:

  • Any commands that you would want to run in each parallel thread go into the (optional) Setup thread.
  • You can add a new parallel thread by actually clicking to add a new parallel thread, instead of adding a command and “scheduling” it into a desired thread number.
  • Same goes for commands you would like to run after each thread — the Post-thread commands, as we call them.

Threads are clearly separated, and you can even give them a name which would later be used on the build page. As a reminder, parallel threads on Semaphore run in different and completely isolated environments. Of course, if your project doesn’t require more than a single thread, none of this is required and the screen remains simple.

There are more handy details that are enabled by this new design:

  • You can drag and drop commands between threads.
  • Want to quickly edit a single command? Just click on it, edit and press Return.
  • If you are editing all thread commands at once, press Command-Return if you are on a Mac to save them quickly.

We hope that you’ll enjoy using the new build commands editor the next time your add or customize a continuous integration project on Semaphore. If you have any feedback or notice any bugs, please let us know.

Happy building!

Platform update on September 22nd

The upcoming platform update is scheduled for September 22nd, 2015.

Cassandra has been updated to version 2.2.1.

Chromedriver gets an update with version 2.19, which supports Chrome versions v43-47.

Git has been updated to version 2.5.2.

PHP receives several updates with versions 5.4.45, 5.5.29, and 5.6.13.

Redis is now on version 3.0.4.

RethinkDB gets an update with version 2.1.3.

New things

Go 1.5 is now available in the platform. It can be selected by adding change-go-version <version> to your build commands. Valid versions are 1.4 and 1.5. After the release candidate period of the platform, the new Go version will be selectable in Project Settings.

Node.js 4.0.0 is added to the platform, along with io.js 2.5.0 and io.js 3.3.0. These versions can be activated by adding nvm use <version> to your build commands. After the final release of this platform update, the selection will be available in Project Settings too. Before switching to Node.js 4.0.0, please see the list of packages which currently don’t work in combination with this version of Node.js.

Trying the new platform

You can evaluate the new platform right away by selecting Ubuntu 14.04 LTS v1509 (release candidate) in the Platform section of Project Settings. We encourage you to try it out, so that we can fix the issues you may encounter until the final release on the next Tuesday.

A full list of changes is available in the platform changelog.

Fine-Grained Control Over Chat Notifications

We are pleased to announce a few highly requested improvements to our chat and email notification system.

Receive notifications only for failed builds and deploys

You can now choose to be notified only after failed builds and deploys, reducing noise in your channels, rooms, and inboxes when continuous delivery is running smoothly. This change applies to both chat and email notifications.

Limit which branches and servers trigger notifications

As your team grows, so does the number of feature branches being worked on, and deploying to staging servers happens more frequently. Receiving notifications about every activity in one central place like Slack usually ends up being too noisy.

Chat notifications can now be filtered by branches (for builds) and servers (for deploys), which means more focused feedback for your team.

For branches, you can use regular expressions to define a whitelist.

For servers, simply tick off those that you would not like to hear from.

Increasing signal, reducing noise

We hope you will find these updates useful. You can access them in your chat notifications settings under “Project settings”.

Just as a reminder, Semaphore supports chat notifications for Campfire, Flowdock, HipChat, and Slack. Be sure to let us know if you have any feedback or suggestions.

Happy building!

Introducing Python Continuous Integration on Semaphore

This week, we’re excited to announce that Semaphore now officially supports continuous integration and deployment of your Python applications.

Python Continuous Integration & Deplyoment

Python: Always look on the bright side of your code

Python, famously named after the BBC comedy series “Monty Python’s Flying Circus”, is a powerful high-level open source programming language supporting multiple programming paradigms including object-oriented, imperative and functional programming. Due to its clean, simple and readable syntax, it is one of the most popular programming languages, ranked as the fifth most popular language in the TIOBE Programming Community Index.

It has a comprehensive standard library which describes all the components which are shipped with the language. This reflects the design philisophy which Python is built on. A small core which is easily extensible. PyPI(Python Package Index) hosts thousands of packages (called wheels), and this number is growing rapidly. PyPI packages make it easy to add a wide variety of functionalities to applications without reinventing the wheel.

True to its name’s origin, Python aims to make the language fun to use on all levels and to maximize user-friendliness.

Setting Up Continuous Integration for a Python project

First, sign up for a free Semaphore account to add a Python project to Semaphore. Semaphore supports both GitHub and BitBucket.

Setting up CI for a Python project on Semaphore

Choosing your project and branch will launch a short analysis in which Semaphore determines the configuration of your project. These suggested build commands and language settings can be changed at any time in Project Settings.

Automatic configuration of a Python repository for continuous integration on Semaphore

Semaphore supports a number of Python versions which are living in their own virtual environment. Every version has a set of packages pre-installed. These include pip, mock, pytest, and nose. Packages installed with pip are cached and reused between builds, drastically improving build times.

Completing the setup will launch the first build of your project.

First continuous integration build of a Python project on Semaphore

To set up continuous deployment for your Python project, you can follow our tutorial that covers continuous deployment of a Django application to Heroku.

We can’t wait to see the things you make with Python and Semaphore. Happy building!

Ruby 2.1.7 Available In a Minor Platform Update

We just released a platform update - v1508.1.

Chromedriver gets an update with version 2.18 which supports Chrome versions v43-46.

Our tool for switching between Java versions change-java-version, now correctly sets the JAVA_HOME environment variable.

New things

Ruby 2.1.7 is now part of the platform which brings a security fix for CVE-2015-3900 and many bug fixes.

A full list of changes is available in the platform changelog.

Platform update on August 25th

The upcoming platform update is scheduled for August 25th, 2015.

Cassandra is now on version 2.2.0. The cassandra-cli tool is deprecated with this release and cqlsh is used instead.

Chromedriver gets an update with version 2.16 which supports Chrome versions v42-45.

Git has been updated to version 2.5.0.

MongoDB is now on version 2.6.11.

MySQL has been updated to version 5.6.26.

Oracle JDK 8 gets and update with version 8u51.

PHP receives several updates with versions 5.4.44, 5.5.28, and 5.6.12.

Qt 5.2.1 gets and update with version 5.4.1.

Redis is now on version 3.0.3.

RethinkDB gets an update with version 2.1.1 which introduces major improvements and support for automatic failover using Raft.

Trying the new platform

You can evaluate the new platform right away by selecting Ubuntu 14.04 LTS v1508 (release candidate) from the Platform section of Project Settings. We encourage you to try it out, so that we can fix the issues you may encounter until the final release on the next Tuesday.

Changes in the final release

Oracle JDK 8 is updated to version 8u60.

Ruby 2.2.3 has been added to the platform.

A full list of changes is available in the platform changelog.

Get future posts like this one in your inbox.

Follow us on