NEW Run your Ruby tests in a few minutes and ship faster with Boosters, one-click auto parallelization · Learn more…
×

Semaphore Blog

News and updates from your friendly continuous integration and deployment service.

Ruby version usage in commercial projects

Get future posts like this one in your inbox.

Follow us on

The chart below shows the distribution of Ruby versions in private projects tested on Semaphore.

Ruby versions used on Semaphore

Currently 80% of all projects on Semaphore are private, ie commercial.

The versions and implementations of Ruby not included are not part of the platform, which also means that no one has ever asked for them yet.

Platform update: Ruby 2.1 preview, locale, new Postgres, MongoDB, Firefox

The latest platform release adds Ruby 2.1.0-preview1 to the list of included rubies. If you’re curious about its new features such as refinements, required keyword arguments and improved method caching, now’s your chance to test them in your app.

The default locale is now en_US.UTF-8, instead of C.UTF-8 which was causing trouble to some users.

Postgres is now at version 9.3, letting you take full advantage of the JSON datatype (available from 9.2).

MongoDB went up to 2.4.7, which enables things like text search and returning symbols from the aggregation framework. See 2.4 release notes.

Firefox has also been upgraded, please run bundle update selenium-webdriver if you haven’t recently.

As usual, this is filed in our platform changelog.

Growing our team - get ready for new greatness

We have awesome news that we would like to share with you. Semaphore team will almost double its development power. New features, speed improvements and even faster support are around the corner. That’s in case you are a fan of TL;DR, otherwise please read on.

A bit about our history. In case you don’t know Rendered Text is the company behind Semaphore. It was born about five years ago around the idea of making great web services primarily with Ruby and other lovely open source tools. We didn’t set out to create a consulting “factory”. We always optimize for happiness and for us that means being a small team.

Well crafted UX has always been huge to us but we also had that nerdy fetish in that we wanted to work on hard technical problems. We believe that Semaphore reflects those two great passions. Over time we were fortunate to have clients who care deeply about UX and appreciate our love for technical challenges. This in turn helped our whole team to develop even beyond our high expectations.

Semaphore was launched by a really tiny team. It was me working half time, then we had a rotation and Marko started working full time. When possible other people from the company contributed chunks of functionality. Six months passed and then it was a team of one and a half developers. As you can see we love to grow slowly and organically. Or it could be that it has been a product of hard constraints of a bootstrapping environment. At the moment it’s Marko, Aleksandar and me, and it’s been like that since May. If you are using Semaphore then you might have met us on support.

We are proud to have bootstrapped successfully. For better or worse we find it hard to change our mindset of going slowly, but we had to reconsider it recently. After a not very long discussion it became pretty obvious that now is the right time to change gears. The customer base is growing fantastically and we have many ideas on how to improve the service. With a large load on our distributed system we are constantly facing challenges that are new to us. We do hate to stop working with clients that we’ve had a long relationship with but we feel we have to start working on Semaphore in full speed. So with Nebojša and Rastko joining, we will be a team of five developers.

We are very excited about this change. The list of features and improvements we would like to implement is long and we are constantly learning as we are listening to your feedback. So prepare yourselves for new and amazing things on Semaphore in the coming months.

RubyGems update and JRuby 1.7.5

As a response to a sudden appearance of intermittent errors on bundle install related to SSL, we’ve upgraded the RubyGems package to the latest version (2.1.9) in all Rubies provided on the Semaphore build platform.

The problem affected users sourcing rubygems.org via https in Gemfile:

source "https://rubygems.org"

The typical error looks like this:

Resolving deltas: 100% (1572/1572), done. 
Could not verify the SSL certificate for https://rubygems.org/. 
There is a chance you are experiencing a man-in-the-middle attack, but most 
likely your system doesn't have the CA certificates needed for verification. For 
information about OpenSSL certificates, see bit.ly/ruby-ssl. To connect without 
using SSL, edit your Gemfile sources and change 'https' to 'http'.

The consensus seems to be that the solution is to upgrade RubyGems and ca-certificates package. Version 2.1.6 of RubyGems from October 8th indeed adds certificates “to follow s3.amazonaws.com certificate change”. The ca-certificates package that we had was already up to date, at least on Ubuntu 12.04 LTS.

The original problem is difficult to debug because it appears randomly. However our tests so far have shown that with the latest RubyGems it appears to be gone. We will continue to monitor the situation, and if necessary investigate if we can backport an even newer version of ca-certificates.

If you do encounter this problem from now again, please let us know via a support request.

Note that a possible general workaround to this kind of a problem is to source "http://rubygems.org" in Gemfile.

In other news, we’ve upgraded JRuby to 1.7.5.

Update 21 Oct: we’re now also setting the SSL_CERT_FILE environment variable. Thanks to Mislav Marohnić for a very good explanation.

Update 23 Oct: created a symlink for the previously nonexisting file returned by OpenSSL::X509::DEFAULT_CERT_FILE.

Update 25 Oct: the issue was acknowledged and fixed on RubyGems infrastructure. It boils down to this commit (via).

Post-thread commands

We’ve rolled out a new feature that lets you run commands at the end of each thread, regardless of whether build commands passed or failed.

Post-thread command on Semaphore

To set it up simply set a command to “Post-thread” in your project’s build settings.

This is great when you need to, for example, upload assets to S3 or tail the test.log when you’re not sure why a certain spec or scenario failed.

There is also a new environment variable available to these commands: SEMAPHORE_THREAD_RESULT. It can hold the value of either “passed” or “failed”.

Continuous delivery via Cloud66

Today we’re very happy to announce our partnership with Cloud 66. Cloud 66 makes it easy to provision, deploy and manage Ruby on Rails applications on major cloud providers.

The integration makes it very easy to have a full continuous delivery pipeline: from pushing to GitHub to testing on Semaphore, and if with all tests passing, deploying to all your servers via Cloud 66.

To get started, head over to the Deployment tab in project settings to add a new server, and select Cloud 66. The seamless integration only requires you to authorize Semaphore with your Cloud 66 account. We assume that you have an application stack ready on Cloud 66, which is the next thing to select in the configuration wizard.

Semaphore and Cloud 66 integration

If you select automatic deployment, Semaphore will send a redeploy request to Cloud 66 whenever your build passes on the corresponding branch. With manual deployment, you choose when to click and launch a redeploy. Note that while Semaphore tracks every deploy in the project timeline, it sends a blank request to Cloud66 to perform the actual deployment work.

Happy deliveries!

Multi-deployment approaches

Capistrano is a great tool for remote server automation and deployment. The standard practice is to have a single Capistrano setup per application, but if you are working on a complex project that consists of several applications or services, managing deployment configuration can become a daunting task.

There are two approaches for tackling deployment of larger projects with multiple applications.

Decentralized multi-deployment approach

This happens by default. With decentralized multi-deployment approach the deployment configuration is located in each application.

~/projects/my-app/my-app-base$ cap production deploy

~/projects/my-app/my-app-billing$ cap production deploy

~/projects/my-app/my-app-importer$ cap production deploy

Drawbacks:

  • No separation of concerns: application code is mixed with deployment code.
  • Duplication of code: code is duplicated across applications.
  • Difficult workflow: multiple directories require more effort to navigate etc.

Centralized multi-deployment approach

With centralized multi-deployment approach the deployment configuration for each application is located in a single repository.

~/projects/my-app-deployment$ cap base:production deploy

~/projects/my-app-deployment$ cap billing:production deploy

~/projects/my-app-deployment$ cap importer:production deploy

Advantages:

  • Separation of concerns: deployment code is located in a single dedicated repository, separate from application code.
  • DRY: configuration can be shared between applications.
  • Increased security: ability to restrict deployment access to authorized team members.

Drawbacks:

  • Having the deployment script outside application code makes continuous delivery a bit more difficult to set up.

CapHub

CapHub is a tool which can help you set up a central deployment repository by generating a Capistrano skeleton suited for deploying to multiple applications.

$ gem install caphub
$ caphub my-app-deployment
$ tree --dirsfirst my-app-deployment my-app-deployment
├── config
│ ├── deploy
│ └── deploy.rb
├── recipes
├── Capfile
├── Gemfile
└── Gemfile.lock

Using centralized deployment on Semaphore

When adding a new server on Semaphore, select deployment option “Using a deploy script” and adapt the following deploy commands:

echo {SERVER_PUBLIC_KEY} >> ~/.ssh/known_hosts

cd /tmp

git clone https://{GITHUB_TOKEN}:x-oauth-basic@github.com/my-company/my-app-deployment.git

cd ./my-app-deployment
bundle install --path vendor/bundle
bundle exec cap base:production deploy:migrations

What we are doing here is pulling and initializing the central deployment repository before calling the deploy task. More information on the GITHUB_TOKEN mentioned above can be found on our documentation site.

Semaphore partners with Code Climate to provide test coverage service

At Semaphore, we want to empower great teams to test and deliver high quality apps. Besides providing top-notch build and deployment services, enabling our customers to get more insights from their test suite is a big part of that. Today we’re proud to announce that Semaphore has partnered with Code Climate to provide test coverage services for Ruby applications.

Working smartly with application and test code determines the team’s ability to ship at a sustainable pace. You need to make sure that your tests exercise enough of your application code. How do you find out which areas of your app are both low quality and untested? Code-to-test ratio is irrelevant if there are many tests for unimportant code or dead code without tests.

After a very quick setup process, your builds on Semaphore will enable Code Climate to surface test coverage information at the repository, class, and source listing level (down to an individual line of code) and provide feedback as metrics change over time in the form of email alerts, activity feeds, chat notifications and RSS.

How it works

Code Climate accepts coverage data sent from Semaphore as it runs your tests. There is a codeclimate-test-reporter RubyGem that you install into your Gemfile. When the build is finished, the gem sends an HTTPS post to Code Climate with a report on which lines of code were executed.

To make that happen, head over to the Integrations screen in your project’s settings. Code Climate has generously agreed to offer Semaphore customers 20% off their first three months.

We are very much looking forward to seeing how your code gets even better with these new features.

Improved repository caching

Over the past week we rolled out an update to our repository caching system. Previously cache was branch based, and all users suffered one disadvantage: first builds of every new branch took more time as the git clone and bundle install had to be done from scratch.

A brief overview of how repository caching works

Our build platform is powered by dedicated hardware and all build servers maintain a cache of repositories they have worked with. The build scheduler tries to assign a job to an available build server which has worked with a given branch before.

Source code is then taken from the cache, advanced to the revision specified by the build and moved to a virtual machine where the actual build or deploy is performed. When it reaches a certain usage quota, cache frees up disk space by removing repositories by iterating through oldest first.

Repository caching is now global

From now on repositories are cached globally and all branches use the same source. One immediate effect is that the cache lives longer. It also uses less disk space, so a build server can accomodate more projects, resulting in overall fewer cache misses as one project can easily be cached on many servers.

So instead of 100% of all first builds of a branch doing a full git clone and bundle install, that number is now down to about 50%. As most teams work in a workflow with feature branches, this results in reduced build times on many projects. The same applies to deploys as well. Of course, we’ll continue to work on improving the platform further.

Get future posts like this one in your inbox.

Follow us on