Semaphore Blog

News and updates from your friendly continuous integration and deployment service.

Capistrano 3 Upgrade Guide

We recently started receiving support requests about Capistrano 3. Of course to provide quality support you have to know the subject, so I set on a quest to upgrade Semaphore’s deployment script from Capistrano 2 to Capistrano 3. As always it took a bit than expected but in the end new code looks nicer.

I have to say that I did have a flashback from couple of years ago when I was setting up Capistrano for the first time: documentation is missing some things and it’s a bit scattered between the readme, wiki and official homepage. But it’s open source so we are all welcome to contribute improvements.

I will try to make the upgrade easier for you by presenting our old vs new Capistrano configuration step by step.

Bootstrap new configuration

Gemfile

As the first step you have to install new gems. Capistrano 2 didn’t have support for multistage configurations so you had to also use capistrano-ext gem. Capistrano 3 has multistage setup included. It is framework agnostic so you would have to use capistrano-rails gem which adds support for deploying Rails applications. Just update your Gemfile like in the example below, run bundle install and you are ready to start the upgrade process.

Capistrano 2:

group :development do
  gem "capistrano"
  gem "capistrano-ext"
end

Capistrano 3:

group :development do
  gem "capistrano-rails"
end

Capify project with Capistrano 3

As official upgrade guide advises, it’s a good idea to just move old Capistrano files to a safe place and than manually move configuration to newly generated files. Here is a tip how to do that:

mkdir old_cap
mv Capfile old_cap
mv config/deploy.rb old_cap
mv config/deploy/ old_cap

After that you are ready to capify your project with new Capistrano:

bundle exec cap install

Capfile

Among other newly generated files you should also have the new Capfile. Below is how our Capfile used to look like and then the new one.

Capistrano 2:

load "deploy"
load "deploy/assets"
Dir["vendor/gems/*/recipes/*.rb","vendor/plugins/*/recipes/*.rb"].each { |plugin| load(plugin) }
load "config/deploy"

Capistrano 3:

require "capistrano/setup"
require "capistrano/deploy"

require "capistrano/bundler"
require "capistrano/rails/assets"
require "capistrano/rails/migrations"
require "whenever/capistrano"

Dir.glob("lib/capistrano/tasks/*.cap").each { |r| import r }

Your new Capfile will also contain two commented lines for rvm and rbenv support. We don’t use any of those tools for managing Ruby versions on our servers so I can’t say much much about that part.

require "capistrano/rvm"
require "capistrano/rbenv"

Multistage configuration

Configuration for stages really hasn’t changed much as you can see below. However there is one thing that you need to pay special attention to. The way of telling Capistrano to deploy specific revision has been changed. If you are using continuous deployment with Capistrano you have probably seen this line:

bundle exec cap -S revision=$REVISION production deploy

REVISION is an environment variable that Semaphore exports during deployment and Capistrano 2 was using it as a parameter. In Capistrano 3 this is gone and you have to take care of that by setting the branch variable to revision or branch that you want to deploy. We already had in our configuration ability to specify branch through environment variable:

set :branch, ENV["BRANCH_NAME"] || "master"

so we just had to prepend ENV["REVISION"] to that chain.

set :branch, ENV["REVISION"] || ENV["BRANCH_NAME"] || "master"

This is one of the things that is not documented and you either have to dig it up in source or ask somewhere. All in all the change should be pretty straightforward.

File below is config/deploy/production.rb.

Capistrano 2:

server "server1.example.com", :app, :web, :db, :primary => true, :jobs => true
server "server2.example.com", :app, :web, :jobs => true

set :branch, ENV["BRANCH_NAME"] || "master"

Capistrano 3:

set :stage, :production

server "server1.example.com", user: "deploy_user", roles: %w{web app db}
server "server2.example.com", user: "deploy_user", roles: %w{web app}

set :branch, ENV["REVISION"] || ENV["BRANCH_NAME"] || "master"

Main configuration - config/deploy.rb

Biggest changes you will have to make will be in this file. I will list the changes that you would need to pay attention to.

  1. You no longer need to require capistrano/ext/multistage or bundler/capistrano. Multistage is included by default and bundler support is included in Capfile.
  2. No need to specify available stages or default_stage.
  3. Variable name for setting repository url has changed from repository to repo_url.
  4. deploy_via :remote_cache is not needed any more. There have been large changes under the hood in a way how Capistrano handles repositories. It now creates a local mirror of the repository on your server.
  5. PTY option is on by default.
  6. ssh_options have changed slightly I think, but basic settings are pretty much the same.
  7. Capistrano will now take care of all symlinks that you need. Just tell it to go through linked_files and linked_dirs.
  8. In case that you are not using rvm or rbenv you will need to override rake and rails commands. (See Capistrano 3 deploy.rb file)

Writing custom tasks has changed significantly and you will have to dig deeper into documentation to write tasks that you need. The library responsible for this is SSHKit. It seems like a quite nice library.

Pro-tip: In Capistrano 2 you could just write var_name and get the value. In new version you always need to write fetch(:var_name). It took me some time to figure this out while I was re-writing a custom task that we use to manage our workers.

Capistrano 2:

require "capistrano/ext/multistage" #1
require "bundler/capistrano"

set :application, "webapp"
set :stages, %w(production staging)
set :default_stage, "staging" #2

set :scm, :git
set :repository,  "git@github.com:example/webapp.git" #3
set :deploy_to, "/home/deploy_user/webapp"
set :deploy_via, :remote_cache #4

default_run_options[:pty] = true #5
set :user, "deploy_user"
set :use_sudo, false

ssh_options[:forward_agent] = true #6
ssh_options[:port] = 3456

set :keep_releases, 20

namespace :deploy do

  desc "Restart application"
  task :restart, :roles => :app, :except => { :no_release => true } do
    run "#{try_sudo} touch #{File.join(current_path,'tmp','restart.txt')}"
  end

  desc "Prepare our symlinks" #7
  task :post_symlink, :roles => :app, :except => { :no_release => true } do
    ["config/database.yml", "config/config.yml"].each do |path|
      run "ln -fs #{shared_path}/#{path} #{release_path}/#{path}"
    end
  end

end

after  "deploy",                   "deploy:post_symlink"
after  "deploy:restart",           "deploy:cleanup"
before "deploy:assets:precompile", "deploy:post_symlink"

Capistrano 3:

set :application, "webapp"

set :scm, :git
set :repo_url,  "git@github.com:example/webapp.git"
set :deploy_to, "/home/deploy_user/webapp"

set :ssh_options, {
  forward_agent: true,
  port: 3456
}

set :log_level, :info

set :linked_files, %w{config/database.yml config/config.yml}
set :linked_dirs, %w{bin log tmp vendor/bundle public/system}

SSHKit.config.command_map[:rake]  = "bundle exec rake" #8
SSHKit.config.command_map[:rails] = "bundle exec rails"

set :keep_releases, 20

namespace :deploy do

  desc "Restart application"
  task :restart do
    on roles(:app), in: :sequence, wait: 5 do
      execute :touch, release_path.join("tmp/restart.txt")
    end
  end

  after :finishing, "deploy:cleanup"

end

Conclusion

The code that you get in the end is cleaner and Capistrano 3 together with SSHKit seems like a powerful combo. However some libraries like whenever and bugsnag don’t have Capistrano 3 support yet, so for now you will have to take care of that part on your own.

Custom Configuration Files

If testing and deploying your project requires some specific configuration files, you can now manage them directly through our new Custom Files feature. This feature allows you to securely create, edit or delete files that are not part of your repository.

For your new custom file you need to specify a target file path. For instance, if you have a project “semaphore_builder” and you want to add a new custom file to your project’s directory, your file path should look like the one on the screenshot below:

For greater security, the content of your custom files can be encrypted. We strongly recommend that you select this option if you are adding sensitive content such as SSH keys. Once encrypted, encrypted file cannot be edited. The identity of your file can determined by an MD5 hash.

Here are some ideas how you could use this feature:

  • Saving one (or more) additional SSH keys used to check out a private project dependency.
  • Creating a database.yml with non-standard attributes.
  • Storing a custom /etc/hosts file for subdomain configuration.

How to use different Gemfiles with Bundler

Normally when you’re working with a Ruby project with Bundler you write a file called Gemfile, run bundle install and then invoke various commands with the bundle exec prefix. However what if you would like to be able to work with different versions of gems over a longer period of time? In that case being able to use multiple Gemfiles within a single branch can help.

Bundler supports passing an environment variable called BUNDLE_GEMFILE to all of its commands. Here is an example how we can tell it to use a file called Gemfile-rails4:

BUNDLE_GEMFILE=Gemfile-rails4 bundle install --path vendor/bundle-rails4

You can then run tests in the similar way:

BUNDLE_GEMFILE=Gemfile-rails4 bundle exec rake spec

On Semaphore, I recommend creating a new project with the same repo and using build commands tailored for the custom Gemfile.

Ruby version usage in commercial projects

The chart below shows the distribution of Ruby versions in private projects tested on Semaphore.

Ruby versions used on Semaphore

Currently 80% of all projects on Semaphore are private, ie commercial.

The versions and implementations of Ruby not included are not part of the platform, which also means that no one has ever asked for them yet.

Platform update: Ruby 2.1 preview, locale, new Postgres, MongoDB, Firefox

The latest platform release adds Ruby 2.1.0-preview1 to the list of included rubies. If you’re curious about its new features such as refinements, required keyword arguments and improved method caching, now’s your chance to test them in your app.

The default locale is now en_US.UTF-8, instead of C.UTF-8 which was causing trouble to some users.

Postgres is now at version 9.3, letting you take full advantage of the JSON datatype (available from 9.2).

MongoDB went up to 2.4.7, which enables things like text search and returning symbols from the aggregation framework. See 2.4 release notes.

Firefox has also been upgraded, please run bundle update selenium-webdriver if you haven’t recently.

As usual, this is filed in our platform changelog.

Growing our team - get ready for new greatness

We have awesome news that we would like to share with you. Semaphore team will almost double its development power. New features, speed improvements and even faster support are around the corner. That’s in case you are a fan of TL;DR, otherwise please read on.

A bit about our history. In case you don’t know Rendered Text is the company behind Semaphore. It was born about five years ago around the idea of making great web services primarily with Ruby and other lovely open source tools. We didn’t set out to create a consulting “factory”. We always optimize for happiness and for us that means being a small team.

Well crafted UX has always been huge to us but we also had that nerdy fetish in that we wanted to work on hard technical problems. We believe that Semaphore reflects those two great passions. Over time we were fortunate to have clients who care deeply about UX and appreciate our love for technical challenges. This in turn helped our whole team to develop even beyond our high expectations.

Semaphore was launched by a really tiny team. It was me working half time, then we had a rotation and Marko started working full time. When possible other people from the company contributed chunks of functionality. Six months passed and then it was a team of one and a half developers. As you can see we love to grow slowly and organically. Or it could be that it has been a product of hard constraints of a bootstrapping environment. At the moment it’s Marko, Aleksandar and me, and it’s been like that since May. If you are using Semaphore then you might have met us on support.

We are proud to have bootstrapped successfully. For better or worse we find it hard to change our mindset of going slowly, but we had to reconsider it recently. After a not very long discussion it became pretty obvious that now is the right time to change gears. The customer base is growing fantastically and we have many ideas on how to improve the service. With a large load on our distributed system we are constantly facing challenges that are new to us. We do hate to stop working with clients that we’ve had a long relationship with but we feel we have to start working on Semaphore in full speed. So with Nebojša and Rastko joining, we will be a team of five developers.

We are very excited about this change. The list of features and improvements we would like to implement is long and we are constantly learning as we are listening to your feedback. So prepare yourselves for new and amazing things on Semaphore in the coming months.

RubyGems update and JRuby 1.7.5

As a response to a sudden appearance of intermittent errors on bundle install related to SSL, we’ve upgraded the RubyGems package to the latest version (2.1.9) in all Rubies provided on the Semaphore build platform.

The problem affected users sourcing rubygems.org via https in Gemfile:

source "https://rubygems.org"

The typical error looks like this:

Resolving deltas: 100% (1572/1572), done. 
Could not verify the SSL certificate for https://rubygems.org/. 
There is a chance you are experiencing a man-in-the-middle attack, but most 
likely your system doesn't have the CA certificates needed for verification. For 
information about OpenSSL certificates, see bit.ly/ruby-ssl. To connect without 
using SSL, edit your Gemfile sources and change 'https' to 'http'.

The consensus seems to be that the solution is to upgrade RubyGems and ca-certificates package. Version 2.1.6 of RubyGems from October 8th indeed adds certificates “to follow s3.amazonaws.com certificate change”. The ca-certificates package that we had was already up to date, at least on Ubuntu 12.04 LTS.

The original problem is difficult to debug because it appears randomly. However our tests so far have shown that with the latest RubyGems it appears to be gone. We will continue to monitor the situation, and if necessary investigate if we can backport an even newer version of ca-certificates.

If you do encounter this problem from now again, please let us know via a support request.

Note that a possible general workaround to this kind of a problem is to source "http://rubygems.org" in Gemfile.

In other news, we’ve upgraded JRuby to 1.7.5.

Update 21 Oct: we’re now also setting the SSL_CERT_FILE environment variable. Thanks to Mislav Marohnić for a very good explanation.

Update 23 Oct: created a symlink for the previously nonexisting file returned by OpenSSL::X509::DEFAULT_CERT_FILE.

Update 25 Oct: the issue was acknowledged and fixed on RubyGems infrastructure. It boils down to this commit (via).

Post-thread commands

We’ve rolled out a new feature that lets you run commands at the end of each thread, regardless of whether build commands passed or failed.

Post-thread command on Semaphore

To set it up simply set a command to “Post-thread” in your project’s build settings.

This is great when you need to, for example, upload assets to S3 or tail the test.log when you’re not sure why a certain spec or scenario failed.

There is also a new environment variable available to these commands: SEMAPHORE_THREAD_RESULT. It can hold the value of either “passed” or “failed”.

Get future posts like this one in your inbox.

Follow us on