NEW Run your Ruby tests in a few minutes and ship faster with Boosters, one-click auto parallelization · Learn more…
×

Semaphore Blog

News and updates from your friendly continuous integration and deployment service.

Heartbleed

Get future posts like this one in your inbox.

Follow us on

As you have probably heard, a severe security bug in OpenSSL dubbed Heartbleed was uncovered on April 7th. It allows anyone to read 64k chunks of memory of the systems using the vulnerable versions of OpenSSL, leaving no trace in common system or server logs.

All Semaphore’s current infrastructure is using OpenSSL and was thus affected and needed to be updated. Here’s what we did and what we recommend you to do as well.

OpenSSL upgrade

We upgraded OpenSSL to a secure version on our front-end system on April 8th 9:47 AM UTC. Immediate next step was to replace SSL certificates. Once that was in place, the platform was secure from any future attacks. We then proceeded to upgrade OpenSSL and certificates on our build platform as well.

Passwords, API tokens and sessions

We have found no evidence of harm, but absence of evidence does not equal evidence of absence, hence we recommend that you change your Semaphore password and reset your API token. We have implemented a one-click way for resetting the API token for this purpose, you can find it in project settings > API. We have also reset all user sessions.

GitHub tokens

GitHub issued their response to Heartbleed. We encourage you to use their 2FA system if you are not already. They implemented an API endpoint for resetting OAuth tokens, which we applied to all Semaphore users.

Heroku tokens

If you are deploying from Semaphore to Heroku, and you changed your Heroku password (which you should do), then the API token that Semaphore is using internally to run db:migrate for you is no longer valid. We have implemented an easy way for you to set a new value of Heroku API token, which you can find on your server settings page.

Contacting us

If you still have any concerns, please do get in touch, we are happy to help and answer any questions. You can reach us via the “Support” link inside the app or via semaphore@renderedtext.com.

A New Way To Read Build Results

We are happy to announce a new state of the art reporting feature!

As we are always in pursuit of greatness and bleeding edge technology, this time we have taken inspiration from the past in order to do something completely new. From now on, for each and every build Semaphore will send you a full test report, printed in crisp dot-matrix font directly to your address. We expect first deliveries to arrive today after 2PM UTC.

This feature is available to all Pro customers. To enable it, simply fill in your mailing address in your account settings.

Your paper reports will be wrapped in a water resistant package envelope, so they will will be saved from all weather hazards.

All modesty aside, we believe that this could be a start of a new era in the CI world and the way we communicate with our customers.

Managing Externals in Ruby Tests

In this post we’ll explore options for dealing with external resources when writing tests. Generally, a common solution is to use a mock instance of the resource that is not part of your system under test. It is however important to make sure that your mock, or stubbed response is a faithful copy and does not get out of date.

Bending time

With Timecop, it is not necessary to create records and then modify their timestamps manually in order to verify some time-based conditions. Consider the following example:

describe ".recent" do

  before do
    2.times do
      post = create(:post)
      post.update_attribute(:created_at, 2.months.ago)
    end

    @recent_post = create(:post)
    @recent_post.update_attribute(:created_at, 10.days.ago)
  end

  it "returns posts from the past month" do
    Post.recent.count.should eql(1)
    Post.recent.should include(@recent_post)
  end
end

Real-life examples of course get more messy, for example when you’re dealing with more elaborate conditions for a leaderboard. Compare the same spec with a version using Timecop:

describe ".recent" do

  before do
    Timecop.travel(2.months.ago) do
      2.times { create(:post)
    end

    Timecop.travel(10.days.ago) { @recent_post = create(:post) }
  end

  it "returns posts from the past month" do
    Post.recent.count.should eql(1)
    Post.recent.should include(@recent_post)
  end
end

Making HTTP requests

Depending on external resources makes the test suite slow and prone to unexpected errors in case of connectivity issues or service outages. For these reasons you should avoid making real HTTP requests in your tests. A common solution is to stub the requests and, optionally, responses. Ruby has very good tools to achieve this.

Webmock lets you stub and set expectations on HTTP requests:

stub_request(:get, "www.example.com").with(:query => {"a" => ["b", "c"]})

RestClient.get("http://www.example.com/?a[]=b&a[]=c")    # ===> Success

When dealing with many HTTP requests, especially if they are important enough that you want to stub the full response, I recommend using VCR. It works with any test framework and lets you record your test suite’s HTTP interactions in YAML files and “replay” them, ie reuse them as stubbed responses, during future test runs. We are using VCR to test Semaphore’s interaction with the GitHub API, for example.

describe HookCreationService do

  let(:project) { FactoryGirl.build(:project, :hash_id => "1a3b5") }

  vcr_options = { :cassette_name => "HookCreationService/connection_working",
                  :record => :new_episodes }

  context "connection working", vcr: vcr_options do

    it "sets project's github_hook_id" do
      project.should_receive(:update_attribute)
        .with(:github_hook_id, instance_of(Fixnum))

      HookCreationService.execute(project)
    end
  end
end

The first time this test runs (in practice on your development machine), the actual HTTP request is made. VCR stores the full response, including headers in fixtures/cassette_library/HookCreationService/connection_working.yml. Future runs of the test will use Webmock under the hood to stub the request and return the saved response. Not only will they be fast, as they won’t reach for the internet, but you can be sure that they’ll be accurate, since they will use a bit-exact copy of response. If we ever hit that URL with different options, VCR will record a new copy. All this is provided by the :new_episodes option. Configuration can be provided globally of course so you don’t repeat yourself.

You can find more information about VCR on its documentation site.

A word of caution: VCR does not insulate you from changes in external APIs. If an API changes and you are not paying attention, your tests will still be passing but the production system will stop working. Ideally your API provider is responsible and is not making any changes without announcing them. In that case, follow the API changes feed and when something does change, delete your cassette files and re-record them.

Testing an OAuth flow

Developers of applications that use OAuth to authenticate users via a third-party service can find it challenging to test that flow. I will present how we test GitHub authentication for Semaphore.

If you are using the excellent OmniAuth gem and one of its many providers, you have an option to use its facilities for integration testing. In short, you can short-circuit the flow to immediately redirect to the authentication callback (that’s a route in your app) with a provided mock hash. This is good because a fairly simple hash is what OmniAuth distills for you in production as well.

Since we do integration testing with Cucumber, this is our features/support/omniauth.rb:

Before("@omniauth_test") do
  OmniAuth.config.test_mode = true

  OmniAuth.config.mock_auth[:github] = {
    "uid" => "1",
    "provider" => "github",
    "info" => { "nickname" => "darkofabijan", "name" => "Darko Fabijan" },
    "credentials" => { "token" => "63146da137f3612f..." }
  }
end

After("@omniauth_test") do
  OmniAuth.config.test_mode = false
end

The first time this was set up, we authenticated Semaphore (a local development instance or production, does not matter) with GitHub and pasted the credentials token above. The Cucumber suite is using VCR, so the next step was to record cassettes with real API responses. A scenario only needs to include the @omniauth_test tag to apply this token to all requests to GitHub API:

Feature: Building projects

  @omniauth_test @javascript
  Scenario: New user signs up, creates a project without collaborators and builds it
    Given I am on the homepage
    #...

Again, first test runs create YAML files with real responses which we can reuse in developing the feature:

---
http_interactions:
- request:
    method: get
    uri: https://api.github.com/user/repos?access_token=63146da137f3612f...
    body:
      encoding: US-ASCII
      string: ""
    headers:
      Accept-Encoding:
      - gzip;q=1.0,deflate;q=0.6,identity;q=0.3
      Accept:
      - "*/*"
      User-Agent:
      - Ruby
  response:
    status:
      code: 200
      message: OK
    headers:
      Server:
      - GitHub.com
      #...

At this point the test suit can run autonomously. The only thing left, as a security measure, is to revoke access on GitHub in order to invalidate this particular API token.

New Build and Deploy Emails

We are very happy to announce that Semaphore is now sending new and more helpful build and deploy emails.

So far, we have been sending out plain text messages to report your build or deploy results, but now you can enjoy our new HTML layout that closely resembles the pages you are used to seeing on Semaphore. Here’s what it looks like:

In case when a command fails, the email will include a tail of its log.

Different people have very different needs and workflow. You can integrate Semaphore with Slack and Campfire, you can even use webhooks, but from now on even emails alone could give you a nice and pleasurable experience.

If you prefer plain text email, it is still available since the new emails are sent as multipart.

Slack Integration

We’re very happy to announce official integration between Semaphore and Slack.

Slack is a new team communication service with apps for all major platforms. We’ve recently started using it ourselves and it’s been a great experience.

The integration lets you receive status messages from Semaphore when a build or deploy finishes right in your Slack room of choice. To set it up, simply go to https://my.slack.com/services/new/semaphore and follow the instructions. You’ll be up and running in a minute.

Big thanks to the Slack team for working with us on this feature.

Upcoming Platform Update on March 5

The upcoming platform update is scheduled for Wednesday, March 5 2014.

Ruby 2.1.1 is added to the stack. Rubies 2.0.0 and 1.9.3 get upgraded patchlevels, so bundle update debugger will likely be necessary.

Firefox goes up to version 27, requiring bundle update selenium-webdriver for most projects that use that gem.

Node and NPM are updated to the latest versions too.

A full list of changes is available in the platform changelog.

Update March 6: ElasticSearch package changed and it no longer runs the server on boot. Users will need to add sudo service elasticsearch start as their first build setup command. Sorry about that, we’ll have it fixed shortly.

Update March 17: ElasticSearch server now runs on boot again.

Building Pull Requests From Forked Repos

We are glad to announce that Semaphore now supports building pull requests from forked repositories.

So far, in order to see pull request status you had to have the forked repository on Semaphore, along with the parent repo. However, GitHub recently removed the ability to push commit status from forked to parent repo and therefore we had to do some work to make our system better.

With this new feature, forked repository does not have to be on Semaphore and still, all its’ pull requests towards the parent repository will be tested and will have build status posted to the corresponding pull request page on GitHub:

If a new commit gets pushed to pull request, it will be tested as well.

We have updated Semaphore’s dashboard to include incoming pull requests and their status. Pull request builds will be shown along other branches, with a slightly different name format:

As with feature branches, once the pull request is merged, it will be removed from your dashboard. Same goes with the closing of the pull request.

Clojure on Semaphore

We’re very excited to announce official support for testing and deploying Clojure projects. Clojure is the third official language supported by Semaphore.

Continuous integration for Clojure projects in 1 minute

The steps for setting up your Clojure project on Semaphore are as simple as for a Ruby or JavaScript project.

If you don’t already have a Semaphore account, sign up for a free 30-day trial for unlimited builds, unlimited number of collaborators, unlimited deployment and two processors for parallel testing.

New accounts will need to authorize Semaphore to access GitHub repositories, in order to be able to pull and receive notifications about code changes, post commit status in pull requests etc.

On your dashboard, click on “Build a New project”. Semaphore will fetch basic information about projects that you have access to on GitHub and ask you to select one.

Semaphore will now analyze your repository and recognize that it is a Clojure project, presetting lein test as your build command. Leiningen 2.3.4 and OpenJDK 7 are preinstalled on the platform.

What remains is to optionally invite your collaborators to Semaphore and – launch the first build. From that point, Semaphore will automatically test your project after every push to your Git repository.

After the first build, your dependencies will be cached for later reuse.

If you are developing a project that works with a database, see our documentation page about database access for information about the engines available and required credentials.

Why Clojure

Clojure is a dynamic, general-purpose programming language that runs on the Java Virtual Machine. It is a dialect of Lisp, sharing the code-as-data philosophy, support for macros and functional programming. It provides immutable, persistent data structures and is designed to support high-performance, concurrent applications.

The heart of Clojure is simplicity. Bottom line is, all code is in small functions grouped in namespaces whose scope of effect is very limited and understandable. This gives you confidence and leads to easy composition of higher-order systems. It does not take long to learn the syntax of the language; the challenge lies in changing the traditional object-oriented mindset used to mutability and side-effects and working with a different set of programming abstractions.

These characteristics sound a lot different if the last language you’ve studied was something like Ruby or Python. You cannot write imperative code in Clojure, design patterns go out of the window, concurrency is easy to implement and reason about and unless you really try you can’t create global, mutable state. There are many use cases where this is a blessing; so far most of them are on the axis between complex systems and data analysis.

Clojure adoption is on the rise, people at conferences are beginning to talk about replacing “legacy Ruby apps” and quite a few smart and experienced engineers are recommending it as the next language developers should master.

Useful resources for learning Clojure

Tutorials

Presentations

Books

When you’re wondering which tools are available for some job, check out Clojure Toolbox.

Webhooks API and More Updates

Our goal is to let our users fit Semaphore to a wide range of workflows, while keeping the core product simple to use. The API is a big part of that, and today I would like to announce some improvements we’ve made.

First, we added some useful information to existing API endpoints:

  • There is now a html_url for projects on Semaphore in Project API response.
  • Webhook payload is richer for event (which can be “build” or “deploy”) and project_hash_id, a unique identifier of the project on which the event occurred.

Second, bigger and more interesting thing to mention is the new webhooks API. Now you can easily create, update and delete webhooks via API, which makes it easy to automate your favourite Semaphore integration.

You can always find the latest API documentation on our documentation site.

We are beginning to plan the next version of Semaphore API and we’re ready to hear any suggestions about features you would like to see. And of course about new use cases or tools that you would like to integrate. Just write a comment below or get in touch via our standard support channel.

Happy integrating!

Rails Testing Antipatterns: Controllers

This is the third post in the series about antipatterns in testing Rails applications. See part 1 for thoughts on fixtures and factories, and part 2 on models.

Skipping controller tests

When starting out with Rails, with all the “magic” going on it can be easy to forget that controllers are just classes that need tests as well. If it seems to be hard to write a test for a controller, you should embrace the pain; it is telling you that your controller is doing too much and should be refactored.

A controller handles the request and prepares a response. So it shouldn’t be subscribing a user to a newsletter or preparing an order. In general we’ve found that controllers should be limited to:

  • Extracting parameters;
  • Checking for permissions, if necessary;
  • Directly calling one thing from the inside of the app;
  • Handling any errors, if they affect the response;
  • Rendering a response or redirecting.

I’ve found that controllers are best tested with a mocking approach, since they tie many things together. The freedom you get from mocking should be used as an advantage in molding the controller method’s code to an ideal shape.

Creating records in controller tests

Because Rails controllers should be designed as “callers” of logic defined - and tested - elsewhere, it is pure overhead to involve the database in controller tests. Use mocks and test doubles to decouple from other application layers and stay focused on the task of specifying the functional behavior of the controller.

This is not really a controller spec but a semi-integration test:

describe "GET show" do
  let(:user) { create(:user) }
  let(:report) { create(:report, :user => user) }

  it "fetches user's report" do
    user_id = user.id
    report_file = report
    get :show, :user_id => user_id, :id => report_file.id
    assigns(:report).id.should eql(report.id)
    response.status.should eql(200)
  end
end

The code above is more like an attempt to write a passable test for an existing implementation. Contrast that with writing a test first, thinking only about interfaces, one assertion at a time, without interacting with the database:

describe "GET show" do
  let(:user) { build_stubbed(User) }
  let(:report) { build_stubbed(Report) }

  before do
    User.stub(:find) { user }
    user.stub(:find_report) { report }
  end

  describe "GET show" do
    it "finds the user's report" do
      user.should_receive(:find_report) { report }

      get :show, :user_id => user.id, :id => report.id
    end

    it "assigns a report" do
      get :show, :user_id => user.id, :id => report.id

      assigns(:report).should eql(report)
    end

    it "renders the 'show' template" do
      get :show, :user_id => user.id, :id => report.id

      response.should render_template("show")
      response.code.should eql(200)
    end
  end
end

Sometimes however, data is sensitive enough and goes through some kind of an API to warrant an integration test with real records. This is a use case for request specs.

# spec/requests/login_attempts_spec.rb
require "spec_helper"

describe "login attempts" do

  context "failed" do

    it "records a new failed login attempt" do
      expect {
        post user_session_path,
          :user => { :email => "user@example.com", :password => "badpassword" }
      }.to change(LoginAttempt, :count).by(1)
    end
  end
end

No controller tests, only integration

One might argue that it is not necessary to test controllers when there are comprehensive integration tests.

In practice, creating an integration test for every possible edge case is much harder and never really done. Integration tests are also much slower, because they always work with the database and may involve UI testing. Controller specs that use mocks and test doubles are extremely fast.

Complex controller spec setup

Epic setup is generally a code smell, telling us that our class is doing too much. See this context for verifying that a welcome email for a newly registered user goes out:

describe UsersController do

  describe "POST create" do

    context "on successful save" do

      before do
        @user = double(User)
        @user.stub(:save).and_return(true)

        User.stub(:new) { @user }

        @mailer = double(UserMailer)
        UserMailer.stub(:welcome) { @mailer }
        @mailer.stub(:deliver)

        ReferralConnector.stub(:execute)
        NewsletterSignupService.stub(:execute)
      end

      it "sends a welcome email" do
        @mailer.should_receive(:deliver)

        post :create, :user => { :email => "jo@example.com" }
      end
    end
  end
end

Difficult tests always go hand in hand with a problematic implementation. The spec above is for a controller that looks something like this:

class UsersController

  def create
    @user = User.new(user_params)

    if @user.save
      UserMailer.welcome(@user).deliver
      ReferralConnector.execute(@user)
      NewsletterSignupService.execute(email, request.ip)

      redirect_to dashboard_path
    else
      render :new, :error => "Registration failed"
    end
  end

  private

  def user_params
    params[:user].permit(*User::ACCESSIBLE_ATTRIBUTES)
  end
end

A developer following the BDD approach and thinking a bit in advance could write a spec for a method handling user registration so that all post-signup things, whatever they may be, are delegated to a PostSignup class, which can be tested in isolation.

describe "POST create" do

  context "on successful save" do

    before do
      User.any_instance.stub(:save) { true }
      PostSignup.any_instance.stub(:run)
    end

    it "runs the post signup process" do
      PostSignup.should_receive(:new) { post_signup }
      post_signup.should_receive(:run)

      post :create
    end

    it "redirects to dashboard" do
      post :create
      response.should redirect_to(dashboard_path)
    end
  end
end

Get future posts like this one in your inbox.

Follow us on