NEW Run your Ruby tests in a few minutes and ship faster with Boosters, one-click auto parallelization · Learn more…
×

Semaphore Blog

News and updates from your friendly continuous integration and deployment service.

Ruby version usage in commercial projects, 2014 edition

Get future posts like this one in your inbox.

Follow us on

Last year we calculated how much each version of Ruby was used by private projects on Semaphore. Given that many people viewed and shared the report, we decided to run the numbers again and compare the results.

To set the context, here are the most notable milestones achieved by the Ruby core team in the meantime:

  • Ruby 2.1.1 and 2.1.2 were released.
  • 1.9.3 entered security maintenance phase.
  • 1.8.7 and 1.9.2 reached end of life.

The chart below shows the distribution of Ruby versions in private projects tested on Semaphore.

Ruby versions used on Semaphore in 2014

The bar chart below shows difference in usage between 2013 and 2014.

Ruby versions used on Semaphore by years

The most notable change is that now the majority of projects is using some version of Ruby 2. If we combine the representation of all 2.x versions, we see a rise from 30.6% to 51.1%.

Semaphore Gets a Free Plan For Private and Open Source Projects

With great pleasure we are announcing that Semaphore is now free to use for private projects, for up to 100 builds per month. This is great for personal sideprojects, early stage MVPs and similar low volume development activities.

Free accounts have two processors available. This allows you to either test two private projects or branches at the same time, or run one build with tests split in two parallel threads. To ensure fair usage we are counting builds using two threads as two free builds.

We will convert all new users who are currently in a 30-day free trial to the new free plan.

Semaphore becomes free for open source

We are not stopping there however. The same free plan you can use for private projects also allows unlimited testing and deployment of open source projects.

Sign up, add your project from GitHub, run your first test suite in a minute without any changes in source code and share it on your profile.

Giving back to the community

Semaphore would not be possible without open source software. From the backbone essentials such as Git and Linux to tools like RSpec and PhantomJS, Semaphore is both built with and made to support development of projects based on open source.

Our goal has always been to make Semaphore free for open source projects. However bootstrapping from an almost anonymous position obliged us to wait until the business is profitable enough. All we had was a team and a goal to keep making the best continuous integration service we can. Now we are happy to share it with the community.

Language support

Semaphore currently has full support for testing and deploying Clojure, JavaScript / Node.js, PHP and Ruby projects. Our plan is to add support for all major open source languages by the end of the year.

Open source organizations

To allow large open source projects to take full advantage of our organization management features, we are introducing a special open source plan which allows an organization to have unlimited projects and unlimited open source builds with two processors for parallel testing.

Changes to existing profiles

All current users and organizations who already have an open source project on Semaphore will find that their project pages have become public. Only collaborators can access settings and trigger new builds and deploys, of course. For example, check out profiles of Perceptive Software or FlatStack.

Reduced price of Starter plan by $10, gets unlimited projects

To streamline our pricing and help you build more, we are making our Starter plan the first available paid plan. With unlimited private projects (instead of five) and one processor for running tests, it now costs $29.

We hope that you are as excited about the new capabilities as we are, and can’t wait to see what you’ll be making next. Happy building!

More Flexibile Connection Between Semaphore Projects And GitHub Repositories

We are glad to introduce a new feature within our project settings - “Repository” settings. The main purpose of this new settings option is to add more flexibility between your Semaphore project and GitHub repository.

Repository tab is placed within project settings. It allows you to see:

  • repository that this project is connected to
  • username of repository owner at GitHub
  • username of the Semaphore user whose token is being used for communication with GitHub
  • status of repository deploy key at GitHub
  • status of webhook that is set on Github.

Besides presenting you this information, the repository tab also lets you take action. For example, if a deploy key or webhook is missing on GitHub, Semaphore will present you an option to re-set them with a single click.

You can also change the repository used for the given project. This is useful if the repository name or owner on GitHub has changed and made the Git clone URL invalid. Or if the person whose OAuth API token Semaphore is currently using is no longer available. This way you will reactivate your project on Semaphore without losing project history, configuration and collaborators.

Introducing User Profiles

Recently we introduced organizations and organization profile pages with a list of projects. Today, we’re launching the first version of profile pages for users.

On the profile page you can see basic information about a user together with a list of her project. You can reach a user’s profile by clicking on her username in the project header on the dashboard. The profile is currently accessible only to the collaborators of one of the user’s projects.

This is only the beginning of the user experience overhaul that we started with organizations. We’re actively working on a new set of features so stay tuned.

Writing Clear Code: A Developer Interview With Agworld

Developer Interview” is a new series here at Semaphore blog. We’ll interview developers from some of the companies using Semaphore to find out how they work and share their insights with you. First up: Agworld, a company building software as a service for farm management. Chief scientist Jason Hutchens tells us their story.

Tell us a little bit about your company and what it does.

Agworld is an agriculture tech startup from Perth, Western Australia. We have customers around the world and offices in several other countries, including the US.

Our customers include both farmers and the agronomists, suppliers and contractors they work with. They use our website, iPad app and iPhone app to plan their season, manage their day-to-day operations, store historical farm records, share relevant information with stakeholders, produce detailed reports and view all of this data in the context of a map of their farm.

“Knowing that the work we do is moving us towards our goal of making global food production profitable and environmentally sustainable is a great motivator”

It may not sound sexy, but there’s a lot of really interesting problems to be solved in Ag, and knowing that the work we do is moving us towards our goal of making global food production profitable and environmentally sustainable is a great motivator.

What does your workflow typically look like, from an idea until it’s shipped?

Typically we speak to customers and potential customers directly to understand the problems that we could help them to solve, even sending developers on field trips to experience all of that first-hand. This allows our product managers to work closely with our development and customer support teams to sketch out a solution, working with our designers to mock UI and often writing user stories in the form of cucumber scenarios.

We embrace Scrum for development, and integrate weekly for internal testing. We release frequently, with each release going through manual QA before being pushed to production in each of our geographical regions.

We maintain separate feature and hotfix branches across dozens of repositories, as our product is heavily service-oriented. Merging is typically done via a pull request on GitHub, giving us peer oversight, and Semaphore integration helps us to perform these merges once the build is green.

What tools and guidelines do you use to write tests?

In general we endeavour to make sure that all new code has sufficient test coverage; this is usually achieved through peer review rather than a formal process, although we’ve recently taken advantage of the Coveralls integration with Semaphore to easily eyeball our code coverage. This has also helped us to remove stale code (our app dates back to 2009, so we refactor as we go to reduce technical debt).

We use the fairly standard combination of RSpec of unit tests and Cucumber, Capybara, PhantomJS for integration tests.

“Your code should reflect the fact that it’s solving some real-world problem for a real-world customer”

What’s the last big realization about programming that you had?

Personally I’ve had two big insights recently:

You should write code as if you’re explaining what you’re doing to another developer. Your goal should be to eliminate developer uncertainty about what the code is doing. Sections of your codebase that are especially defensive (evidenced by lots of conditional code that checks the state of variables) belies this lack of trust, and should be fixed.

You shouldn’t write any code until you properly understand the problem that you’re solving. It’s easy to get distracted by a great solution without respecting the underlying problem. Your code should reflect the fact that it’s solving some real-world problem for a real-world customer. Business logic should be clearly evident; you shouldn’t have to go looking elsewhere to find out why the code exists.

When was the last time Semaphore saved you?

Semaphore saves us in little ways every day; we don’t need to maintain our own in-house build farm, the status of our branches is always visible, and assembling a release candidate from a dozen feature branches is no longer an intractable problem. Also, it’s great that setting up builds for a wide variety of projects across JRuby, MRI 1.9.3 and 2.1, and JavaScript required no management and very little manual configuration.

Introducing Organizations

Today we are very happy to announce Organizations on Semaphore. Organizations simplify project management inside companies and other groups, introduce a fine grained permission system and help you focus on your work while Semaphore scales up the continuous integration process as your operation grows.

If you ever wanted to let somebody else add a project under your billing account or give a group of collaborators read-only access without ability to launch builds and deploy, you’re going to love Organizations.

We’re also introducing some new payment options that are tailored for companies and enterprises, so let’s dive in what’s new.

One account for your organization’s CI needs

Creating an Organization lets you group all projects that belong to your organization under a new billing account. You create an Organization as a regular Semaphore user and become its owner, with ability to invite other users to share the ownership. Being an owner gives you full access to all projects and all organization management features, from billing to permission settings.

Adding a project to an organization

Once you have or belong to an organization, you can choose to add it to the organization account.

You can easily transfer an existing project under your personal account to an organization. To do that, go to your Account, locate your project and select the organization to transfer it to:

All existing collaborators will be copied to a new organization team with access to the given project.

Projects are accessed by teams with permissions

As an organization owner, you can define teams of users which have access to selected projects.

You can invite new members to a team by email. Similarly to the new collaboration model we launched recently for personal projects, team access is not connected to who is a collaborator on GitHub, giving you flexibility to add or remove anyone. This is also one of the reasons we are now requiring all users to have a Semaphore username and using the username as the identity indicator once a user is on board.

Teams can have one of the following permission sets:

  • Access — Team is only able to access projects and read build and deploy results.
  • Edit — In addition to Access level, team can also change project settings, add branches and manually trigger builds and deploys.
  • Admin — In addition to Edit level, team can also add and remove projects from the organization.

You can of course have as many teams as you want.

Besides the interface to manage teams, you can also see and manage individual organization members.

Profiles

Organization profiles are our first step towards making Semaphore more user-centric. Since project’s owner is part of its identity, you can now click on the organization name and view its most recently updated projects.

Pricing

With Organizations we’ve updated our pricing structure.

Personal plans include Solo, Starter and Double, formerly known as Pro 2, now with unlimited projects.

If you’re currently using one of the plans with more processors, you can continue to do so indefinitely. Previous higher plans have their equivalent in organization plans, at the same price, since we believe you will benefit from using Organizations.

Organization plans start with Pro 2 and go up to Pro 32. They have no project limit, and the only variable is the number of processors for concurrent and parallel builds, with higher plans including a discount instead of having a price that is a simple multiplier.

We are introducing an option to pay annually on all plans. This is great if you want to simplify the charge flow and enjoy a discount equivalent to getting one month of free service.

Organizations can also pay annually by invoice. To do this, simply go to your organization’s subscription settings and submit the company information form. We will invoice you on net 30 terms with wire transfer instructions.

Take a look at our new pricing.

Get started with Organizations today

It took us a few internal iterations to get to this solution which we believe will scale for organizations of all size and the time to come. We are happy to grow with you, dear user, and hope that these features will help you and your team deliver some amazing software.

Create a new organization now.

Improved Project Collaboration Management

We are glad to announce a more flexible solution for managing project collaborators. So far, project collaborators were directly mapped from repository collaborators on GitHub. In order to have access to a Semaphore project, user had to have a GitHub account and be a collaborator on the corresponding repository on GitHub. This was a great solution back in the early days since it enabled a fast onramp process, but since then our users and ourselves have grown and a need for something different has emerged.

In the past, if the user wanted to leave the project, she would have to leave or be removed from the collaborators list on GitHub. After this, the project owner on Semaphore still had to trigger a sync of collaborators with GitHub in order to remove the user from project on Semaphore. Same thing goes with joining project. Overall there was not much flexibility in the processes that take place after a project was set up on Semaphore.

From now on, project’s collaborators on Semaphore will not be bound to those on GitHub. You can manage your collaborators on Semaphore without making any changes on GitHub.

The project owner on Semaphore can invite anyone to collaborate via email, including people who are not collaborators on GitHub, for example the QA team.

If the user with that specific email already exists, she will automatically be added to project’s collaborators. Otherwise, an invitation email will be sent. Once the user follows the link in the invitation email and signs up, she will gain access to the project. Also, every user can now leave a project with a single click.

We hope that you find our new project collaborators approach useful and that it will improve your workflow.

TDD Antipatterns: Local Hero

hidden under clouds

This is the second blog post in a series inspired by a Stack Overflow thread. You can read the first post here.

A local hero is defined as:

A test case that is dependent on something specific to the development environment it was written on in order to run. The result is the test passes on development boxes, but fails when someone attempts to run it elsewhere.

Leaving database state aside, this typically arises due to differences in configuration between a developer’s machine and somebody else’s or CI system.

Machines used for CI for web apps usually run in headless mode, which means that they lack a hardware display. Instead, the OS renders the screen as a bitmap in memory. On Linux the most frequently used implementation is Xfvb, which is also used on Semaphore.

In earlier versions of Semaphore’s build platform, the virtual resolution was set to be 1280x800. Eventually we learned that this resolution is not large enough for all users. Some developers that use Selenium to run tests in a real browser were getting mysterious failures due to an expected element not being present on the screen. For example, the navigation bar was implemented as a responsive element, hiding certain links in smaller browser windows. Nowadays the default resolution on Semaphore is 1900x1200.

In case above, the tests were implicitly depending on screen resolution, a rather unexpected component. But sometimes a local hero arises when the application simply was not configured completely. For example, Semaphore’s front-end application tests depend on the existence of a file config/config.yml, which is not stored in version control (because it contains sensitive credentials) but should be copied or created manually during project setup. To make it work, it was necessary to create an encrypted custom configuration file on Semaphore with just enough content to make the tests pass.

A related case is…

The hidden dependency

a unit test that requires some existing data to have been populated somewhere before the test runs. If that data wasn’t populated, the test will fail and leave little indication to the developer what it wanted, or why… forcing them to dig through acres of code to find out where the data it was using was supposed to come from.

Management of database state in tests is best to be to completely automated by your tools. If your tests require you to manually clear the database before each run, look for a tool to do that for you. For example, Ruby developers often use database_cleaner, whose purpose is to maintain a clean state during tests. There’s a similar package for Node.js.

The problem occurs because test runners — or your own code — often wrap tests in a transaction, so other processes, such as the web server, may not be able to access test data which was previously set.

Having established this practice, you also insulate your test suite from the bad practice of requiring some special data to be manually inserted before the tests run.

See also Incidental database state in Rails Testing Antipatterns: Models.

Upcoming Platform Update on July 7th

The upcoming platform update is scheduled for Monday, July 7, 2014.

Elasticsearch is updated to version 1.2.1.

NodeJS gets two updates with 0.8.27 and 0.10.29. These releases of NodeJS address the current OpenSSL vulnerability and introduce a breaking UTF-8 change. Read more about it here.

JRuby is updated to version 1.7.13.

Firefox is bumped up to version 30, requiring bundle update selenium-webdriver for most projects that use that gem. Issues with selenium-webdriver and the latest Firefox can be reported here.

Two PHP version updates are inluded too, namely 5.4.30 and 5.5.14.

Bundler is now updated to version 1.6.3.

As an addition, AWS CLI is now included in our platform.

A full list of changes is available in the platform changelog

TDD Antipatterns: The Free Ride

There is an excellent thread on Stack Overflow TDD Anti-patterns catalogue which contains short descriptions of some common traps we can fall into when writing test-driven code. In the upcoming series of short blog posts, I will illustrate some of them with snippets of Ruby code.

The Free Ride a.k.a Piggyback: rather than write a new test case method to test another/distinct feature/functionality, a new assertion rides along in an existing test case.

In the following spec we are verifying that an event contains a feedback form whenever a user posts some content that indicates that she was present at the event. It comes from a real code base, just a little bit adapted for readability.

describe Event do
  describe "feedback form presence" do

    before do
      @user = create(:user)
      @event = create(:event)
    end

    it "is displayed when user generates an activity" do
      comment = @event.create_comment(:user => @user)
      expect(@event).to include_feedback_form(@user)

      comment.destroy
      expect(@event).not_to include_feedback_form(@user)

      photo = @user.post_photo(@event)
      expect(@event).to include_feedback_form(@user)

      photo.destroy
      expect(@event).not_to include_feedback_form(@user)

      like = @user.like(@event)
      expect(@event).to include_feedback_form(@user)

      like.destroy
      expect(@event).not_to include_feedback_form(@user)
    end
  end
end

What’s the problem here? Aside from unnecessary repetition of expectations that an event without any associated data does not include the feedback form, the description of the test does not match the things it tests. It is vague (“is displayed when user generates an activity”), which naturally comes from the fact that the test contains multiple different expectations. Once we let ourselves go down this path though, it is easy to end up with further complications. This example is actually rather simple; sometimes the expectations may not even appear to be connected when you look at them with fresh eyes, because of a lacking description.

A failure in any of these expectations would then not point to the exact cause of the test failure, which breaks down one of the main benefits of tests — a short feedback loop about our programming decisions. In a moment when more than one expectation would be broken, eg during a refactoring session, you would not be able to get that information because the test runner stops at the first failed expectation.

If we consider this OK, then the logical conclusion would be to make the whole suite a single test with numerous expectations. But the opposite is rather easy to achieve:

describe Event do
  describe "feedback form" do

    before do
      @user = create(:user)
      @event = create(:event)
    end

    context "before all activity" do
      it "is not displayed" do
        expect(@event).not_to include_feedback_form(@user)
      end
    end

    context "user posts a comment" do

      before { @event.create_comment(:user => @user) }

      it "is displayed" do
        expect(@event).to include_feedback_form(@user)
      end
    end

    context "user removes a comment" do

      before do
        comment = @event.create_comment(:user => @user)
        comment.destroy
      end

      it "is not displayed" do
        expect(@event).not_to include_feedback_form(@user)
      end
    end

    context "user posts a photo" do

      before { @user.post_photo(@event) }

      it "is displayed" do
        expect(@event).to include_feedback_form(@user)
      end
    end

    context "user likes it" do

      before { @user.like(@event) }

      it "is displayed" do
        expect(@event).to include_feedback_form(@user)
      end
    end
  end
end

Using different test values in a single test case

A related case, perhaps more a matter of readability, is to avoid grouping different test values in a single test case. For example:

describe City do

  it "generates slug based on name" do
    city = create(:city, :name => "San Sebastián")
    expect(city.slug).to eq("sansebastian")

    city = create(:city, :name => "Maceió")
    expect(city.slug).to eq("maceio")
  end
end

Instead we can follow the approach which would produce the most informative form of test output and feedback during development:

describe City do

  describe "slug, generated from name" do

    context "name is 'San Sebastián'" do
      subject { create(:city, :name => "San Sebastián") }
      its(:slug) { is_expected.to eq("sansebastian") }
    end

    context "name is 'Maceió'" do
      subject { create(:city, :name => "Maceió") }
      its(:slug) { is_expected.to eq("maceio") }
    end
  end
end

Update Jun 25: The examples above could be improved by explicitly stating what they are testing:

describe City do

  describe "slug, generated from name" do

    context "name is 'San Domingo'" do
      subject { create(:city, :name => "San Domingo") }

      it "strips the whitespace and converts to lowercase" do
        expect(subject.slug).to eq("sandomingo") }
      end
    end

    context "name is 'Maceió'" do
      subject { create(:city, :name => "Maceió") }

      it "strips replaces accented character" do
        expect(subject.slug).to eq("maceio") }
      end
    end
  end
end

Get future posts like this one in your inbox.

Follow us on