Introduction
A big obstacle to the mission to treat infrastructure as code is the way we test it. In the past, the practice of writing automated tests for infrastructure did not exist. Instead, the process was manual, repetitive and error-prone: we would log into a remote machine and run various commands to verify the system.
A big drawback of this process is that you must have a list of everything that needs to be checked. When you add to this a team of people, evolving and scaling your system becomes very difficult. On the other hand, if you do have tests, but they are hard to run and aren’t automated, the team usually ends up ignoring them.
With the development of configuration management tools like Chef or Puppet, the process of testing got a new face. Thanks to automatization, tests are now becoming easier to run, more reliable and, most importantly, much faster.
In this tutorial, we will cover writing integration tests for servers managed by Chef. For that purpose, we will use Test Kitchen as a test runner for virtual machines, backed by Vagrant and VirtualBox. The actual test will be written in Ruby using the Serverspec testing framework.
Prerequisites
Since we are working with Chef-managed infrastructure, we need to have installed the standard environment that comes with the The Chef Development Kit. It includes all essential tools for managing the code that runs your infrastructure, including tools for testing it, such as Test Kitchen. For more details about the setup of the Chef development environment, you can always consult the intro tutorial in this series, Infrastructure as Code with Chef: Introduction to Chef.
Overview of the Testing Example
For the purpose of this tutorial, we will use a basic cookbook for installing Vim from source, or from official distribution packages. We will not cover the actual code of this cookbook, since in integration testing we only care about the results of the execution, and the actual implementation is less significant.
We will test two scenarios, one that includes the recipe for installing Vim from packages, and one that will install it from the source.
Test Kitchen
Test Kitchen is a test automatization tool distributed with ChefDK. It manages virtual machines, internally called nodes, applies the Chef configuration and runs tests for you. It can be used in a fully automated way, where all the steps are executed one after another, including destroying a node at the end of a run. Running tasks manually is also an option.
When running integration tests, you can and should use the same Chef configuration as the one you run on a real server – the same run list, recipes, roles and attributes. Optionally, you can provide custom attributes used only in the test environment, like fake data for example.
A Node can be represented with any type of virtualization via Test Kitchen’s plugins, called drivers. In most situations it’s the default Vagrant, but there are also alternatives such as Docker, or one of many cloud providers like Amazon Web Services or DigitalOcean. The full list of supported drivers can be found by running the kitchen driver discover command.
All settings for Test Kitchen are defined inside the .kitchen.yml file in your repository root.
Note that, although we will be using Test Kitchen on a single cookbook, it can generally be used to test a complete run list that is associated with your machine type.
Setting up Test Kitchen
Let’s test our cookbook “tdi-example-vim” and its recipes with the chef_solo provisioner on Ubuntu 14.04 and CentOS 7.0 platforms with Vagrant as driver. Here’s what our configuration file should look like:
# .kitchen.yml
---
driver:
name: vagrant
provisioner:
name: chef_solo
platforms:
- name: ubuntu-14.04
- name: centos-7.0
suites:
- name: source
run_list:
- recipe[tdi-example-vim::default]
attributes:
vim:
install_method: "source"
- name: package
run_list:
- recipe[tdi-example-vim::default]
attributes:
vim:
install_method: "package"
To verify setup and see an overview of all available instances provided by Test Kitchen, just run:
$ kitchen list
Instance Driver Provisioner Last Action
source-ubuntu-1404 Vagrant ChefSolo <Not Created>
source-centos-70 Vagrant ChefSolo <Not Created>
package-ubuntu-1404 Vagrant ChefSolo <Not Created>
package-centos-70 Vagrant ChefSolo <Not Created>
As you can see, Test Kitchen is providing us with four instances, each suite against every platform defined in a configuration file. These instances have not been created yet, so let’s write some tests and apply them to those instances.
Verifying a Chef Run with Serverspec
Now that we have a complete infrastructure for running integration tests, we need to set our expectations and find a way to verify them. As Test Kitchen is solely an automatization tool, it does not provide us with a way to write tests. We need another tool for that purpose.
Serverspec is a testing framework based on RSpec, a testing framework that is popular in the Ruby community. It allows you to write RSpec-style tests for checking infrastructure configuration, no matter how it’s provisioned in the first place. With tools like this, there isn’t much room for human error, and the testing done in a descriptive manner.
Serverspec Setup
To start writing Serverspec tests, you need to create a directory named integration inside the test directory, where all the integration tests live. This directory should contain a subdirectory for each testing framework we’ll use, which means that you are able to use many testing frameworks as you want on the same suite, without any collision.
.
`-- test
`-- integration
|-- #{SUITE 1}
| `-- #{TEST FRAMEWORK}
`-- #{SUITE 2}
`-- #{TEST FRAMEWORK}
...
We intend to use serverspec as the framework of choice, so let’s create a corresponding directory structure for it:
mkdir -p test/integration/source/serverspec
mkdir -p test/integration/package/serverspec
The first thing we need is a Ruby helper script which loads Serverspec and sets the general configuration options, like the path used by the binary while executing tests.
# test/integration/package/serverspec/spec_helper.rb
require 'serverspec'
require 'pathname'
set :backend, :exec
set :path, '/bin:/usr/local/bin:$PATH'
With this setup in place, you can now require it at the beginning of every spec file you write, so you don’t need to repeat the setup steps for every new spec file. Since the tests are written in plain Ruby it’s done in the standard way:
# test/integration/package/serverspec/vim_spec.rb
require 'spec_helper'
Writing Serverspec Tests
The first and the most basic thing that we can test is whether our provisioner has installed the package that we are expecting. If you did this manually, you would probably just run the vim command to check if the program is successfully launched. With Serverspec, there is a much nicer way to check specific resources like commands.
Beside commands, Serverspec can also check the status of packages, which we can use in this case, since our example used packages provided by our distribution of choice. The list of the benefits of using Serverspec does not end there, it provides us with a lot of additional resources and matchers which can test a lot of other stuff, like, for example, services or ports.
You can find the full list of resource types with examples on the Serverspec documentation site.
In the RSpec manner, every test should include the standard describe / it structure for defining test cases. We can describe the status of the package using ‘vim’, and expect it to be installed:
# test/integration/package/serverspec/vim_spec.rb
...
describe package('vim') do
it { should be_installed }
end
Similarly to this, we can also check other resources, like the output of a command, or just its exit status. For our use case, a useful thing to test is the installed version of our application. Just like when doing it manually, you can check the output of the vim --version command and see if it returns the desired version — in this case it’s expected to be 7.4.
# test/integration/package/serverspec/vim_spec.rb
...
describe command('vim --version') do
its(:stdout) { should match /VIM - Vi IMproved 7.4/ }
end
With the first set of tests defined, it’s time to run them. This can be done via the knife test command, followed by the name of the instance that we want to test. A list of all available instances can be found with the knife list command. Let’s check the package installation against Ubuntu 14.04:
$ kitchen test package-ubuntu-1404
-----> Starting Kitchen (v1.2.1)
-----> Cleaning up any prior instances of <package-ubuntu-1404>
-----> Destroying <package-ubuntu-1404>...
==> default: Forcing shutdown of VM...
==> default: Destroying VM and associated drives...
Vagrant instance <package-ubuntu-1404> destroyed.
Finished destroying <package-ubuntu-1404> (0m6.14s).
-----> Testing <package-ubuntu-1404>
-----> Creating <package-ubuntu-1404>...
Bringing machine 'default' up with 'virtualbox' provider...
....
Package "vim"
should be installed
Finished in 0.25537 seconds (files took 0.44389 seconds to load)
1 example, 0 failures
Finished verifying <package-ubuntu-1404> (0m23.01s).
-----> Destroying <package-ubuntu-1404>...
==> default: Forcing shutdown of VM...
==> default: Destroying VM and associated drives...
Vagrant instance <package-ubuntu-1404> destroyed.
Finished destroying <package-ubuntu-1404> (0m5.00s).
Finished testing <package-ubuntu-1404> (2m55.08s).
-----> Kitchen is finished. (2m55.53s)
As you can see, Test Kitchen has created a new virtual machine, provisioned it with a specified run list, applied tests against it, and destroyed the machine after the run.
As previously mentioned, this is a completely automated process, but in the development phase it would just be too slow to run this process again and again, so we can create (or converge, in Test Kitchen terminology) an instance manually, and then only apply tests after each change is made, so we don’t need to wait for converging every single time. When tests are passed and we are done with them, we will just destroy the instances we don’t need.
Let’s converge (create and provision) a new instance manually again, and add some more tests.
$ kitchen converge package-ubuntu-1404
-----> Starting Kitchen (v1.2.1)
-----> Creating <package-ubuntu-1404>...
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'opscode-ubuntu-14.04'...
...
[2015-02-23T09:40:34+00:00] INFO: Chef Run complete in 17.832076516 seconds
[2015-02-23T09:40:34+00:00] INFO: Skipping removal of unused files from the cache
Running handlers:
[2015-02-23T09:40:34+00:00] INFO: Running report handlers
Running handlers complete
[2015-02-23T09:40:34+00:00] INFO: Report handlers complete
Chef Client finished, 1/1 resources updated in 24.579074799 seconds
Finished converging <package-ubuntu-1404> (1m2.91s).
-----> Kitchen is finished. (1m49.79s)
Now that you have a provisioned instance, let’s improve the tests to support multiple platforms we defined in the setup phase. Current tests are not applicable for platforms where the name of the package is not the same as on Ubuntu, for example on CentOS it’s called ‘vim-minimal’. The tests are written in plain Ruby so you can set the conditions against the OS family for example, which can be used to write separate test cases for Ubuntu and CentOS.
# test/integration/package/serverspec/vim_spec.rb
require 'spec_helper'
if os[:family] == 'ubuntu'
describe package('vim') do
it { should be_installed }
end
end
if os[:family] == 'redhat'
describe package('vim-minimal') do
it { should be_installed }
end
describe package('vim-enhanced') do
it { should be_installed }
end
end
describe command('vim --version') do
its(:stdout) { should match /VIM - Vi IMproved 7.4/ }
end
Because the instance is already converged in the step above, you can easily apply new changes to it and check test results.
kitchen verify package-ubuntu-1404
Since the tests passed and you’re done with improving them, it’s safe to destroy the node.
kitchen destroy package-ubuntu-1404
The tests needed for the installation of our favorite editor from packages are done. You can apply similar techniques for testing recipes that compile the application from the source. Tests need to check if the desired version is installed, and which user has compiled it. With vim, this can be done using one command, vim--version, which prints both of these pieces of information.
# test/integration/source/serverspec/vim_spec.rb
require 'spec_helper'
describe command('vim --version') do
its(:stdout) { should match /VIM - Vi IMproved 7.4/ }
its(:stdout) { should match /Compiled by vagrant@source-/ }
end
To verify them, run them in the standard way, against corresponding node.
kitchen test package-ubuntu-1404
Once all the work is finished, you should be able to run all the tests with the kitchen test command, without providing any additional argument, and it will spawn, test and destroy every node defined in the .kitchen.yml file.
If you get stuck in the process, want to test just one command, or manually explore how to test something, you can always log into the converged instance and execute commands. You can do this using the following commands:
kitchen converge package-ubuntu-1404 # converge instance if you do not have one
kitchen login package-ubuntu-1404
Wrapping Up
Test Kitchen provides an easy way to automate the process of describing and testing server infrastructure. With it, you can run integration tests against infrastructure on every change in your infrastructure stack, development machine, or on the CI service, and apply all of the benefits of Continuous Integration to your infrastructure code.
The full source from this tutorial can be found on GitHub. If you’re interested, you can also follow the steps taken through individual commits in the repository.