Introduction
In this tutorial, we will walk through the process of setting up a local environment for Clojure development using the Ansible configuration management tool. Ansible and Clojure are a perfect fit, as both place an emphasis on simplicity and building with small, focused components. While we will not cover the Clojure development process specifically, we will learn how to set up an environment that can be used across a whole host of Clojure projects.
Additionally, we will see how using a tool like Ansible will help us implement the Twelve-Factor App methodology. The Twelve-Factor App is a collection of best practices collected by the designers of the Heroku platform.
By the end of this tutorial, you will have a virtualized environment for running a Clojure web application, as well as all supporting software, including MongoDB and RabbitMQ. You will be able to quickly get up and running on multiple development machines. You can even use the same Ansible configuration to provision a production server.
Prerequisites
For the purposes of this tutorial, we will be using Vagrant to spin up several VMs for our development environment. If you have not used Vagrant before, please make sure to install the following packages for your OS before continuing:
Additionally, you will want to have Leiningen installed locally.
Finally, this tutorial uses an existing chat application as an example, so go ahead and clone the repo with this command:
git clone https://github.com/kendru/clojure-chat.git
Unless otherwise specified, the shell commands in the rest of this tutorial should be run from the directory into which you have cloned the repository.
What Makes Ansible Different
With mature configuration management products such as Puppet and Chef available, why would you choose Ansible? It is something of a newcomer to the field, but it has gained a lot of traction since its initial release in 2012. The three things that set Ansible apart are that it is agentless, it uses data-driven configuration, and it is also good for task automation.
Agentless
Unlike Puppet and Chef, Ansible does not require any client software to be installed on the machines that it manages. It only requires Python and SSH, which are included out of the box on every Linux server.
The agentless model has a couple of advantages. First, it is dead simple. The only machine that needs anything installed on is is the one that you will run Ansible from. Additionally, not having any client software installed on the machines you manage means that there are fewer components in your infrastructure that you need to worry about failing.
The simplicity of Ansible’s agentless model is a good fit in the Clojure community.
Data-Driven Configuration
Unlike Puppet and Chef, which specify configuration using a programming language, Ansible keeps all configuration in YAML files. At first, this may sound like a drawback, but as we will see later, keeping configuration as data makes for much cleaner, less error-prone codebase.
Once again, Clojure programmers are likely to see the value of using data as the interface to an application. Data is easy to understand, can be manipulated programmatically, and working with it does not require learning a new programming language.
When you adopt Chef, you need to know Ruby. With Puppet, you need to learn the Puppet configuration language. With Ansible, you just need to know how YAML works (if you don’t already, you can learn the syntax in about 5 minutes).
Task-Based Automation
In addition to system configuration, Ansible excels at automating repetitive tasks that may need to be run across a number of machines, such as deployments. An Ansible configuration file (called a playbook) is read and executed from top to bottom, as a shell script would be. This allows us to describe a procedure, and then run it on any number of host machines.
For example, you may use something like the following playbook to deploy a standalone Java application that relies on the OS’s process manager, such as Upstart or systemd.
---
# deploy.yml
# Deploy a new version of "myapp"
#
# Usage: ansible-playbook deploy.yml --extra-vars "version=1.2.345"
- hosts: appservers
sudo: yes
tasks:
- name: Download new version from S3
s3: bucket=acme-releases object=/myapp/{{ version }}.jar dest=/opt/bin/myapp/{{ version }}.jar mode=get
- name: Move symlink to point to new version
file: src=/opt/bin/myapp/{{ version }}.jar dest=/opt/bin/myapp/deployed.jar state=link force=yes
notify: Restart myapp
handlers:
- name: Restart myapp
action: service name=myapp state=restarted
This example playbook downloads a package from Amazon S3, creates a symlink, and restarts the system service that runs your application. From this simple playbook, you could deploy your application to dozens — or even hundreds — of machines.
Installing Ansible
If you have not already installed Vagrant and Leiningen, please do so now. The following steps require that both are present on your local machine. We also assume that you already have Python installed. If you are running any flavor of Linux or OSX, you should have Python.
Installing Ansible is a straightforward process. Check out the Ansible docs to see if there is a pre-packaged download available for your OS. Otherwise, you can install with Python’s package manager, pip.
sudo easy_install pip
sudo pip install ansible
Now, let’s verify that the install was successful:
$ ansible --version
ansible 1.9.1
configured module search path = None
Provisioning Vagrant with Ansible
Now that all dependencies are installed, it’s time to get our Clojure environment set up.
The master branch for this tutorial’s git repository contains a completed version of all configuration. If you would like to follow along and build out the playbooks yourself, you can check out the not-provisioned tag:
git checkout -b follow-along tags/not-provisioned
At this point, we want to instruct Vagrant to provision our virtual environment with Ansible. One of the key concepts in Ansible is that of an inventory, which contains named groups of host names or IP addresses so that Ansible can configure these hosts by group name. Thankfully, Vagrant will automatically generate an inventory for us. We just need to specify how to group the VMs by adding the following to our Vagrantfile:
config.vm.provision "ansible" do |ansible|
ansible.groups = {
"application" => ["app"],
"database" => ["infr"],
"broker" => ["infr"],
"common:children" => ["application", "database", "broker"]
}
end
This creates 4 groups of servers, each with a single virtual machine. Notice that both database and broker groups have the same server (infr). This will cause all configuration for both groups to be applied the the same VM.
While we could start up our Vagrant environment now, Ansible would have nothing to do. Let’s fix that by writing some plays to provision our environment.
Writing Ansible Plays
Before we dig into the plays that we need for our application dependencies, let’s write a simple task to place a message of the day (motd) on each of the servers that will be displayed when the user logs in. We will be using a role-based layout for our Ansible configuration, so let’s create a common role and add our config. Your directory structure should look something like the following:
Vagrantfile
...
config/
└── roles
└── common
├── tasks
└── templates
Next, we’ll add a main.yml file to the tasks directory that will define the motd task.
---
# config/roles/common/tasks/main.yml
# Tasks common to all hosts
- name: Install motd
template: src=motd.j2 dest=/etc/motd owner=root group=root mode=0644
Briefly, this file defines a single task that uses the template module built into Ansible to take a file from this role’s templates directory and copy it onto some remote machine, replacing the template variables with data from Ansible.
Along with this task, we’ll create the motd.j2 template.
# config/roles/common/templates/motd.j2
Welcome to {{ ansible_hostname }}
This message was installed with Ansible
When Ansible copies this file to each host, it will replace {{ ansible_hostname }} with the DNS host name of the machine that it is installed on. There are quite a few variables that are available to all templates, and you can additionally define your own on a global, per-host, or per-group basis. The official documentation has very complete coverage of the use of variables.
Finally, we need to create the playbook that will apply the task that we just wrote to each of our servers.
---
# config/provision.yml
# Provision development environment
- name: Apply common configuration
hosts: all
sudo: yes
roles:
- common
In order for Vagrant to use this playbook, we need to add the following line to our Vagrant file in the same block as the Ansible group configuration that we created earlier:
ansible.playbook = "config/provision.yml"
We can now provision our machines. If you have not yet run vagrant up, running that command will download and initialize VirtualBox VMs and provision them with Ansible (this will take a while on the first run). After we run vagrant up initially, we can re-provision the machines with:
$ vagrant provision
# ...
==> app: Running provisioner: ansible...
PLAY [Apply common configuration] *********************************************
GATHERING FACTS ***************************************************************
ok: [app]
TASK: [common | Install motd] ******************************
changed: [app]
PLAY RECAP ********************************************************************
app : ok=2 changed=1 unreachable=0 failed=0
If all was successful, you should see output similar to the above for each of the VMs in our environment.
Ansible Play for Application Server
In our infrastructure, the application server will be dedicated to running only the Clojure application itself. The only dependencies for this server are Java and Leiningen, the Clojure build tool. On a production machine, we would probably not install Leiningen, but it will be helpful for us to build and test our application on the VM.
Let’s go ahead and create two separate roles called “java” and “lein”.
mkdir -p config/roles/{java,lein}/tasks
cat <<EOF | tee config/roles/java/tasks/main.yml config/roles/lein/tasks/main.yml
---
# TODO
EOF
Next, let’s add these roles to a play at the end of our playbook.
# config/provision.yml
- name: Set up app server
hosts:
- application
sudo: yes
roles:
- java
- lein
For the purpose of our application, we would like to install the Oracle Java 8 JDK, which is not available from the standard Ubuntu repositories, so we will add a repository from WebUpd8 and use debconf to automatically accept the Oracle Java license, which is normally an interactive process. Thankfully, there are already Ansible modules for adding apt repositories as well as changing debconf settings. See why they say that Ansible has “batteries included”?
---
# config/roles/java/tasks/main.yml
# Install Oracle Java 8
- name: Add WebUpd8 apt repo
apt_repository: repo='ppa:webupd8team/java'
- name: Accept Java license
debconf: name=oracle-java8-installer question='shared/accepted-oracle-license-v1-1' value=true vtype=select
- name: Update apt cache
apt: update_cache=yes
- name: Install Java 8
apt: name=oracle-java8-installer state=latest
- name: Set Java environment variables
apt: name=oracle-java8-set-default state=latest
Next, we’ll add the task to install Leiningen. Instead of having Ansible download Leiningen directly from the internet, we will download a copy and make it part of our configuration so that we can easily version it:
mkdir config/roles/lein/files
wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein -O config/roles/lein/files/lein
With that done, the actual task for installing Leinengen becomes a one-liner.
---
# config/roles/lein/tasks/main.yml
# Install leiningen
- name: Copy lein script
copy: src=lein dest=/usr/bin/lein owner=root group=root mode=755
Let’s make sure that everything is working:
$ vagrant provision
# ... lots of output
PLAY RECAP ********************************************************************
app : ok=9 changed=6 unreachable=0 failed=0
Ansible Plays for Infrastructure Server
Next up, we’ll add the play that will set up our infr server with MongoDB and RabbitMQ. We’ll create roles for each of these applications, and we’ll create plays to apply the mongodb role to servers in the database group, and the rabbitmq role to the servers in the broker group. If you recall, we only have the infr VM in each of those groups, so both roles will be applied to that same server.
We’ll set up the role skeletons similar to the way we did with the java and lein roles.
mkdir -p config/roles/{mongodb,rabbitmq}/{tasks,handlers}
cat <<EOF | tee config/roles/mongodb/tasks/main.yml config/roles/rabbitmq/tasks/main.yml
---
# TODO
EOF
This time, we’ll add two separate plays to our playbook.
# config/provision.yml
- name: Set up database server
hosts:
- database
sudo: yes
roles:
- mongodb
- name: Set up messaging broker server
hosts:
- broker
sudo: yes
roles:
- rabbitmq
Next, we’ll fill in the main tasks to install MongoDB and RabbitMQ.
---
# config/roles/mongodb/tasks/main.yml
# Install and configure MongoDB
- name: Fetch apt signing key
apt_key: keyserver=keyserver.ubuntu.com id=7F0CEB10 state=present
- name: Add 10gen Mongo repo
apt_repository: >
repo='deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.0 multiverse'
state=present
- name: Update apt cache
apt: update_cache=yes
- name: Install MongoDB
apt: name=mongodb-org state=latest
- name: Start mongod
service: name=mongod state=started
# Allow connections to mongo from other hosts
- name: Bind mongo to IP
lineinfile: >
dest=/etc/mongod.conf
regexp="^bind_ip ="
line="bind_ip = {{ mongo_bind_ip }}"
notify:
- restart mongod
- name: Install pip (for adding mongo user)
apt: name=python-pip state=latest
- name: Install pymongo (for adding mongo user)
pip: name=pymongo state=latest
- name: Add mongo user
mongodb_user: >
database={{ mongo_database }}
name={{ mongo_user }}
password={{ mongo_password }}
state=present
This role is a little more complicated than what we have seen so far, but it’s still not too bad. We again use built-in Ansible modules to fetch 10Gen’s signing key and add their MongoDB repository. We use the service module to start the mongod system service, then we use the lineinfile module to edit a single line in the default MongoDB config file.
You may have noticed that we used several variables in the arguments to a couple of tasks. There are a couple of places that variables can live, but for this tutorial, we will declare these variables globally in config/group_vars/all.yml. The variables in this file will be applied to every group of servers.
mkdir config/group_vars
cat <<EOF > config/group_vars/all.yml
---
# config/group_vars/all.yml
# Variables common to all groups
mongo_bind_ip: 0.0.0.0
mongo_database: clojure-chat
mongo_user: clojure-chat
mongo_password: s3cr3t
EOF
Additionally, we added a notify line that will notify a handler when the task has been run. Handlers are generally used to start and stop system services. Let’s go ahead and create the handler now.
cat <<EOF > config/roles/mongodb/handlers/main.yml
---
# config/roles/mongodb/vars/main.yml
# MongoDB service handlers
- name: restart mongod
service: name=mongod state=restarted
EOF
Since there are no surprises in the configuration for RabbitMQ, we will not go into it here. However, the full config is available in the repo for reference.
Ansible Plays for the Application
In order to follow the 12-factor app pattern, we would like to have all of our configuration stored in environment variables. Our app is already set up to read from HTTP_PORT, MONGODB_URI and RABBITMQ_URI. We’ll just write a simple task to add those variables to the vagrant user’s login shell.
---
# config/roles/clojure_chat/tasks/main.yml
# Add application environment variables
- name: Add env variables
template: src=app_env.j2 dest=/home/vagrant/.app_env owner=vagrant group=vagrant
- name: Include env variables in vagrant's login shell
shell: echo ". /home/vagrant/.app_env" >> /home/vagrant/.bash_profile
And we’ll go ahead and create the template that defines these variables:
# config/roles/clojure_chat/templates/app_env.j2
# {{ ansible_managed }}
export HTTP_PORT="{{ backend_port }}"
# We are including the auth mechanism because Ansible's mongodb_user
# module creates users with the SCRAM-SHA-1 method by default
export MONGODB_URI="mongodb://{{ mongo_user }}:{{ mongo_password }}@{{ database_ip }}/{{ mongo_database }}?authMechanism=SCRAM-SHA-1"
export RABBITMQ_URI="amqp://{{ rmq_user }}:{{ rmq_password }}@{{ broker_ip }}:5672{% if rmq_vhost == '/' %}/%2f{% else %}{{ rmq_vhost }}{% endif %}"
Verify the Configuration
We just wrote a lot of configuration, but now we have a fully-reproduceable development environment for a Clojure application that utilizes a messaging queue and a noSQL database.
Let’s make sure that everything works by provisioning again and firing up the repl.
vagrant provision
vagrant ssh app
cd /vagrant
lein repl
The repl will load our clojure-chat.main namespace, where we can start up the server.
;; REPL
clojure-chat.main=> (-main)
Now a web server should be running inside our VM on port 3000. We can check it out by visiting http://10.0.15.12:3000/ in a browser.
At this point you have successfully set up a complete, multi-machine development environment for Clojure using Ansible!