How to Set Up a Clojure Environment with Ansible

Learn how to provision a Clojure development environment using Ansible, a popular devops automation tool.

Make the testing and deployment of your projects fast, painless, and inexpensive.

Try Semaphore for free

Introduction

In this tutorial, we will walk through the process of setting up a local environment for Clojure development using the Ansible configuration management tool. Ansible and Clojure are a perfect fit, as both place an emphasis on simplicity and building with small, focused components. While we will not cover the Clojure development process specifically, we will learn how to set up an environment that can be used across a whole host of Clojure projects.

Additionally, we will see how using a tool like Ansible will help us implement the Twelve-Factor App methodology. The Twelve-Factor App is a collection of best practices collected by the designers of the Heroku platform.

By the end of this tutorial, you will have a virtualized environment for running a Clojure web application, as well as all supporting software, including MongoDB and RabbitMQ. You will be able to quickly get up and running on multiple development machines. You can even use the same Ansible configuration to provision a production server.

Prerequisites

For the purposes of this tutorial, we will be using Vagrant to spin up several VMs for our development environment. If you have not used Vagrant before, please make sure to install the following packages for your OS before continuing:

Additionally, you will want to have Leiningen installed locally.

Finally, this tutorial uses an existing chat application as an example, so go ahead and clone the repo with this command:

git clone https://github.com/kendru/clojure-chat.git

Unless otherwise specified, the shell commands in the rest of this tutorial should be run from the directory into which you have cloned the repository.

What Makes Ansible Different

With mature configuration management products such as Puppet and Chef available, why would you choose Ansible? It is something of a newcomer to the field, but it has gained a lot of traction since its initial release in 2012. The three things that set Ansible apart are that it is agentless, it uses data-driven configuration, and it is also good for task automation.

Agentless

Unlike Puppet and Chef, Ansible does not require any client software to be installed on the machines that it manages. It only requires Python and SSH, which are included out of the box on every Linux server.

The agentless model has a couple of advantages. First, it is dead simple. The only machine that needs anything installed on is is the one that you will run Ansible from. Additionally, not having any client software installed on the machines you manage means that there are fewer components in your infrastructure that you need to worry about failing.

The simplicity of Ansible's agentless model is a good fit in the Clojure community.

Data-Driven Configuration

Unlike Puppet and Chef, which specify configuration using a programming language, Ansible keeps all configuration in YAML files. At first, this may sound like a drawback, but as we will see later, keeping configuration as data makes for much cleaner, less error-prone codebase.

Once again, Clojure programmers are likely to see the value of using data as the interface to an application. Data is easy to understand, can be manipulated programmatically, and working with it does not require learning a new programming language.

When you adopt Chef, you need to know Ruby. With Puppet, you need to learn the Puppet configuration language. With Ansible, you just need to know how YAML works (if you don't already, you can learn the syntax in about 5 minutes).

Task-Based Automation

In addition to system configuration, Ansible excels at automating repetitive tasks that may need to be run across a number of machines, such as deployments. An Ansible configuration file (called a playbook) is read and executed from top to bottom, as a shell script would be. This allows us to describe a procedure, and then run it on any number of host machines.

For example, you may use something like the following playbook to deploy a standalone Java application that relies on the OS's process manager, such as Upstart or systemd.

---
# deploy.yml
# Deploy a new version of "myapp"
#
# Usage: ansible-playbook deploy.yml --extra-vars "version=1.2.345"

- hosts: appservers
  sudo: yes
  tasks:
    - name: Download new version from S3
      s3: bucket=acme-releases object=/myapp/{{ version }}.jar dest=/opt/bin/myapp/{{ version }}.jar mode=get

   - name: Move symlink to point to new version
     file: src=/opt/bin/myapp/{{ version }}.jar dest=/opt/bin/myapp/deployed.jar state=link force=yes
     notify: Restart myapp

  handlers:
    - name: Restart myapp
      action: service name=myapp state=restarted

This example playbook downloads a package from Amazon S3, creates a symlink, and restarts the system service that runs your application. From this simple playbook, you could deploy your application to dozens — or even hundreds — of machines.

Installing Ansible

If you have not already installed Vagrant and Leiningen, please do so now. The following steps require that both are present on your local machine. We also assume that you already have Python installed. If you are running any flavor of Linux or OSX, you should have Python.

Installing Ansible is a straightforward process. Check out the Ansible docs to see if there is a pre-packaged download available for your OS. Otherwise, you can install with Python's package manager, pip.

sudo easy_install pip
sudo pip install ansible

Now, let's verify that the install was successful:

$ ansible --version
ansible 1.9.1
  configured module search path = None

Provisioning Vagrant with Ansible

Now that all dependencies are installed, it's time to get our Clojure environment set up.

The master branch for this tutorial's git repository contains a completed version of all configuration. If you would like to follow along and build out the playbooks yourself, you can check out the not-provisioned tag:

git checkout -b follow-along tags/not-provisioned

At this point, we want to instruct Vagrant to provision our virtual environment with Ansible. One of the key concepts in Ansible is that of an inventory, which contains named groups of host names or IP addresses so that Ansible can configure these hosts by group name. Thankfully, Vagrant will automatically generate an inventory for us. We just need to specify how to group the VMs by adding the following to our Vagrantfile:

config.vm.provision "ansible" do |ansible|
  ansible.groups = {
    "application" => ["app"],
    "database" => ["infr"],
    "broker" => ["infr"],
    "common:children" => ["application", "database", "broker"]
  }
end

This creates 4 groups of servers, each with a single virtual machine. Notice that both database and broker groups have the same server (infr). This will cause all configuration for both groups to be applied the the same VM.

While we could start up our Vagrant environment now, Ansible would have nothing to do. Let's fix that by writing some plays to provision our environment.

Writing Ansible Plays

Before we dig into the plays that we need for our application dependencies, let's write a simple task to place a message of the day (motd) on each of the servers that will be displayed when the user logs in. We will be using a role-based layout for our Ansible configuration, so let's create a common role and add our config. Your directory structure should look something like the following:

Vagrantfile
...
config/
└── roles
    └── common
        ├── tasks
        └── templates

Next, we'll add a main.yml file to the tasks directory that will define the motd task.

---
# config/roles/common/tasks/main.yml
# Tasks common to all hosts

- name: Install motd
  template: src=motd.j2 dest=/etc/motd owner=root group=root mode=0644

Briefly, this file defines a single task that uses the template module built into Ansible to take a file from this role's templates directory and copy it onto some remote machine, replacing the template variables with data from Ansible.

Along with this task, we'll create the motd.j2 template.

# config/roles/common/templates/motd.j2
Welcome to {{ ansible_hostname }}
This message was installed with Ansible

When Ansible copies this file to each host, it will replace {{ ansible_hostname }} with the DNS host name of the machine that it is installed on. There are quite a few variables that are available to all templates, and you can additionally define your own on a global, per-host, or per-group basis. The official documentation has very complete coverage of the use of variables.

Finally, we need to create the playbook that will apply the task that we just wrote to each of our servers.

---
# config/provision.yml
# Provision development environment

- name: Apply common configuration
  hosts: all
  sudo: yes
  roles:
    - common

In order for Vagrant to use this playbook, we need to add the following line to our Vagrant file in the same block as the Ansible group configuration that we created earlier:

ansible.playbook = "config/provision.yml"

We can now provision our machines. If you have not yet run vagrant up, running that command will download and initialize VirtualBox VMs and provision them with Ansible (this will take a while on the first run). After we run vagrant up initially, we can re-provision the machines with:

$ vagrant provision
# ...
==> app: Running provisioner: ansible...

PLAY [Apply common configuration] *********************************************

GATHERING FACTS ***************************************************************
ok: [app]

TASK: [common | Install motd] ******************************
changed: [app]

PLAY RECAP ********************************************************************
app                        : ok=2    changed=1    unreachable=0    failed=0

If all was successful, you should see output similar to the above for each of the VMs in our environment.

Ansible Play for Application Server

In our infrastructure, the application server will be dedicated to running only the Clojure application itself. The only dependencies for this server are Java and Leiningen, the Clojure build tool. On a production machine, we would probably not install Leiningen, but it will be helpful for us to build and test our application on the VM.

Let's go ahead and create two separate roles called "java" and "lein".

mkdir -p config/roles/{java,lein}/tasks
cat <<EOF | tee config/roles/java/tasks/main.yml config/roles/lein/tasks/main.yml
---
# TODO

EOF

Next, let's add these roles to a play at the end of our playbook.

# config/provision.yml
- name: Set up app server
  hosts:
    - application
  sudo: yes
  roles:
    - java
    - lein

For the purpose of our application, we would like to install the Oracle Java 8 JDK, which is not available from the standard Ubuntu repositories, so we will add a repository from WebUpd8 and use debconf to automatically accept the Oracle Java license, which is normally an interactive process. Thankfully, there are already Ansible modules for adding apt repositories as well as changing debconf settings. See why they say that Ansible has "batteries included"?

---
# config/roles/java/tasks/main.yml
# Install Oracle Java 8

- name: Add WebUpd8 apt repo
  apt_repository: repo='ppa:webupd8team/java'

- name: Accept Java license
  debconf: name=oracle-java8-installer question='shared/accepted-oracle-license-v1-1' value=true vtype=select

- name: Update apt cache
  apt: update_cache=yes

- name: Install Java 8
  apt: name=oracle-java8-installer state=latest

- name: Set Java environment variables
  apt: name=oracle-java8-set-default state=latest

Next, we'll add the task to install Leiningen. Instead of having Ansible download Leiningen directly from the internet, we will download a copy and make it part of our configuration so that we can easily version it:

mkdir config/roles/lein/files
wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein -O config/roles/lein/files/lein

With that done, the actual task for installing Leinengen becomes a one-liner.

---
# config/roles/lein/tasks/main.yml
# Install leiningen

- name: Copy lein script
  copy: src=lein dest=/usr/bin/lein owner=root group=root mode=755

Let's make sure that everything is working:

$ vagrant provision
# ... lots of output
PLAY RECAP ********************************************************************
app                        : ok=9    changed=6    unreachable=0    failed=0

Ansible Plays for Infrastructure Server

Next up, we'll add the play that will set up our infr server with MongoDB and RabbitMQ. We'll create roles for each of these applications, and we'll create plays to apply the mongodb role to servers in the database group, and the rabbitmq role to the servers in the broker group. If you recall, we only have the infr VM in each of those groups, so both roles will be applied to that same server.

We'll set up the role skeletons similar to the way we did with the java and lein roles.

mkdir -p config/roles/{mongodb,rabbitmq}/{tasks,handlers}
cat <<EOF | tee config/roles/mongodb/tasks/main.yml config/roles/rabbitmq/tasks/main.yml
---
# TODO

EOF

This time, we'll add two separate plays to our playbook.

# config/provision.yml
- name: Set up database server
  hosts:
    - database
  sudo: yes
  roles:
    - mongodb

- name: Set up messaging broker server
  hosts:
    - broker
  sudo: yes
  roles:
    - rabbitmq

Next, we'll fill in the main tasks to install MongoDB and RabbitMQ.

---
# config/roles/mongodb/tasks/main.yml
# Install and configure MongoDB

- name: Fetch apt signing key
  apt_key: keyserver=keyserver.ubuntu.com id=7F0CEB10 state=present

- name: Add 10gen Mongo repo
  apt_repository: >
    repo='deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.0 multiverse'
    state=present

- name: Update apt cache
  apt: update_cache=yes

- name: Install MongoDB
  apt: name=mongodb-org state=latest

- name: Start mongod
  service: name=mongod state=started

# Allow connections to mongo from other hosts
- name: Bind mongo to IP
  lineinfile: >
    dest=/etc/mongod.conf
    regexp="^bind_ip ="
    line="bind_ip = {{ mongo_bind_ip }}"
  notify:
    - restart mongod

- name: Install pip (for adding mongo user)
  apt: name=python-pip state=latest

- name: Install pymongo (for adding mongo user)
  pip: name=pymongo state=latest

- name: Add mongo user
  mongodb_user: >
    database={{ mongo_database }}
    name={{ mongo_user }}
    password={{ mongo_password }}
    state=present

This role is a little more complicated than what we have seen so far, but it's still not too bad. We again use built-in Ansible modules to fetch 10Gen's signing key and add their MongoDB repository. We use the service module to start the mongod system service, then we use the lineinfile module to edit a single line in the default MongoDB config file.

You may have noticed that we used several variables in the arguments to a couple of tasks. There are a couple of places that variables can live, but for this tutorial, we will declare these variables globally in config/group_vars/all.yml. The variables in this file will be applied to every group of servers.

mkdir config/group_vars
cat <<EOF > config/group_vars/all.yml
---
# config/group_vars/all.yml
# Variables common to all groups

mongo_bind_ip: 0.0.0.0
mongo_database: clojure-chat
mongo_user: clojure-chat
mongo_password: s3cr3t
EOF

Additionally, we added a notify line that will notify a handler when the task has been run. Handlers are generally used to start and stop system services. Let's go ahead and create the handler now.

cat <<EOF > config/roles/mongodb/handlers/main.yml
---
# config/roles/mongodb/vars/main.yml
# MongoDB service handlers

- name: restart mongod
  service: name=mongod state=restarted
EOF

Since there are no surprises in the configuration for RabbitMQ, we will not go into it here. However, the full config is available in the repo for reference.

Ansible Plays for the Application

In order to follow the 12-factor app pattern, we would like to have all of our configuration stored in environment variables. Our app is already set up to read from HTTP_PORT, MONGODB_URI and RABBITMQ_URI. We'll just write a simple task to add those variables to the vagrant user's login shell.

---
# config/roles/clojure_chat/tasks/main.yml
# Add application environment variables

- name: Add env variables
  template: src=app_env.j2 dest=/home/vagrant/.app_env owner=vagrant group=vagrant

- name: Include env variables in vagrant's login shell
  shell: echo ". /home/vagrant/.app_env" >> /home/vagrant/.bash_profile

And we'll go ahead and create the template that defines these variables:

# config/roles/clojure_chat/templates/app_env.j2
# {{ ansible_managed }}

export HTTP_PORT="{{ backend_port }}"

# We are including the auth mechanism because Ansible's mongodb_user
# module creates users with the SCRAM-SHA-1 method by default
export MONGODB_URI="mongodb://{{ mongo_user }}:{{ mongo_password }}@{{ database_ip }}/{{ mongo_database }}?authMechanism=SCRAM-SHA-1"

export RABBITMQ_URI="amqp://{{ rmq_user }}:{{ rmq_password }}@{{ broker_ip }}:5672{% if rmq_vhost == '/' %}/%2f{% else %}{{ rmq_vhost }}{% endif %}"

Verify the Configuration

We just wrote a lot of configuration, but now we have a fully-reproduceable development environment for a Clojure application that utilizes a messaging queue and a noSQL database.

Let's make sure that everything works by provisioning again and firing up the repl.

vagrant provision
vagrant ssh app
cd /vagrant
lein repl

The repl will load our clojure-chat.main namespace, where we can start up the server.

;; REPL
clojure-chat.main=> (-main)

Now a web server should be running inside our VM on port 3000. We can check it out by visiting http://10.0.15.12:3000/ in a browser.

At this point you have successfully set up a complete, multi-machine development environment for Clojure using Ansible!

5c8e523e613b720fec1291442702a708
Andrew Meredith

A software professional with a passion for simplicity and an interest in functional programming. Blogs at http://kendru.github.io.

on this tutorial so far.
User deleted author {{comment.createdAt}}

Edited on {{comment.updatedAt}}

Cancel

Sign In You must be logged in to comment.