1 Jul 2015 · Software Engineering

    How to Set Up a Clojure Environment with Ansible

    15 min read
    Contents

    Introduction

    In this tutorial, we will walk through the process of setting up a local environment for Clojure development using the Ansible configuration management tool. Ansible and Clojure are a perfect fit, as both place an emphasis on simplicity and building with small, focused components. While we will not cover the Clojure development process specifically, we will learn how to set up an environment that can be used across a whole host of Clojure projects.

    Additionally, we will see how using a tool like Ansible will help us implement the Twelve-Factor App methodology. The Twelve-Factor App is a collection of best practices collected by the designers of the Heroku platform.

    By the end of this tutorial, you will have a virtualized environment for running a Clojure web application, as well as all supporting software, including MongoDB and RabbitMQ. You will be able to quickly get up and running on multiple development machines. You can even use the same Ansible configuration to provision a production server.

    Prerequisites

    For the purposes of this tutorial, we will be using Vagrant to spin up several VMs for our development environment. If you have not used Vagrant before, please make sure to install the following packages for your OS before continuing:

    Additionally, you will want to have Leiningen installed locally.

    Finally, this tutorial uses an existing chat application as an example, so go ahead and clone the repo with this command:

    git clone https://github.com/kendru/clojure-chat.git

    Unless otherwise specified, the shell commands in the rest of this tutorial should be run from the directory into which you have cloned the repository.

    What Makes Ansible Different

    With mature configuration management products such as Puppet and Chef available, why would you choose Ansible? It is something of a newcomer to the field, but it has gained a lot of traction since its initial release in 2012. The three things that set Ansible apart are that it is agentless, it uses data-driven configuration, and it is also good for task automation.

    Agentless

    Unlike Puppet and Chef, Ansible does not require any client software to be installed on the machines that it manages. It only requires Python and SSH, which are included out of the box on every Linux server.

    The agentless model has a couple of advantages. First, it is dead simple. The only machine that needs anything installed on is is the one that you will run Ansible from. Additionally, not having any client software installed on the machines you manage means that there are fewer components in your infrastructure that you need to worry about failing.

    The simplicity of Ansible’s agentless model is a good fit in the Clojure community.

    Data-Driven Configuration

    Unlike Puppet and Chef, which specify configuration using a programming language, Ansible keeps all configuration in YAML files. At first, this may sound like a drawback, but as we will see later, keeping configuration as data makes for much cleaner, less error-prone codebase.

    Once again, Clojure programmers are likely to see the value of using data as the interface to an application. Data is easy to understand, can be manipulated programmatically, and working with it does not require learning a new programming language.

    When you adopt Chef, you need to know Ruby. With Puppet, you need to learn the Puppet configuration language. With Ansible, you just need to know how YAML works (if you don’t already, you can learn the syntax in about 5 minutes).

    Task-Based Automation

    In addition to system configuration, Ansible excels at automating repetitive tasks that may need to be run across a number of machines, such as deployments. An Ansible configuration file (called a playbook) is read and executed from top to bottom, as a shell script would be. This allows us to describe a procedure, and then run it on any number of host machines.

    For example, you may use something like the following playbook to deploy a standalone Java application that relies on the OS’s process manager, such as Upstart or systemd.

    ---
    # deploy.yml
    # Deploy a new version of "myapp"
    #
    # Usage: ansible-playbook deploy.yml --extra-vars "version=1.2.345"
    
    - hosts: appservers
      sudo: yes
      tasks:
        - name: Download new version from S3
          s3: bucket=acme-releases object=/myapp/{{ version }}.jar dest=/opt/bin/myapp/{{ version }}.jar mode=get
    
       - name: Move symlink to point to new version
         file: src=/opt/bin/myapp/{{ version }}.jar dest=/opt/bin/myapp/deployed.jar state=link force=yes
         notify: Restart myapp
    
      handlers:
        - name: Restart myapp
          action: service name=myapp state=restarted

    This example playbook downloads a package from Amazon S3, creates a symlink, and restarts the system service that runs your application. From this simple playbook, you could deploy your application to dozens — or even hundreds — of machines.

    Installing Ansible

    If you have not already installed Vagrant and Leiningen, please do so now. The following steps require that both are present on your local machine. We also assume that you already have Python installed. If you are running any flavor of Linux or OSX, you should have Python.

    Installing Ansible is a straightforward process. Check out the Ansible docs to see if there is a pre-packaged download available for your OS. Otherwise, you can install with Python’s package manager, pip.

    sudo easy_install pip
    sudo pip install ansible

    Now, let’s verify that the install was successful:

    $ ansible --version
    ansible 1.9.1
      configured module search path = None

    Provisioning Vagrant with Ansible

    Now that all dependencies are installed, it’s time to get our Clojure environment set up.

    The master branch for this tutorial’s git repository contains a completed version of all configuration. If you would like to follow along and build out the playbooks yourself, you can check out the not-provisioned tag:

    git checkout -b follow-along tags/not-provisioned

    At this point, we want to instruct Vagrant to provision our virtual environment with Ansible. One of the key concepts in Ansible is that of an inventory, which contains named groups of host names or IP addresses so that Ansible can configure these hosts by group name. Thankfully, Vagrant will automatically generate an inventory for us. We just need to specify how to group the VMs by adding the following to our Vagrantfile:

    config.vm.provision "ansible" do |ansible|
      ansible.groups = {
        "application" => ["app"],
        "database" => ["infr"],
        "broker" => ["infr"],
        "common:children" => ["application", "database", "broker"]
      }
    end

    This creates 4 groups of servers, each with a single virtual machine. Notice that both database and broker groups have the same server (infr). This will cause all configuration for both groups to be applied the the same VM.

    While we could start up our Vagrant environment now, Ansible would have nothing to do. Let’s fix that by writing some plays to provision our environment.

    Writing Ansible Plays

    Before we dig into the plays that we need for our application dependencies, let’s write a simple task to place a message of the day (motd) on each of the servers that will be displayed when the user logs in. We will be using a role-based layout for our Ansible configuration, so let’s create a common role and add our config. Your directory structure should look something like the following:

    Vagrantfile
    ...
    config/
    └── roles
        └── common
            ├── tasks
            └── templates

    Next, we’ll add a main.yml file to the tasks directory that will define the motd task.

    ---
    # config/roles/common/tasks/main.yml
    # Tasks common to all hosts
    
    - name: Install motd
      template: src=motd.j2 dest=/etc/motd owner=root group=root mode=0644

    Briefly, this file defines a single task that uses the template module built into Ansible to take a file from this role’s templates directory and copy it onto some remote machine, replacing the template variables with data from Ansible.

    Along with this task, we’ll create the motd.j2 template.

    # config/roles/common/templates/motd.j2
    Welcome to {{ ansible_hostname }}
    This message was installed with Ansible

    When Ansible copies this file to each host, it will replace {{ ansible_hostname }} with the DNS host name of the machine that it is installed on. There are quite a few variables that are available to all templates, and you can additionally define your own on a global, per-host, or per-group basis. The official documentation has very complete coverage of the use of variables.

    Finally, we need to create the playbook that will apply the task that we just wrote to each of our servers.

    ---
    # config/provision.yml
    # Provision development environment
    
    - name: Apply common configuration
      hosts: all
      sudo: yes
      roles:
        - common

    In order for Vagrant to use this playbook, we need to add the following line to our Vagrant file in the same block as the Ansible group configuration that we created earlier:

    ansible.playbook = "config/provision.yml"

    We can now provision our machines. If you have not yet run vagrant up, running that command will download and initialize VirtualBox VMs and provision them with Ansible (this will take a while on the first run). After we run vagrant up initially, we can re-provision the machines with:

    $ vagrant provision
    # ...
    ==> app: Running provisioner: ansible...
    
    PLAY [Apply common configuration] *********************************************
    
    GATHERING FACTS ***************************************************************
    ok: [app]
    
    TASK: [common | Install motd] ******************************
    changed: [app]
    
    PLAY RECAP ********************************************************************
    app                        : ok=2    changed=1    unreachable=0    failed=0

    If all was successful, you should see output similar to the above for each of the VMs in our environment.

    Ansible Play for Application Server

    In our infrastructure, the application server will be dedicated to running only the Clojure application itself. The only dependencies for this server are Java and Leiningen, the Clojure build tool. On a production machine, we would probably not install Leiningen, but it will be helpful for us to build and test our application on the VM.

    Let’s go ahead and create two separate roles called “java” and “lein”.

    mkdir -p config/roles/{java,lein}/tasks
    cat <<EOF | tee config/roles/java/tasks/main.yml config/roles/lein/tasks/main.yml
    ---
    # TODO
    
    EOF

    Next, let’s add these roles to a play at the end of our playbook.

    # config/provision.yml
    - name: Set up app server
      hosts:
        - application
      sudo: yes
      roles:
        - java
        - lein

    For the purpose of our application, we would like to install the Oracle Java 8 JDK, which is not available from the standard Ubuntu repositories, so we will add a repository from WebUpd8 and use debconf to automatically accept the Oracle Java license, which is normally an interactive process. Thankfully, there are already Ansible modules for adding apt repositories as well as changing debconf settings. See why they say that Ansible has “batteries included”?

    ---
    # config/roles/java/tasks/main.yml
    # Install Oracle Java 8
    
    - name: Add WebUpd8 apt repo
      apt_repository: repo='ppa:webupd8team/java'
    
    - name: Accept Java license
      debconf: name=oracle-java8-installer question='shared/accepted-oracle-license-v1-1' value=true vtype=select
    
    - name: Update apt cache
      apt: update_cache=yes
    
    - name: Install Java 8
      apt: name=oracle-java8-installer state=latest
    
    - name: Set Java environment variables
      apt: name=oracle-java8-set-default state=latest

    Next, we’ll add the task to install Leiningen. Instead of having Ansible download Leiningen directly from the internet, we will download a copy and make it part of our configuration so that we can easily version it:

    mkdir config/roles/lein/files
    wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein -O config/roles/lein/files/lein

    With that done, the actual task for installing Leinengen becomes a one-liner.

    ---
    # config/roles/lein/tasks/main.yml
    # Install leiningen
    
    - name: Copy lein script
      copy: src=lein dest=/usr/bin/lein owner=root group=root mode=755

    Let’s make sure that everything is working:

    $ vagrant provision
    # ... lots of output
    PLAY RECAP ********************************************************************
    app                        : ok=9    changed=6    unreachable=0    failed=0

    Ansible Plays for Infrastructure Server

    Next up, we’ll add the play that will set up our infr server with MongoDB and RabbitMQ. We’ll create roles for each of these applications, and we’ll create plays to apply the mongodb role to servers in the database group, and the rabbitmq role to the servers in the broker group. If you recall, we only have the infr VM in each of those groups, so both roles will be applied to that same server.

    We’ll set up the role skeletons similar to the way we did with the java and lein roles.

    mkdir -p config/roles/{mongodb,rabbitmq}/{tasks,handlers}
    cat <<EOF | tee config/roles/mongodb/tasks/main.yml config/roles/rabbitmq/tasks/main.yml
    ---
    # TODO
    
    EOF

    This time, we’ll add two separate plays to our playbook.

    # config/provision.yml
    - name: Set up database server
      hosts:
        - database
      sudo: yes
      roles:
        - mongodb
    
    - name: Set up messaging broker server
      hosts:
        - broker
      sudo: yes
      roles:
        - rabbitmq

    Next, we’ll fill in the main tasks to install MongoDB and RabbitMQ.

    ---
    # config/roles/mongodb/tasks/main.yml
    # Install and configure MongoDB
    
    - name: Fetch apt signing key
      apt_key: keyserver=keyserver.ubuntu.com id=7F0CEB10 state=present
    
    - name: Add 10gen Mongo repo
      apt_repository: >
        repo='deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.0 multiverse'
        state=present
    
    - name: Update apt cache
      apt: update_cache=yes
    
    - name: Install MongoDB
      apt: name=mongodb-org state=latest
    
    - name: Start mongod
      service: name=mongod state=started
    
    # Allow connections to mongo from other hosts
    - name: Bind mongo to IP
      lineinfile: >
        dest=/etc/mongod.conf
        regexp="^bind_ip ="
        line="bind_ip = {{ mongo_bind_ip }}"
      notify:
        - restart mongod
    
    - name: Install pip (for adding mongo user)
      apt: name=python-pip state=latest
    
    - name: Install pymongo (for adding mongo user)
      pip: name=pymongo state=latest
    
    - name: Add mongo user
      mongodb_user: >
        database={{ mongo_database }}
        name={{ mongo_user }}
        password={{ mongo_password }}
        state=present

    This role is a little more complicated than what we have seen so far, but it’s still not too bad. We again use built-in Ansible modules to fetch 10Gen’s signing key and add their MongoDB repository. We use the service module to start the mongod system service, then we use the lineinfile module to edit a single line in the default MongoDB config file.

    You may have noticed that we used several variables in the arguments to a couple of tasks. There are a couple of places that variables can live, but for this tutorial, we will declare these variables globally in config/group_vars/all.yml. The variables in this file will be applied to every group of servers.

    mkdir config/group_vars
    cat <<EOF > config/group_vars/all.yml
    ---
    # config/group_vars/all.yml
    # Variables common to all groups
    
    mongo_bind_ip: 0.0.0.0
    mongo_database: clojure-chat
    mongo_user: clojure-chat
    mongo_password: s3cr3t
    EOF

    Additionally, we added a notify line that will notify a handler when the task has been run. Handlers are generally used to start and stop system services. Let’s go ahead and create the handler now.

    cat <<EOF > config/roles/mongodb/handlers/main.yml
    ---
    # config/roles/mongodb/vars/main.yml
    # MongoDB service handlers
    
    - name: restart mongod
      service: name=mongod state=restarted
    EOF

    Since there are no surprises in the configuration for RabbitMQ, we will not go into it here. However, the full config is available in the repo for reference.

    Ansible Plays for the Application

    In order to follow the 12-factor app pattern, we would like to have all of our configuration stored in environment variables. Our app is already set up to read from HTTP_PORT, MONGODB_URI and RABBITMQ_URI. We’ll just write a simple task to add those variables to the vagrant user’s login shell.

    ---
    # config/roles/clojure_chat/tasks/main.yml
    # Add application environment variables
    
    - name: Add env variables
      template: src=app_env.j2 dest=/home/vagrant/.app_env owner=vagrant group=vagrant
    
    - name: Include env variables in vagrant's login shell
      shell: echo ". /home/vagrant/.app_env" >> /home/vagrant/.bash_profile

    And we’ll go ahead and create the template that defines these variables:

    # config/roles/clojure_chat/templates/app_env.j2
    # {{ ansible_managed }}
    
    export HTTP_PORT="{{ backend_port }}"
    
    # We are including the auth mechanism because Ansible's mongodb_user
    # module creates users with the SCRAM-SHA-1 method by default
    export MONGODB_URI="mongodb://{{ mongo_user }}:{{ mongo_password }}@{{ database_ip }}/{{ mongo_database }}?authMechanism=SCRAM-SHA-1"
    
    export RABBITMQ_URI="amqp://{{ rmq_user }}:{{ rmq_password }}@{{ broker_ip }}:5672{% if rmq_vhost == '/' %}/%2f{% else %}{{ rmq_vhost }}{% endif %}"

    Verify the Configuration

    We just wrote a lot of configuration, but now we have a fully-reproduceable development environment for a Clojure application that utilizes a messaging queue and a noSQL database.

    Let’s make sure that everything works by provisioning again and firing up the repl.

    vagrant provision
    vagrant ssh app
    cd /vagrant
    lein repl

    The repl will load our clojure-chat.main namespace, where we can start up the server.

    ;; REPL
    clojure-chat.main=> (-main)

    Now a web server should be running inside our VM on port 3000. We can check it out by visiting http://10.0.15.12:3000/ in a browser.

    At this point you have successfully set up a complete, multi-machine development environment for Clojure using Ansible!

    Leave a Reply

    Your email address will not be published. Required fields are marked *

    mm
    Writen by:
    A software professional with a passion for simplicity and an interest in functional programming. Blogs at http://kendru.github.io.