Continuous Deployment with Docker, AWS, and Ansible

Introduction

You’ve built some Docker images and made something locally. Now it’s time to go to production — and you’re stuck. This is not uncommon. There are quite a few articles out there about using Docker in a development environment. However, running Docker in production could still use a detailed explanation. This tutorial will take you from a greenfield web project to an application running in production. The process looks as follows:

1. Build and push Docker images with make,
2. Connect Semaphore CI,
3. Push the image to AWS ECR,
4. Bootstrapp a Docker AWS Elastic Beanstalk application with AWS Cloudformation, and
5. Coordinate infrastructure and application deployment with Ansible.

There are multiple moving parts. That’s what it usually takes to get code into production — especially with Docker. Let’s take a moment to examine other possible solutions and tools before jumping in.

Docker in Production

This area is becoming important as more teams move to Docker and need ways to to put their applications in production. Docker announced, at DockerCon 2016, that Docker 1.12 comes with orchestration primitives built in. Still there are a plethora of other ways to deploy Docker containers to production. They roughly fall into three categories.

1. Scheduling Clusters
   • DCOS, Mesos, Kubernetes, ECS, Docker Universal Control Plane, and others. These systems create a resource pool from a varying number of machines. Users can create tasks/jobs (naming varies from system to system) and the cluster will schedule them to run. Some are Docker specific, others are not. Generally, these things are meant for high-scale environments and don’t make sense for a small number of containers.
2. Hosted PaaS
   • Docker Cloud, Heroku, Elastic Beanstalk. These systems abstract the cluster and scaling from the end user. Users tell the system how to run the application through a configuration file. Then, the system takes care of the rest. These systems usually have a low barrier to entry, integrate with other services, and are a bit pricey compared to other offerings.
3. Self-Managed
   • IaaS (AWS, DigitalOcean, SoftLayer) or bare metal. This category offers the most flexibility with the most upfront work and required technical knowledge. Useful for teams deploying internal facing applications or with the time/knowledge to manage infrastructure and production operations.

Most teams opt for a combination of option one and three. They may use AWS to provision a Mesos cluster where developers can deploy their things to. Option two is best suited for small groups and individuals because of knowledge, time, and resource constraints. This tutorial assumes that you have chosen option number two.

Our Toolchain

Elastic Beanstalk is AWS’s PaaS offering. It can run Docker, PHP, Ruby, Java, Python, Go, and many others as a web application or cronstyle worker applications. We’ll use their Docker support to deploy our application, as well as an example of how to deploy production AWS infrastructure.

Production infrastructure has to come from somewhere. AWS is the default choice for advanced use cases. CloudFormation is the natural choice to provision all the AWS resources. It’s the first partner and will generally have the most full featured support compared to the tools such as Terraform. How can we write the whole process together? Ansible, with its built-in cloudformation module, works well. Ansible is easy to learn and just as powerful, or more powerful, for certain use cases than Chef or Puppet. Ansible is a mix of configuration management and general DevOps style automation. Ansible allows us to deploy the infrastructure, coordinate local calls to build code, and finally make external calls to trigger new deploys.

Step 1: Build and Test a Docker Image

I’ll use a simple Ruby web application built with Sinatra. The language or framework is not specifically relevant for this tutorial. This is just to demonstrate building and testing a Docker image.

The Makefile follows best practices for working with Ruby and Docker. They are summarized below:

• Gemfile lists all the application dependencies,
• make Gemfile.lock uses docker to produce the required dependency manifest file,
• make build Uses the application dependencies to produce a Docker image on top of the official ruby image,
• make test Runs the tests included in the Docker image,
• src/ contains relevant ruby source files, and
• test/ contains the test files.

The complete source is available as well. Here is a snippet from the Dockerfile:

	FROM ruby:2.3

	ENV LC_ALL C.UTF-8

	RUN mkdir -p /app/vendor
	WORKDIR /app
	ENV PATH /app/bin:$PATH

	COPY Gemfile Gemfile.lock /app/
	COPY vendor/cache /app/vendor/cache
	RUN bundle install --local -j $(nproc)

	COPY . /app/

	EXPOSE 80

	CMD [ "bundle", "exec", "rackup", "-o", "0.0.0.0", "-p", "80", "src/config.ru" ]

Let’s take a look at the Makefile :

	RUBY_IMAGE:=$(shell head -n 1 Dockerfile | cut -d ' ' -f 2)
	IMAGE:=cd-example/hello_world
	DOCKER:=tmp/docker

	Gemfile.lock: Gemfile
		docker run --rm -v $(CURDIR):/data -w /data $(RUBY_IMAGE) \
			bundle package --all

	$(DOCKER): Gemfile.lock
		docker build -t $(IMAGE) .
		mkdir -p $(@D)
		touch $@

	.PHONY: build
	build: $(DOCKER)

	.PHONY: test-image
	test-image: $(DOCKER)
		docker run --rm $(IMAGE) \
			ruby $(addprefix -r./,$(wildcard test/*_test.rb)) -e 'exit'

	.PHONY: test-ci
	test-ci: test-image test-cloudformation

	.PHONY: clean
	clean:
		rm -rf $(DOCKER)

After cloning the source, run:

	make clean test-ci

You now have a fully functioning web server that says “Hello World.” You can test this by starting a Docker container.

	docker run --rm cd-example/hello_world

Step 2: Connecting CI

We’ll use Semaphore as our CI. This is a straightforward process. Push code to Github and configure in Semaphore. We’ll have two pipeline steps for now.

1. make clean and
2. make test-ci.

You should now have a green build. Step 3 is pushing this image somewhere, where our infrastructure can use it.

Step 3: Pushing the Image

Amazon provides the Elastic Container Registry service where we can push Docker images. AWS creates a default registry for every AWS account. Luckily for us, Semaphore also provides a transparent integration with ECR. However, ECR does not allow you to push images immediately. Firstly, you have to create the repository where the Docker images will be stored. Now is is a good time to consider any prerequisites for the Elastic Beanstalk application. Elastic Beanstalk requires an S3 bucket to read, what they call, “Application Versions” from. You can think of these as “releases”. Right now, we need two things from AWS:

1. A repository in our registry to push image to and
2. An S3 bucket to push source code to.

We’ll use a combination of Cloudformation and Ansible to coordinate the process. CloudFormation will create the previously mentioned resources. Ansible allows us to deploy the CloudFormation stack. This is powerful because Ansible automatically creates or updates the CloudFormation stacks. Together, they provide continuous deployment.

Let’s start by reviewing the CloudFormation template used to create the resources. We won’t go in-depth into CloudFormation here. Instead, we’ll focus on the high level relevant components in our pipeline and leave the rest to the AWS docs. CloudFormation templates are JSON documents. CloudFormation reads the template, builds a dependency graph, and then creates or updates everything accordingly.

	{
		"AWSTemplateFormatVersion": "2010-09-09",
		"Description": "Pre-reqs for Hello World app",
		"Parameters": {
			"BucketName": {
				"Type": "String",
				"Description": "S3 Bucket name"
			},
			"RepositoryName": {
				"Type": "String",
				"Description": "ECR Repository name"
			}
		},
		"Resources": {
			"Bucket": {
				"Type": "AWS::S3::Bucket",
				"Properties": {
					"BucketName": { "Fn::Join": [ "-", [
						{ "Ref": "BucketName" },
						{ "Ref": "AWS::Region" }
					]]}
				}
			},
			"Repository": {
				"Type": "AWS::ECR::Repository",
				"Properties": {
					"RepositoryName": { "Ref": "RepositoryName" }
				}
			}
		},
		"Outputs": {
			"S3Bucket": {
				"Description": "Full S3 Bucket name",
				"Value": { "Ref": "Bucket" }
			},
			"Repository": {
				"Description": "ECR Repo",
				"Value": { "Fn::Join": [ "/", [
					{
						"Fn::Join": [ ".", [
							{ "Ref": "AWS::AccountId" },
							"dkr",
							"ecr",
							{ "Ref": "AWS::Region" },
							"amazonaws.com"
						]]
					},
					{ "Ref": "Repository" }
				]]}
			}
		}
	}

There are two input parameters and two output parameters. Elastic Beanstalk requires a bucket in a specific region. The templates take the bucket parameter and appends the region to it. Then, it outputs the complete ECR registry URL. You must know your AWS account ID to use ECR. We can use that value available in CloudFormation to output the full registry endpoint. The value can be used programmatically from within Ansible. Time to move onto the Ansible playbook.

Ansible models things with Playbooks. Playbooks contain tasks. Tasks use modules to do whatever is required. Playbooks are YML files. We’ll build up the deploy playbook as we go. The first step is to deploy the previously mentioned CloudFormation stack. The next step is to use the outputs to push the image to our registry.

	---
	- hosts: localhost
		connection: local
		gather_facts: False
		vars:
			aws_region: eu-west-1
			app_name: "semaphore-cd"
			prereq_stack_name: "{{ app_name }}-prereqs"
			bucket_name: "{{ app_name }}-releases"
		tasks:
			- name: Provision Pre-req stack
				cloudformation:
					state: present
					stack_name: "{{ prereq_stack_name }}"
					region: "{{ aws_region }}"
					disable_rollback: true
					template: "cloudformation/prereqs.json"
					template_parameters:
						BucketName: "{{ bucket_name }}"
						RepositoryName: "{{ app_name }}"
				register: prereqs

			- name: Generate artifact hash
				command: "./script/release-tag"
				changed_when: False
				register: artifact_hash

			- name: Push Image to ECR
				command: "make push UPSTREAM={{ prereqs.stack_outputs.Repository }}:{{ artifact_hash.stdout }}"
				changed_when: False

The first task uses the cloudformation module to create or update the stack. Next, a local script is called to generate a Docker image tag. The script uses git to get the current SHA. Finally, it uses a defined make target to push the image to the registry. The new make push target looks like this:

	.PHONY: push
	push:
		docker tag $(IMAGE) $(UPSTREAM)
		docker push $(UPSTREAM)

Ansible provides the UPSTREAM variable on the command line. We can also update our test suite to verify our CloudFormation template. Here’s the relevant snippet.

	.PHONY: test-cloudformation
	test-cloudformation:
		aws --region eu-west-1 cloudformation \
			validate-template --template-body file://cloudformation/prereqs.json

	.PHONY: test-image
	test-image: $(DOCKER)
		docker run --rm $(IMAGE) \
			ruby $(addprefix -r./,$(wildcard test/*_test.rb)) -e 'exit'

	.PHONY: test-ci
	test-ci: test-image test-cloudformation

Now, it’s time to set the whole thing up on Semaphore. There are a few things you need to do there.

1. First, set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables so CI can talk to AWS,
2. Use those access keys to configure the Semaphore CI ECR addon to authorize CI to push Docker images, and
3. Install ansible as part of the CI run.

You should create a separate IAM user for your CI system. This allows you grant access to required services. In this case you should create an IAM user and attach the AWS managed policies for CloudFormation and Elastic Beanstalk. If you already have an existing deploy/automation/ci IAM, then ensure that the appropriate policies are attached.

Get your access keys and run through the bits in your projects settings. Then your build steps should be:

1. sudo pip install ansible,
2. make clean,
3. make test-ci, and
4. ansible-playbook deploy.yml.

Finally, push your code and you should see the deploy playbook push the application to the upstream registry.

Step 4: Deploy Image to Elastic Beanstalk

You’ve made it to the final level. It’s time to put this code into production. This involves a few moving pieces:

1. Creating an “Application Version” containing the configuration required to run our container,
2. Uploading that file to S3, and
3. Creating a Docker Elastic Beanstalk Application authorized to pull images from the Docker registry .In other words, deploying the cloudformation stack with all input parameters.

Step 4.1: CloudFormation

Let’s work backwards from the CloudFormation template. The template is the most complex component. CloudFormation templates generally have at least three sections: Parameters, Resources, and Outputs. Parameters define inputs, e.g. what instance type to use. Resources are AWS managed components, e.g. EC2 Instances, Elastic Loadbalancers, etc. Outputs describe information about the resources/parameters, e.g. the public DNS name for an Elastic LoadBalancer, or EC2 instance IP.

Our templates take the following parameters:

• S3Bucket – Bucket containing source code zip file,
• S3ZipKey – Key for code zip file,
• RackEnv – Used to set the RACK_ENV environment variable on the Docker container, and
• HealthCheckPath – Given the Elastic Loadbalancer to monitor application health.

CloudFormation templates are JSON documents. We’ll go over one section at a time. The complete source is available as well.

	{
		"Description": "Hello World EB application & IAM policies",
		"Parameters": {
			"S3Bucket": {
				"Type": "String",
				"Description": "S3 Bucket holding source code bundles"
			},
			"S3ZipKey": {
				"Type": "String",
				"Description": "Path to zip file"
			},
			"RackEnv": {
				"Type": "String",
				"Description": "Value for RACK_ENV and name of this environment"
			},
			"HealthCheckPath": {
				"Type": "String",
				"Description": "Path for container health check"
			}
		}
	}

CloudFormation templates may include the Decription key. This describes the stack’s purposes and resources. Let’s discuss resources now. Our application will be deployed to Elastic Beanstalk. This requires us to create all the Elastic Beanstalk specific resources and associated IAM, access rules for those not familiar with AWS, policies. We need to create the following:

• An IAM instance profile. The instance profile gives the EC2 instances running our container the required Elastic Beanstalk access and, more importantly, read only access our AWS account’s ECR registry,
• The Elastic Beanstalk application itself. Elastic Beanstalk applications have multiple named environments (e.g. production, staging, qe),
• The Elastic Beanstalk environment. This includes the load balancer and EC2 instances running our containers. Environments can be single instances or horizontally scaled load balanced setups. We’ll opt for a load balanced setup,
• The Environment Configuration Template. This tells Elastic Beanstalk how to configure a particular environment. These are defaults which may be overriden by the environment. Templates may be used to launch multiple environments. This way, you can mirror your production and staging environment configurations. However, we only have a single environment, specified by the RackEnv parameter, so all settings are set on this resource, and
• The Application Version. Elastic Beanstalk calls these “source code bundles”. Every unique code deploy is a new application version. Our CloudFormation template works by creating a single version and continually updating the code included that version.

CloudFormation automatically creates the dependency graph between each resource. This works by its own implicit rules or explicit calls to the Ref function or DependsOn attribute, which you’ll see in the following snippets. Ref refers to named Parameters or Resources. Let’s go through each resource step by step, starting with the Instance Profile. Note that each JSON object is inside the Resources object. This is omitted for formatting reasons.

	"ElasticBeanstalkRole": {
		"Type": "AWS::IAM::Role",
		"Properties": {
			"Path": "/hello-world/",
			"ManagedPolicyArns": [
				"arn:aws:iam::aws:policy/AWSElasticBeanstalkWebTier",
				"arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
			],
			"AssumeRolePolicyDocument": {
				"Version" : "2012-10-17",
				"Statement": [{
					"Effect": "Allow",
					"Principal": {
						"Service": [ "ec2.amazonaws.com" ]
					},
					"Action": [ "sts:AssumeRole" ]
				}]
			},
			"Policies": [ ]
		}
	},
	"ElasticBeanstalkProfile": {
		"Type": "AWS::IAM::InstanceProfile",
		"Properties": {
			"Path": "/hello-world/",
			"Roles": [
				{ "Ref": "ElasticBeanstalkRole" }
			]
		}
	}

The snippet contains two resources. First, the AWS::IAM::Role is granted access to the appropriate AWS components via managed policies. Managed policies are created and maintained by AWS itself. These are useful for declaring something like “full access to EC2” or “read only S3”. arn:aws:iam::aws:policy/AWSElasticBeanstalkWebTier state gives the instances everything required to service web traffic. arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly does what it says on the tin — grants read-only access to the registry. Finally, an AWS::IAM::InstanceProfile is created with the role. CloudFormation knows to use our role through the Ref statement. Normally, it’s not required to create an instance profile. However, it is required for our use case because we need to grant access to our Docker registry.

Let’s move on to the Elastic Beanstalk resources, starting with the application and version declaration.

	"ElasticBeanstalkApplication": {
		"Type": "AWS::ElasticBeanstalk::Application",
		"Properties": {
			"Description": "semaphore-cd-hello"
		}
	},
	"ElasticBeanstalkVersion": {
		"Type": "AWS::ElasticBeanstalk::ApplicationVersion",
		"Properties": {
			"ApplicationName": { "Ref": "ElasticBeanstalkApplication" },
			"Description": "Source Code",
			"SourceBundle": {
				"S3Bucket": { "Ref": "S3Bucket" },
				"S3Key": { "Ref": "S3ZipKey" }
			}
		}
	}

The ElasticBeanstalkApplication is comparatively sparse. It’s only a container for the other more complex resources. Next the AWS::ElasticbeanStalk::ApplicationVersion is created from the S3Bucket and S3ZipKey parameters and associated with the application via Ref. At this point, we have all required resources to move on to the configuration template and environment.

	"ElasticBeanstalkConfigurationTemplate": {
		"Type": "AWS::ElasticBeanstalk::ConfigurationTemplate",
		"DependsOn": [ "ElasticBeanstalkProfile" ],
		"Properties": {
			"Description": "Semaphore CD Configuration Template",
			"ApplicationName": { "Ref": "ElasticBeanstalkApplication" },
			"SolutionStackName": "64bit Amazon Linux 2016.03 v2.1.0 running Docker 1.9.1",
			"OptionSettings": [
				{
					"Namespace": "aws:elasticbeanstalk:environment",
					"OptionName": "EnvironmentType",
					"Value": "LoadBalanced"
				},
				{
					"Namespace": "aws:elasticbeanstalk:application",
					"OptionName": "Application Healthcheck URL",
					"Value": { "Ref": "HealthCheckPath" }
				},
				{
					"Namespace": "aws:autoscaling:launchconfiguration",
					"OptionName": "IamInstanceProfile",
					"Value": { "Fn::GetAtt": [ "ElasticBeanstalkProfile", "Arn" ] }
				},
				{
					"Namespace": "aws:elasticbeanstalk:application:environment",
					"OptionName": "RACK_ENV",
					"Value": { "Ref": "RackEnv" }
				}
			]
		}
	},
	"ElasticBeanstalkEnvironment": {
		"Type": "AWS::ElasticBeanstalk::Environment",
		"Properties": {
			"Description": { "Ref": "RackEnv" },
			"ApplicationName": { "Ref": "ElasticBeanstalkApplication" },
			"TemplateName": { "Ref": "ElasticBeanstalkConfigurationTemplate" },
			"VersionLabel": { "Ref": "ElasticBeanstalkVersion" },
			"Tier": {
				"Type": "Standard",
				"Name": "WebServer"
			}
		}
	}

Let’s break it down, starting with the AWS::ElasticBeanstalk::LaunchConfigurationTemplate. The AppliationName is a Ref to the previously defined resource. Next, the SolutionStackName is set to Docker. Elastic Beanstalk supports many different technologies, e.g. Java, Python, Ruby, Node, Go, Docker. This value declares which technology stack to use. Next, the various settings are specified. The EnvironmentType is set to LoadBalanced. The ELB the healtcheck path is configured. The InstanceProfile is specified. The Fn::GetAtt function can get a specific attribute for a given resource. The Arn is the Amazon Resource Name, which is essentially a UUID. Finally, the RACK_ENV environment variable is set based on the RackEnv parameter.

Next, the AWS::ElasticBeanstalk::Environment is declared with Ref calls to all the other resources. CloudFormation automatically creates environment names, thus the RackEnv parameter is used for Description. This makes it easy to identify in the AWS console.

Lastly, the Outputs section declares the EndpointURL. The value is the public DNS for the Elastic Beanstalk environment. The value can be used to create a CNAME on your own domain or pasting into the browser.

		"Outputs": {
			"EndpointURL": {
				"Description": "Public DNS Name",
				"Value": {
					"Fn::GetAtt": [ "ElasticBeanstalkEnvironment", "EndpointURL" ]
				}
			}
		}
	}

That concludes the CloudFormation template. Time to move on to Ansible playbook changes.

Step 4.2: Ansible Playbook

Elastic Beanstalk Docker deploys use a Dockerrun.aws.json. The configuration file tells Elastic Beanstalk what Docker image to use, which ports to expose, and various other settings. Simply providing this file in a zip file is enough given we are using a pre-built image. Then, the zip file needs to be uploaded to S3. Finally, the CloudFormation stack we previously defined must be deployed with updated parameters, e.g. the new code location.

The first step in this process is to create a temporary scratch directory for creating files.

	- name: Create scratch dir for release artifact
		command: "mktemp -d"
		register: tmp_dir
		changed_when: False

The mktemp -d creates a directory on the temporary file system and writes the path to standard out. The result is registered in the tmp_dir variable. changed_when is set to False, so Ansible knows that result is never expected to change, thus it should always be “OK”.

The next step is creating the Dockerrun.aws.json file. Ansible has rich templating support. We defined template files which are filled in with the appropriate variables at run time. This is important because the Docker image changes every time, since the git SHA is the tag). Here is the template and associated Ansible task:

	{
		"AWSEBDockerrunVersion": "1",
		"Image": {
			"Name": "{{ image }}",
			"Update": "true"
		},
		"Ports": [{
			"ContainerPort": "80"
		}]
	}

The configuration file exposes port 80, the port declared in the Dockerfile. The Image.Update key is set to true. This instructs Elastic Beanstalk to pull a new image every time. The Ansible task to generates the final Dockerrun.aws.json.

	- name: Create Dockerrun.aws.json for release artifact
		template:
			src: files/Dockerrun.aws.json
			dest: "{{ tmp_dir.stdout }}/Dockerrun.aws.json"
		vars:
			image: "{{ prereqs.stack_outputs.Repository }}:{{ artifact_hash.stdout }}"

The next two tasks create the zip file and upload to S3:

	- name: Create release zip file
		command: "zip -r {{ tmp_dir.stdout }}/release.zip ."
		args:
			chdir: "{{ tmp_dir.stdout }}"
		changed_when: False

	- name: Upload release zip to S3
		s3:
			region: "{{ aws_region }}"
			mode: put
			src: "{{ tmp_dir.stdout }}/release.zip"
			bucket: "{{ prereqs.stack_outputs.S3Bucket }}"
			object: "{{ app_name }}-{{ artifact_hash.stdout }}.zip"

Finally, the CloudFormation stack is deployed with updated parameters collected in the previous tasks.

	- name: Deploy application stack
		cloudformation:
			state: present
			stack_name: "{{ app_stack_name }}"
			region: "{{ aws_region }}"
			disable_rollback: true
			template: "cloudformation/app.json"
			template_parameters:
				S3Bucket: "{{ prereqs.stack_outputs.S3Bucket }}"
				S3ZipKey: "{{ app_name }}-{{ artifact_hash.stdout }}.zip"
				RackEnv: "{{ environment_name }}"
				HealthCheckPath: "/ping"

Go ahead and push your code again and wait for a bit. Initial provisioning can be a bit slow, but it will work. Open the Elastic Beanstalk URL, and you should see “Hello World”. Congratulations! Your infrastructure code and source code are now deployed on every commit.

Wrap Up

It’s time to recap the things we’ve covered.

1. How to use make to build and test Docker images,
2. How to use CloudFormation to create an ECR repository and S3 Bucket for application deployment,
3. How to use ansible to coordinate local build process and deploying remote AWS infrastructure,
4. How to use CloudFormation to create and configure a Docker-based Elastic Beanstalk application from scratch, and
5. How to use ansible to coordinate the entire process of deploying a Docker-based Elastic Beanstalk environment.

This process is powerful and follows the same structure independent of the Docker deploy target. The process goes as follows:

1. Build, test, and push a new Docker Image and
2. Use deployment target’s tools to trigger a new deploy.

We’ve implemented the process one way. There a few things you could do next. You could use the EndpointURL output in combination with Ansible’s get_url module to test each deploy. It’s also possible to parameterize the playbook and CloudFormation template to build a new application for each new topic branch. You’re also now equipped to implement a similar pipeline on top of a different deployment target.

Remember to check the complete source files because the tutorial contains annotated versions. Here are the most important links:

• Makefile
• CloudFormation Template
• Deploy Playbook
• The complete source code

If you have any questions or comments, feel free to post them below.

Finally, happy shipping!

P.S. Want to continuously deliver your applications made with Docker? Check out Semaphore’s Docker support.

Read next:

• Sign up to receive a free ebook guide to CI/CD with Kubernetes
• Lightweight Docker Images in 5 Steps
• Continuous Deployment with Google Container Engine and Kubernetes
• Creating a Heroku-like Deployment Solution with Docker