22 Mar 2017 · Software Engineering

    Packaging and Deploying Go Applications to AWS using Semaphore

    19 min read
    Contents

    Introduction

    This tutorial will illustrate how you can use AWS CodeDeploy, a scalable deployment service offered by AWS that lets you automate your deployment process. We will package and continuously deploy a Go application to an EC2 instance. The tutorial will use a sample application and walk you through the process of updating your existing setup, to allow for the continuous deployment of your application.

    Goals

    By the end of this tutorial, you will:

    • Modify an existing EC2 instance to get it ready for continuous deployment,
    • Create users, roles and policies in AWS for use with CodeDeploy, and
    • Set up Semaphore to continuously test and deploy your application.

    Prerequisites

    This tutorial assumes:

    • Familiarity with Go,
    • General familiarity with AWS, and
    • That you have an application running on an EC2 instance.

    The Sample Application

    The application we will use in this tutorial is a very simple application that consists of four files:

    • main.go — Contains the main application,
    • main_test.go — Contains the tests for the application,
    • config.json — A config file used by the application, and
    • index.html — The template used by the application.

    The contents of these files are as follows:

    1. main.go
    // main.go
    
    package main
    
    import (
    	"encoding/json"
    	"html/template"
    	"time"
    
    	"io/ioutil"
    	"log"
    	"net/http"
    )
    
    var t *template.Template
    var c config
    
    type data struct {
    	Weekday  time.Weekday
    	Greeting string
    }
    
    type config struct {
    	Greeting string
    }
    
    func main() {
    	loadConfig()
    	initializeTemplates()
    	http.HandleFunc("/", handler)
    	log.Fatal(http.ListenAndServe(":8000", nil))
    }
    
    func loadConfig() {
    	contentBytes, _ := ioutil.ReadFile("config.json")
    	json.Unmarshal(contentBytes, &c)
    }
    
    func initializeTemplates() {
    	t, _ = template.ParseFiles("index.html")
    }
    func handler(w http.ResponseWriter, r *http.Request) {
    	data := &data{time.Now().Weekday(), c.Greeting}
    
    	t.Execute(w, data)
    }

    The application is a simple web server that responds with the contents of index.html. It passes two values to the template – one from the configuration file and another one that’s the current day of the week.

    1. main_test.go
    // main_test.go
    
    package main
    
    import (
    	"encoding/json"
    	"io/ioutil"
    	"testing"
    )
    
    func TestConfig(t *testing.T) {
    	contentBytes, err := ioutil.ReadFile("config.json")
    	if err != nil {
    		t.Errorf("The config file missing")
    	}
    
    	var c config
    	if err = json.Unmarshal(contentBytes, &c); err != nil {
    		t.Errorf("Error while parsing the config file")
    	}
    
    	if c.Greeting == "" {
    		t.Errorf("The config is missing the Greeting key")
    	}
    }

    This test checks that:

    • The config.json file is present,
    • The config.json file contains valid JSON, and
    • The Greeting key in the config.json file is non empty
    1. index.html
    <!doctype html>
    <html>
    
    <head>
      <title>Sample App for AWS CodeDeploy </title>
      <meta name="viewport" content="width=device-width, initial-scale=1">
      <meta charset="UTF-8">
    </head>
    
    <body>
      <h4>{{.Greeting}} World</h4>
      <p>Day: {{.Weekday}}</p>
    </body>
    
    </html>

    The template uses the Greeting and Weekday values passed in from the application to display some content.

    1. config.json
    {
      "Greeting": "Hello"
    }

    While this is an extremely simple application, it can be considered fairly representative of Go applications, which mostly consist of the main binary, configuration files, and templates.

    Setting up Continuous Testing with Semaphore

    Before we modify our EC2 instance and set up CodeDeploy, let’s begin by setting up the project on Semaphore for continuous testing.

    Note: This step is required because we want to avoid deploying the application if the tests are failing. After setting up testing, Semaphore will only deploy those versions which pass all tests.

    If you haven’t already, sign up with Semaphore. Semaphore can use repositories hosted on Github or Bitbucket, so make sure that you push the application code to one of these hosts.

    Once you have done this and signed into your Semaphore account, click on the button to add a new project:

    Add new project

    Note: If this is your first project on Semaphore, you might see a different screen for adding a new project.

    Now, select the repository from the list:

    Select repository

    Next, select the branch of your repository that you want to run tests against:

    Select branch

    At this point, Semaphore will analyze your project and it will set it up using the default settings applicable to a Go project. This is where you can customize the build and the testing process if you want to. You can also change these settings later in the project’s settings section. We’ll just change the version of Go used to the latest available version and click the Build With These Settings button at the bottom.

    Build settings

    At this point, Semaphore will begin building and testing your project. From now on, whenever you push any changes to this project, Semaphore will automatically build and test your project.

    First build

    Accessing the application in its current state should result in something like the following:

    Application before

    Note: The URL of your application will most likely be different and specific to your application and your EC2 setup.

    Now that we have continuous testing set up, let’s move on and configure AWS to use CodeDeploy.

    Setting Up AWS

    As mentioned in the prerequisites section, this tutorial assumes that you have an application already running on an EC2 instance. With this setup, we need to make the following changes in our AWS account in order to be able to use CodeDeploy:

    1. Create a user,
    2. Create a deployment role for CodeDeploy,
    3. Create a policy and a role for the EC2 instance,
    4. Attach this role to the EC2 instance,
    5. Tag the EC2 instance,
    6. Install the CodeDeploy agent on the EC2 instance, and
    7. Set up CodeDeploy in the AWS console.

    Let’s now go through each of these steps.

    1. Create a User

    We want to create a user whose credentials will be used from Semaphore to continuously deploy the application. This user needs to be able to do the following:

    • Upload the latest version of the application to a bucket in S3, and
    • Use CodeDeploy to deploy and activate the latest version from S3.

    To do this, we need to attach the following policies to the user:

    • AmazonS3FullAccess and
    • AWSCodeDeployDeployerAccess

    Begin by visiting the IAM > Users section in your AWS console:

    IAM Users

    Click on the Add user button.

    Add user

    Enter the name of the user, select Programmatic access for access type and click the Next: Permissions button.

    User permissions

    On this screen, select the option to Attach existing policies directly. From the list of the policies that appear at the bottom, select the AmazonS3FullAccess and AWSCodeDeployDeployerAccess policies, and click the button to continue.

    User permissions confirmation

    You should now see the review screen with the options you have selected. Click the Create user button to complete the process.

    User credentials

    This screen shows the Access key ID and the Secret access key for this user. Make a note of these values, as we’ll need them when setting up continuous deployment on Semaphore.

    2. Create a Deployment Role for CodeDeploy

    When setting up CodeDeploy in a later step, we’ll need to attach a role to it that gives it the appropriate permissions. In this step, we will create the role with the following permissions:

    • AmazonS3FullAccess,
    • AWSCodeDeployDeployerAccess, and
    • AmazonEC2FullAccess.

    To do this, go to the IAM > Roles section in your AWS console.

    Deployment Role

    Click on the Create New Role button.

    New Role

    Enter the name of the role, and click the Next Step button.

    Role type

    On this screen, select Amazon EC2 from the AWS Service Roles section.

    Attach policy

    You should now see a screen with a list of policies. From these policies, select the following:

    • AmazonS3FullAccess,
    • AWSCodeDeployDeployerAccess, and
    • AmazonEC2FullAccess,

    and click on the Next Step button.

    Role review

    Next, you’ll see a review screen with the details of the role you are creating. Click on the Create Role button after you’ve ensured that the settings are correct.

    Role created

    We now need to modify this role to set the appropriate trust relationship. Click on the role name and select the Trust Relationship tab.

    Trust relationship

    Click the Edit Trust Relationship button.

    Policy Document

    Modify the policy document so that the Statement > Principal > Service key is set to codedeploy.amazonaws.com.

    Click on the Update Trust Policy button to complete this process. We now have a role, in our case named semaphore-deploy-role, that we will attach to the CodeDeploy setup.

    3. Create a Policy and a Role for the EC2 Instance

    To get the existing EC2 instance to work with CodeDeploy, we need to create and attach a role with the appropriate permissions to the instance.

    To begin, go to the IAM > Policies section in your AWS console.

    Policies

    Click on the Create Policy button.

    Create policy

    On this screen, select the option to Create Your Own Policy

    Create your own policy

    Enter the name of the policy, this tutorial uses the semaphore-instance-policy name, and enter the following in the policy document:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Action": [
            "s3:Get*",
            "s3:List*"
          ],
          "Effect": "Allow",
          "Resource": [
            "arn:aws:s3:::semaphore-bucket/*",
            "arn:aws:s3:::aws-codedeploy-us-east-1/*"
          ]
        }
      ]
    }

    Note: This tutorial used an EC2 instance in the us-east-1 region. If your instance belongs to another region, you might want to modify the second Resource value above accordingly.

    Click on Create Policy to create the policy.

    Now that you have this policy, create a new role as we did in the previous step. In this case, apply the newly created policy to this new role instead of the three policies applied in the previous step.

    This tutorial uses semaphore-instance-policy as the name for this policy and semaphore-instance-role as the name for the role that uses this policy.

    4. Attach Role to the EC2 Instance

    Now that we have created the role for the existing EC2 instance, let’s attach it.

    Open the EC2 > Instances section of your AWS console. Here, you should see a list of your instances.

    Select the checkbox next to the instance that you’re interested in and click on Actions > Instance Settings > Attach/Replace IAM Role.

    Instance Role

    On this screen, choose the role that was created in the previous step.

    Attach Instance Role

    Click on the Apply button to attach the role to this EC2 instance.

    5. Tag the EC2 Instance

    When setting up CodeDeploy, we need to specify the machines a particular application should be deployed to. This is done by tagging the machines and specifying these tags in CodeDeploy.

    While this might seem like an overkill for a single instance, it can be quite handy if you have multiple instances that you want to deploy to.

    Open the EC2 > Instances section of your AWS console. Here, you should see a list of your instances.

    Select the checkbox next to the instance that you’re interested in and click on Actions > Instance Settings > Add/Edit Tags.

    Instance tag

    On this screen, enter some value in the Key and Value fields. These can be anything. In this case, we have set the key to Name and the value to semaphore-app.

    Add tag

    Click on the save button to apply the tag to this EC2 instance.

    6. Install the CodeDeploy Agent on the EC2 Instance

    To deploy applications, CodeDeploy requires its agent to be present on all the concerned machines. In this section, we will install this agent on our EC2 instance.

    To begin, log in via SSH to your EC2 machine.

    Note: This tutorial uses Ubuntu on the EC2 instance. The commands in this section are specific to Ubuntu. If your instance uses another operating system, you can find the respective instructions here on the AWS documentation page.

    Refresh the package list and sources on your system:

    sudo apt-get update
    

    Install python-pip, ruby and wget:

    sudo apt-get install python-pip ruby wget -y
    

    Fetch the CodeDeploy agent installer:

    wget https://aws-codedeploy-us-east-1.s3.amazonaws.com/latest/install
    

    Note: This tutorial uses the us-east-1 region. If you use a different region, you can modify the above command accordingly.

    Make the installer executable:

    chmod +x ./install
    

    Install the CodeDeploy agent:

    sudo ./install auto
    

    Start the CodeDeploy agent:

    sudo service codedeploy-agent start
    

    After carrying out these instructions, be sure to reboot your instance for the changes to take effect. With this, your instance is all set up to work with CodeDeploy.

    7. Set up CodeDeploy in the AWS console

    As the final part of setting up CodeDeploy, we need to create a CodeDeploy application and a deployment group.

    In your AWS console, go to the CodeDeploy section and click on Get Started Now. You should see the following screen:

    CodeDeploy start

    Choose Custom Deployment and click on Skip Walkthrough.

    Create application

    On this screen, enter the application name and the deployment group name. These can be set to any values of your choosing. We’ve set the application name to semaphore-app and the deployment group name to semaphore-deployment-group.

    Choose In-place deployment as the deployment type. Note that in-place deployment incurs some downtime as your application is replaced with the new version. You can explore Blue/green deployment, which can deploy an application with zero downtime, but, we use in-place deployment in order to to keep things simple.

    Deployment group

    Scroll down a bit, and enter the tag of the EC2 instance, created in step 5 above. You should see your instance in the list of matching instances.

    Choose CodeDeployDefault.OneAtATime as the deployment configuration.

    Finally, choose the role we created in step 2 above, as the service role in the final section and click on the Create application button to complete the process.

    With this, we have completed the setup required on the AWS end to use CodeDeploy. All that’s left for us to do is configure Semaphore for continuous deployment.

    Setting Up Continuous Deployment on Semaphore

    To configure Semaphore to continuously deploy the application, we need to do the following:

    1. Set up the environment variables with AWS credentials,
    2. Create scripts that will be used by CodeDeploy,
    3. Create an appspec.yml file (required by CodeDeploy), and
    4. Add instructions to deploy the application.

    Technically, the files created in step 2 and 3 could be part of your project. However, because these won’t change often, we will create them right within Semaphore, instead of creating them in our project repository. This has the added advantage of preventing deployment details from being publicly available in case you have a publicly visible repository.

    1. Configure the Environment Variables

    When deploying from Semaphore, we will use the credentials of the user created in step 1 of the previous section. We can set these credentials up as encrypted environment variables to keep them secure and prevent them from appearing in logs.

    We want to set the following three environment variables:

    • AWS_ACCESS_KEY_ID,
    • AWS_SECRET_ACCESS_KEY, and
    • AWS_DEFAULT_REGION.

    You should have values for the first two from step 1 of the previous section. We set the value of AWS_DEFAULT_REGION to us-east-1 for this tutorial, but you can change this to suit your setup.

    Open the Project Settings > Environment Variables section of your project on Semaphore.

    Environment variables

    Click the button to add an environment variable.

    Add variable

    Enter AWS_ACCESS_KEY_ID as the Name, the actual access key as the Content and check the Encrypt content checkbox.

    Click the Create Variable button to complete the step.

    After creating the remaining two environment variables in a similar manner, you should see something like this:

    All environment variables

    2. Create CodeDeploy Scripts

    For using CodeDeploy, we will need scripts for the following tasks:

    • To start the application,
    • To stop the application, and
    • To clean up files from a previous deployment.

    Note: This tutorial assumes the use of supervisor to manage the application. Furthermore, this tutorial assumes that the application is installed in the /app directory of the instance. If your setup differs, you will need to modify these scripts accordingly.

    To create these scripts in Semaphore, open the Project Settings > Configuration Files page for your project.

    Config file

    Click on the button to add a configuration file.

    Start.sh

    Enter scripts/start.sh as the File path (so that the complete path then becomes /home/runner/scripts/start.sh).

    Enter the following in the Content section:

    sudo supervisorctl start App
    

    Note: The tutorial assumes that the supervisor configuration for this application uses the name App.

    Finally, save this file.

    Similarly, create two additional files with the following values:

    i. To stop the application:

    File path: scripts/stop.sh

    Content:

    sudo supervisorctl stop App
    

    ii. To clean up files:

    File path: scripts/cleanup.sh

    Content:

    sudo rm /app -rf
    sudo mkdir /app
    sudo chown -R ubuntu:ubuntu /app
    

    Note: ubuntu is the default user if you choose the standard Ubuntu AMI on EC2. If your setup differs, modify this accordingly.

    3. Create appspec.yml

    The appspec.yml file is a specification file in the YAML format that lets you configure the deployment process. It allows you to copy files, manipulate file permissions, and execute scripts at various stages of the deployment process.

    We can create this file just like we created the script files in the previous section. Use deploy/appspec.yml as the file path and the following as the content:

    version: 0.0
    
    os: linux
    
    files:
      - source: /aws-codedeploy
        destination: /app/
      - source: /index.html
        destination: /app/
      - source: /config.json
        destination: /app/
    
    hooks:
      ApplicationStop:
        - location: scripts/stop.sh
          timeout: 180
          runas: ubuntu
      BeforeInstall:
        - location: scripts/cleanup.sh
          timeout: 180
          runas: ubuntu
      ApplicationStart:
        - location: scripts/start.sh
          timeout: 180
          runas: ubuntu

    If your setup differs, modify the values in this file accordingly.

    The files section identifies the files to be copied from the deployment package to the instance. aws-codedeploy is the name of our application’s executable binary.

    Note that for the source files, the package root acts as the root. So, for example, /index.html refers to the index.html file stored at the root of the package, not at the root of the file system.

    The hooks section configures the scripts that should run at various stages of deployment. The content of this section in our case configure the deployment process to:

    • Stop the application before the deployment,
    • Clean up files from the previous deployment before the current deployment, and
    • Start the application after the deployment.

    4. Add Deployment Instructions

    Based on the current configuration, on Semaphore:

    • The /home/runner/project_directory contains the application source files,
    • The /home/runner/scripts contains the scripts, and
    • The /home/runner/deploy contains appspec.yml.

    Note: /home/runner is the home directory on the Semaphore machine running the build, test and deploy tasks.

    With this in mind, let’s now configure Semaphore to start deploying our application on every successful build.

    Go to the your project’s home page on Semaphore.

    Project home page

    Click on the Set Up Deployment (just below the branch name) button to begin the configuration.

    Set up deployment

    Choose Generic Deployment from the list of options.

    Automatic deployment

    Choose Automatic as the deployment strategy.

    Deployment instructions

    Enter the following as the deploy commands:

    go build
    cp aws-codedeploy ~/deploy/
    cp index.html ~/deploy/
    cp config.json ~/deploy/
    cp -r ~/scripts ~/deploy/
    
    aws deploy push --application-name semaphore-app --s3-location s3://semaphore-bucket/semaphore-app.zip --source ~/deploy
    
    aws deploy create-deployment --application-name semaphore-app --s3-location bucket=semaphore-bucket,key=semaphore-app.zip,bundleType=zip --deployment-group-name semaphore-deployment-group --deployment-config-name CodeDeployDefault.OneAtATime
    

    The first command, go build, builds your project and creates the executable binary.

    The next three commands copy the binary, index.html and config.json to the deploy folder, which already contains the appspec.yml file.

    The next command copies the entire scripts folder, which contains the scripts we created earlier, into the deploy folder.

    At this stage, the deploy folder contains:

    • appspec.yml,
    • The project files, and
    • The scripts folder.

    The next command is:

    aws deploy push --application-name semaphore-app --s3-location s3://semaphore-bucket/semaphore-app.zip --source ~/deploy
    

    This command uses AWS’s command line interface aws, which is pre-installed on Semaphore, to:

    • Package the contents of the deploy folder, and
    • Upload this package to S3.

    In this case, we specify that we want to use the bucket named semaphore-bucket and name the package semaphore-app.zip. Be sure to modify these values according to your setup.

    The final command is:

    aws deploy create-deployment --application-name semaphore-app --s3-location bucket=semaphore-bucket,key=semaphore-app.zip,bundleType=zip --deployment-group-name semaphore-deployment-group --deployment-config-name CodeDeployDefault.OneAtATime
    

    This command initiates the deployment of the latest version of your application. Note that:

    • The values for --application-name and --deployment-group-name must match the application name used when setting up CodeDeploy in step 7 of the previous section, and
    • The value for --s3-location must match the values set in the previous command.

    After entering these commands, click the Next Step button.

    Server Name

    Enter the server name, this can be any value, and click the Create Server button.

    Setup complete

    This should complete your setup for continuous deployment. If you want, you can click the Deploy button to manually initiate a deployment.

    Run manual deployment

    With this, you have now configured Semaphore to continuously test and deploy your application.

    Testing the Setup

    To test the setup, let’s begin by introducing a change that breaks one of the tests. Let’s modify the Greeting key in config.json to Greetings and push the change to our repository.

    This will initiate a build on Semaphore, which will result in the following:

    Failed build

    As you can see, Semaphore aborted the process because the build failed and didn’t deploy this version.

    Let’s revert config.json to its original version. Additionally, we’ll also make a couple of other changes will help us use the advantages of continuous deployment.

    Change the value of Greeting in config.json from Hello to Howdy.

    {
      "Greeting": "Howdy"
    }

    In index.html, change

    <p>Day: {{.Weekday}}</p>

    to

    <p style="font-weight: bold; color: red;">Day: {{.Weekday}}</p>

    Now when you commit and push these changes, you should see something like the following on Semaphore:

    Passing build

    This shows that the project has passed all the tests and was successfully deployed. Visiting our application again shows us the following:

    App after

    As you can see, our application is live with the latest changes and we didn’t have to spend any effort on manually deploying these changes.

    Conclusion

    Congratulations! You’ve successfully set up AWS and Semaphore to continuously deploy a Go application to an existing EC2 instance using AWS CodeDeploy.

    If you have any questions and comments, feel free to leave them in the section below.

    Leave a Reply

    Your email address will not be published. Required fields are marked *

    Avatar
    Writen by:
    Kulshekhar is an independent full stack engineer with 10+ years of experience in designing, developing, deploying and maintaining web applications.