Get Insightful Test Reports of Your CI/CD Pipelines. Learn more →

AWS Serverless With Monorepos

Learn how to build and deploy a facial recognition application using serverless functions in AWS. Along the way, we’ll discover the Serverless Application Model, Lambda testing, and optimizing monorepo CI/CDs.

The final serverless app deployment pipeline

Going serverless with AWS and monorepos

We’ll be deploying a face-indexing application built with a serverless backend that consumes AWS services. The components we need are:

  • AWS Lambda: runs serverless functions that process and index face images.
  • API Gateway: presents a single endpoint for the Lambda functions.
  • Rekognition: does all the facial recognition.
  • DynamoDB: stores face metadata.
  • Cloudfront: a global, low-latency CDN to speed up website load times.
  • S3: stores image files and hosts the frontend code.
High level architecture
Application architecture

How does the serverless app work? After populating the database with face images processed with Rekognition, a user can submit an image to find its owner’s name. If the face image has been scanned, the name belonging to it is returned.

The complete code for the project is located here:

The demo is a monorepo, i.e. a repository holding several projects. Monorepos increase transparency across the team and help keep code consistent. They can, however, present some scalability issues, but they shouldn’t affect us as long as we use Semaphore monorepo workflows.

Shall we start?

Preparations

To complete the tutorial, you will need:

First, ensure you’ve run aws configure to connect your machine to the AWS account.

$ aws configure
​
AWS Access Key ID: <Type Your Access Key>
AWS Secret Access Key: <Type Your Secret Key>

While you’re setting things up, also head over to Semaphore and create a secret with the AWS access key. Semaphore will need this information to deploy on your behalf. Be sure to check the guided tour if this is your first time using Semaphore.

Store AWS credentials on Semaphore
A secret to access AWS services

Finally, pick a region for the deployment. All serveless functions and AWS services should run in the same region when possible. In this tutorial, we’re using eu-west-1 (Ireland).

Deploying AWS serverless

AWS popularized serverless apps with their Lambda service back in 2014. Serverless keeps us focused solely on the code without worrying about servers, infrastructure, or containers. There is no maintenance and we enjoy reduced runtime costs (at least at the beginning).

The flip side is that we’re limited to the supported languages and must work within the runtime limits imposed by the platform. Also, their event-driven nature forces us to rethink how applications are built.

Amazon published SAM (Serverless Application Model) as a way of standardizing and easing Lambda development. SAM is an open-source framework that provides shorthand expressions for interacting with the AWS ecosystem. Not only can it deploy serverless functions, it can also test them locally, debug them while running, access remote logs, and even help perform canary deployments.

Let’s see how it works.

Deploy the backend

We’ll start with the backend. The demo includes two choices for this; we have serverless functions in Java and Python. Both implement the same entry points:

  • Upload: adds a new face image to the database.
  • Recognize: scans a face image and looks it up in the database.

For the purposes of this article, we’ll work with the Java version.

Build the backend. Run sam build inside the backend folder in the monorepo.

$ cd java-app-backend/
$ sam build

The build artifacts are stored in the .aws-sam folder, so you may want to gitignore it.

The template file describes everything needed to run the functions in AWS: the API paths to expose, the permissions required, and which services they depend on.

Test serverless functions. SAM uses a Docker-based testing environment for rapid development. Try running sam local start-api to start the development environment. SAM creates a local HTTP server that hosts all your functions.

Alternatively, you can run sam local start-lambda coupled with sam local invoke to invoke a specific serverless function. In most cases, you’ll need to provide an event payload during the invocation, which you can generate with sam local generate-event.

For examples of how this works, check out the developer guide.

Deploy the backend. SAM packages and uploads code to S3. Then, it creates the AWS Lambda and API Gateway definitions so the functions can run on demand. Internally, SAM generates CloudFormation templates to provision everything.

To begin deployment, run the following command:

$ sam deploy --guided \
   --stack-name serverless-web-application-java-backend \
   --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM

The --guided option will walk you through all the steps. With --capabilities you’re granting extra permissions to CloudFormation, specifically to change IAM permissions and expand macros in the template.

When asked “X may not have authorization defined, Is this okay? Enter Y. This is SAM’s roundabout way of warning you that the function will be publicly-accessible.

In some cases, SAM may fail to create the S3 bucket (the error is that it can’t find bucket X). In that case, you’ll need to manually create it:

$ aws s3api create-bucket --bucket BUCKET_NAME --region eu-west-1

If you run into any errors, You can restart the deployment process from scratch by deleting the samconfig.toml file and rerunning sam deploy.

The deployment will take a few minutes. Once done, run the following command and check the id property of the function that has been created. Take note of it, as we’ll need it later on.

$ aws apigateway get-rest-apis

{
   "items": [
      {
           "id": "iaadyd31lk", // <-- this is the unique id of the API
           "name": "serverless-web-application-java-backend",
          ...

Backend build stage

Commit all changes made so far into the repository, as we’ll set up continuous deployment with Semaphore next.

Before continuing, add your repository to Semaphore and complete the basic project setup. Once that’s done, create a block with a single job in the build stage for the Java backend.

Take a look at these commands:

sem-version java 8
wget https://github.com/aws/aws-sam-cli/releases/latest/download/aws-sam-cli-linux-x86_64.zip
unzip aws-sam-cli-linux-x86_64.zip -d sam-installation
sudo ./sam-installation/install
checkout
cd java-app-backend
sam build
cache store build-java-app-backend-$SEMAPHORE_GIT_BRANCH .aws-sam

The build job begins by switching the active Java version with sem-version and installing SAM in the CI machine. After performing a checkout to clone the repository, we build the artifact, located in the .aws-sam folder, and save it in the cache.

Backend continuous integration

Next, scroll down the right pane until you find the Skip/Run conditions section. Change the condition to “Run this block when conditions are met” and enter the following into the When? field:

change_in('/java-app-backend')

change_in is the center of gravity of monorepo workflows because it allows us to tie blocks to folders and files in the repository. The function scans the Git history and figures out which parts of the code were recently changed and runs or skips CI/CD jobs based on what it finds, cutting down build times and costs.

📙 You can learn all about continuous integration for monorepos with our free ebook: CI/CD for Monorepos

Continuous deployment for the backend

First, click on Add First Promotion and enable automatic promotions. Then enter this condition into the When? field, which triggers the serverless app deployment when all tests pass on the master branch and the backend code has been modified.

branch = 'master' AND result = 'passed' AND change_in('/java-app-backend')

Create a deployment job in the new pipeline. By now, most commands should sound familiar. The only change is that we’re using --no-confirm-changeset, so the process is non-interactive.

sem-version java 8
wget https://github.com/aws/aws-sam-cli/releases/latest/download/aws-sam-cli-linux-x86_64.zip
unzip aws-sam-cli-linux-x86_64.zip -d sam-installation
sudo ./sam-installation/install
checkout
cd java-app-backend
cache restore build-java-app-backend-$SEMAPHORE_GIT_BRANCH
sam deploy --no-confirm-changeset --stack-name serverless-web-application-java-backend --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM
Deploy the backend job
Deploy Java pipeline

Finish setting up the CI/CD block by enabling the AWS secret you created at the beginning. Finally, save the changes into the master branch by clicking on Run the workflow > Start.

Deploy the backend pipeline running
Backend pipeline

Deploy the frontend

The serverless app frontend is a React SPA (Single Page Application) that interacts with the Python and Java backends. We’ll run it directly from an S3 Bucket.

In the file frontend/src/GlobalConstants.ts, you’ll find the URLs for both backends. Remember the id property for the serverless functions from earlier? You need it now. Update the URLs so they point to the Lambda functions.

Update the URL in frontend/src/GlobalConstants.ts

Now build the application with:

$ cd frontend/
$ yarn install
$ yarn build

Create the bucket either with the S3 console or with the CLI. Pay attention to the bucket name; it should be unique, so you may need to experiment until you find one that works for you:

$ aws s3api create-bucket \
   --bucket myfrontend-aws-monorepo \
   --region eu-west-1 \
   --create-bucket-configuration LocationConstraint=eu-west-1

Copy the files and then set the bucket to host the website with the following commands:

$ aws s3 cp build s3://myfrontend-aws-monorepo --recursive --acl public-read
$ aws s3 website s3://myfrontend-aws-monorepo --index-document index.html

The website can be visited with this URL (change the region as needed):

http://BUCKET_NAME.s3-website.eu-west-1.amazonaws.com

To complete the deployment, provision Cloudfront and direct it to the S3 website. The CDN will help you quickly deliver the application to end-users all over the world. Replace the URL of the website as needed:

$ aws cloudfront create-distribution --origin-domain-name myfrontend-aws-monorepo.s3-website.eu-west-1.amazonaws.com --default-root-object index.html

{
    "Location": "https://cloudfront.amazonaws.com/2020-05-31/distribution/E26Q8XKJFH6K4G",
    "ETag": "E1E0WE2I8CVJPH",
    "Distribution": {
        "Id": "E26Q8XKJFH6K4G",
        "ARN": "arn:aws:cloudfront::890702391356:distribution/E26Q8XKJFH6K4G",
        "Status": "InProgress",
        "LastModifiedTime": "2021-11-09T20:32:02.716000+00:00",
        "InProgressInvalidationBatches": 0,
        "DomainName": "dk4ef7pcvfhnv.cloudfront.net",  // <--- Cloudfront enabled URL

        . . .

Open a browser at the CloudFront-enabled site (use the returned DomainName value). In the upper right corner of your application, you can switch between the Java and Python backends.

Try registering some face images to see if everything is working as expected.

The frontend running on the cloud

You may check the Lambda remote execution logs with sam logs -n LAMBDA_FUNCTION_NAME --stack STACK_NAME.

Continuous deployment for the frontend

We’ll update our CI/CD pipeline to test and deploy the frontend every time it changes.

Open the workflow editor, add a new block to build the frontend and cache the build files.

sem-version node 16.13.0
checkout
cd frontend
cache restore
yarn install
cache store
yarn build
cache store build-frontend-$SEMAPHORE_GIT_BRANCH build
Building the frontend

Create a test block to run the React unit tests.

sem-version node 16.13.0
checkout
cd frontend
cache restore
yarn test
Testing the frontend
Frontend continuous integration

Then, set the Skip/Run conditions on both new blocks to the following:

change_in('/frontend')

Now, create a promotion and a deploy job. The condition for deployment is:

branch = 'master' AND result = 'passed' AND change_in('/frontend')

Finally, create a deployment job to copy the build files to the S3 bucket.

checkout
cd frontend
cache restore build-frontend-$SEMAPHORE_GIT_BRANCH
aws s3 cp build s3://${BUCKET_NAME} --recursive --acl public-read
Deploy frontend to S3
Frontend deployment pipeline

Enable the AWS secret, then click on Run the workflow > Start.

Serverless app monorepo pipeline
The final workflow

Try the backend and frontend deployments once more to check that everything is in order.

Conclusion

In this post, we’ve learned about a few tools to help us work AWS serverless hosted in a monorepo. We’ve seen how SAM can help us quickly develop and test serverless functions, and how to individually deploy services in a monorepo with Semaphore.

Read Next:

Have a comment? Join the discussion on the forum