Since I discovered the main concepts and benefits of Serverless applications, I’m gradually migrating the few services I run on my personal server(s) towards solutions which require no need for me to keep any (virtual) server running. The reason is simple: there are so many other interesting things to do than keeping an online system in good shape. If that’s not enough, leveraging FaaS (at least for very small workloads) usually brings cheaper bills compared to traditional hosting.
Even if there are quite a few nice tutorials available on how to setup common kind of Serverless projects, I can’t remember one covering all the requirements I had. This time around it’s also true, therefore I’ll try to show here the steps I took for implementing and deploying a simple API backend using Python (v3.6), AWS Lambda, AWS API Gateway and Terraform. This simple diagram shows how the setup is supposed to look like:
The API backend source code for our example is the simplest one could imagine, but it should be trivial to extend it to cover more complex cases. The API Gateway Lambda integration makes use of the relatively new proxy feature, which means all the HTTP requests hitting the gateway endpoint (except OPTIONS, intercepted by the Gateway for serving proper CORS headers) is forwarded to the Lambda as is, including path and method in the metadata, so the routing logic can be extended without changing any API Gateway configuration.
Let’s have a look at the Python code in our example: for each HTTP POST request to /contact_us, after some basic input sanitation, parse the body in search of some special attributes and, if the request is formatted as expected, send a message via the Pushover service:
NOTE: when using API Gateway Lambda proxy feautre, be careful to use the proper response format as shown above at line 2, otherwise you might step into errors hard to debug.
/contact_us endpoint is currently used by a couple of HTML forms and It notifies me on my smartphone when there’s any new submission, with no need for exposing my email address (or any other personal data) in the site HTML/JS code. I found a bunch of articles showing how to setup something very similar, but none fullfilling all my requirements. Here they are, in descending order of importance:
- the whole system is fully menaged (a.k.a. Serverless)
- limit the clicks needed for the setup at the bare minimum
- API endpoint is exposed as a vanity url with valid SSL certificate
- CORS setup should allow requests from any domain
- follow the Principle of least privilege, e.g. IAM permissions are set as tight as possible
- no resource usage surprises (e.g. no infinite expiration date for CloudWatch Logs)
- no Serverless frameworks needed
- low cost, possibly free of charge
I consider reproducible deployments a hard requirement when using cloud services: I don’t want to wander around web consoles pointing and clicking to setup my services, it might be the right approach for prototyping but definitely not for deploying anything to be considered production ready.
I have nothing against the various Serverless frameworks out there, quite the contrary: they are definitely interesting tools fitting various needs, for example when I used AWS Chalice for another project I was quite happy with it, but for something as simple as an HTML form backend, the need for a framework feels like unnecessary overhead. There’s another reason too: frameworks do a good job hiding part of the platform compexities, but at least this time around I want to have the best possible understanding of all the components involved. Not taking any shortcut, better known as the hard way, is in my experience one of the best ways to fully understand the tools I’m dealing with.
I also want to expose the endpoint making use of the domain name I registered in the past: something like
https://api.mycoolservice.com/ looks definitely more polished than
https://987654abcdef.cloudfront.net, not to mention it’s way easier to remember. To make use of our new REST API endpoint from any HTTP client we also need to add some CORS configuration which is also supported by API Gateway.
It’s a bit surprising to find that proper management of AWS resources for Lamba is missing from most of the tutorials I’ve found. I’m referring specifically to CloudWatch Logs expiration (default is no expiration) and IAM permission (default is write permission to every LogGroup in the account). What I generally want is for logs older then one week to expire automatically, and to tighten the default IAM configuration for Lambda to reduce the attack surface . The setup will cover those parts too with just a couple of more Terraform resources. Maybe this Terraform snippet from
infra/lambda.tf better explains what I mean:
Let’s get started
From now on I’ll assume that you already have:
- registered a domain (
mycoolservice.comin our example) and that you can add DNS records to it
- can read emails sent to
firstname.lastname@example.org(or similar recipients, more details will follow)
- an AWS account with administrator rights
- installed Terraform v.0.11.7 (or higher) with its AWS provider v1.27.0 (or higher)
- installed aws-cli v1.15.53 (or higher)
Terraform will use your AWS aws-cli credentials to create the resources, so be careful to have
~/.aws/credentials file properly setup (or whatever other configuration method you’re using for aws CLI, for example env variables if that’s what you’re using).
NOTE: later on you’ll need to edit a configuration file,
config.sh, and one of the variables is called
$profile and it contains the name of the aws-cli profile in said credentials file that Terraform will use, so you’ll be able to specify anything other then default if needed.
First step: create an SSL certificate with AWS Certificate Manager
To begin with, we’ll need an SSL certificate fully managed by ACM. AWS will try to send you an email to a few addresses, like
email@example.com (check out the official documentation for details) to be sure you own the domain. If you prefer, there’s also a DNS verification where you need to add some DNS record. Run this command to request an SSL cert for
api.mycoolservice.com using email validation:
At this point you should check your email from ACM and click on the verify link before proceeding.
Clone the example code and add your configuration
I set up a GIT repository on GitHub with all that’s needed to setup this proof of concept, including the Python Lambda, the Terraform configuration and a simple deployment script. You should clone it in your working directory:
serverless-api folder there’s an example configuration file:
config.sh.example content into a new
config.sh file and edit
config.sh with your details. If you don’t have a Pushover account you can just leave the
$pushover* variables empty (or unchanged, they are bogus anyway), the Lambda will fail without sending any push notification but you’ll still be able to test that everything else is working as expected.
Let’s have a quick look at the Terraform files in the
main.tf there are all the variable declarations and the AWS provider. Having them we can pass those values at runtime with
-var name=value in our deploy script. Nothing really interesting here. Things get a bit more complex for setting up the Lambda properly:
Beside the need to interpolate the
replace function to avoid invalid characters when naming our AWS resources (dot is not valid for Lambda names for example), it should be fairly readable. We obviously need a Lambda, a IAM role to associate with it, some IAM policies to specify what kind of privileges are needed, and a CloudWatch LogGroup to collect logs.
What’s left is the most complex part which is setting up API Gateway. Thankfully, Hashicorp published a good tutorial which I basically copied verbatim. I only had to add a couple of fixes required to make CORS work in my case and the few resources needed to link the SSL certificate and expose it with my domain name:
At this point everything should be ready to be set up in AWS. You simply need to execute
build.sh shell script:
You’ll be prompted by Terraform to type yes to continue, that’s your last chance for checking that everything shown by Terraform plan looks as expected. If Terraform apply execution completes successfully, those new resources will be created in your AWS account:
Fourth step: add a CNAME to enable your vanity URL
If everything has been created successfully, you should be able to see something like this configured in API gateway:
NOTE: adding a custom domain to the CloudFront distribution linked to API Gateway will take some minutes (~20), so be patient if the above command doesn’t show you what’s expected right after Terraform execution has terminated.
distributionDomainName value above is the one you need to add as a DNS record for your
api.mycoolservice.com entry. To test that this step is complete, you could run for example this CLI command and search for a similar output:
Test your new API
If you reached this point everythings should be set and ready to be tested end to end! Terraform took care of everything, including:
- packaging the Lambda with the Python code and uploading it
- configuring API Gataway integration and CORS settings
- linking your custom domain to the API Gateway (Cloudfront) endpoint making use of the SSL certificate registered at the beginning of the process
Let’s check with Curl that everything is actually in place and working as expected:
CORS configuration should allow requests from any origin:
NOTE: we didn’t create any setup for the root path (/), a request there will return a misleading message:
Deploy your own application
In this example we’be been using one of my personal project as template, but it’s easy enough to replace the
src/lambda.py code with your business logic, or even replace Python with one of the other supported runtimes (Node.js, Java, C# and Go).
API Gateway configuration should be now setup correctly without any need to be modified, so each new
deploy.sh execution will only update the content of the Lambda resource with the code in
src/lambda.py. You might also want to change the environment variables provided to the Lambda: to do that you need to edit both the Terraform code (i.e.
infra/lambda.tf) and the
deploy.sh script, and replace/extend with with your data accordingly:
Bring your own source
This tutorial is completed, but you could go on and experiment some more with the setup just editing
src/lambda.py and running
deploy.sh, Terraform will notice the source file has changed and upload the new version.
This is an example of logs generated by a proper request, you can find them all in the CloudWatch Logs LogGroup
Destroy all the things
When you’re done experimenting, you might want to get rid of all the AWS resources we created so far. It’s as easy as replacing apply with destroy in
API Gateway Lambda proxy integration, together with Amazon Certificate Manager, makes it very cheap (basically free for small traffic loads) and easy to setup HTTP REST endpoints and expose them on our custom domains. Here I showed you how to put together all the needed scaffolding with Terraform and a few lines of Bash to have a basic no clicks needed solution.
It should be easy to customize the example code and make this solution fit to more restrictive requirements, for example adding proper testing/staging/production lifecycles, monitoring, alerting, etc.
Please leave a comment for feedback or just ask for help if something is not clear or not working for you.