An Easy Way to Share Secrets with your Lambda Function

Anyone who runs some workload on AWS Lambda has probably come across the limitations with passing secrets into the function (at least in a secure manner). Throughout AWS' documentation you have then probably come across Secret Manager and SSM (Systems Manager) as a way to help you with that. I like to use SSM as it is fairly easy to get started with and to introduce to your team as a centralized place to store secrets.

There are some nice documentations out there:

1. How to Securely Use Secrets in AWS Lambda, which describes a nice solution using an npm package called middy. They even have an official middleware for ssm.

When my team started working with SSM we had a solution similar to (1). We pretty much got tired of seeing this so close to our application code, it was demotivating work of having to mock or disabling this functionality when e.g. running tests or in our local development environment. We also had various Lambda function and using different runtime environments, even just executing a shell command.

Specifying What We Want!

Here is a trimmed down version of what I wanted to achieve in a CloudFormation template

What's most notable is that the environment variable names are prefixed with SSM_. Our view was that this should result in an environment variable SECRET_TOKEN set to the value read from /examples/token:1.

This is also similar to how you'd write that up using dynamic reference (not supported by Lambda environment variables) or pass secrets to an ECS task definition:

Behind the scenes

Runtime Execution Wrapper

In short Lambda exposes a way for you to modify the runtime environment by injecting a script into the Init phase of the Lambda execution environment lifecycle (Here's an example image of how an example lifecycle might look like).

You can use whatever scripting language you are comfortable with as long as it's available in the runtime environment (or it can even just be a binary executable). How you inject it is detailed nicely here. In short you just write your script and direct Lambda to run it by setting a predefined environment variable AWS_LAMBDA_EXEC_WRAPPER to the path of your script.

The SSM Wrapper Script

Let's get our BASH hats on and prepare our browsers with some large amounts of StackOverflow & Google result tabs. Here is the entire script below, and then we'll break it down a bit:

1. We'll loop over a list of lines of ENV_VARIABLE=value and filter only to those that are prefixed with `SSM_`

2. We strip each line of all characters that come after "=" and then strip away the SSM_ prefix. That way we know the variable name to use.

3. We then do the opposite and strip the line all characters that come before "=", effectively getting the name/path of the SSM parameter.

4. Then we use the aws-cli to fetch the parameter, decrypt it and output the value to ENV_VALUE. (Don't worry, the decryption part is skipped if it's not a SecureString type)

5. There is some light-weight error handling, where we just print out the failed paths (if needed we could fail exit immediately) but in the success case we export the new variable name with the value retrieved from SSM.

6. The "exec" part is what makes the script an actual wrapper i.e. replace the current process with the execution of the command line arguments, effectively sharing the newly exported environment variables with that process.

A Brief Mention of IAM

Now there are some basic permission levels that need to be set here (apart from the basic lambda execution policies). It basically boils down to two points:

• You need to give permission to read the target parameters from SSM

• If you're using `SecureString` types then you need to give permission to decrypt secrets using the KMS key which they were encrypted with. Most of the time it's just the default KMS key for SSM which is AWS managed.

Now here's an example policy document:

Let's see it in action

Build an example Lambda

Here's the lambda handler. The function's code is simple and just returns the environment variable values to us.

Let's build and package it with docker (take a moment to appreciate how great it is to deploy lambda code like that now!). Here's the Dockerfile:

Now at the time of writing this I did see an improvement here to make. I feel that installing the entire CLI here can bloat up the image size (if you're concerned about that). Since we're effectively just using the get-parameter from SSM you could probably just use the API directly with something like curl.

Invoke it!

We'll invoke it 3 times and view the logs in CloudWatch.

As you can see the execution environment gets reused across all invocations meaning that the wrapper script is executed once. You can also see the first log statements showing our wrapper script in action! Great!

The return value just like we expected! Cheers! :)

Conclusion

We made a fancy flexible BASH script which is executed during Lambda's Init phase using runtime execution wrapper. It retrieves variables from SSM Parameter Store and exports them as environment variables only to be accessible to the Lambda function handler. As the Lambda runtime execution wrapper is used in all runtime environments and our wrapper script is written in BASH, it's very re-usable!

References

Checkout the full example on GitHub.