Running aws-cli Commands Inside An AWS Lambda Function ~ AwsTechNix

"Even though aws-cli is not available by default in AWS Lambda"

The AWS Lambda environments for each programming language (e.g., Python, Node, Java) already have the AWS client SDK packages pre-installed for those languages. For example, the Python AWS Lambda environment has boto3 available, which is ideal for connecting to and using AWS services in your function.

This makes it easy to use AWS Lambda as the glue for AWS. A function can be triggered by many different service events, and can respond by reading from, storing to, and triggering other services in the AWS ecosystem.

However, there are a few things that aws-cli currently does better than the AWS SDKs alone. For example, the following command is an efficient way to take the files in a local directory and recursively update a website bucket, uploading (in parallel) files that have changed, while setting important object attributes including MIME types guessing:

aws s3 sync --delete --acl public-read LOCALDIR/ s3://BUCKET/

The aws-cli software is not currently pre-installed in the AWS Lambda environment, but we can fix that with a little effort.

Background

The key to solving this is to remember that aws-cli is available as a Python package. Mitch Garnaat reminded me of this when I was lamenting the lack of aws-cli in AWS Lambda, causing me to smack my virtual forehead. Amazon has already taught us how to install most Python packages, and we can apply the same process for aws-cli, though a little extra work is required, because a command line program is involved.
NodeJS/Java/Go developers: Don’t stop reading! We are using Python to install aws-cli, true, but this is a command line program. Once the command is installed in the AWS Lambda environment, you can invoke it using the system command running functions in your respective languages.

Steps

Here are the steps I followed to add aws-cli to my AWS Lambda function. Adjust to suit your particular preferred way of building AWS Lambda functions.

Create a temporary directory to work in, including paths for a temporary virtualenv, and an output ZIP file:

tmpdir=$(mktemp -d /tmp/lambda-XXXXXX)
virtualenv=$tmpdir/virtual-env
zipfile=$tmpdir/lambda.zip

Create the virtualenv and install the aws-cli Python package into it using a subshell:

(
  virtualenv $virtualenv
  source $virtualenv/bin/activate
  pip install awscli
)

Copy the aws command file into the ZIP file, but adjust the first (shabang) line so that it will run with the system python command in the AWS Lambda environment, instead of assuming python is in the virtualenv on our local system. This is the valuable nugget of information buried deep in this article!

rsync -va $virtualenv/bin/aws $tmpdir/aws
perl -pi -e '$_ = "#!/usr/bin/python\n" if $. == 1' $tmpdir/aws
(cd $tmpdir; zip -r9 $zipfile aws)

Copy the Python packages required for aws-cli into the ZIP file:

(cd $virtualenv/lib/python2.7/site-packages; zip -r9 $zipfile .)

Copy in your AWS Lambda function, other packages, configuration, and other files needed by the function code. These don’t need to be in Python.

cd YOURLAMBDADIR
zip -r9 $zipfile YOURFILES

Upload the ZIP file to S3 (or directly to AWS Lambda) and clean up:

aws s3 cp $zipfile s3://YOURBUCKET/YOURKEY.zip
rm -r $tmpdir

In your Lambda function, you can invoke aws-cli commands. For example, in Python, you might use:

import subprocess
command = ["./aws", "s3", "sync", "--acl", "public-read", "--delete",
           source_dir + "/", "s3://" + to_bucket + "/"]
print(subprocess.check_output(command, stderr=subprocess.STDOUT))

Note that you will need to specify the location of the aws command with a leading "./" or you could add /var/task(cwd) to the $PATH environment variable.

Example

This approach is used to add the aws-cli command to the AWS Lambda function used by the AWS Git-backed Static WebsiteCloudFormation stack.

You can see the code that builds the AWS Lambda function ZIP file here, including the installation of the aws-cli command:

https://github.com/alestic/aws-git-backed-static-website/blob/master/build-upload-aws-lambda-function

Notes

• I would still love to see aws-cli pre-installed on all the AWS Lambda environments. This simple change would remove quite a bit of setup complexity and would even let me drop my AWS Lambda function inline in the CloudFormation template. Eliminating the external dependency and having everything in one file would be huge!
• I had success building awscli on Ubuntu for use in AWS Lambda, probably because all of the package requirements are pure Python. This approach does not always work. It is recommended you build packages on Amazon Linux so that they are compatible with the AWS Lambda environment.
• The pip install -t DIR approach did not work for aws-cli when I tried it, which is why I went with virtualenv. Tips welcomed.
• I am not an expert at virtualenv or Python, but I am persistent when I want to figure out how to get things to work. The above approach worked. I welcome improvements and suggestions from the experts.