How to run aws cli command in lambda function

When an user is planning to run aws cli commands, it is best to call AWS API calls via boto3, however there are other some specific use cases where user wants to run aws cli commands directly using lambda function. One such use case is to create an EMR cluster via lambda function ** (we will demonstrate the creation of EMR in a separate blog)

This document provides step by step how to enable aws cli in the lambda function

Step -1: Configuring Python and pip in a linux environment. Here we have used aws ec2 instance to demonstrate the e2e activity.

Most of the Amazon EC2 Linux instances by default have python 3 packages. To know whether python is installed or not, type the command “yum list installed | grep -i python3”. If the python3 is already available, user can see below details.

yum list installed | grep -i python3

If python3 is not installed (the above command did not give any output), run the below command.

sudo yum install python3 -y

Step -2: To know where python is installed, type the command — whereis python3

Step -3: Create a virtual environment under the ec2-user home directory

For easy steps, create some temp variables

# Temporary directory for the virtual environment
export VIRTUAL_ENV_DIR="awscli-virtualenv"
# Temporary directory for AWS CLI and its dependencies
export LAMBDA_LAYER_DIR="awscli-lambda-layer"
# The zip file that will contain the layer
export ZIP_FILE_NAME=”awscli-lambda-layer.zip”

1. To install virtual environment, type python3 -m venv ${VIRTUAL_ENV_DIR}

# Changes current dir to the virtual env directory
cd ${VIRTUAL_ENV_DIR}/bin/
# Activate virtual environment
source activate

2. To install aws cli, type pip install awscli

pip install awscli

Step -3: To use aws cli in lambda function, need to change the path of the python in the aws file ( which is created after installing aws cli).

Open the file aws in vi mode and delete the first line and paste the line — #!/var/lang/bin/python

Step -4: Deactivate the virtual env by the command deactivate

Step -5: Next step is to package the aws cli packages so that we can be used it in lambda

# Changes current directory back to where it started
cd ../..
# Creates a temporary directory to store AWS CLI and its dependencies
mkdir ${LAMBDA_LAYER_DIR}
# Changes the current directory into the temporary directory
cd ${LAMBDA_LAYER_DIR}
# Copies aws and its dependencies to the temp directory
cp ../${VIRTUAL_ENV_DIR}/bin/aws .
cp -r ../${VIRTUAL_ENV_DIR}/lib/python3.7/site-packages/. .
# Zips the contents of the temporary directory
zip -r ../${ZIP_FILE_NAME} *

Step -6: Copy the zip file to a s3 path so that we can use it as a layer in lambda function. Change your S3 path :)

aws s3 cp awscli-lambda-layer.zip s3://sanjeeb-poc-lab-001/input/

Step -7: Now we can create a lambda layer , to do the same go the Lambda console, click on Layers in the left navigation and click create layer to create a new layer.

Step -8:

· Give the name of the layer — aws-cli-layer

· Select the upload a file from S3 as we have copy the zip file to S3

· Select the compatible architecture ( in this case we select x86_64)

· Select the python version ( since the virtual environment is created in Python3.7, so we selected python 3.7)

Step -9: Once the layer is created, we can see the details.

Step -10: To test whether the layer is working fine or not, create a lambda function

Go to Lambda, click on function and create a function

Provide the basic information of the lambda and click create.

After the function is created, to add the layer, click Layer in the design tab

Step -11: In the lambda function, paste the below program ( NOTE — This is a test program to test whether you lambda is able to run aws cli commands or not)

import subprocess

import logging

logger = logging.getLogger()

logger.setLevel(logging.INFO)

def run_command(command):

command_list = command.split(‘ ‘)

try:

logger.info(“Running shell command: \”{}\””.format(command))

result = subprocess.run(command_list, stdout=subprocess.PIPE);

logger.info(“Command output:\n — -\n{}\n — -”.format(result.stdout.decode(‘UTF-8’)))

except Exception as e:

logger.error(“Exception: {}”.format(e))

return False

return True

def lambda_handler(event, context):

run_command(‘/opt/aws — version’)

Note — This is NOT my code, I followed the blog — https://bezdelev.com/hacking/aws-cli-inside-lambda-layer-aws-s3-sync/

Step -12: Click deploy to save the lambda function, create a test event to test the function. Click on test to check the lambda function output.

Well done, you are able to run aws cli command via lambda function. All the credit goes to the wonderful blog — https://bezdelev.com/hacking/aws-cli-inside-lambda-layer-aws-s3-sync/