AWS Use Case 2— Serverless Lambda — S3 Event Trigger with multiple folders.

Problem Statement: Write an event trigger program using AWS Lambda so that when a file is uploaded in the source folder of source bucket, it will copy to require destination folder.

Solution Step:

Step-1: Create a source bucket called aws-datalake-sanjeeb-test-04062022
Step-2: Create 3 folders (for simple use case) i.e source1, source2,source3So the source bucket will look like below.

Step -3: Create a target bucket called aws-datalake-sanjeeb-target-test-04062022
Note- You can give any bucket name but it has to unique across AWS.

Step -4: Create an IAM role which will have S3 full access ( You can restrict S3 access but poc, I select the full access)
and Lambda basic execution and full lambda access. See the below screenshot

Step -5 : Create a lambada function with below details
Function Name: aws-datalake-event-trigger-src-tgt
Run time: Python 3.9
Attach the IAM role ( which is created in Step -4) by selecting Change default execution role option and click function

Step 6: In the function Code source tab, put the below code and click deploy.

#################### Code Snippet

import json
import sys
import boto3
import urllib

tgt_bucket_path=’aws-datalake-sanjeeb-target-test-04062022'

def lambda_handler(event, context):
#print(event)
bucket = event[‘Records’][0][‘s3’][‘bucket’][‘name’]
print(“Bucket Name: “ + bucket)
key = urllib.parse.unquote_plus(event[‘Records’][0][‘s3’][‘object’][‘key’],encoding=’utf-8')
print(“Folder and File Name:” + key)
print(“Target Bucket Path : “ + tgt_bucket_path)
# Copy the source to Target
s3 = boto3.resource(‘s3’)
copy_source = {‘Bucket’: bucket,’Key’: key}
s3.meta.client.copy(copy_source, tgt_bucket_path, ‘target/’+ key)
return {
‘statusCode’: 200,
‘body’: json.dumps(‘Hello from Lambda!’)
}

Step -7: Hold on Event trigger needs to be enabled.
To do the same , in the lambda page — Function overview — Add trigger ( See the below screenshot), Select S3 as Trigger and give the bucket name and event type ( you can select as per your choice) and select the recurisve methos ( This is just to ensure your source and target should not be in same bucket or else it is an infinite loop )

Step -8: Validate whether the trigger is added in the source bucket (Go to Source bucket -> Properties -> Event Notifications). So far so good.

Now its time to test e2e of the lambda execution .

Create 3 files source1.txt, source2.txt, source3.txt and upload in source bucket and respective folders.

The lambda function automatically trigger it and you can see it via cloudwatch logs. see the below screenshot

Step -9: You can also validate the destination bucket and folder and validate whether the files are copied or not.

Note-

  1. This is a simple poc, if you want to change the target location, folder you can always customize your lambda code.
  2. The source and target folders are also removed after the poc.

Keep learning.. keep enjoying.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store