Trigger Lambda On S3 Events Using CDK

AWS Lambda Trigger Using CDK & S3

Lambda functions can be called or invoked directly from the Lambda console, the Lambda API, the AWS SDK, the AWS CLI, and AWS toolkits. You can also configure other AWS services to invoke your function, or you can configure Lambda to read from a stream or queue and invoke your function.

The interesting thing about Lambda is that it has native integrations with 200 service offerings from AWS. This means you can use any of these serices to trigger the lambda functions.

If you are an AWS enthusiast and wanting to learn more about Lambdas and S3 for your project or an upcoming interview, you have come to the right place. You can either follow along with the code presented in this article or you can get the code from the GitHub — CDK Lamba S3 Event Listener.

If you are using this GitHub repo, follow the readme for setting up the project in your PC.

Lambda Trigger

A trigger is a Lambda resource or a resource in another service that you configure to invoke your function in response to lifecycle events, external requests, or on a schedule. Your function can have multiple triggers.

I will show you, one of the ways you can trigger or invoke the lambda functions using S3 events. For a brief understanding of this article, have a look at the workflow image below.

> Lambda S3 Event Trigger Workflow

Yes, it is going to be that simple, and not many components are required for us to implement this workflow using S3 triggers.

Each trigger acts as a client invoking your function independently. Each event that Lambda passes to your function only has data from one client or trigger.

Objectives

1. Setup AWS CLI and CDK for the development environment

2. Generate a CDK app

3. Write some python code for creating a Lambda function and S3 Bucket.

4. Attaching S3 events as a trigger for the created Lambda function

5. Deploy and run the app in AWS

Prerequisites for following this article

1. AWS free tier account

2. NPM and Node v12+ installed

3. Python v3.6+ version installed

4. Your Favorite code editor to write some lambda handler code

NOTE: This article will not cost you any money if you follow along. We will be utilizing the free tier usage only for this article.

Image Source—AWS

You can learn more about the free tier by visiting this link — AWS Free Tier

Setup AWS and CDK for the development environment

Before starting to write some code, you will need to do two things —

1. Set up AWS CLI

2. Configure AWS CLI with IAM Credentials

3. Set up AWS CDK

Follow the above links to set up your IAM credentials and AWS CDK for the project. If you face any difficulties while setting up IAM credentials or CDK, comment on this article so that I can help you resolve the issues.

Generate a CDK app

Once you have successfully set up CDK, it is time to get started with creating the project using CDK. Run the below command to initialize a CDK project —

$ mkdir cdk-lambda && cd cdk-lambda
$ cdk init app --language=python

cdk init initializes the project and the structure will look something like as shown below —

.
├── README.md
├── app.py
├── cdk.json
├── cdk-lambda
│ ├── __init__.py
│ └── cdk-lambda_stack.py
├── requirements.txt
├── setup.py
└── source.bat

As an additional step, create a folder with the name — lambda inside the project directory and create a python file with the name — LambdaListener.py inside the lambda directory. After this, the project structure will look something like the below —

.
├── README.md
├── app.py
├── lambda
│ └── LambdaListener.py
├── cdk.json
├── cdk-lambda
│ ├── __init__.py
│ └── cdk-lambda_stack.py
├── requirements.txt
├── setup.py
└── source.bat

After the init process completes and the virtualenv is created, you can use the following step to activate your virtualenv for Linux and MacOS.

$ source .venv/bin/activate

If you are working on a Windows platform, you would activate the virtualenv like this:

% .venv\Scripts\activate.bat

Now, let's add some dependencies that we will be needing to successfully run this project. Place following code inside requirements.txt file —

aws_cdk.core
aws_cdk.aws_s3
aws_cdk.aws_s3_notifications
aws_cdk.aws_lambda

Once added, run the below command to install the required dependencies —

$ pip install -r requirements.txt

Create Lambda function and S3 Bucket

Lambda is a compute service that lets you run code without provisioning or managing servers.

You organize your code into Lambda functions. Lambda runs your function only when needed and scales automatically, from a few requests per day to thousands per second. You pay only for the compute time that you consume — there is no charge when your code is not running.

You can invoke your Lambda functions using the Lambda API, or Lambda can run your functions in response to events from other AWS services. For example, you can use Lambda to:

• Build data-processing triggers for AWS services such as Amazon Simple Storage Service (Amazon S3) and Amazon DynamoDB.

• Process streaming data stored in Amazon Kinesis.

• Create your own backend that operates at AWS scale, performance, and security.

In our project, we will be using Amazon Simple Storage Service (Amazon S3) events to trigger the lambda function and record the event data. Following is our CDK stack code — cdk-lambda_stack.py

from aws_cdk import (aws_iam as iam,
             aws_lambda as _lambda,
             aws_s3 as s3,
             aws_s3_notifications as s3_notify,
             core as cdk)

class CdkLambdaStack(cdk.Stack):
    def __init__(self, scope: cdk.Construct,
                       construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # create new IAM group and use
        group = iam.Group(self, "VSGroup")
        user = iam.User(self, "VSUser")

        # Add IAM user to the group
        user.add_to_group(group)

        # Create S3 Bucket
        bucket = s3.Bucket(self, 'vs-bucket')
        bucket.grant_read_write(user)

        # Create a lambda function
        lambda_func = _lambda.Function(self, 'LambdaListener',
                       runtime=_lambda.Runtime.PYTHON_3_8,
                       handler='LambdaListener.handler',
                       code=_lambda.Code.asset('cdk_lambda\lambda'),
                       environment={'BUCKET_NAME':
                                    bucket.bucket_name})

        # Create trigger for Lambda function using suffix
        notification = s3_notify.LambdaDestination(lambda_func)
        notification.bind(self, bucket)

        # Add Create Event only for .jpg files
        bucket.add_object_created_notification(
           notification, s3.NotificationKeyFilter(suffix='.jpg'))

This stack will create an S3 bucket and a Lambda function. A notification handler will be created that will listen to the .jpg file create events in the s3 bucket. Once create event is fired, lambda function will get triggered and log the event inside cloudwatch logs. Below is the python code of the lambda function — LambdaListener.py

import boto3
import os

s3 = boto3.client('s3')

def handler(event, context):
    bucket_name = (os.environ['BUCKET_NAME'])
    key = event['Records'][0]['s3']['object']['key']

    try:

        # Log the event
        print("[LambdaListenet] New file with name {} created in bucket {}".format(key, bucket_name))

        response = {'status': 'success', 'key': key}
        return response

    except Exception as e:
        print(e)
        print("[Error] :: Error processing file {} from bucket {}.   ".format(key, bucket_name))
        raise e

In this lambda handler, we are going to receive the S3 event details and we will log that detail inside cloudwatch logs. Soon I will show this in action after we deploy this app to AWS.

Deploy and run the app in AWS

Now let us deploy this CDK project to AWS and see it in action.

All CDK Toolkit commands start with cdk, which is followed by a subcommand (list, synthesize, deploy, etc.).

Some subcommands have a shorter version (ls, synth, etc.) that is equivalent. Options and arguments follow the subcommand in any order. The available commands are summarized here.

Image Source—AWS

For us, only two commands are required. Staying in the outermost directory of the project i.e., cdk_lambda, run the below-mentioned command to deploy the app on the cloud —

$ cdk bootstrap

Since this CDK app contains an S3 bucket and other resources, we have to bootstrap it using this command as per guidelines mentioned in AWS documentation.

The process of provisioning these initial resources is called bootstrapping.

The command cdk bootstrap will create minimal AWS resources required to run the app. Now run the following command to finally deploy our app on the cloud —

$ cdk deploy

This command will log few cloud formation logs that are created on resource creation or initiation. You should see something like below once it is finally completed —

✅  CdkLambdaStack ARN:
arn:aws:cloudformation:<region>:<acc-id>:stack/CDKLambda/123

Two stacks are created as visible under the CloudFormation console for that region —

_CDK Stack in AWS CloudFormation

Now, if you upload a .jpg file inside the s3 bucket that was created using CDK stack, inside your cloudwatch log groups, you can see that the event is successfully logged.

Cloudwatch log group created for the Lambda Function

In the log groups shown above, you can select the latest one (usually the topmost log group), based on the date and time it was created, and you can see the log as shown below.

Event Logs inside Cloudwatch log group

How To Utilize This Project

There are many ways you can utilize this project.

For example, you can invoke an ECS task to initiate long-running processing on a video that was uploaded in an S3 bucket, you can invoke a glue job that will load the data inside the data catalog for data processing, etc.

This article gives you the easiest approach to get started with Lambda and S3 events and how you invoke a lambda using triggers from S3.

Find the GitHub link for this project here — https://github.com/varun-singh-01/lambda-s3-event-listener.

If you enjoyed this article, give me a clap and follow me for more updates.

More About The Author😄

I am a full-time software engineer with 4+ years of experience in Python, AWS, and many more technologies. I have started writing recently as it helps me to read more. I am looking forward to sharing technical knowledge and my life experiences with people out there.

Register Here for my Coders Weekly newsletter where I share interesting tutorials, python libraries, was, and tools, etc.

Github | LinkedIn | Twitter | Facebook | Quora | Coders Weekly