Using S3 Event Notifications in AWS CDK - Complete Guide

avatar

Borislav Hadzhiev

Sun May 09 20216 min read

Table of Contents #

  1. Using S3 Event Notifications in AWS CDK
  2. Lambda Destination for S3 Bucket Notifications in AWS CDK
  3. SQS Destination for S3 Bucket Notifications in AWS CDK
  4. SNS Destination for S3 Bucket Notifications in AWS CDK

Using S3 Event Notifications in AWS CDK #

Bucket notifications allow us to configure S3 to send notifications to services like Lambda, SQS and SNS when certain events occur.

In order to add an event notifications to an S3 bucket in AWS CDK, we have to call the addEventNotification method on an instance of the Bucket class.

In this article we're going to add Lambda, SQS and SNS destinations for S3 Bucket event notifications.

Let's start with invoking a lambda function every time an object in an S3 bucket is created.

Lambda Destination for S3 Bucket Notifications in AWS CDK #

In order to define a lambda destination for an S3 bucket notification, we have to instantiate the LambdaDestination class, passing in a lambda function.

The code for this article is available on GitHub

Let's define a lambda function that gets invoked every time we upload an object to an S3 bucket:

lib/cdk-starter-stack.ts
import * as s3n from '@aws-cdk/aws-s3-notifications'; import * as lambda from '@aws-cdk/aws-lambda'; import * as s3 from '@aws-cdk/aws-s3'; import * as cdk from '@aws-cdk/core'; import * as path from 'path'; export class CdkStarterStack extends cdk.Stack { constructor(scope: cdk.App, id: string, props?: cdk.StackProps) { super(scope, id, props); // ๐Ÿ‘‡ define lambda const lambdaFunction = new lambda.Function(this, 'lambda-function', { runtime: lambda.Runtime.NODEJS_14_X, handler: 'index.main', code: lambda.Code.fromAsset(path.join(__dirname, '/../src/my-lambda')), }); // ๐Ÿ‘‡ create bucket const s3Bucket = new s3.Bucket(this, 's3-bucket', { removalPolicy: cdk.RemovalPolicy.DESTROY, autoDeleteObjects: true, }); // ๐Ÿ‘‡ invoke lambda every time an object is created in the bucket s3Bucket.addEventNotification( s3.EventType.OBJECT_CREATED, new s3n.LambdaDestination(lambdaFunction), // ๐Ÿ‘‡ only invoke lambda if object matches the filter // {prefix: 'test/', suffix: '.yaml'}, ); new cdk.CfnOutput(this, 'bucketName', { value: s3Bucket.bucketName, }); } }

Let's go over the code snippet.

  1. we created a lambda function, which we'll use as a destination for an s3 event
  2. we created an s3 bucket, passing in clean up props that will allow us to delete the resources when we destroy the cdk stack later
  3. we invoked the addEventNotification method on our bucket. The method takes 3 parameters:
NameDescription
eventthe s3 event, on which the notification is triggered
destinationthe destination for the notification. The supported services are Lambda, SQS and SNS.
filtersallow us to only send an event to the destination if the filter matches. For example, if filter had suffix of .yaml, any other file suffixes would not trigger a notification.
  1. we created an output for the bucket name to easily identify it later on when we test the integration

In our case we have subscribed a lambda function to object creation events of the bucket and we haven't specified a filter. Every time an object is uploaded to the bucket, the lambda function will get invoked.

The code for this article is available on GitHub

Let's add the code for the lambda at src/my-lambda/index.js:

src/my-lambda/index.js
async function main(event) { console.log('event is ๐Ÿ‘‰', JSON.stringify(event, null, 4)); return { body: JSON.stringify({message: 'Success! ๐ŸŽ‰๐ŸŽ‰๐ŸŽ‰'}), statusCode: 200, }; } module.exports = {main};

The function logs the S3 event, which will be an array of the files we uploaded to S3, and returns a simple success message.

Let's execute a deployment, redirecting the bucket name output to a file:

shell
npx cdk deploy \ --outputs-file ./cdk-outputs.json

The stack created multiple lambda functions, because CDK created a custom resource for us behind the scenes.

If we locate our lambda function in the management console, we can see that the S3 trigger has been set up to invoke the function on events of type ObjectCreated:

s3 event object created

CDK also automatically attached a resource-based IAM policy to the lambda function, that allows our S3 bucket to invoke it.

Let's manually upload an object to the S3 bucket using the management console and see if the lambda function gets invoked.

You can find the bucket name in the cdk-outputs.json file in the root directory of your project.

After I've uploaded an object to the bucket, the CloudWatch logs show that the lambda function got invoked with an array of s3 objects:

s3 bucket notification lambda

When manipulating S3 objects in lambda functions, on create events, be careful not to cause an infinite loop. If your lambda function gets triggered on object creation and uploads an object to the same S3 bucket, the process would repeat an infinite number of times.

We were able to successfully set up a lambda function destination for S3 bucket notifications triggered on object creation events.

SQS Destination for S3 Bucket Notifications in AWS CDK #

The process for setting up an SQS destination for S3 bucket notification events is the same. We are going to create an SQS queue and pass it as the destination parameter to the addEventNotification method on the S3 bucket.

The code for this article is available on GitHub
lib/cdk-starter-stack.ts
import * as lambda from '@aws-cdk/aws-lambda'; import * as s3 from '@aws-cdk/aws-s3'; import * as s3n from '@aws-cdk/aws-s3-notifications'; import * as sqs from '@aws-cdk/aws-sqs'; import * as cdk from '@aws-cdk/core'; import * as path from 'path'; export class CdkStarterStack extends cdk.Stack { constructor(scope: cdk.App, id: string, props?: cdk.StackProps) { super(scope, id, props); // ... rest const queue = new sqs.Queue(this, 'sqs-queue'); s3Bucket.addEventNotification( s3.EventType.OBJECT_REMOVED, new s3n.SqsDestination(queue), // ๐Ÿ‘‡ only send message to queue if object matches the filter // {prefix: 'test/', suffix: '.png'}, ); new cdk.CfnOutput(this, 'queueName', { value: queue.queueName, }); } }

Let's go over what we did in the code snippet.

  1. we created an SQS queue

  2. we invoked the addEventNotification method on the s3 bucket. This time we are subscribing to the OBJECT_REMOVED event, which is triggered when one or multiple objects are removed from the S3 bucket.

    For the destination we've passed our SQS queue, and we haven't specified a filter for the names of the objects that have to be deleted to trigger the event.

  3. we created an output with the name of the queue

Let's deploy the resources:

shell
npx cdk deploy \ --outputs-file ./cdk-outputs.json

If we look at the access policy of the created SQS queue, we can see that CDK has automatically set up permissions, that allow the S3 bucket to send messages to the queue:

sqs access policy

Let's delete the object we placed in the S3 bucket to trigger the OBJECT_REMOVED event and make S3 send a message to our queue.

You can either delete the object in the management console, or via the CLI:

shell
aws s3api delete-object \ --bucket YOUR_BUCKET_NAME \ --key file.txt

After I've deleted the object from the bucket, I can see that my queue has... 2 messages.

s3 event notification sqs

Keep in mind that, in rare cases, S3 might notify the subscriber more than once.

We've successfully set up an SQS queue destination for OBJECT_REMOVED S3 bucket events.

SNS Destination for S3 Bucket Notifications in AWS CDK #

Lastly, we are going to set up an SNS topic destination for S3 bucket notifications.

The code for this article is available on GitHub
lib/cdk-starter-stack.ts
import * as lambda from '@aws-cdk/aws-lambda'; import * as s3 from '@aws-cdk/aws-s3'; import * as s3n from '@aws-cdk/aws-s3-notifications'; import * as sns from '@aws-cdk/aws-sns'; import * as sqs from '@aws-cdk/aws-sqs'; import * as cdk from '@aws-cdk/core'; import * as path from 'path'; export class CdkStarterStack extends cdk.Stack { constructor(scope: cdk.App, id: string, props?: cdk.StackProps) { super(scope, id, props); // ... rest const topic = new sns.Topic(this, 'sns-topic'); s3Bucket.addEventNotification( s3.EventType.REDUCED_REDUNDANCY_LOST_OBJECT, new s3n.SnsDestination(topic), // ๐Ÿ‘‡ only send message to topic if object matches the filter // {prefix: 'test/', suffix: '.png'}, ); new cdk.CfnOutput(this, 'topicName', { value: topic.topicName, }); } }

Let's go over what we did in the code snippet.

  1. we created an SNS topic
  2. we invoked the addEventNotification method on the S3 bucket. The event we've used is irrelevant, but our destination is the SNS topic we created.
We can only subscribe 1 service (lambda, SQS, SNS) to an event type. For example, we couldn't subscribe both lambda and SQS to the object create event.

There's no good way to trigger the event we've picked, so I'll just deploy to see if CDK has set up the necessary permissions for the integration.

shell
npx cdk deploy \ --outputs-file ./cdk-outputs.json

If we take a look at the access policy of the SNS topic, we can see that CDK has automatically set up permissions for our S3 bucket to publish messages to the topic.

sns topic access policy

Clean up #

To delete the resources we have provisioned, execute the destroy command:

shell
npx cdk destroy

Further Reading #

Join my newsletter

I'll send you 1 email a week with links to all of the articles I've written that week

Buy Me A Coffee