Using S3 Event Notifications in AWS CDK - Complete Guide

# Table of Contents

# Using S3 Event Notifications in AWS CDK

Bucket notifications allow us to configure S3 to send notifications to services like Lambda, SQS and SNS when certain events occur.

In order to add event notifications to an S3 bucket in AWS CDK, we have to call the addEventNotification method on an instance of the Bucket class.

We're going to add Lambda, SQS and SNS destinations for S3 Bucket event notifications.

Let's start with invoking a Lambda function every time an object is uploaded to an S3 bucket.

# Lambda Destination for S3 Bucket Notifications in AWS CDK

In order to define a lambda destination for an S3 bucket notification, we have to instantiate the LambdaDestination class, passing it a Lambda function.

The code for this article is available on GitHub

Let's define a Lambda function that gets invoked every time we upload an object to an S3 bucket:

lib/cdk-starter-stack.ts

Copied!
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as s3 from 'aws-cdk-lib/aws-s3';
import * as s3n from 'aws-cdk-lib/aws-s3-notifications';
import * as sns from 'aws-cdk-lib/aws-sns';
import * as sqs from 'aws-cdk-lib/aws-sqs';
import * as cdk from 'aws-cdk-lib';
import * as path from 'path';

export class CdkStarterStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // 👇 define lambda
    const lambdaFunction = new lambda.Function(this, 'lambda-function', {
      runtime: lambda.Runtime.NODEJS_18_X,
      handler: 'index.main',
      code: lambda.Code.fromAsset(path.join(__dirname, '/../src/my-lambda')),
    });

    // 👇 create bucket
    const s3Bucket = new s3.Bucket(this, 's3-bucket', {
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      autoDeleteObjects: true,
    });

    // 👇 invoke lambda every time an object is created in the bucket
    s3Bucket.addEventNotification(
      s3.EventType.OBJECT_CREATED,
      new s3n.LambdaDestination(lambdaFunction),
      // 👇 only invoke lambda if object matches the filter
      // {prefix: 'test/', suffix: '.yaml'},
    );

    new cdk.CfnOutput(this, 'bucketName', {
      value: s3Bucket.bucketName,
    });
  }
}

Let's go over the code snippet.

We created a Lambda function, which we'll use as a destination for an S3 event
We created an S3 bucket, passing it clean-up props that will allow us to delete the resources when we destroy the CDK stack later
We invoked the addEventNotification method on our bucket. The method takes 3 parameters:

Name	Description
`event`	the S3 event, on which the notification is triggered
`destination`	the destination for the notification. The supported services are Lambda, SQS and SNS.
`filters`	allow us to only send an `event` to the `destination` if the filter matches. For example, if the filter had `suffix` of `.yaml`, any other file suffixes would not trigger a notification.

We created an output for the bucket name to easily identify it later on when we test the integration

We subscribed a Lambda function to object creation events of the bucket and we haven't specified a filter. Every time an object is uploaded to the bucket, the Lambda function will get invoked.

The code for this article is available on GitHub

Let's add the code for the lambda at src/my-lambda/index.js:

src/my-lambda/index.js

Copied!
async function main(event) {
  console.log('event is 👉', JSON.stringify(event, null, 4));
  return {
    body: JSON.stringify({message: 'Success! 🎉🎉🎉'}),
    statusCode: 200,
  };
}

module.exports = {main};

The function logs the S3 event, which will be an array of the files we uploaded to S3, and returns a simple success message.

Let's run the deploy command, redirecting the bucket name output to a file.

shell

Copied!
npx aws-cdk deploy \
  --outputs-file ./cdk-outputs.json

The stack created multiple lambda functions because CDK created a custom resource for us behind the scenes.

If we locate our Lambda function in the management console, we can see that the S3 trigger has been set up to invoke the function on events of type ObjectCreated:

s3 event object created

CDK also automatically attached a resource-based IAM policy to the Lambda function that allows our S3 bucket to invoke it.

Let's manually upload an object to the S3 bucket using the management console and see if the Lambda function gets invoked.

You can find the bucket name in the cdk-outputs.json file in the root directory of your project.

After I've uploaded an object to the bucket, the CloudWatch logs show that the lambda function got invoked with an array of s3 objects:

s3 bucket notification lambda

When manipulating S3 objects in Lambda functions on create events be careful not to cause an infinite loop. If your lambda function gets triggered on object creation and uploads an object to the same S3 bucket, the process would repeat an infinite number of times.

We were able to successfully set up a lambda function destination for the S3 bucket notifications triggered on object creation events.

# SQS Destination for S3 Bucket Notifications in AWS CDK

The process for setting up an SQS destination for S3 bucket notification events is the same. We are going to create an SQS queue and pass it as the destination parameter to the addEventNotification method on the S3 bucket.

The code for this article is available on GitHub

lib/cdk-starter-stack.ts

Copied!
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as s3 from 'aws-cdk-lib/aws-s3';
import * as s3n from 'aws-cdk-lib/aws-s3-notifications';
import * as sns from 'aws-cdk-lib/aws-sns';
import * as sqs from 'aws-cdk-lib/aws-sqs';
import * as cdk from 'aws-cdk-lib';
import * as path from 'path';

export class CdkStarterStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // ... rest

    const queue = new sqs.Queue(this, 'sqs-queue');

    s3Bucket.addEventNotification(
      s3.EventType.OBJECT_REMOVED,
      new s3n.SqsDestination(queue),
      // 👇 only send message to queue if object matches the filter
      // {prefix: 'test/', suffix: '.png'},
    );

    new cdk.CfnOutput(this, 'queueName', {
      value: queue.queueName,
    });
  }
}

Let's go over what we did in the code sample:

We created an SQS queue.
We invoked the addEventNotification method on the S3 bucket. This time we are subscribing to the OBJECT_REMOVED event, which is triggered when one or multiple objects are removed from the S3 bucket.
For the destination, we passed our SQS queue, and we haven't specified a filter for the names of the objects that have to be deleted to trigger the event.
we created an output with the name of the queue.

Let's deploy the resources:

shell

Copied!
npx aws-cdk deploy \
  --outputs-file ./cdk-outputs.json

If we look at the access policy of the created SQS queue, we can see that CDK has automatically set up permissions that allow the S3 bucket to send messages to the queue:

sqs access policy

Let's delete the object we placed in the S3 bucket to trigger the OBJECT_REMOVED event and make S3 send a message to our queue.

You can either delete the object in the management console or via the CLI:

shell

Copied!
aws s3api delete-object \
  --bucket YOUR_BUCKET_NAME \
  --key file.txt

After I've deleted the object from the bucket, I can see that my queue has... 2 messages.

s3 event notification sqs

Keep in mind that, in rare cases, S3 might notify the subscriber more than once.

We've successfully set up an SQS queue destination for OBJECT_REMOVED S3 bucket events.

# SNS Destination for S3 Bucket Notifications in AWS CDK

Lastly, we are going to set up an SNS topic destination for S3 bucket notifications.

The code for this article is available on GitHub

lib/cdk-starter-stack.ts

Copied!
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as s3 from 'aws-cdk-lib/aws-s3';
import * as s3n from 'aws-cdk-lib/aws-s3-notifications';
import * as sns from 'aws-cdk-lib/aws-sns';
import * as sqs from 'aws-cdk-lib/aws-sqs';
import * as cdk from 'aws-cdk-lib';
import * as path from 'path';

export class CdkStarterStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // ... rest

    const topic = new sns.Topic(this, 'sns-topic');

    s3Bucket.addEventNotification(
      s3.EventType.REDUCED_REDUNDANCY_LOST_OBJECT,
      new s3n.SnsDestination(topic),
      // 👇 only send message to topic if object matches the filter
      // {prefix: 'test/', suffix: '.png'},
    );

    new cdk.CfnOutput(this, 'topicName', {
      value: topic.topicName,
    });
  }
}

Let's go over what we did in the code sample:

We created an SNS topic.
We invoked the addEventNotification method on the S3 bucket. The event we used is irrelevant, but our destination is the SNS topic we created.

We can only subscribe 1 service (lambda, SQS, SNS) to an event type. For example, we couldn't subscribe both lambda and SQS to the object create event.

There's no good way to trigger the event we've picked, so I'll just deploy to see if CDK has set up the necessary permissions for the integration.

shell

Copied!
npx aws-cdk deploy \
  --outputs-file ./cdk-outputs.json

If we take a look at the access policy of the SNS topic, we can see that CDK has automatically set up permissions for our S3 bucket to publish messages to the topic.

sns topic access policy

# Clean up

To delete the resources we have provisioned, run the destroy command:

shell

Copied!
npx aws-cdk destroy

# Further Reading

I wrote a book in which I share everything I know about how to become a better, more efficient programmer.

You can use the search field on my Home Page to filter through all of my articles.

Using S3 Event Notifications in AWS CDK - Complete Guide

# Table of Contents