Copy Files and Folders between S3 Buckets

Table of Contents #

1. Copy all files between S3 Buckets with AWS CLI
2. Copy Files under a specific Path between S3 Buckets
3. Filtering which Files to Copy between S3 Buckets

Copying all files between S3 Buckets with AWS CLI #

To copy files between S3 buckets with the AWS CLI, run the s3 sync command, passing in the names of the source and destination paths of the two buckets. The command recursively copies files from the source to the destination bucket.

Let's run the command in test mode first. By setting the --dryrun parameter we can verify the command produces the expected output, without actually running it.

Copied!
aws s3 sync s3://SOURCE_BUCKET s3://DESTINATION_BUCKET --dryrun

The output of the command shows that without the --dryrun parameter, it would have copied the contents of the source bucket to the destination bucket.

The s3 sync command copies the objects from the source to the destination bucket, if:

• the size of the objects differs
• the last modified time of the source is newer than the last modified time of the destination
• the s3 object does not exist under the specified prefix in the destination bucket

This means that if we had an image.png file in both the source and the destination buckets, it would only get copied if:

• the size of the image differs
• the last modified time of the image in the source is newer than the last modified time of the destination image.

Copying Files under a specific Path between S3 Buckets #

To copy files under a specific prefix, between s3 buckets, run the s3 sync command, passing in the complete source and destination bucket paths.

Copied!
aws s3 sync s3://SOURCE_BUCKET/my-folder s3://DESTINATION_BUCKET/my-folder

This time I ran the command without the --dryrun parameter. The output shows that the files from source-bucket/folder have been copied to destination-bucket/folder.

To verify the files got copied successfully, run the s3 ls command.

Copied!
aws s3 ls s3://DESTINATION_BUCKET/YOUR_FOLDER --recursive --human-readable

Filtering which Files to Copy between S3 Buckets #

To filter which files should get copied from the source to the destination bucket, run the s3 sync, passing in the exclude and include parameters.

Let's look at an example where we copy all of the files with .png extension, under a specific folder, from the source to the destination bucket.

Copied!
aws s3 sync s3://SOURCE_BUCKET s3://DESTINATION_BUCKET --exclude "*" --include "my-folder-1/*.png"

This means that if we reverse the order and pass the --include "my-folder-1/*.png" parameter first, followed by --exclude "*", we would exclude all files from the command and not copy anything.

You can repeat the --include and --exclude parameters as many times as necessary. For example, the following command copies all files with the .txt and .png extensions, from the source to the destination bucket.

Copied!
aws s3 sync s3://SOURCE_BUCKET s3://DESTINATION_BUCKET --exclude "*" --include "my-folder-1/*.png" --include "my-folder-1/*.txt"

The output of the command shows that all files with .png and .txt extensions, under the my-folder-1 directory were successfully copied to the destination bucket.

Further Reading #

• AWS CDK Tutorial for Beginners - Step-by-Step Guide
• How does AWS CDK work
• What is a Token in AWS CDK
• CDK Constructs - Complete Guide
• What is an identifier (id) in AWS CDK
• How to use Context in AWS CDK
• How to use Parameters in AWS CDK