Copy Files and Folders between S3 Buckets

Table of Contents #

Copying all files between S3 Buckets with AWS CLI #

To copy files between S3 buckets with the AWS CLI, run the s3 sync command, passing in the names of the source and destination paths of the two buckets. The command recursively copies files from the source to the destination bucket.

Let's run the command in test mode first. By setting the --dryrun parameter we can verify the command produces the expected output, without actually running it.

shell

Copied!
aws s3 sync s3://SOURCE_BUCKET s3://DESTINATION_BUCKET --dryrun

copy files between buckets test mode

The output of the command shows that without the --dryrun parameter, it would have copied the contents of the source bucket to the destination bucket.

Once you are sure the command does what you intend, exit test mode, by removing the --dryrun parameter.

The s3 sync command copies the objects from the source to the destination bucket, if:

the size of the objects differs
the last modified time of the source is newer than the last modified time of the destination
the s3 object does not exist under the specified prefix in the destination bucket

This means that if we had an image.png file in both the source and the destination buckets, it would only get copied if:

the size of the image differs
the last modified time of the image in the source is newer than the last modified time of the destination image.

Copying Files under a specific Path between S3 Buckets #

To copy files under a specific prefix, between s3 buckets, run the s3 sync command, passing in the complete source and destination bucket paths.

shell

Copied!
aws s3 sync s3://SOURCE_BUCKET/my-folder s3://DESTINATION_BUCKET/my-folder

copy files under prefix between buckets

This time I ran the command without the --dryrun parameter. The output shows that the files from source-bucket/folder have been copied to destination-bucket/folder.

To verify the files got copied successfully, run the s3 ls command.

shell

Copied!
aws s3 ls s3://DESTINATION_BUCKET/YOUR_FOLDER --recursive --human-readable

files copied successfully

Filtering which Files to Copy between S3 Buckets #

To filter which files should get copied from the source to the destination bucket, run the s3 sync, passing in the exclude and include parameters.

Let's look at an example where we copy all of the files with .png extension, under a specific folder, from the source to the destination bucket.

shell

Copied!
aws s3 sync s3://SOURCE_BUCKET s3://DESTINATION_BUCKET --exclude "*" --include "my-folder-1/*.png"

copy files between buckets with filter

Note that the order the --exclude and --include parameters matters. Parameters passed later in the command have higher precedence.

This means that if we reverse the order and pass the --include "my-folder-1/*.png" parameter first, followed by --exclude "*", we would exclude all files from the command and not copy anything.

You can repeat the --include and --exclude parameters as many times as necessary. For example, the following command copies all files with the .txt and .png extensions, from the source to the destination bucket.

shell

Copied!
aws s3 sync s3://SOURCE_BUCKET s3://DESTINATION_BUCKET --exclude "*" --include "my-folder-1/*.png" --include "my-folder-1/*.txt"

copy files with multiple filters

The output of the command shows that all files with .png and .txt extensions, under the my-folder-1 directory were successfully copied to the destination bucket.

Copy Files and Folders between S3 Buckets

Table of Contents #

Copying all files between S3 Buckets with AWS CLI #

Copying Files under a specific Path between S3 Buckets #

Filtering which Files to Copy between S3 Buckets #

Further Reading #

Borislav Hadzhiev