Borislav Hadzhiev
Reading time·3 min
Photo from Unsplash
To copy files between S3 buckets with the AWS CLI, run the s3 sync
command,
passing in the names of the source and destination paths of the two buckets. The
command recursively copies files from the source to the destination bucket.
Let's run the command in test mode first. By setting the --dryrun
parameter we
can verify the command produces the expected output, without actually running
it.
aws s3 sync s3://SOURCE_BUCKET s3://DESTINATION_BUCKET --dryrun
The output of the command shows that without the --dryrun
parameter, it would
have copied the contents of the source bucket to the destination bucket.
--dryrun
parameter.The s3 sync command copies the objects from the source to the destination bucket, if:
This means that if we had an image.png
file in both the source and the
destination buckets, it would only get copied if:
To copy files under a specific prefix, between s3 buckets, run the s3 sync
command, passing in the complete source and destination bucket paths.
aws s3 sync s3://SOURCE_BUCKET/my-folder s3://DESTINATION_BUCKET/my-folder
This time I ran the command without the --dryrun
parameter. The output shows
that the files from source-bucket/folder
have been copied to
destination-bucket/folder
.
To verify the files got copied successfully, run the s3 ls command.
aws s3 ls s3://DESTINATION_BUCKET/YOUR_FOLDER --recursive --human-readable
To filter which files should get copied from the source to the destination
bucket, run the s3 sync
, passing in the exclude
and include
parameters.
Let's look at an example where we copy all of the files with .png
extension,
under a specific folder, from the source to the destination bucket.
aws s3 sync s3://SOURCE_BUCKET s3://DESTINATION_BUCKET --exclude "*" --include "my-folder-1/*.png"
--exclude
and --include
parameters matters. Parameters passed later in the command have higher precedence.This means that if we reverse the order and pass the
--include "my-folder-1/*.png"
parameter first, followed by --exclude "*"
, we
would exclude all files from the command and not copy anything.
You can repeat the --include
and --exclude
parameters as many times as
necessary. For example, the following command copies all files with the .txt
and .png
extensions, from the source to the destination bucket.
aws s3 sync s3://SOURCE_BUCKET s3://DESTINATION_BUCKET --exclude "*" --include "my-folder-1/*.png" --include "my-folder-1/*.txt"
The output of the command shows that all files with .png
and .txt
extensions, under the my-folder-1
directory were successfully copied to the
destination bucket.