Mon Sep 20 2021·3 min read
Photo by Holly Mandarich
To download an entire bucket to your local file system, use the AWS CLI
command, passing in the s3 bucket as a source and a directory on your file
system as a destination.
The sync command recursively copies the contents of the source to the destination.
Let's first run the
sync command in test mode, by setting the
parameter. This demonstrates which files and directories would be downloaded to
your local file system once the command is executed.
aws s3 sync s3://YOUR_BUCKET . --dryrun
YOUR_BUCKETplaceholder, otherwise you would get an
. characters signifies the current directory. Create a directory that will
store the files and folders of the S3 bucket and navigate to it before running
sync command in real mode.
Let's run the
sync command to download all of the files and folders of the S3
bucket to the current directory of our local file system:
aws s3 sync s3://YOUR_BUCKET .
To filter the files that are downloaded from the bucket to our local file
system, make use of the
--exclude parameters, when executing
By default all files are included. To exclude all
.png images from the
downloaded files, run the following command:
aws s3 sync s3://YOUR_BUCKET . --quiet --exclude "*.png"
The above command downloads all of the bucket's files, except for files with the
In the call to the
sync command we also used the
--quiet parameter to
suppress its output.
--quietparameter would also suppress any errors the command throws, which can be confusing and hard to debug.
We can also exclude an entire directory from being downloaded:
aws s3 sync s3://YOUR_BUCKET . --exclude "images/*"
In the command above, we've excluded all of the files located in the
folder, for example
Conversely, to only download
aws s3 sync s3://YOUR_BUCKET . --exclude "*" --include "*.pdf"
It's important to note that the order is important when specifying both the
Filters that appear later in the command have higher precedence, for example the
following command excludes all files, because the
aws s3 sync s3://YOUR_BUCKET . --include "*.png" --exclude "*"
However, if we specify the
--include parameter second, we would download all
.png images to our local file system:
aws s3 sync s3://YOUR_BUCKET . --exclude "*" --include "*.png"
Lastly, you can set the
--include parameters as many times as
you want, for example the following command downloads all
aws s3 sync s3://YOUR_BUCKET . --exclude "*" --include "*.txt" --include "*.pdf"
I'll send you 1 email a week with links to all of the articles I've written that week