Download an Entire S3 Bucket - Complete Guide
Borislav Hadzhiev
Last updated: Jul 26, 2022
Photo from Unsplash
Table of Contents #
Downloading an Entire S3 Bucket #
To download an entire bucket to your local file system, use the AWS CLI sync
command, passing it the s3 bucket as a source and a directory on your file
system as a destination, e.g. aws s3 sync s3://YOUR_BUCKET ..
The sync command recursively copies the contents of the source to the destination.
Let's first run the sync command in test mode by setting the --dryrun
parameter. This demonstrates which files and directories would be downloaded to
the local file system once the command is run.
aws s3 sync s3://YOUR_BUCKET . --dryrun
YOUR_BUCKET placeholder, otherwise you would get an AccessDenied error.The . character signifies the current directory. Create a directory that will
store the files and folders of the S3 bucket and navigate to it before running
the sync command in real mode.
Let's run the sync command to download all of the files and folders of the S3
bucket to the current directory on the local file system.
aws s3 sync s3://YOUR_BUCKET .
Downloading an S3 Bucket with Filters #
To filter the files that are downloaded from the bucket to the local file
system, make use of the --include and --exclude parameters when running the
sync command, e.g.
aws s3 sync s3://YOUR_BUCKET . --quiet --exclude "*.png".
By default, all files are included. To exclude all .png images from the
downloaded files, run the following command:
aws s3 sync s3://YOUR_BUCKET . --quiet --exclude "*.png"
The command downloads all of the bucket's files, except for files with the
.png extension.
We also used the --quiet parameter to suppress the sync command's output.
--quiet parameter would also suppress any errors the command throws, which can be confusing and hard to debug.We can also exclude an entire directory from being downloaded.
aws s3 sync s3://YOUR_BUCKET . --exclude "images/*"
We excluded all of the files located in the images/ folder, e.g.
images/cat.jpg, images/dog.png, etc.
Conversely, to download only .pdf files to the local file system, you would
exclude everything and include only files with the .pdf extension.
aws s3 sync s3://YOUR_BUCKET . --exclude "*" --include "*.pdf"
Note that the order is important when specifying both the --exclude and
--include parameters.
Filters that appear later in the command have higher precedence. For example,
the following command excludes all files because the --exclude parameter
overrides --include.
aws s3 sync s3://YOUR_BUCKET . --include "*.png" --exclude "*"
However, if we specify the --include parameter second, we would download all
.png images to our local file system:
aws s3 sync s3://YOUR_BUCKET . --exclude "*" --include "*.png"
Lastly, you can set the --exclude and --include parameters as many times as
necessary. For example, the following command downloads all .txt and .pdf
files from the bucket to the local file system.
aws s3 sync s3://YOUR_BUCKET . --exclude "*" --include "*.txt" --include "*.pdf"
Further Reading #
- Download a Folder from AWS S3
- List all Files in an S3 Bucket with AWS CLI
- Get the Size of a Folder in AWS S3 Bucket
- How to Get the Size of an AWS S3 Bucket
- Add a Bucket Policy to an AWS S3 Bucket
- Configure CORS for an AWS S3 Bucket
- Allow Public Read access to an AWS S3 Bucket
- Copy a Local Folder to an S3 Bucket
- How to Rename a Folder in AWS S3
- Copy Files and Folders between S3 Buckets
- How to Delete a Folder from an S3 Bucket
- Count Number of Objects in S3 Bucket
- AWS CDK Tutorial for Beginners - Step-by-Step Guide
- How to use Parameters in AWS CDK
