Download a Folder from AWS S3

avatar

Borislav Hadzhiev

Last updated: Jul 25, 2022

banner

Photo from Unsplash

How to Download a Folder from AWS S3 #

Use the s3 cp command with the --recursive parameter to download an S3 folder to your local file system. The s3 cp command takes the S3 source folder and the destination directory as inputs and downloads the folder.

Create a folder on your local file system where you'd like to store the downloads from the bucket, open your terminal in that directory and run the s3 cp command.

Let's set the --dryrun parameter to run the command in test mode to verify everything looks good.

shell
aws s3 cp s3://YOUR_BUCKET/YOUR_FOLDER . --recursive --dryrun

download s3 folder test mode

The output of the dry run command shows that everything works as expected.

Now let's run the s3 cp command in real mode to download the contents of the S3 folder to our local file system.

shell
aws s3 cp s3://YOUR_BUCKET/YOUR_FOLDER . --recursive

download s3 folder

Replace the YOUR_BUCKET and YOUR_FOLDER placeholders with the bucket and directory names.

We set the --recursive parameter in the command. It makes the s3 cp command applicable to all files under the specified directory.

The . character signifies that the destination of the downloads is the current directory.

Set Filters when Downloading a Folder from AWS S3 #

Set the --include and --exclude parameters when running the cp command to filter the files that are downloaded to your local file system from a bucket.

The default behavior is that all files in the folder are included. For example, to exclude all files with a .jpg extension from being downloaded, run the following command:

shell
aws s3 cp s3://YOUR_BUCKET/YOUR_FOLDER . --recursive --exclude "*.jpg"

exclude jpg files from downloads

We can also exclude an entire directory from being downloaded:

shell
aws s3 cp s3://YOUR_BUCKET/YOUR_FOLDER/ . --recursive --exclude "movies/*"

The command excludes all files located in the movies/ directory, e.g. movies/movie-1.mp4 and movies/movie-2.mp5.

Conversely, to only download .txt files, we have to exclude everything and include only files with the .txt extension.

shell
aws s3 cp s3://YOUR_BUCKET/YOUR_FOLDER/ . --recursive --exclude "*" --include "*.txt"

only download txt files

Note that the order is important when setting both the --exclude and --include parameters.

The filters that appear later in the command have higher precedence. For instance, the following command excludes all files because the --exclude parameter overrides --include.

shell
aws s3 cp s3://YOUR_BUCKET/YOUR_FOLDER/ . --recursive --include "*.jpg" --exclude "*"

However, if we specify the --include parameter second, we would download all .jpg files from the specified directory.

shell
aws s3 cp s3://YOUR_BUCKET/YOUR_FOLDER/ . --recursive --exclude "*" --include "*.jpg"

Lastly, we can set the --exclude and --include parameters as many times as necessary.

For instance, the following command downloads all .jpg and .png files from the bucket:

shell
aws s3 cp s3://YOUR_BUCKET/YOUR_FOLDER/ . --recursive --exclude "*" --include "*.jpg" --include "*.png"

Further Reading #

I wrote a book in which I share everything I know about how to become a better, more efficient programmer.
book cover
You can use the search field on my Home Page to filter through all of my articles.