How to merge text files in Python [5 simple Ways]

# Table of Contents

1. How to merge text files in Python
2. Merging smaller text files in Python
3. Merging text files in Python using shutil.copyfileobj()
4. Merging text files in Python by using fileinput
5. Using the glob() method to merge text files in Python

# How to merge text files in Python

To merge text files in Python:

1. Store the paths to the text files in a list.
2. Use the with open() statement to open the output file for writing.
3. Use a for loop to iterate over the file paths.
4. On each iteration, open the current file for reading and write its contents to the output file.

The example project has the following folder structure.

Copied!
my-project/
  └── main.py
  └── file-1.txt
  └── file-2.txt

Here are the contents of file-1.txt.

Copied!
bobby
hadz
.
com

And here are the contents of file-2.txt.

Copied!
one
two
three
four

This is the code for the main.py file.

Copied!
file_paths = ['file-1.txt', 'file-2.txt']

with open('output-file.txt', 'w', encoding='utf-8') as output_file:
    for file_path in file_paths:
        with open(file_path, 'r', encoding='utf-8') as input_file:
            for line in input_file:
                output_file.write(line)

If there is no end-of-line character at the end of each file, you'd have to manually add one.

Copied!
file_paths = ['file-1.txt', 'file-2.txt']

with open('output-file.txt', 'w', encoding='utf-8') as output_file:
    for file_path in file_paths:
        with open(file_path, 'r', encoding='utf-8') as input_file:
            for line in input_file:
                output_file.write(line)
            # ✅ Manually add a newline character at the end of each file
            output_file.write('\n')

The last line adds a newline (\n) character at the end of each file.

If I run the script with python main.py, the following output-file.txt file is produced.

Copied!
bobby
hadz
.
com
one
two
three
four

We used the with open() statement to open the output file for writing.

The next step is to iterate over the list of file paths.

On each iteration, we open the file at the current file path for reading.

Copied!
file_paths = ['file-1.txt', 'file-2.txt']

with open('output-file.txt', 'w', encoding='utf-8') as output_file:
    for file_path in file_paths:
        with open(file_path, 'r', encoding='utf-8') as input_file:
            for line in input_file:
                output_file.write(line)

The next step is to use a for loop to iterate over the lines of each input file.

On each iteration, we write the current line to the output file.

This approach works well for large files because we iterate over the lines of each file instead of reading the entire file in memory.

# Merging smaller text files in Python

If the files you are merging are small in size, you don't have to iterate over the lines of each file.

You can directly read the contents of each file in memory.

Copied!
file_paths = ['file-1.txt', 'file-2.txt']

with open('output-file.txt', 'w', encoding='utf-8') as output_file:
    for file_path in file_paths:
        with open(file_path, 'r', encoding='utf-8') as input_file:
            output_file.write(input_file.read())

The code sample is very similar to the one from the previous subheading.

However, instead of using a nested for loop to iterate over the lines of each input file, we use the file.read() method to read the entire file in memory.

Once we've read the file, we write its contents to the output file and repeat the process for each of the file paths.

# Merging text files in Python using shutil.copyfileobj()

You can also use the shutil.copyfileobj() method to merge text files in Python.

The example project has the following folder structure.

Copied!
my-project/
  └── main.py
  └── file-1.txt
  └── file-2.txt

Here are the contents of file-1.txt.

Copied!
bobby
hadz
.
com

And here are the contents of file-2.txt.

Copied!
one
two
three
four

This is the code for the main.py file.

Copied!
import shutil

file_paths = ['file-1.txt', 'file-2.txt']

with open('output-file.txt', 'wb') as output_file:
    for file_path in file_paths:
        with open(file_path, 'rb') as input_file:
            shutil.copyfileobj(input_file, output_file)

If there is no end-of-line character after the last line of each file, you will have to manually add one.

Copied!
import shutil

file_paths = ['file-1.txt', 'file-2.txt']

with open('output-file.txt', 'wb') as output_file:
    for file_path in file_paths:
        with open(file_path, 'rb') as input_file:
            shutil.copyfileobj(input_file, output_file)
            # ✅ add end-of-line character manually
            output_file.write(b'\n')

The last line adds a newline character after copying the contents of each file.

We opened the output-file in wb (write binary) mode and used a for loop to iterate over the file paths.

On each iteration, we open the current file path in rb (read binary) mode and copy its contents into the output file.

The shutil.copyfileobj() method copies the contents of the source file-like object to the output file-like object.

The method optionally takes a third argument - an integer length to specify the buffer size.

By default, the data is read in chunks to avoid uncontrolled memory consumption.

# Merging text files in Python by using fileinput

You can also use the fileinput.input() method to merge text files in Python.

Copied!
import fileinput

file_paths = ['file-1.txt', 'file-2.txt']

with open('output-file.txt', 'w', encoding='utf-8') as output_file, fileinput.input(file_paths) as input_file:
    for line in input_file:
        output_file.write(line)

The code sample uses the with open() statement to open the output file for writing and the fileinput.input() method to create an instance of the FileInput class.

The instance can be used as a context manager in the with statement.

The FileInput object also automatically closes the input files even if an exception occurs.

Running the python main.py command produces a file called output-file.txt with the following contents.

Copied!
bobby
hadz
.
com
one
two
three
four

# Using the glob() method to merge text files in Python

If you need to merge all .txt files that match a given pattern, use the glob() method.

The example project has the following folder structure.

Copied!
my-project/
  └── main.py
  └── text-files/
    └── file-1.txt
    └── file-2.txt

Here are the contents of text-files/file-1.txt.

Copied!
bobby
hadz
.
com

And here are the contents of text-files/file-2.txt.

Copied!
one
two
three
four

This is the code for the main.py file.

Copied!
import glob

file_paths = glob.glob('text-files/*.txt')

# 👇️ ['text-files/file-1.txt', 'text-files/file-2.txt']
print(file_paths)

with open('output-file.txt', 'w', encoding='utf-8') as output_file:
    for file_path in file_paths:
        with open(file_path, 'r', encoding='utf-8') as input_file:
            output_file.write(input_file.read() + '\n')

The glob.glob() method returns a list of path names that match the given pattern.

Once we have a list of the file paths, we use the with open() statement to open the output file.

The next steps are to:

1. Iterate over the file paths.
2. Open each input file for reading.
3. Write its contents to the output file.

Notice that we also append a newline (\n) character at the end of each input file.

This is not necessary if your input fields end with a newline character.

# Additional Resources

You can learn more about the related topics by checking out the following tutorials:

• Count number of unique Words in a String or File in Python
• Create a file name using Variables in Python
• How to get the File path of a Class in Python
• Taking a file path from user input in Python
• How to check if a File is Empty in Python
• NameError: name '__file__ is not defined in Python [Fixed]
• Write a String to a File on a New Line every time in Python
• How to unzip a .gz file using Python [5 simple Ways]
• How to merge multiple JSON files in Python [3 Ways]
• OSError [Errno 22] invalid argument in Python [Solved]