Run multiple Python files concurrently / one after the other

# Table of Contents

# Run multiple Python files concurrently

You can use the subprocess.Popen() class to run multiple files concurrently in Python.

The class takes a sequence of program arguments and runs a child program in a new process.

Suppose, we have the following a.py, b.py and c.py files.

This is a.py.

a.py

Copied!
print('A file contents')

This is b.py.

b.py

Copied!
print('B file contents')

And this is c.py.

c.py

Copied!
print('C file contents')

Here is the main.py file that runs the a.py, b.py and c.py files concurrently.

main.py

Copied!
import subprocess

# Create and start the processes
proc1 = subprocess.Popen(['python', 'a.py'])
proc2 = subprocess.Popen(['python', 'b.py'])
proc3 = subprocess.Popen(['python', 'c.py'])

# Wait for the processes to finish
proc1.wait()
proc2.wait()
proc3.wait()

run multiple files concurrently in python

The code for this article is available on GitHub

Notice that the files are run concurrently, therefore the print() statements run in a different order.

If you use python3 or py instead of python, you might have to change the first argument in the args list that is passed to subprocess.Popen().

main.py

Copied!
# python3 (macOS and Linux)
proc1 = subprocess.Popen(['python3', 'a.py'])

# or py alias (Windows)
proc1 = subprocess.Popen(['py', 'a.py'])

The list we passed to the subprocess.Popen() class is a sequence of program arguments.

By default, the program to execute is the first item in the args list.

The Popen.wait() method waits for the child process to terminate.

main.py

Copied!
# Wait for the processes to finish
proc1.wait()
proc2.wait()
proc3.wait()

You can optionally pass a timeout argument to the method.

main.py

Copied!
proc1.wait(timeout=10)
proc2.wait(timeout=10)
proc3.wait(timeout=10)

If the process doesn't terminate after timeout seconds, a TimeoutExpired exception is raised.

# Run multiple Python files concurrently via the command line

If you want to run multiple files concurrently via the command line, issue the following command.

shell

Copied!
python a.py & python b.py & python c.py

Make sure to replace a.py, b.py and c.py with the names of the files you want to run concurrently.

run multiple python files concurrently via command line

The ampersand syntax is valid in bash and zsh.

You can also add an ampersand & after the last filename to have the files running in the background.

shell

Copied!
python a.py & python b.py & python c.py &

# Creating a bash script to run multiple Python files concurrently

You can also create a bash script to run multiple Python files concurrently.

Create a file named my-script.sh with the following contents.

my-script.sh

Copied!
#!/bin/bash

python a.py &
python b.py &
python c.py &

To run your script, open your terminal in the same directory and issue the ./my-script.sh command.

run multiple files concurrently with bash script

# Creating a bash script to run multiple Python files one after the other

If you want to run the files one after the other remove the ampersand & characters.

my-script.s

Copied!
#!/bin/bash

python a.py
python b.py
python c.py

run multiple files one after the other with bash script

# Run multiple Python files one after the other

If you need to run multiple Python files one after the other, use the subprocess.run() method.

main.py

Copied!
import subprocess

subprocess.run('python a.py', shell=True, check=False)
subprocess.run('python b.py', shell=True, check=False)
subprocess.run('python c.py', shell=True, check=False)

run multiple python files one after the other with subprocess run

The code for this article is available on GitHub

The subprocess.run() method runs the supplied command, waits for the command to complete and returns a CompletedPorcess instance.

Depending on your Python installation, you might also have to use the python3 or py commands.

main.py

Copied!
# python3 (macOS and Linux)
subprocess.run('python3 a.py', shell=True, check=False)

# py alias (Windows)
subprocess.run('py a.py', shell=True, check=False)

If the shell argument is True, the specified command is executed through the shell.

If the check argument is True and the process exits with a non-zero exit code, a CalledProcessError exception is raised.

The attributes of the exception hold the arguments, the exit code and stdout and stderr.

# Running multiple Python files concurrently with `subprocess.run()`

You can also use the subprocess.run() method to run multiple Python files concurrently.

main.py

Copied!
import subprocess

subprocess.run(
    'python a.py & python b.py & python c.py',
    shell=True,
    check=False
)

run multiple python files concurrently with subprocess run

The code for this article is available on GitHub

Notice that the filenames are separated by an ampersand &.

# Run multiple Python files one after the other using `os.system()`

You can also use the os.system method to run multiple files one after the other in Python.

main.py

Copied!
import os


os.system('python a.py')
os.system('python b.py')
os.system('python c.py')

run multiple files one after the other using os system

The os.system() method executes the supplied command in a subshell.

As with the previous example, depending on your Python installation, you might have to use python3 or py instead of python.

# Wrapping the code in your files in a `main()` function

Alternatively, you can wrap the code in your a.py, b.py and c.py files in main functions, so it doesn't automatically run the module's code when it is imported.

Here is the updated code for a.py.

a.py

Copied!
def main():
    print('A file contents')

Here is the code for b.py.

b.py

Copied!
def main():
    print('B file contents')

And here is the code for c.py.

c.py

Copied!
def main():
    print('C file contents')

Here is the updated main.py module.

main.py

Copied!
import a
import b
import c

a.main()
b.main()
c.main()

wrapping each module in main function

The example runs the files one after the other.

Alternatively, you can use multiple threads.

main.py

Copied!
import a
import b
import c
from threading import Thread

Thread(target=a.main).start()
Thread(target=b.main).start()
Thread(target=c.main).start()

using multiple threads

The code for this article is available on GitHub

# Additional Resources

You can learn more about the related topics by checking out the following tutorials:

I wrote a book in which I share everything I know about how to become a better, more efficient programmer.

You can use the search field on my Home Page to filter through all of my articles.

Run multiple Python files concurrently / one after the other

# Table of Contents