Multiprocessing

Last updated on 2026-06-23 | Edit this page

Estimated time: 12 minutes

Overview

Questions

  • How can I speed up my code by running multiple processes at the same time?
  • What is multiprocessing and when should I use it?

Objectives

  • Explain the concept of multiprocessing and how it can improve performance in certain situations.
  • Demonstrate how to use the multiprocessing module in Python to run multiple processes concurrently.
  • Show how to use Queue for communication between processes and Pool for managing a group of worker processes.

multiprocessing

Normally we run our programs one after another in Python.

PYTHON 

program1()
program2()

So program2 waits for program1 to finish in order to execute. But in some cases, it might take very long for program1 to run:

PYTHON 

pull_data()
calculate_some_operations()

In this example, we could choose to use multiprocessing instead of waiting for the pull_data function to finish executing.

When exactly to use multiprocessing?

Multiprocessing is useful for CPU-bound programs.

Callout

CPU-bound programs are those whose performance is limited by the CPU’s speed.

Which processes are CPU-bound?

When we want to process large amounts of data or images, the program’s performance will depend largely on the CPU. It is also useful for mathematical/scientific calculations, as these can take some time to execute as well.

Caution

Multiprocessing is not suitable for every job and creating a new process is costly!

The multiprocessing module in Python includes several modules. One of these is the Queue module, which enables communication between processes. Normally, the memory spaces of two processes are separate; therefore, they cannot access each other’s variables, etc.

PYTHON 

def process1():
    x = 1

def process2():
    y = 2

process1 cannot see y, and vice versa.

So we use the multiprocessing.Queue module to communicate safely. process1() can pass its data using multiprocessing.Queue():

A process can put data into a queue using put(), and another process can get it using get().

PYTHON 

def process1(queue):
    x = 1
    queue.put(x)

def process2(queue):
    value_process2 = queue.get()
    print("Received: ", value_process2)
    y = 2

Let’s say we have a restaurant and we take orders using this take_order function.

PYTHON 

import time
from multiprocessing import Queue

def take_orders(queue, start_time):
    # waiter puts orders into the queue one by one
    orders = ["Pizza", "Burger", "Pasta", "Sushi", "Salad"]

    for order in orders:
        print(f"{time.time() - start_time:.2f}s - Order received: {order}")

        queue.put(order)
        time.sleep(1) # assume the order is taken in 1 second

    queue.put(None)  # signal: no more orders coming

We put a time check to see when the order is received.

And we also need to prepare the orders that we took from our guests.

PYTHON 

def prepare_orders(queue, start_time):
    while True:
        order = queue.get()
        if order is None:
            break

        print(f"{time.time() - start_time:.2f}s - Preparing: {order}")
        time.sleep(2) # we assume it's being prepared in 2 seconds

In these two functions we used Queue because we needed to exchange the orders safely between the function that takes them and prepares them.

Let’s call these two functions in main() to see when they run.

PYTHON 

if __name__ == "__main__":
    queue = Queue()

    start_time = time.time()

    take_orders(queue, start_time)
    prepare_orders(queue, start_time)

Output:

    0.00s - Order received: Pizza
    1.00s - Order received: Burger
    2.00s - Order received: Pasta
    3.00s - Preparing: Pizza
    5.00s - Preparing: Burger
    7.00s - Preparing: Pasta

The timestamps show that all orders are received first. The preparation doesn’t start until after that.

Now we can try to call the functions with the help of the Process module.

With the help of Process, we can start our program as a separate process instead of waiting for the main program to finish.

PYTHON 

if __name__ == "__main__":
    queue = Queue()
    start_time = time.time()

    waiter = Process(target=take_orders, args=(queue, start_time))
    kitchen = Process(target=prepare_orders, args=(queue, start_time))

    waiter.start()
    kitchen.start()

    # we make the main program wait for these processes to finish
    waiter.join()
    kitchen.join()

Output:

0.01s - Order received: Pizza
0.02s - Preparing: Pizza
1.01s - Order received: Burger
2.01s - Order received: Pasta
2.02s - Preparing: Burger
4.03s - Preparing: Pasta

Here, take_orders and prepare_orders run in separate processes.

We can see from the timestamps that the kitchen does not wait until all orders are received. While the waiter continues taking orders, the kitchen already starts preparing them.

This shows the main idea of multiprocessing: independent tasks can make progress during the same time period.

As you can see, we created the processes manually here, using Process. But that’s not always necessary. Instead, we can create a group of worker processes that automatically share the work, like hiring several cooks to prepare multiple orders instead of assigning each order manually.

To do that, we could use Pool from multiprocessing. We also need to make a small change to our prepare_orders function.

PYTHON 

from multiprocessing import Pool

def prepare_orders(args):
    order, start_time = args  # we unpack tuple, since pool.map sends single argument

    print(f"{time.time() - start_time:.2f}s - Preparing: {order}")
    time.sleep(2)
    return f"{order} ready"  # pool.map collects return values

if __name__ == "__main__":
    start_time = time.time()

    orders = ["Pizza", "Burger", "Pasta", "Sushi", "Salad"]
    with Pool(processes=3) as pool:
        order_status = pool.map(
            prepare_orders, [(order, start_time)
            for order in orders])

        print(order_status)

Output:

0.05s - Preparing: Pizza
0.05s - Preparing: Burger
0.05s - Preparing: Pasta
2.06s - Preparing: Sushi
2.06s - Preparing: Salad

You can see that the first 3 orders are being prepared at the same time, because Pool(processes=3) creates 3 worker processes. After first 3 orders start, the remaining orders wait in line until one of the workers becomes available. Since it takes 2 seconds to prepare an order, we can see that after 2 seconds the 4th and 5th orders start being prepared.

The important thing here is that we did not create each process manually. After creating the pool, it distributes the jobs between the worker processes automatically.

Challenge

Challenge 1:

Key Points
  • Multiprocessing allows us to run multiple processes concurrently, which can improve performance for CPU-bound tasks.