Python: run functions in parallel with a multiprocessing wrapper function

One common way to run functions in parallel with Python is to use the multiprocessing module which is powerful, it has many options to configure and a lot of things to tweak.

But what about if we want just a very simple functionality like running a number of functions in parallel and nothing else?

We can use the following wrapper function for this:

def run_parallel(*functions):
    '''
    Run functions in parallel
    '''
    from multiprocessing import Process
    processes = []
    for function in functions:
        proc = Process(target=function)
        proc.start()
        processes.append(proc)
    for proc in processes:
        proc.join()

This function allows us to input as parameter any number of functions with or without their parameters and executed in parallel.

Example

#!/usr/bin/env python3def task_a():
    print('this is task a')def task_b():
    print('this is task b')def hello(msg):
    print('Hello: %s'%(msg))def run_parallel(*functions):
    '''
    Run functions in parallel
    '''
    from multiprocessing import Process
    processes = []
    for function in functions:
        proc = Process(target=function)
        proc.start()
        processes.append(proc)
    for proc in processes:
        proc.join()if __name__ == '__main__':
    run_parallel(task_a(),task_b(),hello(msg="konstantinos"))

Executing this will bring the following results:

this is task a
this is task b
Hello: konstantinos

This is nice and simple, but how we will get results back from the functions?

To do this we need to use a shared variable like a list and append the results of the function in the list:

#!/usr/bin/env python3
results = []
def task_a():
    res = {}
    res['task_a'] = 1 + 1
    results.append(res)def task_b():
    res = {}
    res['task_b'] = 2 + 2
    results.append(res)def task_c(name):
    res = {}
    res['task_c'] = "Name: %s"%(name)
    results.append(res)def run_parallel(*functions):
    '''
    Run functions in parallel
    '''
    from multiprocessing import Process
    processes = []
    for function in functions:
        proc = Process(target=function)
        proc.start()
        processes.append(proc)
    for proc in processes:
        proc.join()if __name__ == '__main__': run_parallel(task_a(),task_b(),task_c(name="konstantinos"))
    print(results)

Results:

[{'task_a': 2}, {'task_b': 4}, {'task_c': 'Name: konstantinos'}]