[SOLVED] Set niceness of each process in a multiprocessing.Pool

Set niceness of each process in a multiprocessing.Pool

How can I set the niceness for each process in a multiprocessing.Pool? I understand that I can increment niceness with os.nice(), but how do call it in the child process after creating the pool? If I call it in the mapped function it will be called every time the function executes, rather than once when the process is forked.

import multiprocessing as mp    

NICENESS = 19
DATA = range(100000)

def foo(bar):
    return bar * 2

pool = mp.Pool(100)
# Somehow set niceness of each process to NICENESS

pool.map(foo, DATA)

Solution

What about using an initializer for that? https://docs.python.org/3.8/library/multiprocessing.html#multiprocessing.pool.Pool The function is called once when the pool is started so the os.nice() call in the initializer sets the niceness for the proces after that.

I've added some additional statements to show that it works in your worker function but the os.nice() calls should obviously be removed since you want a static niceness value.

import multiprocessing as mp
import os

NICENESS = 3
DATA = range(6)


def foo(bar):
    newniceness = os.nice(1) # remove this
    print('Additional niceness:', newniceness) # remove this
    return bar * 2


def set_nicesness(val): # the initializer
    newval = os.nice(val) # starts at 0 and returns newvalue
    print('niceness value:', newval)



pool = mp.Pool(3, initializer=set_nicesness, initargs=(NICENESS,))
# Somehow set niceness of each process to NICENESS
pool.map(foo, DATA)

As you can see from the prints the niceness now starts at 3 (I've set this for NICENESS) and starts incrementing from there.

Or as a useable snippet

import multiprocessing as mp
import os

NICENESS = 3


def mp_function(bar: int) -> int:
    return bar * 2


if __name__ == '__main__':
    pool = mp.Pool(3, initializer=os.nice, initargs=(NICENESS,))
    data = range(6)
    pool.map(mp_function, data)