I am using tensorflow eager to do calculation. The aim is to allocate the work on all gpus. However, I find that this cannot be done using multiprocessing.
Following is the code (it is indeed very short, except for some extra work):
import os,sys
import multiprocessing
import numpy as np
# clear folder
folder = os.getcwd()+'/temp/'
for the_file in os.listdir(folder):
file_path = os.path.join(folder, the_file)
if os.path.isfile(file_path):
os.unlink(file_path)
# process
p={}
n_batches=4
# kernel to be called in each process
# here, the example is just to return i_batch
def kernel(i_batch):
import tensorflow as tf
from tensorflow.python.eager.context import context, EAGER_MODE, GRAPH_MODE
def switch_to(mode):
ctx = context()._eager_context
ctx.mode = mode
ctx.is_eager = mode == EAGER_MODE
switch_to(EAGER_MODE)
assert tf.executing_eagerly()
with tf.device("GPU:"+str(i_batch)):
tf.tile([1],[10])
r=tf.constant(i_batch).numpy()
return r
# multiprocessing loop
for i_batch in range(n_batches):
def multi_processing():
result=kernel(i_batch)
np.save(os.getcwd()+'/temp/result'+str(i_batch), result)
# start multi-processing to allocate
p[i_batch] = multiprocessing.Process(target=multi_processing)
p[i_batch].daemon = True
p[i_batch].start()
# wait
for i_batch in range(n_batches):
p[i_batch].join()
result=0.
for i_batch in range(n_batches):
result+=np.load(os.getcwd()+'/temp/result'+str(i_batch)+'.npy')
result
the function kernel is to be called by the main loop which distributes the work on four gpus. But it yielded the error:CUDA_ERROR_OUT_OF_MEMORY.
This is actually very short, it should not take many resources.
Does anyone know how to fix this?
Since Tensorflow greedily allocates memory, one process might consume all resources. Reference: https://stackoverflow.com/a/34514932/10111931
If you take a look at GPUOptions, apart from setting per_process_gpu_memory_fraction suggested in the answer above, you could take a look at using allow_growth=True to request memory as and when required.
The second thing that you can try is to use the CUDA_VISIBLE_DEVICES option to let each process operate with only a subset of the GPU's. Reference: https://stackoverflow.com/a/37901914/10111931