make
have -j
flag, which makes make
process more faster.
This flag tells make
information of allowed to spawn the provided amount of 'threads'.
As the same manner, is there any simple way to apply -j
-like option on normal execution?
For example, Assuming that I want execute my python script more faster.
$ python myprogram.py -j4 // <--?
Is there any useful utils in linux to control the amount of thread like -j
do?
Parallelizing a program has to be done by the programmer, not the user.
make
computes a dependency tree for the target. Most targets will depend on more than one input, like an executable file that's built from several parts, like .c files compiled into .o files. The developers of make
understood this, and using the dependency tree, they wrote make
so it can figure out which parts can be prepared independent of each other, and -j4
tells it to prepare 4 in parallel, for instance starting 4 compiler processes (not threads!) in parallel.
To accelerate your Python program, you yourself need to identify portions that can be executed independent of each other, which will totally depend on the specifics of the problem your Python program solves; there is no general solution, and many problems are very hard to parallelize.
Parallelization comes in two forms: processes and threads. Threads share their memory (except for the stack), and in Python are limited by the Global Interpreter Lock (GIL), so getting more computing power using threads is often not possible in Python. (The situation is different in C, C++ and Java, for instance, where threads can get you speedup.) Processes (as utilized by make
) on the other hand have a much harder time talking to each other (using shared memory, semaphores, sockets etc), because they are truly independent of each other.
In Python, the modules multiprocessing
and threading
provide functionality for working with multiple processes and threads respectively.
Be advised that under Unix/Linux/POSIX, creating new processes from a program that has already created threads might easily give you deadlocks unless you are very careful.