I seem unable to fix my space in filenames issue using switches like -print0 for gnu-find and -0 for gnu-parallel, gnu-xargs in this scenario as is usually recommended.
I succeeded in combining find, parallel in pipe mode and xargs to run commands in parallel in "blocks" for 100k+ files. I use echo and ls in the examples below but I plan to use my own python command. Note that I want to run each command instance on more than one file due to overhead in starting my program hence the use of parallel in --pipe mode and --block etc. The command
find ./dirNames/ -type f | parallel --pipe --block 100 -j4 --round-robin "echo \"Start *****\"; cat ; echo \"Done *****\""
results in
Start *****
./dirNames/bbbbbbbbbbbbbbbb
./dirNames/dddddddddddddddddddd
./dirNames/aaaaaaaaaaaaaaaa
Done *****
Start *****
./dirNames/cccccccc cccccccc
./dirNames/eeeeeeeeeeeeeeeeeeee
Done *****
as desired. gnu-echo is run twice, in one instance it is run with 3 files and in the other instance with 2 files. If I try this with xargs and ls I run into the classic space in filename problem ...
find dirNames/ -type f | parallel --pipe --block 40 -j4 --round-robin "echo \"Start *****\"; xargs ls -l ; echo \"Done *****\""
Resulting in this
Start *****
-rw-rw-r-- 1 robert robert 0 Jun 24 10:10 dirNames/bbbbbbbbbbbbbbbb
-rw-rw-r-- 1 robert robert 0 Jun 25 16:11 dirNames/eeeeeeeeeeeeeeeeeeee
Done *****
Start *****
-rw-rw-r-- 1 robert robert 0 Jun 24 10:10 dirNames/aaaaaaaaaaaaaaaa
Done *****
Start *****
-rw-rw-r-- 1 robert robert 0 Jun 25 16:11 dirNames/dddddddddddddddddddd
Done *****
Start *****
Done *****
ls: cannot access 'dirNames/cccccccc': No such file or directory
ls: cannot access 'cccccccc': No such file or directory
which in this scenario I seem unable to fix using switches like -print0 for find and -0 for parallel and xargs as is usually recommended for this problem. parallel seems confused by the output of find with -print0. Please advise as I have truly run out of ideas :(
This is the answer I posted to the GNU Parallel mailing list.
I think you need to use --recstart '\0' instead of --null on parallel. And I think you'll run into problems when you run your python script with the file names on the command line - I've used ls here to demonstrate a possible solution to that also.
$ find -type f -print0 | parallel --keep-order --no-run-if-empty --pipe --blocksize 15 --recstart '\0' --roundrobin \
"echo start {#}; xargs -0r ls -Q 2>&- | xargs -rt ls --fu; echo end {#}"
start 1
-rw-r--r-- 1 larry wheel 0 2021-06-27 12:39:02.916427000 -0700 ./a
-rw-r--r-- 1 larry wheel 0 2021-06-27 12:40:33.076957000 -0700 ./g
-rw-r--r-- 1 larry wheel 0 2021-06-27 12:40:33.096995000 -0700 ./i
end 1
ls --fu ./a ./g ./i
start 2
-rw-r--r-- 1 larry wheel 0 2021-06-27 12:39:02.916552000 -0700 ./b c
-rw-r--r-- 1 larry wheel 0 2021-06-27 12:40:33.076553000 -0700 ./f
-rw-r--r-- 1 larry wheel 0 2021-06-27 12:40:33.077123000 -0700 ./h
end 2
ls --fu './b c' ./f ./h
start 3
-rw-r--r-- 1 larry wheel 0 2021-06-27 12:39:02.916633000 -0700 ./d
-rw-r--r-- 1 larry wheel 0 2021-06-27 12:40:33.076273000 -0700 ./e
end 3
ls --fu ./d ./e
Note the suppression of stderr on the first ls - without it there are error messages from ls about not being able to list file attributes on a null file name.