bashparallel-processinggnu-parallel

How to read a 2 column file and pass each column as a paired input for GNU parallel

I have a list of paired files in a directory that I put inside a file (using paste)

cat list.files.r12.txt | head -n3
./ID1_L2_R1_001.fastq.gz    ./ID1_L2_R2_001.fastq.gz
./ID2_L3_R1_001.fastq.gz    ./ID2_L3_R2_001.fastq.gz
./ID3_L2_R1_001.fastq.gz    ./ID3_L2_R2_001.fastq.gz

You can see that the pair is between R1 and R2.

This is just for testing as this is going to be passed to another command. The command below just paste line by line.

parallel -j 1 "echo {}" ::::  <(cat list.files.r12.txt | head -n3)

This one duplicates the files

parallel -j 1 "echo {} {}" ::::  <(cat list.files.r12.txt | head -n3)

This doesn't work:

parallel -j 1 "echo {1} {2}" ::::  <(cat list.files.r12.txt | head -n3)

Basically, what I want is that each line of the list.files.r12.txt file could be executed in parallel. This is the desired output:

program ... ./ID1_L2_R1_001.fastq.gz ./ID1_L2_R2_001.fastq.gz
program ... ./ID2_L2_R1_001.fastq.gz ./ID2_L2_R2_001.fastq.gz
program ... ./ID3_L2_R1_001.fastq.gz ./ID3_L2_R2_001.fastq.gz
Solution 
  • cat list.files.r12.txt | head -n3 |
   parallel --colsep '\t' echo

    cat list.files.r12.txt | head -n3 |
   parallel --colsep '\t' echo one:{1} two:{2}

    When you have paired end reads you can normally do:

    parallel --plus echo read:{} paired:{/_R1_/_R2_} ::: *_R1_*

    This way you avoid the pasting.