bashdebianpv

Bash/pv Outputting "^M" at the end of each line

Super simple, but I have NO IDEA how to fix this. I have a super simple pv line (or so I thought) that writes to a file, but the output appends "^M" to the end of each line which the tail command doesn't interpret properly.

To explain, I'm writing an image to a compute module (CM3 or CM4) with pv and outputting the stderr to a temp file I can read from a dialog box lower in the code.

For some reason, the tailboxbg I'm using doesn't realize it's changing and just keeps the first line posted (since it's appended with "^M" which is not a newline) but I never specify anywhere to add that to the end of the line.

The code:

pv -teabf -s "$copySize" "$copyImage" > "devCompMod" 2> _temp &

The _temp file:

 429MiB 0:00:01 [ 429MiB/s] ETA 0:00:33^M 819MiB 0:00:02 [ 408MiB/s] ETA 0:00:34^M 880MiB 0:00:03 [ 292MiB/s] ETA 0:00:47^M 939MiB 0:00:04 [ 234MiB/s] ETA 0:00:59^M 999MiB 0:00:05 [ 199MiB/s] ETA 0:01:09^M 1.34GiB 0:00:06 [ 229MiB/s] ETA 0:00:59^M
pv: write failed: No space left on device

Any help would be greatly appreciated, I'm losing my mind here.

I tried different redirections, but I have no idea how to pipe stderr through something else without completely screwing up my fileread, since it's running in a subshell ("&").

EDIT: A combination of two suggestions on the accepted answer ended up working for me. It's my fault, I didn't add enough of the surrounding code, it was way too precise.

Anyways for anyone who would need this in the future, I ended up using:

pv -F $'%b %t %a %e\n' -f -s "$cpSize" "$cpImage" > "$devTofu" 2> _temp &

And reading the lines directly from _temp (instead of using _pvoutput) by using:

lastLines="$(tail -n 2 _temp | tr -d $'\r\n')"

I ended up using -n 2 because tail had a hard time getting the last line since it always began with an ^M and ended with a \n.

Thank you all for your help!

Solution

  • This has nothing to do with dos or mac. It is just that it really intends to add a \r to have a status line updating, instead of repeating them many times. I do that often in my own code.

    One solution could be to add another command to replace \r by \n.

    pv -teabf -s "$copySize" "$copyImage" 2>&1 > "devCompMod" | tr $'\r' $'\n'

    Btw, that's how you pipe stderr. Just redirect it to stdout. But be sure, of course to redirect stdout on something else. Which you already did.

    The thing that appears unnatural to most people with this is the order. Instinct could say that I should cmd > stdout.txt 2>&1 | stderrfilter. That is, first get take the original content from stdout to a file, and now that this content has be "displaced", use the now empty stdout to hold stderr with 2>&1, and then pipe it.

    But that is not what redirections mean. It is not content that you displace. It is filedescriptors. a>&b, is the bash version of dup2(b,a)

    So see it more some sort of variables holding the destination of outputs. Whose initial values are dest1 = STDOUT, dest2 = STDERR. Everything that is printed to stdout goes to dest1. Everything that is printed to stderr goes to dest2. Unless your redirect something. > outfile.txt, changes dest1 = outfile.txt. 2>&1 changes dest2 to whatever is in dest1, like dest2 = dest1.

    So, doing cmd > stdout.txt 2>&1 | filter, is like dest1=stdout.txt ; dest2=dest1. So that order doesn't work. It means that every outputs goes (mixed) to stdout.txt.

    Instead cmd 2>&1 > stdout.txt | filter, is like dest2=dest1; dest1=stdout.txt so first changes dest2 to initial value of dest1, that is STDOUT then dest1 to stdout.txt. So std output of cmd goest to stdout.txt, and its err output goes to STDOUT, that is, to the pipe.

    Of course, that is not really how it works. But that is schematically how to understand in which order to use the pipes and redirections. Again, think of them as variables, holding destinations. That you have to assign in a logical order. And not as content, that you have to displace.

    And anyway, the correct order to redirect standard output to a file and standard error to a pipe is pv ... 2>&1 > outfile | filter

    Another way

    If you don't want to struggle with redirection, another way could be to tell pv to add a \n itself, by creating the output string with format option.

    pv -F $'%t %e %a %b\n' -f -s "$copySize" "$copyImage" > "devCompMod"

    But it is not ideal. First of all, it doesn't remove the \r (on a terminal, the only consequence is that the cursor is at the beginning of the line, not the end. I suppose most tool parsing lines wouldn't care neither. But you never know). It just add a \n.

    And secondly, I suppose that in reality you would like to redirect the result anyway, since I take your objective is to feed another tool with the progress indication, that is stderr of pv. Your _temp was probably a fix because you couldn't redirect stderr in a pipe... but now that you know how to, I surmise you wouldn't use that _temp anymore.

    But, well, if it works for you, then it is probably preferable over adding another command in your pipe chain (besides, I don't know if tr doesn't add a latency)

    So, tl;dr

    pv -F $'%t %e %a %b\n' -f -s "$copySize" "$copyImage" > "devCompMod" 2> _temp

    or

    pv -teabf -s "$copySize" "$copyImage" 2>&1 > "devCompMod" | tr $'\r' $'\n' > _temp

    if you want the progress into _temp. Preferably the first; the 2nd if the \r that are still (in addition to added \n) in the 1st output are a problem.

    And if you want to redirect that output to a dialog command, instead of using a temp file.

    pv -F $'%t %e %a %b\n' -f -s "$copySize" "$copyImage" 2>&1 > "devCompMod" | dialog

    or

    pv -teabf -s "$copySize" "$copyImage" 2>&1 > "devCompMod" | tr $'\r' $'\n' | dialog