I want to filter lines of oc rsh du -shc output like this:
oc rsh broker-amq-1-2-fsbnd du -shc / 2>&1 | awk '$2=="total" {print $1}'
But I got no results. For local du -shc /some/dir 2>&1 I get desired output.
# locally
$ du -shc ~ 2>&1
du: cannot read directory '/home/xxxx/.cache/doc': Permission denied
du: cannot read directory '/home/xxxx/.cache/dconf': Permission denied
du: cannot read directory '/home/xxxx/.gvfs': Permission denied
52G /home/xxxx/
52G total
And filtering:
# filtering works; search the 2nd arg "total" and print arg 1
$ du -shc ~ 2>&1 | awk '$2=="total" {print $1}'
52G
And printing:
$ du -shc ~ 2>&1 | awk '{print $1}'
du:
du:
du:
52G
52G
And $2:
$ du -shc ~ 2>&1 | awk '{print $2}'
cannot
cannot
cannot
/home/xxx
total
But remotely:
oc rsh broker-amq-1-2-fsbnd du -shc / 2>&1 | awk '$2=="total" {print $1}'
# no output
And if I don't use awk:
$ oc rsh broker-amq-1-2-fsbnd du -shc / 2>&1
du: cannot read directory '/proc/tty/driver': Permission denied
du: cannot read directory '/proc/acpi': Permission denied
du: cannot read directory '/proc/scsi': Permission denied
du: cannot access '/proc/130224/task/130224/fd/3': No such file or directory
du: cannot access '/proc/130224/task/130224/fdinfo/3': No such file or directory
du: cannot access '/proc/130224/fd/4': No such file or directory
du: cannot access '/proc/130224/fdinfo/4': No such file or directory
du: cannot read directory '/run/cryptsetup': Permission denied
du: cannot read directory '/run/secrets/rhsm': Permission denied
du: cannot read directory '/sys/firmware': Permission denied
du: cannot read directory '/var/lib/yum/history/2021-12-02/1': Permission denied
du: cannot read directory '/var/lib/yum/history/2021-12-02/2': Permission denied
du: cannot read directory '/var/lib/yum/history/2021-12-02/4': Permission denied
du: cannot read directory '/var/lib/yum/history/2021-12-02/3': Permission denied
du: cannot read directory '/var/lib/machines': Permission denied
du: cannot read directory '/var/cache/ldconfig': Permission denied
1.8G /
1.8G total
command terminated with exit code 1
And, if I only print $1:
$ oc rsh broker-amq-1-2-fsbnd du -shc / 2>&1 | awk '{print $1}'
du:
du:
du:
du:
du:
du:
du:
du:
du:
du:
du:
du:
du:
du:
du:
du:
1.8G
1.8G
command
Why there are extra indentations? Seems only line break is done, but no return carriage to the beginning of the line???
If I print $2, we can see the 2 lines at the end are aligned; what is wrong here?
$ oc rsh broker-amq-1-2-fsbnd du -shc / 2>&1 | awk '{print $2}'
cannot
cannot
cannot
cannot
cannot
cannot
cannot
cannot
cannot
cannot
cannot
cannot
cannot
cannot
cannot
cannot
/
total
terminated
Local awk version is:
GNU Awk 4.1.4, API: 1.1 (GNU MPFR 4.0.1, GNU MP 6.1.2)
Copyright (C) 1989, 1991-2016 Free Software Foundation.
, and remote, openshift awk version is
GNU Awk 4.0.2
Copyright (C) 1989, 1991-2012 Free Software Foundation.
and local du version is
du (GNU coreutils) 8.28
Copyright (C) 2017 Free Software Foundation, Inc.
, while remote, openshift pod du version is
du (GNU coreutils) 8.22
Copyright (C) 2013 Free Software Foundation, Inc.
Seems remote versions are behind local versions a lot, notice the years of copyright.
As per request of @Ed Morton:
$ oc rsh broker-amq-1-15-snd64 du -shc / 2>/dev/null | od -c
0000000 d u : c a n n o t r e a d
0000020 d i r e c t o r y ' / p r o c
0000040 / t t y / d r i v e r ' : P e
0000060 r m i s s i o n d e n i e d \r
0000100 \n d u : c a n n o t r e a d
0000120 d i r e c t o r y ' / p r o
0000140 c / a c p i ' : P e r m i s s
0000160 i o n d e n i e d \r \n d u :
0000200 c a n n o t r e a d d i r e
0000220 c t o r y ' / p r o c / s c s
0000240 i ' : P e r m i s s i o n d
0000260 e n i e d \r \n d u : c a n n o
0000300 t a c c e s s ' / p r o c /
0000320 2 8 7 2 7 / t a s k / 2 8 7 2 7
0000340 / f d / 3 ' : N o s u c h
0000360 f i l e o r d i r e c t o r
0000400 y \r \n d u : c a n n o t a c
0000420 c e s s ' / p r o c / 2 8 7 2
0000440 7 / t a s k / 2 8 7 2 7 / f d i
0000460 n f o / 3 ' : N o s u c h
0000500 f i l e o r d i r e c t o r
0000520 y \r \n d u : c a n n o t a c
0000540 c e s s ' / p r o c / 2 8 7 2
0000560 7 / f d / 4 ' : N o s u c h
0000600 f i l e o r d i r e c t o
0000620 r y \r \n d u : c a n n o t a
0000640 c c e s s ' / p r o c / 2 8 7
0000660 2 7 / f d i n f o / 4 ' : N o
0000700 s u c h f i l e o r d i
0000720 r e c t o r y \r \n d u : c a n
0000740 n o t r e a d d i r e c t o
0000760 r y ' / r u n / c r y p t s e
0001000 t u p ' : P e r m i s s i o n
0001020 d e n i e d \r \n d u : c a n
0001040 n o t r e a d d i r e c t o
0001060 r y ' / r u n / s e c r e t s
0001100 / r h s m ' : P e r m i s s i
0001120 o n d e n i e d \r \n d u : c
0001140 a n n o t r e a d d i r e c
0001160 t o r y ' / s y s / f i r m w
0001200 a r e ' : P e r m i s s i o n
0001220 d e n i e d \r \n d u : c a n
0001240 n o t r e a d d i r e c t o
0001260 r y ' / v a r / l i b / y u m
0001300 / h i s t o r y / 2 0 2 1 - 0 1
0001320 - 2 6 / 1 ' : P e r m i s s i
0001340 o n d e n i e d \r \n d u : c
0001360 a n n o t r e a d d i r e c
0001400 t o r y ' / v a r / l i b / y
0001420 u m / h i s t o r y / 2 0 2 1 -
0001440 0 1 - 2 6 / 2 ' : P e r m i s
0001460 s i o n d e n i e d \r \n d u :
0001500 c a n n o t r e a d d i r
0001520 e c t o r y ' / v a r / l i b
0001540 / y u m / h i s t o r y / 2 0 2
0001560 1 - 0 1 - 2 6 / 4 ' : P e r m
0001600 i s s i o n d e n i e d \r \n d
0001620 u : c a n n o t r e a d d
0001640 i r e c t o r y ' / v a r / l
0001660 i b / y u m / h i s t o r y / 2
0001700 0 2 1 - 0 1 - 2 6 / 3 ' : P e
0001720 r m i s s i o n d e n i e d \r
0001740 \n d u : c a n n o t r e a d
0001760 d i r e c t o r y ' / v a r
0002000 / l i b / m a c h i n e s ' :
0002020 P e r m i s s i o n d e n i e
0002040 d \r \n d u : c a n n o t r e
0002060 a d d i r e c t o r y ' / v
0002100 a r / c a c h e / l d c o n f i
0002120 g ' : P e r m i s s i o n d
0002140 e n i e d \r \n 3 . 7 G \t / \r \n 3
0002160 . 7 G \t t o t a l \r \n
0002173
xxxxxxx@elxag5zs8d3:~
and:
$ oc rsh broker-amq-1-15-snd64 du -shc / 2>/dev/null | awk '{print $1}' | od -c
0000000 d u : \n d u : \n d u : \n d u : \n
*
0000100 3 . 7 G \n 3 . 7 G \n
0000112
If I change RS, I got stranger results.
$ oc rsh broker-amq-1-15-snd64 du -shc / 2>/dev/null | awk 'BEGIN {RS="\r\n";} {print $1}'
du:
du:
du:
du:
du:
du:
du:
du:
du:
du:
du:
du:
du:
du:
du:
du:
3.7G
3.7G
xxxxxx@elxag5zs8d3:~
It's very odd that your oc rsh broker-amq-1-15-snd64 du -shc / 2>/dev/null | od -c output shows no blanks or tabs, e.g. between cannot and read in:
0000000 d u : c a n n o t r e a d
while without od -c, i.e. when you run oc rsh broker-amq-1-2-fsbnd du -shc / 2>&1, there clearly are blanks:
du: cannot read directory '/proc/tty/driver': Permission denied
Anyway, as shown in the oc rsh broker-amq-1-15-snd64 du -shc / 2>/dev/null | od -c output your output lines end in \r\n, e.g. (emphasis mine):
0000000 d u : c a n n o t r e a d
0000020 d i r e c t o r y ' / p r o c
0000040 / t t y / d r i v e r ' : P e
0000060 r m i s s i o n d e n i e d **\r
0000100 \n** d u : c a n n o t r e a d
0000120 d i r e c t o r y ' / p r o
0000140 c / a c p i ' : P e r m i s s
0000160 i o n d e n i e d **\r \n**
You mentioned that locally du outputs \n-terminated lines so that just means it's probably rsh or oc that's changing that to \r\n - try it with some other command like oc rsh broker-amq-1-15-snd64 date to verify.
In any case, to handle that simply and portably, and assuming you don't actually want \r\n line endings in your final output, change your awk script from this:
awk '$2=="total" {print $1}'
to this which removes the \r at the end of each line before doing anything else with the input:
awk '{sub(/\r$/,"")} $2=="total" {print $1}'
The reason you weren't getting output with $2=="total" is that given input like:
foo total\r\n
$2 isn't "total", it's "total\r", and so the comparison fails.
You mentioned setting ORS="\r\n" to get the desired output - no, that just propagates the problem to the next command.
You could set RS="\r\n" but that would only work in an awk that accepts multi-char RS, e.g. GNU awk, in other awks that'd be treated the same as RS="\r" per the POSIX standard that says RS is a single char.
There are some platforms out there where when \r\n input is detected either:
\r automatically gets stripped by the underlying C primitives that awk calls to read input, or\r\n.so just be aware of that and check what input awk is actually getting and what RS and ORS are set to if the sub() above doesn't do what you want (awk 'NR==1{ print "$0=<"$0">" ORS "RS=<"RS">" ORS "ORS=<"ORS">"; exit }' | od -c should tell you all you need to know).
For more information see Why does my tool output overwrite itself and how do I fix it?.