I'm streaming data using netcat and piping the output to gawk. Here is an example byte sequence that gawk will receive:
=AAAA;=BBBB;;CCCC==DDDD;
The data includes nearly any arbitrary characters, but never contains NULL characters, where = and ; are reserved to be delimiters. As chunks of arbitrary characters are written, each chunk will always be prefixed by one of the delimiters, and always be suffixed by one of the delimiters, but either delimiter can be used at any time: = is not always the prefix, and ; is not always the suffix. It will never write a chunk without also writing an appropriate prefix and suffix. As the data is parsed, I need to distuingish between which delimiter was used, so that my downstream code can properly interpret that information.
Since this is a network stream, stdin remains open after this sequence is read, as it waits for future data. I'd want gawk to read until either delimiter is encountered, and then execute the body of my gawk script with whatever data was found, while ensuring that it properly handles the continuous stream of stdin. I explain this in more detail below.
Here is what I have attempted thus far (zsh script, using gawk, on macOS). For this post, I simplified the body to just print the data - my full gawk script has a much more complicated body. I also simplified the netcat stream to instead just cat a file (along with cat'ing stdin in order to mimic the stream behavior).
cat example.txt - | gawk '
BEGIN {
RS = "=|;";
}
{
if ($0 != "") {
print $0;
fflush();
}
}
'
example.txt
=AAAA;=BBBB;=CCCC;=DDDD;
My attempt successfully handles most of the data......up until the most-recent record. It hangs waiting for more data from stdin, and fails to execute the body of my script for the most-recent record, despite an appropriate delimiter clearly being available in stdin.
Current output: (fails to process the most-recent record of DDDD)
AAAA
BBBB
CCCC
[hang here, waiting for future data]
Desired output: (successfully processes all records, including the most-recent)
AAAA
BBBB
CCCC
DDDD
[hang here, waiting for future data]
What, exactly, could be the cause of this problem, and how can I potentially address it? I recognize that this seems to be somewhat of an edge-case scenario. Thank you all very much for your help!
Here are some misc observations I found during debugging, both before and after I originally made this post. These edits also clarify some questions that came up in the comments, and consolidate the info scattered across various comments into a single place. Also includes some realizations I made about how gawk works internally, based on the extremely insightful information in the comments. Info in this edit supersedes any potentially conflicting info that may have been discussed in the comments.
I briefly investigated whether this could be a pipe buffering issue imposed by the OS. After messing with the stdbuf tool to disable all pipe buffering, it seems that buffering is not the problem at all, at least not in the traditional sense (see item #3).
I noticed that if stdin is closed and a regex is used for RS, no problems occur. Conversely, if stdin remains open and RS is not a regex (i.e. a plaintext string), no problems occur either. The problem only occurs if both stdin remains open and RS is a regex. Thus, we can reasonably assume that it's something related to how regex handles having a continuous stream of stdin.
I noticed that if my RS with regex (RS = "=|;";) is 3 characters long...and stdin remains open...it stops hanging after exactly 3 additional characters appear in stdin. If I adjust the length of my regex to be 5 chars (RS = "(=|;)"), the amount of additional characters necessary to return from hanging adjusts accordingly. Combined with the extremely insightful discussion with Kaz, this establishes that the hanging is an artifact of the regex engine itself. Like Kaz said, when the regex engine parses RS = "=|;";, it ends up trying to read additional characters from stdin in order to be sure that the regex is a match, despite this additional read not being strictly necessary for the regex in question, which obviously causes a hang waiting on stdin. I also tried adding lazy quantifiers to the regex, which in theory means the regex engine can return immediately, but alas it does not, as this is an implementation detail of the regex engine.
The gawk docs here and here state that when RS is a single character, it is treated as a plaintext string, and causes RS to match without invoking the regex engine. Conversely, if RS has 2 or more characters, it is treated as a regex, and the regex engine will be invoked (subsequently bringing the problem discussed in item #3 into play). However, this seems to be slightly misleading, which is an implementation detail of gawk. I tried RS = "xy"; (and adjusted my data accordingly), and re-tested my experiment from #3. No hanging occurred and the correct output was printed, which must mean that despite RS being 2 characters, it is still being treated as a plaintext string - the regex engine is never invoked, and the hanging problem never occurs. So, there seems to be some further filtering on whether RS is treated as plaintext or as a regex.
So....now that we've figured out the root cause of the problem....what do we do about it? An obvious idea would be to avoid usage of the regex engine....but my script still needs to match either delimiter using some sort of OR clause....so that seems to necessitate writing a custom data parser as a C program or otherwise. Although I could do this, and it would certainly solve the problem, but considering the task at hand, I'd rather prefer not to go down this path of weeds.
This brings us to Ed Morton's workaround, which is most probably the best way to go, or perhaps some slight derivative thereof. Summarizing his approach below:
Basically use other CLI tools, or shell reading loops, or perhaps even multiple invocations of gawk, to do a conversion ahead-of-time, before the data is given to the primary gawk invocation. Like Ed said, replace each delimiter to all be suffixed by the NULL character. Since that is done before gawk sees any data, gawk could be set to use the NULL character as RS, which would be treated as a plaintext string and not a regex, which means the regex hanging problem never comes into play. From there, the real delimiter and data chunk could be decoded and processed in whatever way you want.
gawk itself might even be able to do the ahead-of-time conversions....as long as each delimiter in question can be found using a plaintext RS and not a regex RS. Be careful of delimiters that contain characters that are special to a regex, as this might cause gawk to treat it as a regex when you aren't expecting it to.
Although I have now marked Ed's answer as the solution, I think that my final end-all solution will be a hybrid of Ed's approach, Kaz's insight, and some subsequent realizations I made thanks to them. Wish I could mark two answers as solutions! Thank you, everyone, for your help, especially Ed Morton and Kaz!
A workaround inserting a shell read loop into the pipeline to carve the original awk input (the OPs actual netcat output) up into individual characters and then feed them to awk one at a time:
cat example.txt - |
while IFS= read -r -d '' -N1 char; do printf '%s\0' "$char"; done |
awk -v RS='\0' '
/[;=]/ { if (rec != "") { print rec; fflush() }; rec=""; next }
{ rec=rec $0 }
'
AAAA
AAAA
AAAA
AAAA
That requires GNU awk or some other that can handle a NUL character as the RS as that's non-POSIX behavior. It does assume your input can't contain NUL bytes, i.e. it's a valid POSIX text "file".
Read on for how we got there if interested...
I thought there was at least 1 bug here as I found multiple oddities (see below) so I opened a gawk bug report at https://lists.gnu.org/archive/html/bug-gawk/2024-07/msg00006.html but per the gawk provider, Arnold, the differences in behavior in this case are just implementation details of having to read ahead to ensure the regexp matches the right string.
It seems there are 3 issues at play here, e.g. using GNU awk 5.3.0 on cygwin:
$ printf 'A;B;C;\n' > file
$ cat file - | awk -v RS='(;|=)' '{print NR, $0}'
1 A
$ cat file - | awk -v RS=';|=' '{print NR, $0}'
1 A
2 B
$ cat file - | awk -v RS='[;=]' '{print NR, $0}'
1 A
2 B
3 C
(;|=), ;|= and [;=] should be equivalent but clearly they aren't in this case.
The good news is you can apparently work around that problem using a bracket expression as in the 3rd case above instead of an "or".
;:$ printf 'A;B;C;' > file
$ cat file - | awk -v RS='(;|=)' '{print $0; fflush()}'
$ cat file - | awk -v RS=';|=' '{print $0; fflush()}'
A
$ cat file - | awk -v RS='[;=]' '{print $0; fflush()}'
A
B
The bad news is that that impacts the OPs example:
$ printf ';AAAA;BBBB;CCCC;DDDD;' > file
With a literal character RS:
$ cat file - | awk -v RS=';' '{print $0; fflush()}'
AAAA
BBBB
CCCC
DDDD
With a regexp RS that should also make that char literal:
$ cat file - | awk -v RS='[;]' '{print $0; fflush()}'
AAAA
BBBB
CCCC
$ printf ';AAAA;BBBB;CCCC;DDDD;x' > file
$ cat file - | awk -v RS='[;]' '{print $0; fflush()}'
AAAA
BBBB
CCCC
DDDD
$ printf 'A;B;C;\n' > file
$ cat file - | awk -v RS='[;|=]' '{print $0; fflush()}'
A
$ cat file - | awk -v RS='[;a=]' '{print $0; fflush()}'
A
B
C
FWIW I tried setting a timeout:
$ cat file - | awk -v RS='[;]' 'BEGIN{PROCINFO["-", "READ_TIMEOUT"]=100} {print $0; fflush()}'
A
B
awk: cmd. line:1: (FILENAME=- FNR=3) fatal: error reading input file `-': Connection timed out
$ cat file - | awk -v RS='[;]' -v GAWK_READ_TIMEOUT=1 '{print $0; fflush()}'
A
B
and stdbuf to disable buffering:
$ cat file - | stdbuf -i0 -o0 -e0 awk -v RS='[;]' '{print $0; fflush()}'
A
B
and matching every character (thinking I could then use RT ~ /[=;]/ to find the separator):
$ cat file - | awk -v RS='(.)' '{print RT; fflush()}'
A
;
B
;
C
but none of them would let me read the last record separator so at this point I don't know what the OP could do to successfully read the last record of continuing input using a regexp other than something like this:
$ printf 'A;B;C;' > file
$ cat file - |
while IFS= read -r -d '' -N1 char; do printf '%s\0' "$char"; done |
awk -v RS='\0' '/[;=]/ { print rec; fflush(); rec=""; next } { rec=rec $0 }'
A
B
C
and using the OPs sample input but with different text per record to make the mapping of input to output records clearer:
$ printf '=AAAA=BBBB;CCCC;DDDD=' > example.txt
$ cat example.txt - |
while IFS= read -r -d '' -N1 char; do printf '%s\0' "$char"; done |
awk -v RS='\0' '/[;=]/ { print rec; fflush(); rec=""; next } { rec=rec $0 }'
AAAA
BBBB
CCCC
DDDD
We're using NUL chars as the delimiters and various options above to make the shell read loop robust enough to handle blank lines and other white space in the input, see https://unix.stackexchange.com/a/49585/133219 and https://unix.stackexchange.com/a/169765/133219 for details on those issues. We're additionally using a NUL char for the awk RS so it can distinguish between newlines coming from the original input vs a newline as a terminating character being added by the shell printf, otherwise rec in the awk script could never contain a newline as they'd ALL be consumed by matching the default RS.
We're using a pipe to/from the while-read loop instead of process substitution just to ease clarity since the OP is already using pipes.