bash nginx fastcgi server-sent-events buffering

Can't disable buffering on a cgi using nginx, fastcgi and bash

I want to make a CGI that launches a program and displays the output in the web-browser in real-time. Real-time here means as soon as a line of output is produced by the program it should be displayed on the browser page.

I choose to write it in bash to wrap the program execution so I can process request parameters.

webbrowser -> nginx+fastcgi -> bash -> program

I have a testing program that outputs one line every half a second, 10 times.

I thought I could declare a plain text context type in response header and then exec the program.

Unfortunately the output appears in the browser only at the end of the execution all at once. I tested it in Firefox and curl.

I have tested many options and any combination of them to workaround the issue :

disable buffering in nginx with fastcgi_buffering off directive
add X-Accel-Buffering header
use stdbuf -oL program
use xml http request (XHR) + server side event (SSE) instead of plain text.

Nothings works.

I guess the buffering issue is between nginx and bash but I find no way to disable it.

What can I try ?

Solution

I found no way to fix buffering issue in my chain. I tried perl instead of bash with no luck.

So I choose to fill the buffers : after each line of output of the controlled program I echo a bunch of '\0'. Since this content can not be process as a plain text by the web browser, I use the server sent event approach.

#!/bin/sh

printf "HTTP/1.0 200 OK\r\n"
printf "Content-type: text/event-stream\r\n"
printf "Cache-Control: no-cache\r\n"
printf "X-Accel-Buffering: no\r\n"
printf "\r\n"

flush() {
    padding=4100
    dd if=/dev/zero bs=$padding count=1 2>/dev/null
}

subprogram | while read l;
do
    printf "data: ${l}\n\n"
    flush
done

The wrapping page looks like that :

<html>
<head>
   <meta charset="UTF-8">
   <title>Server-sent events demo</title>
</head>
<body>
  <pre></pre>
<script>
var evtSource = new EventSource('/sse.sh?subprogram');
var pre = document.querySelector('pre');
evtSource.onmessage = function(e) {
  pre.textContent += e.data + '\n';
}
</script>
</body>
</html>

The web browser, in that case, takes care of removing extra '\0'.

The drawback is that the cgi output is far larger than the program output.