I have very large sets of data on mobile and desktop clients that I am trying to bi-directionally sync with my website.
I have currently setup a stream, where I am sending data up the server in chunks. When each chunk is completed processing and the server has successfully registered that chunk of data, I then need to send back response data which contains registry information that helps keep the server and the client in sync.
After receiving that response, the client will upload another chunk and the process begins again. The data from the client is streamed to the server to keep the client's memory footprint low.
I have tried many combinations to see if I can get this to work. In all cases, if the server writes the response and the client has kept the stream open (awaiting the response from the server to continue) then I don't receive the servers response until it's connection timeouts.
So is there a way to do this within a stream?
Send data - receive response Send more data - receive response And so on...?
Below is a small contrived sample on how I've been able to successfully read the streamed data, but not send multiple responses back. NOTE: The actual implementation doesn't send a response on every newline, it's just a simplification for demonstration purposes
func UserSyncHandler(db *psql.Database, cache *serverCache.Cache) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
reader := bufio.NewReader(r.Body)
for {
line, err := reader.ReadBytes('\n')
if err != nil {
if err == io.EOF {
logger.Info("Received EOF")
break // naturally reached end of stream
}
writeError(w, "Failed to read from sync stream", err, http.StatusInternalServerError)
return
}
// Send response here
writeData([]byte("confirmation data"))
}
}
}
Thank you so much to LaGec and Cerise Limón
Below is a modified, contrived example showing how I was able to get it to work.
func UserSyncHandler(db *psql.Database, cache *serverCache.Cache) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
// Added as noted in a comment
controller := http.NewResponseController(w)
_ = controller.EnableFullDuplex() // no error reported
reader := bufio.NewReader(r.Body)
for {
line, err := reader.ReadBytes('\n')
if err != nil {
if err == io.EOF {
logger.Info("Received EOF")
break // naturally reached end of stream
}
writeError(w, "Failed to read from sync stream", err, http.StatusInternalServerError)
return
}
// Send response here
writeData([]byte("confirmation data"))
// Added as noted in another comment
controller.Flush()
}
}
}