First to what works: (the description of my problem further down)
A client application sends a long data-stream to the server WebAPI.
The request is chunked (request.Headers.TransferEncodingChunked = True) and the data gets written to the stream "on the fly".
The data consists of serialized objects, each representing the custom class Chunk.
Everything works fine, the server can start to read the stream before the client is done writing (server input buffering disabled for the controller).
Starting the request:
Public Async Function Transfer(authCookie As Cookie) As Task
Using handler As New HttpClientHandler()
handler.CookieContainer = New Net.CookieContainer
handler.CookieContainer.Add(authCookie)
'Client instance
Using client As New HttpClient(handler), content As New Content(AddressOf WriteToStream)
'Send stream asynchronously
Dim request As New HttpRequestMessage(HttpMethod.Post, "https://myserver.net/api/datastream")
request.Content = content
request.Headers.TransferEncodingChunked = True
Await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead).ContinueWith(
Async Function(requestTask)
Dim response As HttpResponseMessage = Nothing
Try
response = Await requestTask
Catch ex As Exception
'Exception writing the stream
Exit Function
End Try
Try
response.EnsureSuccessStatusCode()
Catch ex As Exception
'Exception on server ocurred
End Try
response.RequestMessage.Dispose()
response.Dispose()
End Function)
End Using
End Using
End Function
Public Sub WriteToStream(str As IO.Stream)
'Head-chunk
head.attachedChunks.CompleteAdding()
'Write head-chunk to stream
Task.Run(
Async Function()
Await head.WriteToStream(str)
End Function).Wait()
End Sub
HttpContent Class:
Public Class Content
Inherits Net.Http.HttpContent
Protected ReadOnly actionOfStream As Action(Of IO.Stream)
Public Sub New(action As Action(Of IO.Stream))
If action Is Nothing Then Throw New ArgumentNullException("action")
actionOfStream = action
End Sub
Protected Overrides Function SerializeToStreamAsync(stream As IO.Stream, context As Net.TransportContext) As Task
Return Task.Factory.StartNew(
Sub(obj)
Dim target As IO.Stream = DirectCast(obj, IO.Stream)
actionOfStream(target)
End Sub,
stream)
End Function
Protected Overrides Function TryComputeLength(ByRef length As Long) As Boolean
length = -1
Return False
End Function
End Class
The method WriteToStream() calls the Head-chunks WriteToStream(), which calls the same method for its sub-chunks:
Public Class Chunk
'More properties here
Public Property attachedChunks As BlockingCollection(Of Chunk)
Public Overridable Async Function WriteToStream(str As IO.Stream) As Task
Using serializationStream As New IO.MemoryStream
'Serialize
Serializer.Serialize(Of Chunk)(serializationStream, Me)
'Write length
Dim cnt As Integer = CInt(serializationStream.Length)
Dim cntBuffer() As Byte = BitConverter.GetBytes(cnt)
Await str.WriteAsync(cntBuffer, 0, cntBuffer.Length)
'Write chunk
serializationStream.Seek(0, IO.SeekOrigin.Begin)
Await serializationStream.CopyToAsync(str)
End Using
Await str.FlushAsync()
'Clearing and disposing stuff
'...
'Write sub-chunks
If attachedChunks IsNot Nothing Then
For Each chunk As Chunk In attachedChunks.GetConsumingEnumerable
Await chunk.WriteToStream(str)
Next
attachedChunks.Dispose()
End If
'Write ending mark
If attachedChunksIndefiniteCount Then
Await str.WriteAsync({0, 0, 0, 0}, 0, 4)
End If
End Function
End Class
So far so good.
The problem comes when I start using a compression like GZipStream to compress my Chunk: the request gets buffered on client side, so it seams.
Here's the WriteToStream() method of the Chunk-class:
Public Class Chunk
'More properties here
Public Property attachedChunks As BlockingCollection(Of Chunk)
Public Overridable Async Function WriteToStream(str As IO.Stream) As Task
Using serializationStream As New IO.MemoryStream
'Serialization and compression
Using gZipStream As New IO.Compression.GZipStream(serializationStream, IO.Compression.CompressionMode.Compress, True)
Using bufferStream As New IO.BufferedStream(gZipStream, 64 * 1024)
'Serialize
Serializer.Serialize(Of Chunk)(bufferStream, Me)
End Using
End Using
'Write length
Dim cnt As Integer = CInt(serializationStream.Length)
Dim cntBuffer() As Byte = BitConverter.GetBytes(cnt)
Await str.WriteAsync(cntBuffer, 0, cntBuffer.Length)
'Write chunk
serializationStream.Seek(0, IO.SeekOrigin.Begin)
Await serializationStream.CopyToAsync(str)
End Using
Await str.FlushAsync()
'Clearing and disposing stuff
'...
'Write sub-chunks
If attachedChunks IsNot Nothing Then
For Each chunk As Chunk In attachedChunks.GetConsumingEnumerable
Await chunk.WriteToStream(str)
Next
attachedChunks.Dispose()
End If
'Write ending mark
If attachedChunksIndefiniteCount Then
Await str.WriteAsync({0, 0, 0, 0}, 0, 4)
End If
End Function
End Class
It does not send the request until all the data is written. Does the request buffer the stream? Does the GZipStream have effect on some kind of underling context? I can't figure out what the problem is...
I found out what was buggin' me: Testing with a larger amount of data, the request started to send data after a coulple of chunks.
The request has a certain buffer (which you can't set working with HttpClient). The uncompressed content reached that limit quite soon, the compressed content obviously later.
It just looked like the whole request was buffered because my testing-data was too short.