Given a shared HTTP manager, it seems that if the requestBody is of type requestBodySource and if wrong length is supplied for the request body, then subsequent requests will crap out on same HTTP manager for about 20 seconds. There seems to be something about interaction of shared state and GivesPopper perhaps that is causing this issue. Here is a sample code that reproduces it - we use requestb.in for sending wrong length upload, and then try to read another valid URL on requestb.in.
{-# LANGUAGE OverloadedStrings #-}
import Data.Conduit.Binary (sourceFile)
import Network.HTTP.Conduit
import Network.HTTP.Types
import qualified Data.ByteString.Lazy as LBS
import System.IO
import Control.Monad.Trans.Resource (runResourceT)
import Control.Concurrent.Async (async,waitCatch)
import Control.Exception (displayException)
main :: IO ()
main = do
{- Set up a ResourceT region with an available HTTP manager. -}
httpmgr <- newManager tlsManagerSettings
httpmgr2 <- newManager tlsManagerSettings
let file ="out" -- some byte contents with length > 1
lenb <- System.IO.withFile file ReadMode hFileSize
let inbytes = sourceFile file
initReq <- parseUrl "http://requestb.in/saxbx3sa"
putreq <- async $ runResourceT $ do
let req = initReq { method = "POST",
-- let us send wrong length in requestBodySource
requestBody = (requestBodySource (fromIntegral $ lenb - 1) inbytes)}
resp <- httpLbs req httpmgr
return (statusCode . responseStatus $ resp)
putreqRes <- waitCatch putreq
case putreqRes of
Left e -> print $ displayException $ e
Right r -> print $ r
getreq <- async $ runResourceT $ do
-- Let us do a GET on a different resource to see if it works
initReq <- parseUrl "http://requestb.in/1l15sz21"
let req = initReq { method = "GET"}
resp <- httpLbs req httpmgr
return (statusCode . responseStatus $ resp)
getreqRes <- waitCatch getreq
case getreqRes of
Left e -> print $ displayException $ e
Right r -> print $ r
Output - first bad upload goes through as HTTP 200, and subsequent GET request immediately causes HTTP 400 error:
*Main> main
200
"StatusCodeException (Status {statusCode = 400, statusMessage = \"Bad Request\"})
[(\"Date\",\"Wed, 29 Jun 2016 11:54:59 GMT\"),(\"Content-Type\",\"text/html\"),
(\"Content-Length\",\"177\"),(\"Connection\",\"close\"),(\"Server\",\"-nginx\"),
(\"CF-RAY\",\"-\"),(\"X-Response-Body-Start\",\"<html>\\r\\n<head><title>400 Bad
Request</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>400 Bad
Request</h1></center>\\r\\n<hr><center>cloudflare-
nginx</center>\\r\\n</body>\\r\\n</html>\\r\\n\"),(\"X-Request-URL\",\"GET
http://requestb.in:80/saxbx3sa\")] (CJ {expose = []})"
Using a different http manager instead for GET request will return HTTP 200. So, shared state in http manager seems to be the problem here.
Has anyone else observed it? I went through github issues for HTTP Manager but haven't seen this reported. On wrong streaming length, the behavior shouldn't be to corrupt the HTTP manager as seems to happen here.
I have also simulated a source file for requestBodySource where length is correct, but the source aborts mid-way due to a simulated failure (to simulate network issues). In that case, there are no errors. So, it seems we have just one case where sending wrong length without any failures will cause some kind of shared state to become corrupt here, which gets released within 25 seconds.
If anyone has any insights on what is going on here, it will be very helpful. I have a workaround of enforcing right streaming length. However, I will like to understand what is going on so that I can avoid running into this situation in production.
This is an issue with http-client as reported here. It leaves it up to the caller to make sure that the passed content length is correct. It is the shared connection to the server that seems to be in the bad state. Depending on the actual length vs expected length, the beginning of the next request might be treated as the end of the previous request body, causing the next request to be misinterpreted by the server.
This has been fixed and merged into the trunk through a pull request. The solution was to add a simple length validation.