I'm working on a server that accepts a URL from a user and downloads it (and does other stuff to it ofc like uploading it back but that's irrelevant here). The maximum file size it should accept is 4 GB which is why the Content-Length MUST exist for the URL that the user provides.
But what happens if say, a malicious server gives a Content-Length of say 2 GB, and ends up transferring 6 GB instead? Are there are mechanisms in place to stop that? I'm using the Rust library reqwest but answers for other HTTP clients would be great too.
A common implementation will just take the Content-length and read as much data as specified - leaving the remaining data in the socket buffer (or maybe some user space buffer). Thus it likely works for this specific request.
But this might actually cause trouble in case of HTTP persistent connection. For a request with a too short Content-length the remaining data will be interpreted as another HTTP request on the same connection. For a response with a similar problem the remaining data will be interpreted as the response to the next request on the connection. In the best case this will be treated as an error due to malformed data and the request will be abandoned. In the worst case it might lead to a security issue though - see also HTTP request and response splitting as a related attack.
... which is why the Content-Length MUST exist for the URL
Please note that Content-length is not actually required in the request or response. The message header might have no indication on the ultimate size of the response, since it might use Transfer-Encoding: chunked or just end with the close of a TCP connection.