I am trying to upload a 0 bytes file with requests to owncloud. I'd like to use a file-like object for that. Normally I'd do it like this:
file_obj = io.BytesIO(b'')
response = requests.put('http://localhost/remote.php/webdav',
auth=('xxx', 'xxx'),
data=file_obj)
But it freezes. If I interrupt the process, I see where it hangs with the stack trace:
Traceback (most recent call last):
File "/home/julian/cc/client/.venv/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 376, in _make_request
httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/julian/cc/client/cc/storage/webdav.py", line 360, in <module>
main()
File "/home/julian/cc/client/cc/storage/webdav.py", line 351, in main
data=file_obj)
File "/home/julian/cc/client/.venv/lib/python3.5/site-packages/requests/api.py", line 120, in put
return request('put', url, data=data, **kwargs)
File "/home/julian/cc/client/.venv/lib/python3.5/site-packages/requests/api.py", line 53, in request
return session.request(method=method, url=url, **kwargs)
File "/home/julian/cc/client/.venv/lib/python3.5/site-packages/requests/sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "/home/julian/cc/client/.venv/lib/python3.5/site-packages/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/home/julian/cc/client/.venv/lib/python3.5/site-packages/requests/adapters.py", line 376, in send
timeout=timeout
File "/home/julian/cc/client/.venv/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
body=body, headers=headers)
File "/home/julian/cc/client/.venv/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 378, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib64/python3.5/http/client.py", line 1174, in getresponse
response.begin()
File "/usr/lib64/python3.5/http/client.py", line 282, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.5/http/client.py", line 243, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib64/python3.5/socket.py", line 575, in readinto
return self._sock.recv_into(b)
KeyboardInterrupt
Wireshark tells me requests sendet the following request, which seems to be fine, but never gets an answer:
PUT /remote.php/webdav/test.txt HTTP/1.1
Host: localhost
Connection: keep-alive
Accept-Encoding: gzip, deflate
Authorization: Basic ***********
Transfer-Encoding: chunked
User-Agent: python-requests/2.9.1
Accept: */*
Content-Length: 0
If I send an empty string, it works:
response = requests.put('http://localhost/remote.php/webdav/test.txt',
auth=('xxx', 'xxx'),
data='')
HTTP Stream:
PUT /remote.php/webdav/test.txt HTTP/1.1
Host: localhost
Content-Length: 0
Accept-Encoding: gzip, deflate
Connection: keep-alive
User-Agent: python-requests/2.9.1
Accept: */*
Authorization: Basic ******
HTTP/1.1 204 No Content
Date: Thu, 24 Mar 2016 16:14:28 GMT
Server: Apache/2.4.10 (Debian) PHP/5.6.19
X-Powered-By: PHP/5.6.19
Set-Cookie: xxx=xxx; path=/; HttpOnly
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: oc_sessionPassphrase=xxxx; path=/; httponly
Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; frame-src *; img-src * data: blob:; font-src 'self' data:; media-src *; connect-src *
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Frame-Options: Sameorigin
X-Robots-Tag: none
X-Download-Options: noopen
X-Permitted-Cross-Domain-Policies: none
Set-Cookie: xxx=xxx; path=/; HttpOnly
OC-FileId: xxxxx
Content-Length: 0
ETag: "xxx"
OC-ETag: "xxx"
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=UTF-8
This works as well:
def chunker(file_obj):
buf = None
while buf != b'':
print('iter')
buf = file_obj.read(16*1024)
yield buf
file_obj = io.BytesIO(b'')
response = requests.put('http://localhost/remote.php/webdav/test.txt',
auth=('xxx', 'xxx'),
data=chunker(file_obj))
Any ideas why this is not working with file-like objects? I am using the latest version of requests (2.9.1) and Python 3.5.
There seems to be a bug in requests. Without specifying auth the request uses Transfer-Encoding: chunked and sends the correct last-chunk at the end of the request, but with auth no last-chunk is sent and the headers are confused.
From https://www.rfc-editor.org/rfc/rfc7230#section-4.1:
chunked-body = *chunk
last-chunk
trailer-part
CRLF
chunk = chunk-size [ chunk-ext ] CRLF
chunk-data CRLF
chunk-size = 1*HEXDIG
last-chunk = 1*("0") [ chunk-ext ] CRLF
chunk-data = 1*OCTET ; a sequence of chunk-size octets
and https://www.rfc-editor.org/rfc/rfc7230#section-3.3.2:
A sender MUST NOT send a Content-Length header field in any message
that contains a Transfer-Encoding header field.
Without the auth argument
f = io.BytesIO(b'')
requests.put('http://localhost:8000/asdf', data=f)
the request sent is
PUT /asdf HTTP/1.1⏎
Host: localhost:8000⏎
User-Agent: python-requests/2.9.1⏎
Transfer-Encoding: chunked⏎
Accept-Encoding: gzip, deflate⏎
Connection: keep-alive⏎
Accept: */*⏎
⏎
0⏎
⏎
(⏎ signifies CRLF above). But if you specify auth
requests.put('http://localhost:8000/asdf', auth=('asdf', 'fdsa'), data=f)
the request is
PUT /asdf HTTP/1.1⏎
Host: localhost:8000⏎
Transfer-Encoding: chunked⏎
Accept-Encoding: gzip, deflate⏎
User-Agent: python-requests/2.9.1⏎
Authorization: Basic YXNkZjpmZHNh⏎
Content-Length: 0⏎
Connection: keep-alive⏎
Accept: */*⏎
⏎
Both Transfer-Encoding and Content-Length are specified, which it should not do, and the last-chunk is not sent, and so the server sits waiting for more chunks to come and requests waits for a response.