javascriptpythonajaxsizeserver

Python local server issue: rfile gets truncated when posting through ajax

I am having an issue when receiving a large string through an Ajax POST to a local python http.server.

I am trying to create a webapp that runs off of the localhost. The webapp load a JSON file when it starts up and uses this to load the data into the app. This data ends up being manipulated while using the app.

When saving, I want to send the entire modified data to a python script, and then write that data to the aforementioned JSON file, so when the webapp re-opens, it will have the most recent data.

The server works properly and saving smaller data to other files works without any issue. The issue is due to the size of this data that I want to save. When sending the data (as a JSON.stringified string) to the python script, the string gets truncated, so the python script is not able to parse it to recreate the JSON object.

I modified the do_POST function to return the content-length header and then the actual size of the data received:

    def do_POST(self):
      length = int(self.headers['Content-Length'])
      print(length)
      post_data = self.rfile.read(length)
      print(len(post_data))

When sending the un-modified data from the browser, the Content-length header is 1013727, but the rfile length is only 23684.

I printed the rfile data and, indeed, it is not the entire string that I tried to send and stops in the middle of an array.

I checked using the Chrome Dev Tools, and the request payload is also the full string that I expected it to be, so I am assuming the issue might be with the file size limitations of the http.server. Unfortunately, I cannot find any information about what this may be or how to change it.

I am using Python 3.4 and Google Chrome as my browser.

Any help will be appreciated. Thanks!

Solution

  • I created a work around by digging through the socket and http.server code

    For CGIHTTPRequestHandler (in http.server), the buffering for the rfile is enabled, and then everything after the first buffer seems to be thrown away.

    What (I think) I did is disabled the buffering, and then set the "first buffer" value to be the entire string by adding the following code to the Handler class:

    rbufsize = -1

def do_POST(self):
  self.rfile._sock = self.rfile
  return CGIHTTPRequestHandler.do_POST(self)

    So it works for now, but it might cause issues in the future