python xml qt curl basehttprequesthandler

Bad request type PUT when sending PUT request through Qt, works with curl

I have subclassed BaseHTTPRequestHandler and implemented do_GET(), do_POST() and do_PUT().

I thought that everything works fine until I put my Qt application to the test. The GET request works and the server sends an XML document with some data that the application processes. The POST request also works and is used to generate test cases through REST.

As for the PUT things are looking weird.

Here is the PUT handler on the server side:

def do_PUT(self):
        """
        Processes PUT request. It can be tested using wget, curl or any
        other tool with support for http PUT. Use this to send reports
        about the current status of device and all of its slave devices

        Example:
            curl -X PUT -d "status=downloading" http://127.0.0.1:8090/so/updateProgress
        """
        print('Processing PUT request...')

        params_parsed = urlparse(self.path)
        params = parse_qs(params_parsed.query)
        if len(params) > 0 or '/so/updateProgress' not in params_parsed.path:
            print('Use "/so/updateProgress" to report on update progress')
            self.__set_headers(data_type='text/html')
            self.send_error(501)
            return

        self.__set_headers(data_type='text/xml')
        report_raw = self.rfile.read(int(self.headers.getheader('Content-length')))
        print('Received report')
        print('Parsing report...')
        # Generate XML from the raw data and validate it
        report = SoRequestHandler.Report.from_xml(report_raw)
        print('Parsing of report complete')
        report.write_to_file(log_path='report_log', append=True)
        self.send_response(200)

In my Qt application I have a class called ReportPublisher, which takes some reports from all the devices it is connected to (through MQTT), aggregates the reports into a single one and sends it to the server, which logs it in a file:

void ReportPublisher::publishHttpReport(HttpReport report)
{
    QString reportXml = report.xml().toString();
    if (reportXml.isEmpty())
    {
        LOG(ERROR) << "Something went wrong with the generation of HTTP report";
        return;
    }

    LOG(INFO) << "Generated report for SO server: " << reportXml;
    QByteArray reportRaw = reportXml.toUtf8();

    QUrl target(this->urlServer);
    target.setPath(this->urlPath);
    QNetworkRequest req(target);
    req.setHeader(QNetworkRequest::ContentTypeHeader, QString("application/xml"));
    this->reply = this->nam->put(req, reportRaw);

    // TODO Read back response (expected code 200)?
}

I have to be honest. I have never done a PUT request in Qt and all I've done so far were GET requests so there might be some really fundamental yet easy to spot error in the code above.

The data that the server receives looks like this:

<updateProgress xmlns='service.so.de/so'>
    <deviceTypeId>...</deviceTypeId>
    <packageId>...</packageId>
    <version>...</version>
    <status>...</status>
</updateProgress>

If I use curl like this

root@obunit:~$ curl -X PUT -i 'http://192.168.120.61:8090/so/updateProgress' -d "<updateProgress xmlns='service.so.de/so'><deviceTypeId>...</deviceTypeId><packageId>...</packageId><version>...</version><status>...</status></updateProgress>"

where 192.168.120.61:8090 is the IP address and the port the server is located at/listening to for incoming requests I have no problem.

However with data coming from my Qt application I get

192.168.120.172 - - [11/Apr/2018 15:32:37] code 400, message Bad HTTP/0.9 request type ('PUT')
192.168.120.172 - - [11/Apr/2018 15:32:37] "PUT  HTTP/1.1" 400 -

in my log (with 192.168.120.172 being the IP address of the system where my software is running.

From my scarce knowledge code 400 means invalid syntax, which can be due to following 2 reasons (at least what I can think of right now):

malformed data (invalid XML, JSON etc.)
incorrect encoding of the data, which is basically equivalent to the malformed data but coming from a different source.

I have tried converting the XML document that I'm generating to QByteArray using QDomDocument::toByteArray(int i). I have also tried (as you can see in the code) to convert the document to QString and then to a UTF-8 QByteArray but I can't get my server to process the data.

What's even stranger is that (as you can see in my do_PUT()) implementation my PUT handler starts with printing a message, which never appears in the logs hence the issue arises from somewhere deeper in the code of BaseHTTPRequestHandler.

Solution

The problem was rather simple yet very annoying.

Following the advice of @MrEricSir I did use Wireshark just to find out that get some weird TCP retramsission. Retransmission usually occurs due to network congestion, which in my case was not a possibility however it did indicate a network issue that has nothing to do with the XML content I was sending.

So I started my remote debugging session again and look closer to where I was emitting the PUT request. And there it was! There was a / missing between the base URL (IP:PORT) and the path. So instead of

PUT 192.168.120.61:8090/so/updateProgress

I was doing

PUT 192.168.120.61:8090so/updateProgress

which ultimately led to the PUT request not succeeding at all (hence the code 400).

Just a hint for anyone reading this in the future - whenever you manually set that path using QUrl::setPath(QString)

QUrl target(this->urlServer);
target.setPath(this->urlPath);

ALWAYS make sure that the string you pass as a path starts with / because Qt isn't going to add one!