[SOLVED] node request module: parsing XML as JSON

node request module: parsing XML as JSON

Until recently I've been fetching XML data using the node request module, and then running that XML through an XML to JSON converter. I discovered by accident that if I set json: true as an option (even knowing the endpoint returns XML, not JSON), I was actually getting back JSON:

var request = require('request');
var options = { gzip: true, json: true, headers: { 'User-Agent': 'stackoverflow question (https://stackoverflow.com/q/52609246/4070848)' } };
options.uri = 'https://api.met.no/weatherapi/locationforecast/1.9/?lat=40.597&lon=-74.26';
request(options, function (error, response, body) {
    console.log(`body for ${options.uri}: ${JSON.stringify(body)}`);
});

The above call returns JSON, whereas the raw URL is actually sending XML. Sure enough, with json: false the returned data is XML:

var request = require('request');
var options = { gzip: true, json: true, headers: { 'User-Agent': 'stackoverflow question (https://stackoverflow.com/q/52609246/4070848)' } };
options.uri = 'https://api.met.no/weatherapi/locationforecast/1.9/?lat=40.597&lon=-74.26';
options.json = false; // <<--- the only difference in the request
request(options, function (error, response, body) {
    console.log(`body for ${options.uri}: ${body}`);
});

So I thought "that's handy", until I tried the same trick with a different URL that also returns XML, and in this case the returned data is still XML despite using the same request options:

var request = require('request');
var options = { gzip: true, json: true, headers: { 'User-Agent': 'stackoverflow question (https://stackoverflow.com/q/52609246/4070848)' } };
options.uri = 'https://graphical.weather.gov/xml/SOAP_server/ndfdXMLclient.php?whichClient=NDFDgen&lat=40.597&lon=-74.26&product=time-series&temp=tempSubmit=Submit';
request(options, function (error, response, body) {
    console.log(`body for ${options.uri}: ${body}`);
});

What is the difference here? How do I get the latter request to return the data in JSON format (so that I can avoid the step of converting XML to JSON myself)? Maybe the endpoint in the first example can detect that JSON is requested and it does in fact return JSON rather than XML?

EDIT weirdly, the first request is now returning XML rather than JSON even with json: true. So maybe this behaviour was down to what was being sent from the endpoint, and they've changed this even since I posted a few hours ago

Solution

So now that the behavior is unrepeatable, the answer is less useful for your particular problem, but I think it's worth pointing out that when you set json:true on the request module, it does a few things under the hood for you:

Sets the Accept header to 'application/json'
Parses the response body using JSON.parse()
Request types with a body also get the body automatically serialized as JSON
Request types with a body also get the Content-Type header added as 'application/json'

So perhaps they did change it, but there are plenty of web services I've seen that will detect the content-type to send based on the Accept header and respond appropriately for some set of types that make sense (usually XML or JSON, but sometimes CSV, TXT, HTML, etc).