I want to download a json file from a website. Using the chrome dev tool, when I am copying the cURL command of the file downloading url, it is giving me the required json response. But when copying the fetch (nodejs) command, and running through "node-fetch" module, it is redirecting me to the access-denied page.
Using Fetch(copied exactly from that website) through async/await
(async ()=>{
try
{
var res=await fetch("url/getjson?file", {
"headers": {
"accept": "application/json, text/plain, */*",
"accept-language": "en-GB,en-US;q=0.9,en;q=0.8",
"content-type": "application/json",
"sec-fetch-dest": "empty",
"sec-fetch-mode": "cors",
"sec-fetch-site": "same-origin",
"sec-gpc": "1",
"cookie": "cooookiiieee=correctcookie; AuthToken=gibberishh; UserName=blahblah; EntityRefId=pleasehelp"
},
"referrer": "referrerurl",
"referrerPolicy": "strict-origin-when-cross-origin",
"body": null,
"method": "GET",
"mode": "cors"
});
const content = await res.json();
console.log(content);
}
catch(err){
console.log(err)
}
})();
Fetch Response
FetchError: invalid json response body at url/error/accessdenied reason: Unexpected token < in JSON at position 1
at /home/user/Documents/node_modules/node-fetch/lib/index.js:272:32
at processTicksAndRejections (internal/process/task_queues.js:93:5)
at async /home/user/Documents/test/app.js:25:19 {
type: 'invalid-json'
}
whereas the response with cURL command was the correct json file as it is not redirecting me.
Curl Command
curl 'url/getjson?file'
-H 'Connection: keep-alive'
-H 'Accept: application/json, text/plain, */*'
-H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.101 Safari/537.36'
-H 'Content-Type: application/json'
-H 'Sec-GPC: 1' -H 'Sec-Fetch-Site: same-origin'
-H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Dest: empty'
-H 'Referer: referrerurl'
-H 'Accept-Language: en-GB,en-US;q=0.9,en;q=0.8'
-H 'Cookie:
cooookiiieee=correctcookie; AuthToken=gibberishh; UserName=blahblah; EntityRefId=pleasehelp'
--compressed
The correct response was received after I added User-Agent in the headers as it was considering the default node agent as bot. It must be added before cookie i.e before sending AuthToken.