I have an experimental PHP based web crawler and I noticed that it cannot read some pages, for example on some particular domains curl says it failed after following 50 redirects but wget reads that same domain just fine:
curl 'netflix.com' -L -o 'output.txt'
Result:
curl: (47) Maximum (50) redirects followed
No data in output.txt file.
While this command works fine:
wget netflix.com
Any ideas on what can cause this? I doubt that remote server handles requests based on the two different user agents.
This is probably because you didn't tell curl to use cookies, which it doesn't do unless you ask it to - while wget enables them by default.
Use the --cookie or --cookie-jar options to enable cookies.