I've noticed that many pages of my single page application (SPA) website are not indexed by Google because they supposedly return "Soft 404" errors. But when I visit these pages in a browser everything works fine.
I tried to inspect these pages with Live Test tool on Google Search console, and I've noticed that the pages are rendered with an error message because of a failed XHR request to my backend. When I go to "More Info" -> "Page resources", it says that the XHR request failed because of "Other error" and no other details are provided. JavaScript console messages are empty too.
I inspected the logs of my backend to see what HTTP error is returned. Turns out these requests are not even made by my website when it's rendered in Google's URL Live Test tool.
What could cause XHR requests to fail with "Other error" message in Google's URL Live Test tool?
After some research, I figured out what was the problem.
Turns out that before making an XHR request to any server, Googlebot makes an HTTP request to the '/robots.txt' URL of that server (not necessary the original website server).
My backend server wasn't able to handle these '/robots.txt' requests properly, and consequently Googlebot was cancelling them because of a timeout. And when that happens, Googlebot don't even make the original XHR request, causing my site to show an error message.
The solution was to make my backend server to handle '/robots.txt' requests properly. In my case, just returning a 404 HTTP error was enough.