[SOLVED] Getting page content via XMLHTTPRequest

Getting page content via XMLHTTPRequest

I am using Protractor for my testing. I have 100 links on a page, rather than clicking on each link and checking the page content I am trying to get the page content via XMLHTTPRequest. I am passing all the contents in the request header that I see on the browser for that page. However, when I run the below code I am getting the log in page, apparently because I do not have active session. However, my UI tests in the browser continues to work fine so I am not logged out on my UI tests. My questions are:

Is there a way for passing current browser session to the XMLHTTPRequest so that I am not redirected to the log in page?
Is there any other way I can get the page content without having to manually open each of the pages?

Here is my code

            var request = new XMLHttpRequest();
            let cookies = await browser.manage().getCookies();
            let cookie = "";
            for(let obj of cookies){
                cookie+= Object.keys(obj).map(() => obj.name + "=" + obj.value);
            }
            cookie.replaceAll(",", ";")
            request.open('GET', 'https://mybaseURL.com/member/XX001', false);
            request.withCredentials = true;
            request.setRequestHeader("sec-ch-ua", "\"Chromium\";v=\"112\", \"Google Chrome\";v=\"112\", \"Not:A-Brand\";v=\"99\"");
            request.setRequestHeader("sec-ch-ua-mobile", "?0");
            request.setRequestHeader("sec-ch-ua-platform", "Windows");
            request.setRequestHeader("Sec-Fetch-Dest", "empty");
            request.setRequestHeader("Sec-Fetch-Mode", "cors");
            request.setRequestHeader("Sec-Fetch-Site", "same-origin");
            request.setRequestHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36");
            request.setRequestHeader("Accept", "application/json");
            // request.setRequestHeader("Accept-Encoding", "gzip, deflate, br");
            request.setRequestHeader("Accept-Language", "en-US,en;q=0.9");
            // request.setRequestHeader("Connection", "keep-alive");
            request.setRequestHeader("Content-Type", "application/json");
            request.setRequestHeader("Cookie", cookie);
            request.send(null);
            console.log(request.responseText)

Response:

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="https://mybaseurl.com/signon.do?TYPE=123456&amp;REALMOID=06-123456-1234-1234-a702-3a6f0a060000&amp;GUID=&amp;SMAUTHREASON=0&amp;METHOD=GET&amp;SMAGENTNAME=$SM$vO344545Q%245454545543C%2bOm645545en5s3&amp;TARGET=$SM$https%3a%2f%2fmybaseurl%2ecom%2fmember%2fXX001">here</a>.</p>
</body></html>

Solution

I found the way to get the DOM of the page without physically opening the page using Fetch. It worked like a charm for me.

            var cookies = await browser.manage().getCookies();
            var myCookie = "";

            for(let obj of cookies){
                myCookie+= Object.keys(obj).map(() => {
                    obj.name + "=" + obj.value
                });
            }
            await browser.executeScript(function() {
                var url = 'https://mybaseURL.com/member/XX001';
                var headers = new Headers();
                headers.append('Cookie', window.myCookie);
                var request = new Request(url, {
                  method: 'GET',
                  headers: headers,
                  credentials: 'include'
                });
              
                return fetch(request).then(function(response) {
                    return response.text();
                });
              }).then(function(html) {
                console.log(html);
            });
    ```