Send HTTP Requests after browsing to a page as separate requests for Web Crawling (Angular site)

My company has recently upgraded to new version of iManage (a file archive system) and it no longer has libraries exposed to VBA. Due to company policy, I can run VBA but can't create VSTO/.NET addins.

I'm trying to fix an addin tool that inventories all items within a folder/subfolders.

The solution I'm currently exploring is to navigate to the folder in the new web portal, and then inventory from there. I can probably do classic webcrawl and click through links in the browser but that's going to be slow and pretty ugly. As it's an Angular application, I think I should be able to fire REST requests and parse the responses without waiting for page loads.

I'm having an issue with InvalidToken coming back as failed.

{
  "error": {
    "code": "InvalidToken",
    "message": "X-Auth-Token is invalid or missing"
  }
}

The current solution is to create a WebBrowser object within a UserForm in Excel VBA. This userform navigates to our iManage portal. I can then navigate the site and click a button to launch the requests.

Private Sub CommandButton1_Click()
    Debug.Print WebBrowser1.Busy
    
    Dim Doc As HTMLDocument
    Set Doc = WebBrowser1.Document
    Debug.Print Doc.cookie

    Dim Request As New WinHttpRequest
    Request.Open "GET", Url:="https://imanage.xxxx.com/work/web/api/v2/customers/1/libraries/CLIENT-JOB/tabs/CLIENT-JOB!9975487/children?limit=500&offset=0&total=true", ASync:=False
    Request.setRequestHeader "Content-Type", "application/json"
    Request.setRequestHeader "Accept-Encoding", "gzip, deflate, br"
    Request.setRequestHeader "Accept-Language", "en-GB,en-US;q=0.9,en;q=0.8"
    Request.setRequestHeader "Connection", "keep-alive"
    Request.setRequestHeader "Host", Doc.Location.host
    Request.setRequestHeader "Referer", Doc.Location.href
    'Request.setRequestHeader "Cookie", WebBrowser1.Document.cookie
    Request.setRequestHeader "Set-Cookie", WebBrowser1.Document.cookie
    Request.setRequestHeader "X-XSRF-TOKEN", Split(Split(WebBrowser1.Document.cookie, ";")(2), "=")(1)
    Request.send

    Dim Result As String
    Result = Request.responseText
    Debug.Print Result
    
End Sub

Private Sub UserForm_Initialize()
     WebBrowser1.Navigate2 "https://imanage.XXXXX.com/work/web/r/custom2/recent-custom2?exclude_emails=true&scope=Admin,AdminArchive,Client-Job,JobArchive&p=1"
End Sub

Which I feel is replicating the request calls I can see in Chrome.

I think a big part of the issue is the HTMLDocument I see within the WebBrowser never lists all of the same cookies that I can see in Chrome.

Cookies flagged as "HttpOnly" in that last screenshot are not retrievable using document.cookie

https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies

A cookie with the HttpOnly attribute is inaccessible to the JavaScript Document.cookie API; it is sent only to the server. For example, cookies that persist server-side sessions don't need to be available to JavaScript, and should have the HttpOnly attribute. This precaution helps mitigate cross-site scripting (XSS) attacks.

Maybe you can try: Retrieve ALL cookies from Internet Explorer