javascriptwebsockethtmlunitwebresponse

Get scripts communications responses in HtmlUnit


I want to get data actualization in a dynamic web. Watching into Chrome Dev Tool I see that the web makes continuous URL get requests to obtain json data or it gets them using websockets.

I was trying to obtain all the responses that the web obtains to actualize its data or the messages recieved by the WebSockets but with my code I only could to get the response of the URL request of the web.

Its possible to make what I need with HtmlUnit?

      /* turn off annoying htmlunit warnings */
        java.util.logging.Logger.getLogger("com.gargoylesoftware").setLevel(java.util.logging.Level.OFF);



        WebClient client = new WebClient(BrowserVersion.CHROME);  
        client.getOptions().setCssEnabled(false);  
        client.getOptions().setJavaScriptEnabled(true);
        client.getOptions().setThrowExceptionOnScriptError(false);

        HtmlPage page = client.getPage("https://mobile.bet365.com/Default.aspx?lng=3");
        client.waitForBackgroundJavaScript(10000);

        List<NameValuePair> response =page.getWebResponse().getResponseHeaders();
        for (NameValuePair header : response) {
             System.out.println(header.getName() + " = " + header.getValue());
         }
        System.out.println(page.asText());

        client.close();
  }

Solution

  • I found the solution to get Html responses in HtmlUnit documentation. I leave here the code, it is possible to make the same with the requests:

    new WebConnectionWrapper(webClient) {
    
            public WebResponse getResponse(WebRequest request) throws IOException {
                WebResponse response = super.getResponse(request);
                if (request.getUrl().toExternalForm().contains("my_url")) {
                    String content = response.getContentAsString();
    
                    //change content
    
                    WebResponseData data = new WebResponseData(content.getBytes(),
                            response.getStatusCode(), response.getStatusMessage(), response.getResponseHeaders());
                    response = new WebResponse(data, request, response.getLoadTime());
                }
                return response;
            }
        };