rubyseleniumwatirbrowsermob-proxyhar

Browsermob Proxy + Watir not capturing traffic continuously


I have the BrowserMob Proxy set up correctly with Watir and it is capturing traffic and saving the HAR file; however, what it's not doing is that it's not capturing the traffic continuously. So following is what I'm trying to achieve:

  1. Go to homepage
  2. Click on a link to go to another page where I need to wait for some events to happen
  3. Once on the second page, start capturing traffic after the event happens and wait for a specific call to occur and capture its contents.

What I'm noticing however, is that it's following all of the above steps, but on step 3 the proxy stops capturing traffic before that call is even made on that page. The HAR that is returned doesn't have that call in it hence the test fails before it even does its job. Following is how the code looks like.

class BMP
attr_accessor :server, :proxy, :net_har, :sel_proxy

 def initialize
    bm_path = File.path(Support::Paths.cucumber_root + "/browsermob- 
    proxy-2.1.4/bin/browsermob-proxy")
    @server = BrowserMob::Proxy::Server.new(bm_path, {:port => 9999, 
       :log => false, :use_little_proxy => true, :timeout => 100})
    @server.start
    @proxy = @server.create_proxy
    @sel_proxy = @proxy.selenium_proxy

    @proxy.timeouts(:read => 50000, :request => 50000, :dns_cache => 
      50000)

    @net_har = @proxy.new_har("new_har", :capture_binary_content => 
      true, :capture_headers => true, :capture_content => true)
end

def fetch_har_entries(target_url)

  har_logs = File.join(Support::Paths.har_logs, "har_file # . 
  {Time.now.strftime("%m%d%y_%H%M%S")} .har")
  @net_har.save_to har_logs

  index = 0
  while (@net_har.entries.count > index) do

    if @net_har.entries[index].request.url.include?(target_url) && 
    entry.request.method.eql?("GET")
      logs = JSON.parse(entry.response.content.text) if not 
          entry.response.content.text.nil?
      har_logs = File.join(Support::Paths.har_logs, "json_file_# . 
          {Time.now.strftime("%m%d%y_%H%M%S")}.json")
      File.open(har_logs, "w") do |json|
         json.write(logs)
      end
      break
    end 
  index += 1
  end
 end
end

In my test file I have following

Then("I navigate to the homepage") do
 visit(HomePage) do |page|
  page.element.click
 end
end

And("I should wait for event to capture traffic") do 
 visit(SecondPage) do |page|
  page.wait_until{page.element2.present?)
  BMP.fetch_har_entries("target/url")
 end
end

What am I missing that is causing the proxy to not capture traffic in its entirety?


Solution

  • In case anyone gets here from a google search, I figured out how to resolve this on my own (thanks stackoverflow community for nothing, lol). So to resolve the issue, i used a custom retriable loop called eventually method.

     logs = nil
    eventually(timeout: 110, interval: 1) do
      @net_har = @proxy.new_har("har", capture_binary_content: true, capture_headers: true, capture_content: true)
      @net_har.entries.each do |entry|
        begin
          break if @net_har.entries.index entry == @net_har.entries.count
          next unless entry.request.url.include?(target_url) &&
                      entry.request.post_data.text.include?(target_body_text)
    
          logs = entry.request.post_data.text
          break
        rescue TypeError
          fail("Response body for the network call came back empty")
        end
      end
      raise EOFError if logs_hash.nil?
    end
    logs
    end
    

    Basically I'm assuming what was happening was the BMP would only cache or capture 30 seconds worth of har logs, and if my network event didn't occur during those 30 secs, i was SOL. So the what above code is doing is that's it's waiting for the logs variable to be not nil, if it is, it raises an EOFError and goes back to the loop initializes the har again and looks for the network call again. It keeps on doing that until it find the call or 110 seconds are up. Following is the eventually method I'm using

    def eventually(options = {})
      timeout = options[:timeout] || 30
      interval = options[:interval] || 0.1
      time_limit = Time.now + timeout
      loop do
        begin
         yield 
        rescue EOFError => error
      end
      return if error.nil?
      raise error if Time.now >= time_limit
    
      sleep interval
     end
    end