watir

Watir Scroll For All Comments


I'm trying to scroll until there are no more contents on a Youtube video. I'm testing a counter versus the current number of comments displayed.

mycount = 0
1.upto(20) do
  thiscount = browser.div(id: "contents").divs(id: "comment-content").size
  puts "#{ mycount } : #{ thiscount }"
  break if mycount == thiscount
  mycount = thiscount
  browser.driver.execute_script("window.scrollBy(0,1000)")
  sleep 10
end

After the first pagedown, the count of comments should increase. It isn't. I've entered a sleep 10 for the comments to load and the count of those comments to also update. It doesn't update. I keep getting 20 : 20 so it breaks and leaves this iteration after a single iteration.

I'm not sure why that valuation isn't updating. How can I fix this so that it can get to the end of the comments?


Solution

  • I ran into 2 problems when running your script:

    1. browser.div(id: "contents").divs(id: "comment-content") returned nothing. There were many "contents" divs, with the first one not including any comments. I removed this locator.
    2. Scrolling by 1000 is not enough to get to the bottom of the page. Therefore the loading of additional comments does not get triggered.

    From the following script:

    scroll_by = 1000000000000
    
    browser.goto('https://www.youtube.com/watch?v=u9DPBxiZZfo&ab_channel=America%27sTestKitchen')
    browser.driver.execute_script("window.scrollBy(0,#{scroll_by})")
    sleep 15
    mycount = 0
    1.upto(20) do
      thiscount = browser.divs(id: "comment-content").size
      puts "#{ mycount } : #{ thiscount }"
      break if mycount == thiscount
      mycount = thiscount
      browser.driver.execute_script("window.scrollBy(0,#{scroll_by})")
      sleep 10
    end
    

    You can see that having a larger scroll gives the expected results:

    0 : 20
    20 : 40
    40 : 60
    60 : 80
    80 : 100
    100 : 120
    120 : 140
    140 : 160
    160 : 180
    180 : 185
    185 : 185
    

    In contrast, setting the scroll_by to just "1000" did not trigger the comments. The output was just:

    0 : 0