watir

Watir Scroll For All Comments

I'm trying to scroll until there are no more contents on a Youtube video. I'm testing a counter versus the current number of comments displayed.

mycount = 0
1.upto(20) do
  thiscount = browser.div(id: "contents").divs(id: "comment-content").size
  puts "#{ mycount } : #{ thiscount }"
  break if mycount == thiscount
  mycount = thiscount
  browser.driver.execute_script("window.scrollBy(0,1000)")
  sleep 10
end

After the first pagedown, the count of comments should increase. It isn't. I've entered a sleep 10 for the comments to load and the count of those comments to also update. It doesn't update. I keep getting 20 : 20 so it breaks and leaves this iteration after a single iteration.

I'm not sure why that valuation isn't updating. How can I fix this so that it can get to the end of the comments?

Solution

  • I ran into 2 problems when running your script:

    1. browser.div(id: "contents").divs(id: "comment-content") returned nothing. There were many "contents" divs, with the first one not including any comments. I removed this locator.
    2. Scrolling by 1000 is not enough to get to the bottom of the page. Therefore the loading of additional comments does not get triggered.

    From the following script:

    scroll_by = 1000000000000

browser.goto('https://www.youtube.com/watch?v=u9DPBxiZZfo&ab_channel=America%27sTestKitchen')
browser.driver.execute_script("window.scrollBy(0,#{scroll_by})")
sleep 15
mycount = 0
1.upto(20) do
  thiscount = browser.divs(id: "comment-content").size
  puts "#{ mycount } : #{ thiscount }"
  break if mycount == thiscount
  mycount = thiscount
  browser.driver.execute_script("window.scrollBy(0,#{scroll_by})")
  sleep 10
end

    You can see that having a larger scroll gives the expected results:

    0 : 20
20 : 40
40 : 60
60 : 80
80 : 100
100 : 120
120 : 140
140 : 160
160 : 180
180 : 185
185 : 185

    In contrast, setting the scroll_by to just "1000" did not trigger the comments. The output was just:

    0 : 0