I'm running a selenium script in Python with the Firefox browser, where the task is to scroll down until the end of the page. The process is composed by scroll down the page, click a button at the bottom which will trigger the page to load more content. This steps should repeat until there's no button anymore.
It's a slow process that takes several hours to complete and, although this script worked in some other pages, for a longer page, I'm stuck in a problem where the geckodriver.log
points to the following error:
JavaScript error: line 0: uncaught exception: out of memory
What I'm not understanding is that my computer has a lot of available memory (RAM), when I check with htop
only 40%
of memory is used.
I have two hypothesis:
As shown in the comments, this is a situation where the page is badly designed. Therefore, to avoid the out of memory
problem I installed the uBlock
extension on Firefox geckodriver, which resolved my problem as it blocked 25% of the page's content, mainly ads.
The implementation is explained in this stack overflow answer