pythonselenium-webdriverfirefoxout-of-memorygeckodriver

Selenium script using Firefox `geckodriver` is running out of memory when there still available


I'm running a selenium script in Python with the Firefox browser, where the task is to scroll down until the end of the page. The process is composed by scroll down the page, click a button at the bottom which will trigger the page to load more content. This steps should repeat until there's no button anymore.

It's a slow process that takes several hours to complete and, although this script worked in some other pages, for a longer page, I'm stuck in a problem where the geckodriver.log points to the following error:

JavaScript error: line 0: uncaught exception: out of memory

What I'm not understanding is that my computer has a lot of available memory (RAM), when I check with htop only 40% of memory is used.

I have two hypothesis:

  1. The memory allocated to the firefox browser is limited, if this is true, how could I increase the amount of memory?
  2. My script is really exploding my memory, and the system kills some parts when this occurs and, when i go check the memory usage, its had been free, tricking me out.

Solution

  • As shown in the comments, this is a situation where the page is badly designed. Therefore, to avoid the out of memory problem I installed the uBlock extension on Firefox geckodriver, which resolved my problem as it blocked 25% of the page's content, mainly ads.

    The implementation is explained in this stack overflow answer