Scraping with requests-html on Raspberry Pi 4 fails with OSError: [Errno 8] Exec format error

Here's what I try to achieve: I have a Raspberry Pi 4 with a Pimoroni Inky Frame. I want to use it for a sponsoring event (Tour for Life) to show the recent amount of gifted money. This gifted amount is found on this page: https://www.tourforlife.nl/de-rollende-rijders

On my Macbook I've used Python code using requests-html to scrape the site and get the gifted-amount class. So scraping works.

Then I brought the code to my Raspberry Pi and then the problems began.

This is a snippet of my Python code:

import requests
from bs4 import BeautifulSoup
from requests_html import HTMLSession
from PIL import Image, ImageFont, ImageDraw

def get_donation_amount():
    """ Get the donation amount from the Tour for Life website

    Returns:
        float: The donation amount
    """
    url = "https://www.tourforlife.nl/de-rollende-rijders"

    session = HTMLSession()
    response = session.get(url)
    response.html.render()
    gift_amount = response.html.find('.gift-amount')[0].text
    # remove euro sign and convert to float
    gift_amount = float(gift_amount[2:].replace(",", "."))
    return gift_amount

And this was the output:

python3 pidonationtracker.py
[INFO] Starting Chromium download.
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 183M/183M [00:17<00:00, 10.4Mb/s]
[INFO] Beginning extraction
[INFO] Chromium extracted to: /home/inky3/.local/share/pyppeteer/local-chromium/1181205
Traceback (most recent call last):
  File "/home/inky3/PythonProjects/pidonationtracker/src/pidonationtracker/pidonationtracker.py", line 59, in <module>
    donation_amount = get_donation_amount()
  File "/home/inky3/PythonProjects/pidonationtracker/src/pidonationtracker/pidonationtracker.py", line 23, in get_donation_amount
    response.html.render()
  File "/home/inky3/PythonProjects/pidonationtracker/.venv/lib/python3.9/site-packages/requests_html.py", line 586, in render
    self.browser = self.session.browser  # Automatically create a event loop and browser
  File "/home/inky3/PythonProjects/pidonationtracker/.venv/lib/python3.9/site-packages/requests_html.py", line 730, in browser
    self._browser = self.loop.run_until_complete(super().browser)
  File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/home/inky3/PythonProjects/pidonationtracker/.venv/lib/python3.9/site-packages/requests_html.py", line 714, in browser
    self._browser = await pyppeteer.launch(ignoreHTTPSErrors=not(self.verify), headless=True, args=self.__browser_args)
  File "/home/inky3/PythonProjects/pidonationtracker/.venv/lib/python3.9/site-packages/pyppeteer/launcher.py", line 307, in launch
    return await Launcher(options, **kwargs).launch()
  File "/home/inky3/PythonProjects/pidonationtracker/.venv/lib/python3.9/site-packages/pyppeteer/launcher.py", line 148, in launch
    self.proc = subprocess.Popen(  # type: ignore
  File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.9/subprocess.py", line 1823, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: '/home/inky3/.local/share/pyppeteer/local-chromium/1181205/chrome-linux/chrome'

Now I've read some older posts (like: OSError when running puppeteer on raspberry pi) that say that the chromium browser is not the correct version for Raspberry Pi's ARM chipset. A lot of those posts are from 2019 and before, so I'm not sure if their solutions are still accurate.

I've tried installing the Chrome driver again from here: https://sites.google.com/chromium.org/driver/

Still the same error. Probably because Puppeteer is looking elsewhere.

Then I read this post: https://github.com/miyakogi/pyppeteer/issues/250 This says that I should install chromium-codecs-ffmpeg and chromium-browser (from https://launchpad.net/ubuntu/trusty/armhf/chromium-codecs-ffmpeg and https://launchpad.net/ubuntu/trusty/armhf/chromium-browser). This also is 2018 material. But I got to try something.

As soon as I tried installing these with these commands:

wget http://launchpadlibrarian.net/361669486/chromium-codecs-ffmpeg_65.0.3325.181-0ubuntu0.14.04.1_armhf.deb
wget http://launchpadlibrarian.net/361669485/chromium-browser_65.0.3325.181-0ubuntu0.14.04.1_armhf.deb

sudo dpkg -i chromium-codecs-ffmpeg_65.0.3325.181-0ubuntu0.14.04.1_armhf.deb

.. I ran into dependency problems:

Selecting previously unselected package chromium-codecs-ffmpeg.
dpkg: considering removing chromium-codecs-ffmpeg-extra in favour of chromium-codecs-ffmpeg ...
dpkg: no, cannot proceed with removal of chromium-codecs-ffmpeg-extra (--auto-deconfigure will help):
 rpi-chromium-mods depends on chromium-codecs-ffmpeg-extra
  chromium-codecs-ffmpeg-extra is to be removed.

dpkg: regarding chromium-codecs-ffmpeg_65.0.3325.181-0ubuntu0.14.04.1_armhf.deb containing chromium-codecs-ffmpeg:
 chromium-codecs-ffmpeg conflicts with chromium-codecs-ffmpeg-extra
  chromium-codecs-ffmpeg-extra (version 120.0.6099.102-rpt1) is present and installed.

dpkg: error processing archive chromium-codecs-ffmpeg_65.0.3325.181-0ubuntu0.14.04.1_armhf.deb (--install):
 conflicting packages - not installing chromium-codecs-ffmpeg
Errors were encountered while processing:
 chromium-codecs-ffmpeg_65.0.3325.181-0ubuntu0.14.04.1_armhf.deb

So here's a list of things I had to uninstall:

sudo dpkg --remove --force-remove-reinstreq rpi-chromium-mods
sudo dpkg --remove --force-remove-reinstreq chromium-chromedriver
sudo dpkg --remove --force-remove-reinstreq chromium-browser-l10n
sudo dpkg --remove --force-remove-reinstreq chromium-browser

But when I tried to uninstall chromium-browser I got a whole list of dependencies:

dpkg: dependency problems prevent removal of chromium-browser:
 libreoffice-help-en-us depends on firefox-esr | epiphany-browser | konqueror | chromium | firefox; however:
  Package firefox-esr is not installed.
  Package epiphany-browser is not installed.
  Package konqueror is not installed.
  Package chromium is not installed.
  Package chromium-browser which provides chromium is to be removed.
  Package firefox is not installed.
 libreoffice-help-en-gb depends on firefox-esr | epiphany-browser | konqueror | chromium | firefox; however:
  Package firefox-esr is not installed.
  Package epiphany-browser is not installed.
  Package konqueror is not installed.
  Package chromium is not installed.
  Package chromium-browser which provides chromium is to be removed.
  Package firefox is not installed.

Okay this is getting ridiculous. Do I really have to uninstall Firefox, Konqueror and a whole list of other things to be able to install an old version of chromium codec, to have Puppeteer take the correct version of Chromium. Or am I going the wrong direction.

I've also tried to use Selenium instead. I you want more details of my failures there I can provide them. They were also Chromium related.

Or is this even possible? It seems such a simple task. Just scrape a float from a webpage..

I've found the solution myself.

I tried to install the chromium codecs with this command:

sudo dpkg -i chromium-codecs-ffmpeg_65.0.3325.181-0ubuntu0.14.04.1_armhf.deb

But this really needs a force-all option. Like this:

wget https://launchpad.net/ubuntu/trusty/armhf/chromium-codecs-ffmpeg_65.0.3325.181-0ubuntu0.14.04.1_armhf.deb

sudo dpkg --force-all -i chromium-codecs-ffmpeg_65.0.3325.181-0ubuntu0.14.04.1_armhf.deb

wget https://launchpad.net/ubuntu/trusty/armhf/chromium-browser_65.0.3325.181-0ubuntu0.14.04.1_armhf.deb

sudo dpkg --force-all -i chromium-browser_65.0.3325.181-0ubuntu0.14.04.1_armhf.deb

So yes, all dependencies need to be updated to use older versions.

Then you're not done. You need to link the chrome browser that puppeteer uses to the correct ARM version of chromium.

So check where the chromium-browser is.

whereis chromium-browser

Results might look like this:

chromium-browser: /usr/bin/chromium-browser /usr/lib/chromium-browser /etc/chromium-browser /usr/share/chromium-browser /usr/share/man/man1/chromium-browser.1.gz

Now go to that path found in the OSError: [Errno 8] error:

cd /home/inky3/.local/share/pyppeteer/local-chromium/1181205/chrome-linux/

And link to the chromium-browser:

ln -s /usr/bin/chromium-browser chrome

Suddenly my scraping command works.

After solving a Inky - numpy related issue, there it is: