I configured a Python script (called test_fire7.py), which uses Selenium and Firefox to be executed by cron in a Python virtual environment called e1. The Python script runs correctly, if executed manually from the console. It runs correctly both with the virtual environment activated as well without activating the environment explicitly, but using a command like this: /root/pye/e1/bin/python3 /root/pye/test_fire7.py
Here is the entry in crontab:
30 20 * * * cd /root/pye && /root/pye/e1/bin/python3 /root/pye/test_fire7.py command arg 2>&1 | logger -t mycmd
And this is the header of test_fire7.py (I omit the rest of the script, because it runs correctly by itself):
from selenium import webdriver
from selenium.webdriver import FirefoxOptions
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
opts = FirefoxOptions()
opts.add_argument("--headless")
driver = webdriver.Firefox(options=opts)
And here is the error, after cron runs the script, captured in syslog:
root@wi-master:~/pye# grep 'mycmd' /var/log/syslog
May 27 20:30:01 wi-master mycmd: Traceback (most recent call last):
May 27 20:30:01 wi-master mycmd: File "/root/pye/test_fire7.py", line 15, in <module>
May 27 20:30:01 wi-master mycmd: driver = webdriver.Firefox(options=opts)
May 27 20:30:01 wi-master mycmd: File "/root/pye/e1/lib/python3.8/site-packages/selenium/webdriver/firefox/webdriver.py", line 201, in __init__
May 27 20:30:01 wi-master mycmd: super().__init__(command_executor=executor, options=options, keep_alive=True)
May 27 20:30:01 wi-master mycmd: File "/root/pye/e1/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 286, in __init__
May 27 20:30:01 wi-master mycmd: self.start_session(capabilities, browser_profile)
May 27 20:30:01 wi-master mycmd: File "/root/pye/e1/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 378, in start_session
May 27 20:30:01 wi-master mycmd: response = self.execute(Command.NEW_SESSION, parameters)
May 27 20:30:01 wi-master mycmd: File "/root/pye/e1/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 440, in execute
May 27 20:30:01 wi-master mycmd: self.error_handler.check_response(response)
May 27 20:30:01 wi-master mycmd: File "/root/pye/e1/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 245, in check_response
May 27 20:30:01 wi-master mycmd: raise exception_class(message, screen, stacktrace)
May 27 20:30:01 wi-master mycmd: selenium.common.exceptions.SessionNotCreatedException: Message: Expected browser binary location, but unable to find binary in default location, no 'moz:firefoxOptions.binary' capability provided, and no binary flag set on the command line
Other scripts run correctly using exactly the same kind of entries in crontab. So I assume the problem is with Selenium in some way.
I have Firefox installed:
root@wi-master:~/pye# firefox -v
Mozilla Firefox 113.0.2
And also geckodriver:
root@wi-master:~/pye# geckodriver --version
geckodriver 0.33.0 ( 2023-05-22)
Again, the script passes without errors, if executed manually. The error message in syslog given above suggests, that Firefox is not installed. But this is not the case!
Update: I tried adding the executable path of geckodriver when calling the webdriver, but with no success. I'm not sure if I locate the executable path correctly
root@wi-master:~# which geckodriver
/snap/bin/geckodriver
I also passed the Firefox binary explicitely like this, but also without success.
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
driver=webdriver.Firefox(firefox_binary=FirefoxBinary(getoutput("find /snap/firefox -name firefox").split("\n")[-1]), options=opts)
With this the Selenium error changes to
selenium.common.exceptions.WebDriverException: Message: Service /snap/firefox/2667/usr/lib/firefox/firefox unexpectedly exited. Status code was: 255
I finally got a solution. It happend through very much trial and error after all suggestions I found on the internet didn't work for me.
For completeness, here is my system:
I use virtual environments and run the scripts from root.
First, I observed where Firefox and geckodriver reside:
root@wikijs-master:~# which firefox
/snap/bin/firefox
root@wikijs-master:~# which geckodriver
/snap/bin/geckodriver
Based on that I added export PATH=$PATH:/snap/bin/; to the crontab entry from above:
* * * * * export PATH=$PATH:/snap/bin/; cd /root/pye && /root/pye/e1/bin/python3 /root/pye/test_fire7.py command arg 2>&1 | logger -t mycmd
And here is a sample python script that cron can execute!
from selenium import webdriver
from selenium.webdriver import FirefoxOptions
opts = FirefoxOptions()
opts.add_argument("--headless")
driver=webdriver.Firefox(options=opts)
driver.get("https://google.com")
print(driver.current_url)
I want to note that other solutions like adding executable paths, absolute paths or binary locations within the webdriver attributes, changing the working directory or modifying the permissions for the cron user didn't work for me.