pythonschedulingpython-importapschedulerimp

How to integrate APScheduler and Imp?


I have built a plugin-based application where "plugins" (python modules) can be loaded by imp and then scheduled for later execution by APScheduler, I was able to successfully integrate them but I want to implement persistence in case of crashes or application reestarts, so I changed the default memory job store to the SqlAlchemyJobStore, it works quite well the first time you execute the program: tasks are loaded, scheduled, saved at the database and executed at the right time.

Problem is when I try to load the application again I get this traceback:

ERROR:apscheduler.jobstores.default:Unable to restore job "d3e0f0068df54d15986e9b7b6757f665" -- removing it
Traceback (most recent call last):
  File "/home/jesus/.local/lib/python2.7/site-packages/apscheduler/jobstores/sqlalchemy.py", line 126, in _get_jobs
jobs.append(self._reconstitute_job(row.job_state))
  File "/home/jesus/.local/lib/python2.7/site-packages/apscheduler/jobstores/sqlalchemy.py", line 114, in _reconstitute_job
job.__setstate__(job_state)
File "/home/jesus/.local/lib/python2.7/site-packages/apscheduler/job.py", line 228, in __setstate__
self.func = ref_to_obj(self.func_ref)
  File "/home/jesus/.local/lib/python2.7/site-packages/apscheduler/util.py", line 257, in ref_to_obj
raise LookupError('Error resolving reference %s: could not import module' % ref)
LookupError: Error resolving reference __init__:run: could not import module

So it is obvious that there is a problem when attempting to import the function again

Here is my scheduler initialization:

    executors = {'default': ThreadPoolExecutor(5)}
    jobstores = {'default': SQLAlchemyJobStore(url='sqlite:///jobs.sqlite')}
    self.scheduler = BackgroundScheduler(executors = executors,jobstores=jobstores)

I have a "tests" dictionary containing the "plugins" that should be loaded and some parameters, "load_plugin" uses imp to load a plugin by it's name.

for test,parameters in tests.items():
        if test in pluggins:
            module=load_plugin(pluggins[test])
            self.jobs[test]=self.scheduler.add_job(module.run,"interval",seconds=parameters["interval"],name=test)

Any idea about how can I handle reconstituting jobs?


Solution

  • Something in the automatic detection of the module name is going wrong. Hard to say what, but the alternative is to manually give it the proper lookup path as a string (e.g. "package.module:function"). If you can do this, you can avoid this problem.