pythonmultithreadingthread-safetyrestrictedpython

Restrictedpython code evaluation executed in thread pool - function invoked twice


i'm using restrictedpython in my app in order to allow the "mid-skilled-user" to develop some pieces of additional code to execute as modular on-the-fly add-on.

For some architectural constraints, i have to use a thread pool in order to parallelize the code execution.

Those threads are created by group of 10, then started and wait to be join on the main execution process.

I'm experiencing some weird issues...sometimes, i see that the restrictedpython thread is runned twice!...i've looked around and i'm a bit surprised that no-one was talking about that, i don't know if there is something wrong in my code or if the restrictedpython is not thread safe.

Anyway, you can see that in my RestrictedPythonThread i'm also implementing Mutex lock after compiling the restrictedpython code and then releasing the Mutex lock before i have the compiled code.

Then, the compiled code is executed without any lock using a simple eval.

Here is my code:

import logging
import threading
from threading import Thread, Lock

from RestrictedPython import compile_restricted
from RestrictedPython.Eval import default_guarded_getiter, default_guarded_getitem
import os
import psycopg2


def _hook_getattr(obj, attr):
    if obj is os:
        raise RuntimeError('Restricted, cannot use os')
    if obj is psycopg2:
        raise RuntimeError('Restricted, Cannot use psycopg2')
    return getattr(obj, attr)


def _hook_writable(obj):
    if obj.__class__.__name__ == "MY_CLASS_NAME":
        return obj
    if isinstance(obj, dict):
        return obj
    if isinstance(obj, list):
        return obj
    raise RuntimeError('Restricted, Cannot write outside restricted defined class/objects')


def _get_compiled_restricted_python(src):
    return compile_restricted(src, '<string>', 'exec')


_restricted_python_globals = {
    '__metaclass__': type,
    '_getattr_': _hook_getattr,
    '_getiter_': default_guarded_getiter,
    '_write_': _hook_writable,
    '_getitem_': default_guarded_getitem,
    'smart_device': None
}

class RestrictedPythonThread(Thread):
    '''
    A Custom Thread inherited class that manages the restricted python execution using mutex mechanism
    '''

    def __init__(self, dev):
        Thread.__init__(self)
        self.dev = dev

    def run(self):
        global mutex
        mutex.acquire()

        code = _get_compiled_restricted_python(
            self.dev.device_epm.epm)
        code_globals = _restricted_python_globals
        code_globals['dev'] = self.dev
        mutex.release()

        exec(code, code_globals)

Solution

  • After digging around i've find that using python redis queue the issue went away...so this was definively an issue related to the shared memory space of threads.

    Maybe using built-in multiprocess module can be enough, but i haven't tested it because in my application i have already python-rq in use.