pythonsocketsmultiprocessingimaplib

using imaplib.IMAP4_SSL in multiple processes


I want to reuse imaplib.IMAP4_SSL instance over many processes so I don't have to login multiple times. Here is some code:

import imaplib
from multiprocessing import Process

def fetch(mail_client):
    mail_client.uid('fetch', b'1', 'BODY[TEXT]')

def main():
    c = imaplib.IMAP4_SSL('imap.gmail.com')
    c.login(user='**', password='***')
    c.select('inbox')

    procs = [Process(target=fetch, args=(c,)) for _ in range(100)]
    for p in procs:
        p.start()

    for p in procs:
        p.join()

if __name__ == '__main__':
    main()

But I get the errors that are related to socket:

imaplib.IMAP4.abort: socket error: [Errno 32] Broken pipe

I thought that this is because processes is writing to the same socket, that imaplib.IMAP4_SSL has, so i tried to add multiprocessing.Lock to prevent simultaneous access:

import imaplib
from multiprocessing import Process, Lock


def fetch(mail_client, lock):
    with lock:
        mail_client.uid('fetch', b'1', 'BODY[TEXT]')


def main():
    c = imaplib.IMAP4_SSL('imap.gmail.com')
    c.login(user='engineering@epallet.com', password='Qwe=1dSAzxc+%')
    c.select('inbox')
    lock = Lock()

    procs = [Process(target=fetch, args=(c, lock)) for _ in range(100)]
    for p in procs:
        p.start()

    for p in procs:
        p.join()


if __name__ == '__main__':
    main()

But the error persists. Some further investigation has shown that the first process calls mail.uid successfully, but second process gets imaplib.IMAP4.abort: command: UID => socket error: EOF anyway.

I am using Ubuntu 16.04. Any suggestions are highly appreciated.

Update: found another exception in stacktrace, maybe it causes all other exceptions:

ssl.SSLError: [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2217)`

Seems like this issue is related Python ssl problem with multiprocessing


Solution

  • You can't. The SSL context isn't shared between processes; without it, the encryption state gets out of sync.

    If you need to perform multiple concurrent operations on the same IMAP inbox, you will need to make multiple connections to the server.