pythonpython-3.xpython-zipfile

ZipFile thread-safe across multiple output files?


It's well covered that writing multiple source files to a single target is not thread safe i.e M:1

reference question

Is python zipfile thread-safe?

Does this same limitation hold M:M?

file_1.txt -> file_1.zip
...
file_m.txt -> file_m.zip

psuedo code

orig_to_zip_name = [
  ['file1.txt','zipped_file_1.zip'],
  ...
  ['filem.txt','zipped_file_m.zip']
]

def single_zip(file_pairs):
  zipped_name, original_name = file_pairs
  with ZipFile(zipped_name, 'w') as zipf:
    zipf.write(original_name,compress_type = ZIP_DEFLATED)

with concurrent.futures.ThreadPoolExecutor() as executor:
  results = executor.map(single_zip, orig_to_zip_name )  

Solution

  • It's perfectly safe. None of the threads are contending on the same files, the same data, or any of it, and ZipFile is not some hack put together by amateurs that has global state that would cause issues simply from the process of compressing different data to different files in parallel.

    Your own link to the other question already told you that, in the top answer.