parsing sdf file, performance issue

I've wrtien a script which read different files and search molecular ID in big sdf databases (about 4.0 GB each).

the idea of this script is to copy every molecules from a list of id (about 287212 molecules) from my original databases to a new one in a way to have only one single copy of each molecule (in this case, the first copy encountered)

I've writen this script:

import re
import sys
import os

def sdf_grep (molname,files):
    filin = open(files, 'r')
    filine= filin.readlines()
    for i in range(0,len(filine)):
        if filine[i][0:-1] == molname and filine[i][0:-1] not in past_mol:
            past_mol.append(filine[i][0:-1])
            iterate = 1
            while iterate == 1:
                if filine[i] == "$$$$\n":
                    filout.write(filine[i])
                    iterate = 0
                    break
                else:
                    filout.write(filine[i])
                i = i+1
        else:
            continue
    filin.close()

mol_dock = os.listdir("test")
listmol = []
past_mol = []
imp_listmol = open("consensus_sorted_surflex.txt", 'r')
filout = open('test_ini.sdf','wa')

for line in imp_listmol:
    listmol.append(line.split('\t')[0])
print 'list ready... reading files'
imp_listmol.close()

for f in mol_dock:
    print 'reading '+f
    for molecule in listmol:
        if molecule not in past_mol:
            sdf_grep(molecule , 'test/'+f) 

print len(past_mol)
filout.close()

it works perfectely, but it is very slow... too slow for what I need. Is there a way to rewrite this script in a way that can reduce the computation time?

thank you very much.

The main problem is that you have three nested loops: molecular documents, molecules and file parsing in the inner loop. That smells like trouble - I mean, quadratic complexity. You should move huge files parsing outside of inner loop and use set or dictionary for molecules. Something like this:

1. For each sdf file
2. For each line, if it is molecule definition
3. Check in dictionary of unfound molecules
4. If present, process it and remove from dictionary of unfound molecules

This way, you will parse each sdf file exactly once, and with each found molecule, speed will further increase.