I am using lmdb python to load the data
def create_dataset():
img_db_fn = 'data/image_train.lmdb'
img_env = lmdb.Environment(img_db_fn, map_size=1099511627776)
img_txn = img_env.begin(write=True, buffers=True)
keys = np.arange(100000)
np.random.shuffle(keys)
img_fns = glob.glob('data/positive/*.jpg')
img_fns += glob.glob('data/negtive/*.jpg')
print len(img_fns) , len(jnt_fns)
for i, img_fn in enumerate( img_fns ):
img_datum = get_img_datum(img_fn)
key = '%06d' % keys[i]
img_txn.put(key, img_datum.SerializeToString())
if i % 10000 == 0:
print 'commit',i
img_txn.commit()
img_txn = img_env.begin(write=True, buffers=True)
img_txn.commit()
img_env.close()
I got a error saying 'img_env = lmdb.Environment(img_db_fn, map_size=1099511627776) AttributeError: 'module' object has no attribute 'Environment''
In case anyone like me find this article on top of your Google Search,
If you simply call
pip install lmdb
on Linux, it wont actually install lmdb
correctly.
To install it, you will also need to install some package:
apt-get install libffi-dev python-dev build-essential
Document on LMDB website: https://lmdb.readthedocs.io/en/release/#installation-unix