Suppose there's a large LMDB file, Normally, I would get its contents sequentially using mdb_cursor_get(cursor, &key, &data, MDB_NEXT) through a cursor.
And now, To increase the read speed,I create a queue, dividing the LMDB into 10 equal parts and starting 10 threads to read each piece of content separately ,then put what each thread reads into the queue.
Is that OK? I don't care about order just care content.
If yes,how do I synchronize the cursor on each thread?
In my tests, multi-threading did not improve the efficiency of reading LMDB, but doubled (or more)the reading time.
Test method:
Generate an LMDB with 10000 pieces of data and record the complete read time: 365s
Divide the LMDB of this 10000 pieces of data into two equal parts and four equal parts, and read one equal part independently: only read one 5000_LMDB time: 170s only read one 2500_LMDB time: 81s
Start multiple threads to read the LMDB : Two threads, total time to read two 5000_LMDB : 596x2s Four threads, total time to read four 2500_LMDB : 301x4s