I am finding that with monetdbe (embedded, Python), I can import data to two tables simultaneously from two processes, but I can't do two SELECT queries.
For example, if I run this in Bash:
(python stdinquery.py < sql_examples/wind.sql &); (python stdinquery.py < sql_examples/first_event_by_day.sql &)
then I get this error from one process, while the other finishes its query fine:
monetdbe.exceptions.OperationalError: Failed to open database: MALException:monetdbe.monetdbe_startup:GDKinit() failed (code -2)
I'm a little surprised that it can write two tables at once but not read two tables at once. Am I overlooking something?
My stdinquery.py is just:
import sys
import monetdbe
monet_conn = monetdbe.connect("dw.db")
cursor = monet_conn.cursor()
query = sys.stdin.read()
cursor.executescript(query)
print(cursor.fetchdf())
You are starting multiple concurrent Python processes. Each of those tries to create or open a database on disk at the dw.db location. That won't work because the embedded database processes are not aware of each others.
With the core C library of monetdbe, it is possible to write multi-threaded applications where each connecting application thread uses its own connection object. See this example written in C here. Again this only works for concurrent threads within a single monetdbe process, not multiple concurrent monetdbe processes claiming the same database location.
Unfortunately it is not currently possible with the Python monetdbe module to setup something analogous to the C example above. But probably in the next release
it is going to be possible to use e.g. concurrent.futures to write something similar in Python.