[SOLVED] Why are random tables created?

Why are random tables created?

I’m using the latest version with he Go client on an AWS EC2 instance. I see more and more random tables created based on what looks like some of the payload/column values. This is happening random.

Here is the Go code or the main thread:

ctx = context.TODO()
qb, err = qdb.LineSenderFromConf(ctx, CONNECTION_STRING)
if err != nil {
    panic("Failed to create QuestDB client")
}

In another thread is this code:

err = qb.Table(QUESTDB_TABLE).
    Symbol("Symbol", delta.Data.Symbol).
    Float64Column("lastPrice", lastPrice).
    At(ctx, t)

if err != nil {
    logger.Error("Failed to insert data")
}

err = qb.Flush(ctx)
if err != nil {
    logger.Error("Failed to flush data")
}

And in another thread tis this code:

err = qb.Table(QUESTDB_TABLE).
    Symbol("Ticker", msg.Ticker).
    Symbol("Exchange", msg.Exchange).
    Symbol("Indicator", msg.Indicator).
    Int64Column("Interval", msg.Interval).
    Float64Column("Close", msg.Close).
    Float64Column("Open", msg.Open).
    Float64Column("High", msg.High).
    Float64Column("Low", msg.Low).
    StringColumn("Message", msg.Message).
    Float64Column("VWAP", msg.VWAP).
    Float64Column("MoneyFlow", msg.MoneyFlow).
    Float64Column("BlueWave", msg.BlueWave).
    At(ctx, msg.Timenow)

if err != nil {
    fmt.Println("Failed to insert data")
}

err = qb.Flush(ctx)
if err != nil {
    fmt.Println("Failed to flush data")
}

The constant are shared with these values:

const QUESTDB_TABLE = "bybit_all"
const CONNECTION_STRING = "http::addr=localhost:9000;username=admin;password=quest;"

As you can see, the table name is defined as a const. Why would QuestDB create new, random tables?

Solution

I found a similar issue a while ago using threads on Python. The QuestDB client libraries are not thread-safe, so when you concurrently insert from different threads weird things might happen. In my case I saw a few tables also with the wrong name, and eventually after a while the client would crash.

I tried then using locks, but it was tricky and performance was not great, so I eventually ended up using Python multiprocessing to make sure each client was independent.

I am not sure what would be the Go equivalent, but hopefully this will help solve the issue.