node.jsmultithreadingsqliteelectronbetter-sqlite3

Electron: Perform sqlite (better-sqlite) db operations in another thread


I'm developing a desktop application using Electron framework and I've to use sqlite database for app data. I decided to use better-sqlite3 because of:

  1. Custom SQL function support (It's very important for me)
  2. It's much faster than node-sqlite3 in most cases
  3. It is simple to use.
  4. It's synchronous API (in most cases I need to get data serialized)

but in some cases, when I perform a query that takes a while to response, the application UI won't responses to user until the query ends.

how can I run some db queries in another thread? or run them asyncronized (like node-sqlite3)?

sorry for bad english


Solution

  • Node allows you a separate process out-of-the-box. ( Threads are a different matter - alas no WebWorkers :( though you can prob find a thread add-on lib somwhere.

    EDIT: Node has added worker_threads since I originally posted this answer. Haven't tried it yet / dunno if they work with better-sqlite.END EDIT

    I've had the same issue as you - needing synchronous code to run without blocking the main thread and I used a child process. It was for better-sqlite too !

    Problem is that how to handle io streams and sigints etc for control is not immediately obvious and differs depending on whether you're running on windows or posix.

    I use a forked child process with silent option set to true to do the synchronous db work.

    If you need control of that process or progress update reports back to your main process for your gui during sync ops ; I control/communicate with the child process by reading/writing on the child process stdin/out using fileSystem writeFileSync / readFileSync at various points in my child process code ( you can't use the normal inter-process comms api during sync ops as that's event driven and can't operate while synchronous code is running. Though you can mix and match the two types of io)

    example of forked child process ;

    //parent.js and child.js in same folder
    
    //parent.js
    process.on('exit', (code) => {
      console.log(`Parent to exit with code: ${code}`);
    });
    
    const readLine = require("readline") ;
    const cp = require('child_process');
    
    var forkOptions = {
        //execArgv:['--inspect-brk'], // uncomment if debugging the child process
        silent:true // child gets own std pipes (important) whch are piped to parent
    };
    var childOptions = [] ;
    const child = cp.fork(`./child.js`,childOptions,forkOptions);
    
    
    //for messages sent from child via writeSync
    const childChannel = readLine.createInterface({
        input: child.stdout
    }).on("line",function(input){
        console.log("writeSync message received from child: " + input) ;
    });
    
    //for messages sent from child via process.send
    child.on('message', (m) => { 
        console.log("process.send message received from child: " + m) ;
    });
    
    
    // Child.js
    process.on('exit', (code) => {
      console.log(`Child to exit with code: ${code}`);
    });
    
    const fs = require('fs');
    
    function doSyncStuff(){    
        for(let i = 0 ; i < 20 ; i++){
            //eg. sync db calls happening here
            process.send(`Hello via process.send from child. i = ${i} \n`); // async commms . picked up by parent's "child.on" event
            fs.writeFileSync(process.stdout.fd,`Hello via writeFileSync from child. i = ${i} \n`) ; // sync comms. picked up by parent's readLine listener ("process" here is the child )           
        }
    }
    
    doSyncStuff();