bashsqliteshellpipe

Pipe string, file contents, string into SQLite without an intermediate file?


I've got this shell script which takes a set of INSERT statements and puts a 'BEGIN TRANSACTION' on the start and a 'COMMIT' on the end. There are a lot of INSERT statements and so this makes the insertion a lot faster.

The script I've used successfully looks like this:

file_name_improved_sql=foo-improved.sql
file_name_original_sql=foo.sql
file_name_sqlite_db=bar.db
path_to_insert_error=/tmp/fooerrors.log
#
echo 'BEGIN TRANSACTION;' | cat - $file_name_original_sql  > $file_name_improved_sql
echo "COMMIT;" >> $file_name_improved_sql
#
cat $file_name_improved_sql | sqlite3 $file_name_sqlite_db 2> $path_to_insert_error

I'd like to improve this so that instead of producing the intermediate file file_name_improved_sql, the two strings and the contents of the file are piped in directly.

Something like this (except this doesn't work)...

export string1="BEGIN TRANSACTION"
export string2="COMMIT"
cat $string1 $file_name_improved_sql $string2 | sqlite3 $file_name_sqlite_db 2> $path_to_insert_error

Solution

  • Either pipe a group of commands to sqlite3:

    {
      printf %s\\n 'BEGIN TRANSACTION;'
      cat "$file_name_original_sql"
      printf %s\\n 'COMMIT;'
    } | sqlite3 "$file_name_sqlite_db" 2> "$path_to_insert_error"
    

    Either use a here-document as input to sqlite3:

    sqlite3 "$file_name_sqlite_db" 2> "$path_to_insert_error" <<EOF
    BEGIN TRANSACTION
    $(cat "$file_name_original_sql")
    COMMIT;
    EOF
    

    Or use printf to combine the elements piped to sqlite3:

    printf 'BEGIN TRANSACTION;\n%sCOMMIT;\n' "$(cat "$file_name_original_sql")" |
      sqlite3 "$file_name_sqlite_db" 2> "$path_to_insert_error"
    

    Or create a function to frame an SQL input stream into a transaction:

    sql_transaction() {
      printf %s\\n 'BEGIN TRANSACTION;'
      cat
      printf %s\\n 'COMMIT;'
    }
    
    sql_transaction <"$file_name_original_sql" |
      sqlite3 "$file_name_sqlite_db" 2> "$path_to_insert_error"