inserting data into Couchbase lite using cblite, one document per line in JSON file - Kotlin

I'm working on Couchbase lite, and I want to prepare a pre built database from a json file. Each time I import the file which has an array of 12 keys, that has 100 line, each line is a single array.

Cblite imports all the file into a single document, which won't allow me to use the content of the document based on the keys, because all the 12 keys are duplicated 100 times. But I found out that i can import the single file into several documents (each line/array) and become a separate document.

But i’m not able to do that with the commands provided in cblite, I got error from cblite including (Can't parse json)

DESTINATION : Database path, replication URL, or JSON file path:
    *.cblite2 :  Copies local database file, and assigns new UUID to target
    *.cblite2 :  With --replicate flag, runs local replication [EE]
    ws://*    :  Networked replication
    wss://*   :  Networked replication, with TLS
    *.json    :  Imports/exports JSON file (one doc per line)
    */        :  Imports/exports directory of JSON files (one per doc)

I can rearrange the file of json anyway that fits… but I can’t create a single file for each array.

The arrays look like that

{"_id":13,"id":"51333591918","owner":"126542062@N08","secret":"16f1758279","server":"65535","farm":66,"title":"dreamland","ispublic":1,"isfriend":0,"isfamily":0,"url_s":"https://live.staticflickr.com/65535/51333591918_16f1758279_m.jpg","height_s":148,"width_s":240}
{"_id":14,"id":"51333156717","owner":"97978797@N03","secret":"82f0fb71aa","server":"65535","farm":66,"title":"Tagpfauenauge","ispublic":1,"isfriend":0,"isfamily":0,"url_s":"https://live.staticflickr.com/65535/51333156717_82f0fb71aa_m.jpg","height_s":160,"width_s":240}

cblite -tool version: cblite tool 2.8 EE
Couchbase lite version : 2.8.6
Error: NullPointerException

I tried several files. one JSON which is public file ( i used it as a test also) is the flickr json response Flickr Photos Json File

also I used the files of the company which are items for POS.

i tried to change the structure of the json files manually even. but all leading to the same result... (putting all the arrays in a single document)

in the db query i used this code

 val query = QueryBuilder
            .select(
                SelectResult.expression(Meta.id),
                SelectResult.property("owner"),
                SelectResult.property("secret"),
                SelectResult.property("server"),
                SelectResult.property("farm"),
                SelectResult.property("title"),
                SelectResult.property("ispublic"),
                SelectResult.property("isfriend"),
                SelectResult.property("isfamily"),
                SelectResult.property("url_s"),
                SelectResult.property("height_s"),
                SelectResult.property("width_s"),
            )
            .from(DataSource.database(database))

        val result = query.execute().allResults()

        for (rs in result) {
           // val doc = database.getDocument(photo.id)
            val id = rs.getString("id")
            val owner = rs.getString("owner")
            val secret = rs.getString("secret")
            val server = rs.getString("server")
            val farm = rs.getInt("farm")
            val title = rs.getString("title")
            val ispublic = rs.getInt("ispublic")
            val isfriend = rs.getInt("isfriend")
            val isfamily = rs.getInt("isfamily")
            val url_s = rs.getString("url_s")
            val height_s = rs.getInt("height_s")
            val width_s = rs.getInt("width_s")

            photoList.add(
                photo(
                    id!!,
                    owner!!,
                    secret!!,
                    server!!,
                    farm,
                    title!!,
                    ispublic,
                    isfriend,
                    isfamily,
                    url_s!!,
                    height_s,
                    width_s
                )
            )

i noticed in Kotlin the option of SelectResult.all() doesn't bring anything, i have to specify the keys one by one to be fetched. but that is not an issue.

i often managed to print the whole document in the console, but i will never be able to use any key within. always getting the NullPointer as an error.

I'm still learning on couchbase and i looking to learn about N1QL, i rarely find something for Kotlin unfortunately

Any help?

Solution

Please see the example repo in GitHub: https://github.com/biozal/cbmobile-split-data-example

I made the assumption that all records in the photo array that is a child object of the photos object is what you wanted to save to the database and that each object you wanted to be a new document.

Given this I wrote a very simple shell script (using "bash") that does the following:

get the count of how many items are in the document
loop through the number of items I need to create documents for
used the open source jq command (https://github.com/stedolan/jq) to get the content and save it into a variable named json
each document needs a unique key so I used the id field for this and pull it from the json
finally I call the cblite tool and have it create the document based on the id I pulled for the item and the json value of it

A screenshot of this query is provided named query-database.png. To query I did the following:

./cblite sampleDb.cblite2
 query --limit 10 ["=", [".owner"], "138459774@N07"]

That query will return the document that the value is in.