I'm making an app that grabs an user's comments from a reddit API and loads them in a local database. I'm using NeDB for the database. The reddit API returns an array of 25 elements(comment) and then you have to paginate using the date of the last element to grab the next array of elements. I map every array and insert each element individually. I don't want to insert the whole array because then the elemens get indexed and I would be left with a database full of different indexes. I'm usually grabbing a few hundred arrays. I'm also using nedb-promises.
A whole array insert goes from an average speed of 3 seconds to 10 seconds
The thing is, the array inserts take too long, and the process gets slower at the end. Is there something wrong with my code? Would using SQLite be much faster? Thanks
const asyncForEach = async function (array, callback) {
for (let index = 0; index < array.length; index++) {
await callback(array[index], index, array);
}
}
const fetch = async (paginationHelper) => {
//response tree: response.data.data[n].body
const {data:{data}} = await pushshift.get(`?before=${paginationHelper !==null ? paginationHelper : ''}&author=${author}`)
return data.map(comment => {
return ({
body: comment.body,
created: comment.created_utc,
subreddit: comment.subreddit,
author: comment.author,
postId: comment.link_id.split('t3_')[1],
parentId: comment.parent_id.split('t1_')[1],
linkId: comment.id
})
})
}
const load = async (comments) => {
let i = 0
await asyncForEach(comments, async comment => {
await db.insert(comment).then(res => {
console.log(res);
i++
console.log(i);
})
})
}
const loadAll = async () => {
let comments = await fetch(null)
//Pagination is made using the comments' date when they were created
let paginationHelper = comments[comments.length-1].created
while(paginationHelper){
[comments] = await Promise.all( [fetch(paginationHelper), load(comments)] )
paginationHelper = comments.length !== 0 ? comments[comments.length-1].created : null
console.log(paginationHelper);
}
}
I'm an idiot, you can just insert the whole array without any issues and its crazy fast.