javascriptnode.jsexpressmongoosebulk

Mongoose bulk insert misses critical information


image

The picture above shows an example of a request getting sent to the following route:

image

and the following picture shows what have been inserted to the database in compass (notice how there are three entries):

image

As we know, Model.create() accepts an array of objects, or an object.

In this example, I am sending an array of objects, to insert them.

Model.create([]) will insert the documents one by one to the database, it doesn't skip the validation part, which is why I chose it. and when it finds a document with a validation error, it skips it, and moves to the next one. until it finishes, then it reports the errors it encounters.

That's what it should be, However it's not exactly working like that.

Note that I have two documents which holds validation errors:

image

However, mongoose is only reporting the first one, it's not reporting the second one, even though it passes by it, and it sees it.

Why this information is critical?

Because on the client side, I have to know which documents got inserted, and which did not.

In this case, (when I will know which are the ones got inserted and the ones that did not), I can show the client for example that the documents x, y, z has been inserted, while the documents f, g, h did not. So the user can correct his mistake and send the request again.

The current error report is useless, because it only tells "there was a validation error", but it doesn't tell you the "where"

The error report should include all the documents which refused to be written to the database in an array.

Update

I realized that

const data = await User.insertMany(req.body)

Has exactly the same behavior. It doesn't only apply to Model.create(). Model.insertMany() has the same problem as well.

How to make mongoose report the full errors?


since we don't have a fold code option yet, I will include the code shown in the images, down here. and I hope this isn't going to polute the question.

mongoose.connect('mongodb://localhost:27017/temp', (err) => {
  if (err) return log.error(log.label, 'an error occured while connecting to DB!')
  log.success(log.label, 'Successfully connected to the database!')
})

app.use(express.json())

app.post('/users', async (req, res, next) => {
  try {
    const data = await User.collection.insertMany(req.body)
    res.json({
      message: 'success!',
      data,
    })
  } catch (error) {
    console.log(error)
    res.json({
      message: 'an error',
      data: error,
    })
  }
})

Solution

  • I checked the source code of create() method. It indeed only saves the first error and not all the errors.

    However, since the create() method will send one request for each item anyway, you can implement your own logic where you will wrap all the items with Promise.allSettled() and use the create() method for each item. That way, you will know exactly which item was successfully added, and which threw an error:

    app.post('/users', async (req, res, next) => {
      try {
        const items = req.body;
    
        const results = await Promise.allSettled(
          items.map((item) => User.create(item))
        );
    
        constole.log(results.map((result) => result.status);
        // Items that were successfully created will have the status "fulfilled", 
        // and items that were not successfully created will have the status "rejected".
       
        return res.status(200).json({ message: 'success', results })
      } catch (error) {
        console.log(error)
        return res.status(400).json({ message: 'an error', data: error })
      }
    })