ruby-on-railsmongodb

Lost cursor error from MongoDB aggregate statement. Why? Rails 5.2, Mongoid 6.4


I have been running a Rails 5.2 project for years without problems. In one place, it uses several multi-stage aggregations. Some of these have now started giving me a "[43:CursorNotFound]: cursor id 283353284060432170 not found (on db:27017)" error. I can post one of the aggregations:

Myclass.collection.aggregate([ {'$match' => {'meta_info.store_id' => store_id_string }},
                               {'$match' => {'meta_info.invalid' => {'$in': [nil, false]} }} ,
                               {'$match' => {'test_participation' => false }} ,
                               {'$match' => {'exit_status' => 'finished' }} ,
                               datehash,
                               {'$unwind' => '$answers'},
                               {'$match' => {'answers.question_id' => q.id.to_i }},
                               {'$project' => {'answers.data.value' => 1}},
                               {'$group' => { '_id' => 0 , 'data' => { '$push' => '$answers.data.value'  }  } }
                              ]).to_a.first

where datehash is yet another match statement that is calculated dynamically. I assign the result to a variable and then process the contents of that variable. My database content has been growing for years. I know nothing about cursors, unfortunately. So I have two questions:

  1. Is it the aggregation itself that takes too long, or is my program too slow in accessing the result?
  2. It is a very complex aggregation and it makes sense it takes so long. How can I specify "no timeout" (or at least, a timeout longer than the default 10 secs)? I have read in various docs that it should be possible to supply MongoDB with an argument to that effect, but I haven't been able to find out how.

I'd also appreciate if you could give me any suggestions about how to streamline the aggregation.

(The original reason my aggregation is so complex is that it is still much, much faster than fetching the raw data and then doing the calculations in Rails.)


Solution

  • It has turned out that the real problem was elsewhere. The cursor lost exception was caused by another process running in parallel which also used the database heavily and did complex computations with the data. When I had changed the cron time to avoid overlaps the problem went away.

    I have still optimised my aggregation as aneroid suggested (combine all the match stages into one, add an _id => 0 field to the project stage and give the answers.data.value field a name), and it really improves the running time. I am now in the process of optimising the Ruby calculations, and it's much faster now.