I have been running a Rails 5.2 project for years without problems. In one place, it uses several multi-stage aggregations. Some of these have now started giving me a "[43:CursorNotFound]: cursor id 283353284060432170 not found (on db:27017)" error. I can post one of the aggregations:
Myclass.collection.aggregate([ {'$match' => {'meta_info.store_id' => store_id_string }},
{'$match' => {'meta_info.invalid' => {'$in': [nil, false]} }} ,
{'$match' => {'test_participation' => false }} ,
{'$match' => {'exit_status' => 'finished' }} ,
datehash,
{'$unwind' => '$answers'},
{'$match' => {'answers.question_id' => q.id.to_i }},
{'$project' => {'answers.data.value' => 1}},
{'$group' => { '_id' => 0 , 'data' => { '$push' => '$answers.data.value' } } }
]).to_a.first
where datehash
is yet another match
statement that is calculated dynamically.
I assign the result to a variable and then process the contents of that variable. My database content has been growing for years. I know nothing about cursors, unfortunately. So I have two questions:
I'd also appreciate if you could give me any suggestions about how to streamline the aggregation.
(The original reason my aggregation is so complex is that it is still much, much faster than fetching the raw data and then doing the calculations in Rails.)
It has turned out that the real problem was elsewhere. The cursor lost exception was caused by another process running in parallel which also used the database heavily and did complex computations with the data. When I had changed the cron time to avoid overlaps the problem went away.
I have still optimised my aggregation as aneroid suggested (combine all the match stages into one, add an _id => 0 field to the project stage and give the answers.data.value field a name), and it really improves the running time. I am now in the process of optimising the Ruby calculations, and it's much faster now.