mongodbjoinmeteorcollectionsmeteor-collections

Meteor: Speeding up MongoDB Join's for Big Data?


I have two collections: Data and Users. In the Data collection, there is an array of user IDs consisting of approximately 300 to 800 users.

I need to join together the countries of all of the users for each row in the Data collection, which hangs my web browser due to there being too much data being queried at once.

I query for about 16 rows of the Data collection at once, and also there are 18833 users so far in the Users collection.

So far I have tried to make both a Meteor method and a transform() JOIN for the Meteor collection which is what's hanging my app.

Mongo Collection:

UserInfo = new Mongo.Collection("userInfo")
GlyphInfo = new Mongo.Collection("GlyphAllinOne", {
    transform: function(doc) {
        doc.peopleInfo = doc.peopleInfo.forEach(function(person) {  
            person.code3 = UserInfo.findOne({userId: person.name}).code3;
            return person;
        })
        return doc;
    }
});

'code3' designates user's country.

Publication:

Meteor.publish("glyphInfo", function (courseId) {
    this.unblock();
    var query = {};
    if (courseId) query.courseId = courseId;
    return [GlyphInfo.find(query), UserInfo.find({})];
})

Tested Server Method:

Meteor.methods({
    'glyph.countryDistribution': function(courseId) {
        var query = {};
        if (courseId) query.courseId = courseId;
        var glyphs = _.map(_.pluck(GlyphInfo.find(query).fetch(), 'peopleInfo'), function(glyph) {
            _.map(glyph, function(user) {
                var data = Users.findOne({userId: user.name});
                if (data) {
                    user.country = data ? data.code3 : null;
                    console.log(user.country)
                    return user;
                }
            });
            return glyph;
        });
        return glyphs;
    }
});

Collection Data:

GlyphAllInOne Collection userInfo Collection

There is an option of preprocessing my collection so that countries would already be included, however I'm not allowed to modify these collections. I presume that having this JOIN be done on startup of the server and thereafter exposed through an array as a Meteor method may stall startup time of the server for far too long; though I'm not sure.

Does anyone have any ideas on how to speed up this query?

EDIT: Tried out MongoDB aggregation commands as well and it appears to be extremely slow on Meteor's minimongo. Took 4 minutes to query in comparison to 1 second on a native MongoDB client.

var codes = GlyphInfo.aggregate([

        {$unwind: "$peopleInfo"},
        {$lookup: {
            from: "users",
            localField: "peopleInfo.name",
            foreignField: "userId",
            as: "details"
        }
        },
        {$unwind: "$details"},
        {$project: {"peopleInfo.Count": 1, "details.code3": 1}}
    ])

Solution

  • Solved the problem by creating a huge MongoDB aggregation call, with the biggest factor in solving latency being indexing unique columns in your database.

    After carefully implementing indices into my database with over 4.6 million entries, it took 0.3 seconds on Robomongo and 1.4 seconds w/ sending data to the client on Meteor.

    Here is the aggregation code for those who'd like to see it:

    Meteor.methods({
        'course.countryDistribution': function (courseId, videoId) {
            var query = {};
            if (courseId) query.courseId = courseId;
    
            var data = GlyphInfo.aggregate([
    
                {$unwind: "$peopleInfo"},
                {$lookup: {
                    from: "users",
                    localField: "peopleInfo.name",
                    foreignField: "userId",
                    as: "details"
                }
                },
                {$unwind: "$details"},
                {$project: {"peopleInfo.Count": 1, "details.code3": 1}},
                {$group: {_id: "$details.code3", count: {$sum: "$peopleInfo.Count"}}}
            ])
    
            return data;
        }
    });
    

    If anyone else is tackling similar issues, feel free to contact me. Thanks everyone for your support!