node.jsasynchronousgetgroupme

Sequentially combine data from multiple GET requests in node.js


I am writing a node.js app with the http module and am trying to combine data from multiple GET requests to produce a result. I checked similar problems but they all involve async.map to init requests in parallel. In my case, I need the result from the first GET request to get the URL for the next one, etc. Here is a sample of what I am dealing with:

// First GET request
https.get('URL', (resp) => {
    let data = '';
    resp.on('data', (chunk) => {
        data += chunk;
    });

    resp.on('end', () => {
        let parsedData = JSON.parse(data);
        let names = parsedData.name;
        let before_id = parsedData.messages[99].id;
    });
});

I then want to pass names and before_id to another GET request like this:

// Second GET request
https.get('URL&before_id=' + before_id, (resp) => {
    let data = '';
    resp.on('data', (chunk) => {
        data += chunk;
    });

    resp.on('end', () => {
        let parsedData = JSON.parse(data);
        names = //names from previous request + this parsedData.name
        let before_id = parsedData.messages[99].id;
    });
});

For context: I am using the GroupMe API, which only allows me to retrieve 100 messages at a time. I would like to retrieve 1000 messages, which is done by done by finding the message id of the last message in a batch of 100 and calling another GET request, passing that message id as a parameter in the URL. (This will request the 100 messages immediately before the specified id). Not only then do I need to pass the id from each request to the next one, but I also need to store the data I need from each batch of requests, combine them, then serve them together. How do I do this?


Solution

  • there will be better ways to structure code than what I'm going to present in answer. But it's the bare minimum example of what you want to do.

    To use the result of one asynchronous operation to initiate some other task, you have to embed that task in the callback of the asynchronous operation.

    // First GET request
    https.get('URL', (resp) => {
        let data = '';
        resp.on('data', (chunk) => {
            data += chunk;
        });
    
        resp.on('end', () => {
            let parsedData = JSON.parse(data);
            let names = parsedData.name;
            let before_id = parsedData.messages[99].id;
            // Second GET request
            https.get('URL&before_id=' + before_id, (resp) => {
                let data = '';
                resp.on('data', (chunk) => {
                    data += chunk;
                });
    
                resp.on('end', () => {
                    let parsedData = JSON.parse(data);
                    names = //names from previous request + this parsedData.name
                    let before_id = parsedData.messages[99].id;
                });
            });
        });
    });