phpfirebasegoogle-cloud-tasks

Google cloud task divide into subtasks by firebase data


I have a google task which gets all companies from my firebase database. I then go through each of those companies in a loop and call additional task for updating each specific company. My problem is that my companies count is increasing and when doing foreach like this i can get into memory limit issues. Here is the actual code for calling the tasks and subtasks:

$router->get('companies', function () use ($router) {

    $slackDataHelpersService = new \App\Services\SlackDataHelpersService();
    $companiesDocuments = $slackDataHelpersService->getCompanies();

    foreach ($companiesDocuments->documents() as $document) {
        $cid = $document->id();
        createTask('companies', 'updateCompany', "{$cid}");
    }

    return res(200, 'Task done');
});

How can i separate my initial companies documents into chunks and call a task for each of those chunks? For example, a task that will go through every 100 documents instead of the whole list?

Here is what i tried without success(i used members in this case):

$router->get('test2', function () use ($router) {

    $db = app('firebase.firestore')->database();

    $membersRef = $db->collection('companies')->document('slack-T01L7H2NDPB')->collection('members');
    $query = $membersRef->orderBy('created', 'desc')->limit(10);

    $perPage = 10;
    $batchCount = 10;
    $lastCreated = null;

    while ($batchCount == $perPage) {

        $loopQuery = clone $query;
        if ($lastCreated != null) {
            $loopQuery->startAfter($lastCreated);
        }
        $docs = $loopQuery->documents();
        $docsRows = $docs->rows();
        $batchCount = count($docsRows);

        if ($batchCount > 1) {
            $lastCreated = $docsRows[$batchCount - 1];
        }
        echo $lastCreated['created'];
        //createTasksByDocs($docs);
    }
    //return res(200, 'Task done');
});

Solution

  • I ended up making a function which uses a while loop and loops until it reaches the limit:

    function paginateCollections($ref, $limit, $functionName)
    {
        $query = $ref->orderBy('created', 'desc')->limit($limit);
    
        $perPage = $limit;
        $batchCount = $limit;
        $lastCreated = null;
    
        while ($batchCount == $perPage) {
    
            $loopQuery = clone $query;
            if ($lastCreated != null) {
                $loopQuery = $loopQuery->startAfter([$lastCreated]);
            }
            $docs = $loopQuery->documents();
            $docsRows = $docs->rows();
            $batchCount = count($docsRows);
    
            if ($batchCount > 1) {
                $lastCreated = $docsRows[$batchCount - 1]['created'];
            }
    
            if (function_exists($functionName)) {
                $functionName($docs);
            }
        }
    }