phplaravellaravel-excel

Counter resets to zero after every chunk in Laravel Excel import


I am importing a CSV using maatwebsite/excel package. The CSV contains over 100 million rows, so I'm using WithChunkReading. I'm also logging each row insertion.

protected $rowCount = 0;

public function collection(Collection $rows): bool
{
    foreach ($rows as $row)
    {
        // Insertions here

        ++$this->rowCount;
        Log::debug('Inserted row #'.$this->rowCount);
    }
    return false;
}

public function chunkSize(): int
{
    return 10;
}

The problem is that after every chunk, $rowCount resets to zero.

This is what laravel.log outputs:

[2020-07-13 10:06:10] local.DEBUG: Inserted row #1
[2020-07-13 10:06:11] local.DEBUG: Inserted row #2
[2020-07-13 10:06:11] local.DEBUG: Inserted row #3
[2020-07-13 10:06:11] local.DEBUG: Inserted row #4
[2020-07-13 10:06:11] local.DEBUG: Inserted row #5
[2020-07-13 10:06:12] local.DEBUG: Inserted row #6
[2020-07-13 10:06:12] local.DEBUG: Inserted row #7
[2020-07-13 10:06:12] local.DEBUG: Inserted row #8
[2020-07-13 10:06:12] local.DEBUG: Inserted row #9
[2020-07-13 10:06:13] local.DEBUG: Inserted row #10
[2020-07-13 10:06:16] local.DEBUG: Inserted row #1
[2020-07-13 10:06:16] local.DEBUG: Inserted row #2
[2020-07-13 10:06:16] local.DEBUG: Inserted row #3

I also tried using $rowCount by reference in a closure

public function collection(Collection $rows): bool
{
    $rowCount = 0;
    foreach ($rows as $row)
    {
        // Insertions here

        $counter = function() use(&$rowCount){
            return ++$rowCount;
        };
        Log::debug('Inserted row #'.$counter->call($row));
    }
    return false;
}

But the result is still the same.

Update: I tried accessing the counter from outside the collection method -

protected $rowCount = 0;

public function collection(Collection $rows): bool
{
    foreach ($rows as $row)
    {
        // Insertions here

        $rowCount = $this->incrementCounter();
        Log::debug('Inserted row #'.$rowCount);
    }
    return false;
}

protected function incrementCounter(){
    return ++$this->rowCount;
}

public function chunkSize(): int
{
    return 10;
}

This, however, did not help either. The counter still resets back.

How can I persist the value of $rowCount so that it does not reset back to 0 after every chunk?
TIA!


Solution

  • A trait called RemembersRowNumber is now introduced to the library to help keep track of row count.

    It can be used as follows:

    namespace App\Imports;
    
    use App\User;
    use Maatwebsite\Excel\Concerns\ToModel;
    use Maatwebsite\Excel\Concerns\RemembersRowNumber;
    use Maatwebsite\Excel\Concerns\WithChunkReading;
    
    class UsersImport implements ToModel, WithChunkReading
    {
        use RemembersRowNumber;
    
        public function model(array $row)
        {
            $currentRowNumber = $this->getRowNumber();
    
            return new User([
                'name' => $row[0],
            ]);
        }
        
        public function chunkSize(): int
        {
            return 1000;
        }
    }
    

    Please refer to the docs here for more detail.