I am importing a CSV using maatwebsite/excel package. The CSV contains over 100 million rows, so I'm using WithChunkReading
. I'm also logging each row insertion.
protected $rowCount = 0;
public function collection(Collection $rows): bool
{
foreach ($rows as $row)
{
// Insertions here
++$this->rowCount;
Log::debug('Inserted row #'.$this->rowCount);
}
return false;
}
public function chunkSize(): int
{
return 10;
}
The problem is that after every chunk, $rowCount
resets to zero
.
This is what laravel.log
outputs:
[2020-07-13 10:06:10] local.DEBUG: Inserted row #1
[2020-07-13 10:06:11] local.DEBUG: Inserted row #2
[2020-07-13 10:06:11] local.DEBUG: Inserted row #3
[2020-07-13 10:06:11] local.DEBUG: Inserted row #4
[2020-07-13 10:06:11] local.DEBUG: Inserted row #5
[2020-07-13 10:06:12] local.DEBUG: Inserted row #6
[2020-07-13 10:06:12] local.DEBUG: Inserted row #7
[2020-07-13 10:06:12] local.DEBUG: Inserted row #8
[2020-07-13 10:06:12] local.DEBUG: Inserted row #9
[2020-07-13 10:06:13] local.DEBUG: Inserted row #10
[2020-07-13 10:06:16] local.DEBUG: Inserted row #1
[2020-07-13 10:06:16] local.DEBUG: Inserted row #2
[2020-07-13 10:06:16] local.DEBUG: Inserted row #3
I also tried using $rowCount
by reference in a closure
public function collection(Collection $rows): bool
{
$rowCount = 0;
foreach ($rows as $row)
{
// Insertions here
$counter = function() use(&$rowCount){
return ++$rowCount;
};
Log::debug('Inserted row #'.$counter->call($row));
}
return false;
}
But the result is still the same.
Update: I tried accessing the counter from outside the collection
method -
protected $rowCount = 0;
public function collection(Collection $rows): bool
{
foreach ($rows as $row)
{
// Insertions here
$rowCount = $this->incrementCounter();
Log::debug('Inserted row #'.$rowCount);
}
return false;
}
protected function incrementCounter(){
return ++$this->rowCount;
}
public function chunkSize(): int
{
return 10;
}
This, however, did not help either. The counter still resets back.
How can I persist the value of $rowCount
so that it does not reset back to 0
after every chunk?
TIA!
A trait called RemembersRowNumber
is now introduced to the library to help keep track of row count.
It can be used as follows:
namespace App\Imports;
use App\User;
use Maatwebsite\Excel\Concerns\ToModel;
use Maatwebsite\Excel\Concerns\RemembersRowNumber;
use Maatwebsite\Excel\Concerns\WithChunkReading;
class UsersImport implements ToModel, WithChunkReading
{
use RemembersRowNumber;
public function model(array $row)
{
$currentRowNumber = $this->getRowNumber();
return new User([
'name' => $row[0],
]);
}
public function chunkSize(): int
{
return 1000;
}
}
Please refer to the docs here for more detail.