How make Laravel's Collection behave like a stream?

I have an export process that runs out of memory sometimes. I use Laravel-Excel package that can chunk export process (using queue() or ShouldQueue), but the thing is that I have to transform my data before passing to Laravel-Excel, so I cant pass query object, only fully formed Collection.

My case looks this way:

• I make a query, applying filters,
• then I get all the data and transform it using collection's transform() method; here I append some extra data that I can't easily append just via joins because this data is collected by Repository object with a lot of extra calculations,
• and then I pass it to exporter (by returnung from collection() method of my export class)

So, the most expensive part here is reading, and seems like I should use collection's chunk() to chunk it, but! I can't pass those chunks to exporter (afaik). The only way to chunk export process is to pass query object (in this case exporter is able to queue and chunk the whole reading/writing process).

So, the question is: can I pass the Collection object, but to chunk it's reading/transforming proces? The same way as streamed file reading/writing works.

Or maybe I miss some laravel-excel functionality that allows me to achieve what I want?

The biggest problem is that Laravel-Excell doesnt work well with Memory. It has no support for streams.

You can reduce memory of your laravel query by not using eloquent models:

$EloquentQueryBuilder
->toBase() // +-10x speed improvement, skip making laravel models
->cursor() // avoid loading whole result set into memory.

But Laravel-Excell will but each value in the Excel file and thus put it all in memory before writing it to disc.

Memory efficient approach.

1. Use the above query builder methods.
2. use CSV (and don't use the Laravel-Excel library)
3. use a fileHandle/stream

I use the following code to make it memory and cpu efficient:

$cursor = $this->getQuery()
            ->toBase() // +-10x speed improvement, skip making laravel models
            ->cursor() // avoid loading whole result set into memory.
            ->map(fn ($item) => $this->map((array) $item));
        // writes all data (1 by 1) to a temporary file then we can then stream to external disk.
        $stream = StreamHelper::streamCsvAssoc($cursor);
        $disk->writeStream($filePath, $stream);
        fclose($stream);

The code behind:

    /**
     * Turn a key=>value array into csv file, with header row.
     *
     * eg: [['key1' => 'value1', 'key2' => 'value2]];
     * results into:
     * key1,key2
     * value1,value2
     *
     * @param iterable $iterator enumerable list that provides data set to write to csv.
     * @param string $separator csv field delimiter
     * @param string $enclosure csv string enclosure
     * @param string $escape csv escape character
     *
     * @return resource csv file as a stream.
     */
    public static function streamCsvAssoc(iterable $iterator, string $separator = ',', string $enclosure = '"', string $escape = '\\')
    {
        $first = true;

        return self::stream($iterator, function ($stream, $value) use (&$first, $escape, $enclosure, $separator) {
            if ($first) {
                fputcsv($stream, array_keys($value), $separator, $enclosure, $escape);
                $first = false;
            }
            fputcsv($stream, array_values($value), $separator, $enclosure, $escape);
        });
    }

    /**
     * @param iterable $iterator
     * @param $writeFunction (stream, value)
     *
     * @return resource
     */
    public static function stream(iterable $iterator, callable $writeFunction)
    {
        $stream = tmpfile();
        if (false === $stream) {
            throw new \Exception('failed making stream');
        }
        foreach ($iterator as $value) {
            $writeFunction($stream, $value);
        }
        rewind($stream);

        return $stream;
    }

Only concern: we first write to a temporary file handle. in the Ideal situation you have an open file handle and stream it directly there. But this is not well supported in both PHP and Laravel. (in say C#/.NET this has much more support)