typescriptasync-iteratorfilewalker

Is there any directory walker in ts / js using an async iterator?


I found plenty of walkers on npm but none is using an asynchronous iterator. Most of them are either using a callback or a promise leading to memory leaks on huge directories.

Is there any recent library using the following pattern:

async function* walk(dirPath) {
    // some magic…
    yield filePath;
}

To then use it like:

for await (const filePath of walk('/dir/path')) {
    console.log('file path', filePath);
}

Solution

  • Okay, I simply made this walker using the synchronous readdir, it is very fast and memory efficient, I listed 2.5 millions of entries in around 3 minutes without any memory leak.

    import path from 'path';
    import fs, {Dirent} from 'fs';
    
    function* walk(path:string):IterableIterator<string> {
    
        const entries:Dirent[] = fs.readdirSync(path, {withFileTypes: true});
    
        for (const entry of entries) {
            const entryPath:() => string = () => `${path}/${entry.name}`;
    
            if (entry.isFile()) {
                yield entryPath();
            }
    
            if (entry.isDirectory()) {
                yield* walk(entryPath());
            }
        }
    }
    

    Example of usage:

    for (const path of walk(directoryPath)) {
        console.log(path);
    }