batch-filecmddirectorypause

Run x5 .bat file, pause, then run next 5 and so on

Background: OS: win10. A single root folder: "U:\11Web\gallery-dl".

Within the root I have 1400+ sub-folders (there are no sub-folders beneath those (and never will be)). (There are also no individual/extraneous files in root (and never will be)).

Each sub-folder has its own .bat file, aGallery-dl.bat.

@echo off
setlocal EnableExtensions DisableDelayedExpansion
pushd "%~dp0" || exit /B
move /Y "Folder2.jpg" "Folder.jpg"
for %%I in (.) do set "FOLDER=%%~nxI"
"%ProgramFiles(x86)%\gallery-dl\gallery-dl.exe" -d "U:\11Web\gallery-dl" --download-archive "%~dp0zzzGDB.sqlite3" "https://www.deviantart.com/%FOLDER%/gallery/all"
if not errorlevel 1 if exist "zzzGDB.sqlite3" del "Folder.jpg"
popd
endlocal

I normally run aGallery-dl.bat manually; open the next folder, double-click to run; next folder run, etc., until I'm running between 5 and 10 cmd windows. Later when they've all completed, I might go thru another 5-10 folders, and so it goes. As each aGallery-dl.bat is executed, the requested files are downloaded and when complete, some cleanup is done on Folder.jpg, and each cmd window closes.

Problem: Looking to automate the running of these a little. Would like a single batch file at the root folder that when run would

  1. Create/update a list.txt/database file of all sub-folders. A list would work here, something simple like dir/l>list.txt but then don't know how count would be kept?

  2. Run the aGallery-dl.bat in the first 5 sub-folders of the list, take a pause of say 15 mins (timeout?), loop and then hit the next 5 folders, and so on until 1400+ are done... Doesn't matter that if I have to reboot, or come back a week later and have to re-run that same batch file, that it starts at the first folder all over again...there is a .sqlite3 database file in each folder that retains all the previously downloaded file information. Starting at the first folder every time would be a feature, downloading only those files that have been updated.

I have no starting code/example as I can't wrap my head around it sufficiently to really begin.

Thanks in advance.

Solution

  • The following batch file can be used to run up to five aGallery-dl.bat parallel.

    @echo off
setlocal EnableExtensions DisableDelayedExpansion
for /D %%I in ("U:\11Web\gallery-dl\*") do if exist "%%I\aGallery-dl.bat" (
    start "Gallery Download %%~nxI" /MIN %ComSpec% /c "%%I\aGallery-dl.bat"
    call :CheckDownloads
)
exit /B

:CheckDownloads
%SystemRoot%\System32\timeout.exe /T 1 >nul
for /F "delims=" %%J in ('%SystemRoot%\System32\tasklist.exe /FI "IMAGENAME eq cmd.exe" /NH /V ^| %SystemRoot%\System32\find.exe /C "Gallery Download "') do if %%J GEQ 5 goto CheckDownloads
goto :EOF

    The subroutine CheckDownloads is called after starting one more command process running with a minimized window parallel to the command process which is processing this batch file for the execution of aGallery-dl.bat with a window title being Gallery Download  and the folder name appended.

    The subroutine waits one second before it runs one more command process in background to run TASKLIST to get a list of running cmd.exe processes which is filtered with FIND for just those command processes with Gallery Download  in window title.

    FIND outputs just the number of lines with the searched string which is the number of command processes running currently to do a gallery download. If this number is (greater or) equal five, the subroutine waits again one second before doing again this downloads task check.

    A slightly modified version would be:

    @echo off
setlocal EnableExtensions DisableDelayedExpansion
for /D %%I in ("U:\11Web\gallery-dl\*") do if exist "%%I\aGallery-dl.bat" if exist "%%I\Folder.jpg" (
    start "Gallery Download %%~nxI" /MIN %ComSpec% /c "%%I\aGallery-dl.bat"
    call :CheckDownloads
)
exit /B

:CheckDownloads
%SystemRoot%\System32\timeout.exe /T 1 >nul
for /F "delims=" %%J in ('%SystemRoot%\System32\tasklist.exe /FI "IMAGENAME eq cmd.exe" /NH /V ^| %SystemRoot%\System32\find.exe /C "Gallery Download "') do if %%J GEQ 5 goto CheckDownloads
goto :EOF

    There is additionally checked if the file Folder.jpg exists in the subfolder to omit all the subfolders on which the gallery download was done already once in the past and so the file Folder.jpg does not exist anymore in the subfolder.

    The timeout value can be increased from 1 to 5 or 10 or 30 seconds to reduce the CPU usage for checking the downloads tasks.

    There should not be started too many gallery downloads parallel as this could be counterproductive depending on CPU cores and download bandwidth.

    The batch file posted here finishes already on up to last four gallery downloads are still running parallel.

    To understand the commands used and how they work, open a command prompt window, execute there the following commands, and read the displayed help pages for each command, entirely and carefully.