PLEASE SEE BELOW THE ORIGINAL QUESTION FOR SOME TEST COMPARISONS OF DIFFERENT WAYS:
So I tried 2 ways so far:
1.Iterate through directory using the code from Get Folder Size from Windows Command Line :
@echo off
set size=0
for /r %%x in (folder\*) do set /a size+=%%~zx
echo %size% Bytes
2.Save output of a
'dir %folder% /s /a'
into a text file, and then read in the size at the bottom
3.The last way I am trying right now is using du (disk utility tool from MS - https://technet.microsoft.com/en-us/sysinternals/bb896651.aspx ).
Now with exception of #3 both of those ways seem way too slow for what I need (100s of thousands of files). So the question is which one of these is the fastest / should be fastest, and if there are any other fast(er) ways to get size of folder contents that has 100k+ files (and there are 100s of folders)
Below is my very hacky way of doing the comparison (butchered my program to see some outputs)
There are some small bugs with some parts like option 3 will fail because it tries to handle a number bigger than 32-bit limit, and I'm sure there is some more issues, but the general timings I think are evident unless I really messed up on my logic.
Option I: Iterate through directories, using VB script to read in the text output from 'dir' and look for the size at the end + convert it to MB (originally got it from somewhere else that I actually lose the place where I got it from) Option II: Iterate, with findstr pipe and output the result directly (no converstion to MB) - from @MC ND Option III: use the compact command to iterate - from @npocmaka Option IV: from @user1016274 - using robocoby
(There are some more answers, but these are the ones I've been able to incorporate)
These are the results i got, and they are pretty consistent in relevance to each other, robocopy blows them away
Option I and Option II were usually close, with Option II slightly better (anywhere from 1min 10sec to 2min 10 secs for both, not sure where the difference is coming from) Part III - 16-17 mins Part IV - 10-20 seconds
@echo OFF
setlocal enabledelayedexpansion
REM OPTION I - directory iteration
REM OPTION II - iteration with findstr pipe
REM OPTION III - compact
:MAIN
REM Initialize log filename
for /f "delims=" %%a in ('echo %date:~10,4%%date:~4,2%%date:~7,2%%time:~0,2%%time:~3,2%%time:~6,2%') do @set LOGFILEPOSTFIX=%%a
set LOGFILEPOSTFIX=%date:~10,4%%date:~4,2%%date:~7,2%%time:~0,2%%time:~3,2%%time:~6,2%
set TIMESTAMP=%date:~10,4%_%date:~4,2%_%date:~7,2%_%time:~0,2%_%time:~3,2%_%time:~6,2%
echo %TIMESTAMP%
set "LOGFILE=Proj_not_in_db_%LOGFILEPOSTFIX%.log"
set option=1
set TIMESTAMP=%date:~10,4%_%date:~4,2%_%date:~7,2%_%time:~0,2%_%time:~3,2%_%time:~6,2%
echo %TIMESTAMP% - PART I ---- Directory Listing into file, iterate through the sizes of all files inside folder >> %LOGFILE%
echo %TIMESTAMP% - PART I
call :PROCESSFOLDER
set TIMESTAMP=%date:~10,4%_%date:~4,2%_%date:~7,2%_%time:~0,2%_%time:~3,2%_%time:~6,2%
echo %TIMESTAMP% - PART I ---- END >> %LOGFILE%
echo %TIMESTAMP% - PART I - END
set option=2
set TIMESTAMP=%date:~10,4%_%date:~4,2%_%date:~7,2%_%time:~0,2%_%time:~3,2%_%time:~6,2%
echo %TIMESTAMP% - PART II findstr pipe ---- >> %LOGFILE%
echo %TIMESTAMP% - PART II
call :PROCESSFOLDER
set TIMESTAMP=%date:~10,4%_%date:~4,2%_%date:~7,2%_%time:~0,2%_%time:~3,2%_%time:~6,2%
echo %TIMESTAMP% - PART II ---- END>> %LOGFILE%
echo %TIMESTAMP% - PART II - END
set option=3
set TIMESTAMP=%date:~10,4%_%date:~4,2%_%date:~7,2%_%time:~0,2%_%time:~3,2%_%time:~6,2%
echo %TIMESTAMP% - PART III compact ---- >> %LOGFILE%
echo %TIMESTAMP% - PART III
call :PROCESSFOLDER
set TIMESTAMP=%date:~10,4%_%date:~4,2%_%date:~7,2%_%time:~0,2%_%time:~3,2%_%time:~6,2%
echo %TIMESTAMP% - PART III ---- END>> %LOGFILE%
echo %TIMESTAMP% - PART III - END
set option=4
set TIMESTAMP=%date:~10,4%_%date:~4,2%_%date:~7,2%_%time:~0,2%_%time:~3,2%_%time:~6,2%
echo %TIMESTAMP% - PART IV robocopy ---- >> %LOGFILE%
echo %TIMESTAMP% - PART IV
call :PROCESSFOLDER
call :CLEANUP
echo FINAL
pause
goto :EOF
:PROCESSFOLDER
echo C:\Windows
echo Processing C:\Windows >> %LOGFILE%
break > projects_in_folder.tmp
for /f "tokens=1-4,* SKIP=7" %%b IN ('dir "C:\Windows" /Q /TW /AD') do (
set _folder=%%f
REM Don't write the 2 lines at the end displaying summary information
if NOT "%%e" EQU "bytes" (
SET _folder=!_folder:~23!
echo !_folder!,%%b>> projects_in_folder.tmp
)
)
set "folder_path=C:\Windows"
call :COMPARE
goto :EOF
:COMPARE
set file_name=%folder_path:\=_%
break > "%file_name%.txt"
if %option%==4 (
set "full_path=C:\Windows"
call :GETFOLDERINFO4
set TIMESTAMP=%date:~10,4%_%date:~4,2%_%date:~7,2%_%time:~0,2%_%time:~3,2%_%time:~6,2%
echo %TIMESTAMP% - PART IV ---- END>> %LOGFILE%
echo %TIMESTAMP% - PART IV - END
)
for /f "tokens=1,2* delims=," %%a in (projects_in_folder.tmp) do (
for /f "tokens=1,* delims=_" %%x in ("%%a") do (
set "projcode=%%x"
)
set full_path=%folder_path%\%%a
if %option%==1 call :GETFOLDERINFO
if %option%==2 call :GETFOLDERINFO2
if %option%==3 call :GETFOLDERINFO3
echo PROJ: %%a SIZE: !totalsize! LASTMODIFIED: %%b >> %LOGFILE%
)
goto :EOF
:GETFOLDERINFO2
set "size=0"
set target=!full_path!
for /f "tokens=3,5" %%a in ('
dir /a /s /w /-c "%target%"
^| findstr /b /l /c:" "
') do if "%%b"=="" set "size=%%a"
echo %size%
set totalsize=%size%
goto :EOF
:GETFOLDERINFO4
pushd "%full_path%" || goto :EOF
setlocal
for /f "tokens=1-10,* delims= " %%a in ('
robocopy %full_path% %TEMP% /S /L /BYTES /XJ /NFL /NDL /NJH ^| find "Bytes"
') do echo %full_path%: %%c
popd
goto :EOF
:GETFOLDERINFO
set totalsize=0
dir "%full_path%" /s /a > size.txt
REM Run VBScript that outputs size in MB which is saved
pushd %~dp0
start /b "" cscript /nologo foldersize.vbs
FOR /F "usebackq tokens=*" %%r in (`CSCRIPT "foldersize.vbs"`) DO SET totalsize=%%r
echo bla > nul
goto :EOF
:GETFOLDERINFO3
set "last=#"
set "_size="
for /f "tokens=1 delims= " %%s in ('compact /s:"%full_path%" /q ') do (
set "_size=!last!"
set "last=%%s"
)
set "_size=%_size: =%"
set "_size=%_size: =%"
set "_size=%_size:.=%"
set "_size=%_size:,=%"
set "_size=%_size: =%"
echo folder size is : %_size% bytes
set totalsize=%_size%
goto :EOF
:CLEANUP
DEL /Q /S projects_in_folder.tmp
DEL /Q /S size.txt
goto :EOF
After some testing and comparing the performance of
dir /s
compact /s
and powershell GetChild-Item
I found that using robocopy is much faster. One additional advantage is that even very long paths do not cause an error (> 256 characters in path), for instance in deeply nested folders.
And if you prefer to not count data behind junctions that can easily be included with robocopy like this:
@echo off
pushd "%~1" || goto :EOF
for /f "tokens=2 delims= " %%a in ('
robocopy "%CD%" "%TEMP%" /S /L /BYTES /XJ /NFL /NDL /NJH /R:0 ^| find "Bytes"
') do echo %CD%: %%a
popd
If you leave out the /BYTES option you'll get the size value formatted in MB or GB. One would have to print the dimension (k,m,g,t denoting kilo, mega, giga, tera) as well in this case, using another loop variable:
@echo off
setlocal ENABLEDELAYEDEXPANSION
pushd "%~1" || goto :EOF
set "folder=%CD%"
if NOT "%folder: =%"=="%folder%" set folder="%folder%"
for /f "tokens=2-3 delims= " %%a in (
'robocopy %folder% %folder% /S /L /XJ /NFL /NDL /NJH /R:0 ^| findstr /I "Bytes"'
) do (
set dim=%%b
set "dim=!dim:k=KB!" & set "dim=!dim:m=MB!" & set "dim=!dim:g=GB!" & set "dim=!dim:t=TB!"
if !dim! EQU %%b set dim=B
echo ^ %CD%: %%a !dim!
)
popd
The robocopy command here does not actually copy anything (due to the '/L' list option) but prints a summary line containing the sum of the filesizes which then is parsed. As robocopy still expects valid paths for the source and destination folders, the folder name is used twice.
The folder name may or may not contain spaces and thus eventually needs to be quoted. That is taken care of in the first lines.
%%b holds either the dimension letter or a numeric value. This is tested by substitution to avoid the 32bit limit of set /A.