Faster Dir Size calculations in Powershell!

Get-ChildItem is probably the command that’s most used when working in PowerShell console. Next to file-system operations the command is also excellent to browse objects accessible via PSDrive(s).

Most sysadmin’s are probably familiar with tools such as TreeSize and/or WinDirStat. Did you ever wonder why these tools are so much faster than Get-ChildItem? There is even a PowerShell TreeSize implementation available on the PSGallery, however it’s pretty slow when running on large file-servers.

The downside of Get-ChildItem is that it will not let you grab a sub-set of properties. It seems likely that this is one of the reasons Get-ChildItem is slow.

The following 3 commands will all produce the same result, which is the total Dir Size in bytes:

#Use erroraction silentlycontinue to ignore access denied errors
(`Get-ChildItem` $home -Recurse -force -erroraction SilentlyContinue | Measure-Object length -Sum).sum

#Good old Dir, recursively
((cmd /c dir $home /-C /S /A:-D-L)[-2] -split '\s+')[3]

#RoboCopy in list only mode:
(robocopy.exe $home c:\fakepathduh /L /XJ /R:0 /W:1 /NP /E /BYTES /NFL /NDL /NJH /MT:64)[-4] -replace '\D+(\d+).*','$1'

17996994274
17996994274
17995815085

Dir size in Windows Explorer

All commands come close enough with only a 1MB deviation in the RoboCopy.exe version. On the right-hand-side you will find a screenshot showing the properties of my homefolder. This also shows a small deviation. This deviation seems to be the result of a growing VHD file in my homefolder. I’ll list to RoboCopy parameters for reference

/L : List only
/XJ : Do not follow NTFS junctions
/R:0 : Retries 0
/W:1 : Wait 1 sec before retry
/NP : Don’t display progress
/E : Include empty folders
/BYTES : Display output in bytes (not supported on pre Win7 RoboCopy versions)
/NFL /NDL : No file and No directory listing
/MT:64 : Use 64 threads

Measuring the Dir Size calculations

So how fast are these commands actually? Which command should we use? Let’s use Measure-Command to find out:

Measure-Command {
    (1..10).foreach{
      (gci $home -Recurse -force -ea 0 |
      Measure-Object length -Sum).sum /1MB -as [int]
    }
} | Select TotalMilliseconds

Measure-Command {
    (1..10).foreach{
      ((cmd /c dir $home /-C /S /A:-D-L)[-2] -split '\s+')[3] /1MB -as [int]
    }
} | Select TotalMilliseconds

Measure-Command {
    (1..10).foreach{
      ((robocopy.exe $home c:\fakepath /L /XJ /R:0 /W:1 /NP /E /BYTES /nfl /ndl /njh /MT:64)[-4] -replace '\D+(\d+).*','$1') /1MB -as [int]
    }
} | Select TotalMilliseconds

Result:

TotalMilliseconds
-----------------
       55468.3775
       14617.8298
        2941.6903

There you go Dir.exe is 3.79x faster than Get-ChildItem on my notebook with SSD. RoboCopy.exe is 18.86x faster.

Warning BUGS

I found a bug when using DIR over PowerShell Remoting. After some fiddling around it appears as DIR get’s into an infinite loop when recursively listing ‘C:\Users\Username\AppData\Local\Application Data'. When running in a local PowerShell session DIR will not try to follow the NTFS Junction. We’ve been able to reproduce this issue on Windows 2008, 2012R2 and on 2016TP5.

When executing locally:

cmd.exe /c dir /A:H
Volume in drive C has no label.
Volume Serial Number is E653-EF16

Directory of C:\Users\jdekoning\AppData\Local\Application Data

File Not Found

When executing over PSR:

cmd.exe /c dir /A:H
Volume in drive C has no label.
Volume Serial Number is E653-EF16

Directory of C:\Users\jdekoning\AppData\Local\Application Data

05/04/2016  10:47 AM    <JUNCTION>     Application Data [C:\Users\jdekoning\AppData\Local]
05/04/2016  10:47 AM    <JUNCTION>     History [C:\Users\jdekoning\AppData\Local\Microsoft\Windows\History]
05/19/2016  08:34 AM            23,699 IconCache.db
05/04/2016  10:47 AM    <JUNCTION>     Temporary Internet Files [C:\Users\jdekoning\AppData\Local\Microsoft\Windows\INetCache]
               1 File(s)         23,699 bytes
               3 Dir(s)  127,835,623,424 bytes free