TLDR; LastWriteTime only updates when the file is read?
I have a log, being written to continuously. Occasionally, it rolls over and creates a new log with a new filename. I wrote a little scriptlet to get me the latest file that matches the mask:
gci $mask | Sort-Object LastWriteTime -Descending | Select-Object -First 1
Only it doesn't work, because, when the file rolls over to a new file, it opens the new file, then writes a few lines in the old one to close out, ending it up with a LastWriteTime later than the creation time of the new one. Powershell then continues to see the new file as a 0-byte file, and doesn't update its' LastWriteTime ever. Despite the file being continuously written to. For hours. If I open that new file and read it in a text editor, it updates the timestamp and size immediately every time. If I call Refresh() on that file object, or its containing directory, it does not update the data.
PS C:\Users\user\Documents> gci $mask |Sort-Object LastWriteTime -Descending | Select-Object -First 1
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 16/10/2024 13:58:33 623766 ResponderService_F0711118_7268_20241016.log
PS C:\Users\user\Documents> gci $mask |Sort-Object LastWriteTime -Descending | % {$_.Refresh()}
PS C:\Users\user\Documents> gci $mask |Sort-Object LastWriteTime -Descending | Select-Object -First 1
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 16/10/2024 13:58:33 623766 ResponderService_F0711118_7268_20241016.log
But if I read the contents of the file and dump them to $null it updates:
PS C:\Users\user\Documents> gci $mask |Sort-Object LastWriteTime -Descending | gc > $null
PS C:\Users\user\Documents> gci $mask |Sort-Object LastWriteTime -Descending | Select-Object -First 1
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 16/10/2024 14:02:56 636468 ResponderService_F0711118_7268_20241016.log
Despite the fact that it was being written to - I assure you - every few seconds. It's not just a matter of a write buffer in the logging process being flushed intermittently either; as I say, this can go on for hours and megabytes worth of logging without updating the file properties, or update seconds apart if I read the contents of the file to force it to update. Every time I read the file I can always see timestamps in the file from the last few seconds.
Can anyone explain to me what's going on here, and why it isn't utter madness? Or tell me what I'm doing that's dumb? What's the use of a LastWriteTime that only updates when the file is read?
EDIT:
SeanH and Eugene both explain that this is NTFS' fault, thanks. On the odd chance someone ends up here looking for something similar, my current script looks like:
foreach ($s in $mr_servers) {
$mask = "\\$s\share$\ResponderService_*";
# Ridiculously, read the files first so NTFS updates the LastWriteTimes to reflect reality
gci $mask | gc -TotalCount 1 > $null;
$f = (gci $mask |Sort-Object LastWriteTime -Descending |Select-Object -First 1).FullName;
# stuff with $f
}
LastWriteTime
isn't updating on write because Window's Buffered I/O and File Caching prioritize file content over metadata.
Metadata is only updated when a file is closed, flushed, or a user or system action (ex. read, disk write) forces a cache refresh.
For your reading pleasure:
Windows File Caching
Windows Buffered I/O
Is this sane? Not for continuous write or real-time analytics. That's why such systems don't use NTFS.