33

I have a server which runs Windows Server 2008 R2 x64 with 4GB of RAM which hosts around 2-3 million files, the majority of which are image files.

Over a course of a week, I have noticed that applications on the server were slowing to a crawl due to excessive paging to the disk due to low memory, which has a knock-on effect to all services currently running on it, causing a major performance issue.

Upon investigation in Task Manager, I noticed that almost all 4GB was in-use but when you look in the Processes tab, the sum of all the memory usage there do not add up and at most only 1.5GB is supposed to be in use.

Using Google to find a solution, it appears that most of the RAM was used in the "Metafile" which is a cache of NTFS information for files on the file system so that the system does not have to query the MFT for information again. This cache is never cleared or marked as "cache" in Task Manager or as "Standby" in Sysinternal's RamMap.

There was a suggestion to install the KB979149 hotfix but upon trying to install it, it says "This update is not applicable to your computer".

The only temporary fixes I have so far found are:

  1. Use RAMmap from Sysinternals to "Empty System Working Set" every 1-3 days which marks the cache as "standby" and "cache" in Task Manager so the RAM can be used by other applications.
  2. Reboot the machine, which is undesirable as this server is serving public websites.

At the moment I am having to perform the 2. fix every few days to prevent it reaching bottleneck levels.

Before: (800 MB RAM used - other applications cannot use this RAM)

enter image description here

After: (800 MB RAM marked as cache - available for other applications)

So my question to you all is: Does any method exist out there to limit the RAM usage of this metafile?

4
  • 4
    4GB RAM on a server wich hosts 2-3 million files is absurd. Upgrade your RAM or upgrade your RAM.
    – pauska
    Oct 27, 2011 at 12:04
  • 1
    Never mind, CacheSet by Sysinternals lets me set the cache size, currently running this at intervals and doing so has solved the issue for me!
    – al2k4
    Oct 27, 2011 at 14:50
  • 6
    Adding RAM doesn't solve the problem. The metafile cache will fill that up too. I've tried doing this on a VMware guest that started with 4 GB and increased it to 12 GB and the same thing happens. The problem is that this memory is for cache purposes, but is not marked as cache according to Windows. It is incorrectly marked as Active/In Use memory, and unfortunately, as it grows it crowds out REAL Active/In Use memory used by real programs and starts paging to disk. When physical RAM fills up everything slows down and you have to do one of the two solutions as the original post mentioned.
    – user132293
    Aug 15, 2012 at 19:12
  • @al2k4 If your question has been resolved, please mark it as solved by clicking the check mark next to the correct answer. Welcome to Server Fault! Jan 6, 2014 at 17:40

7 Answers 7

16

The best method for dealing with this issue is to use the SetSystemFileCacheSize API as MS KB976618 instructs used to instruct.

Don't periodically clear the cache

Using the SetSystemFileCacheSize function rather than clearing the cache periodically improves performance and stability. Clearing the cache periodically will result in too much metafile and other info being purged from memory, and Windows will have to re-read the required info back into RAM from HDD. This creates a sudden and severe drop in performance for several seconds whenever you clear the cache, followed by good performance that slowly degrades as memory fills with metafile data.

Using the SetSystemFileCacheSize function sets minimum and maximum that will result in Windows flagging excess old metafile data as standby memory that the normal caching functions can use or discard according to the current resource demands and normal cache priorities. This also allows more metafile data than the active memory maximum you set, to be in memory as standby data if Windows is not using the memory for anything else, while maintaining plenty of available memory. This is the ideal situation keeping the performance characteristics of the system good all the time.

Third Party Programs are Unsupported by MS

If you are like me and don't want to run a binary from some unknown third party on your production servers, you want an official MS tool or some code you can inspect before running on those servers. The DynCache tool for 2008 R2 is practically impossible to obtain from M$ without paying for a support case and quite frankly, based on the code for 2008, it seems overly bloated for the task as Windows already has the built in logic needed to dynamically size the cache—it just needs to know an appropriate maximum for your system.

Solution to all of the above

I wrote a Powershell script that works on 64 bit machines. You need to run it as an administrator with elevated privileges. You should be able to run it, as is, on any x64 windows Vista / Server 2008 up to and including 10 / Server 2012 R2 with any amount of RAM. You do not need to install any additional software, and as a result keep your server/workstation fully supported by MS.

You should run this script at every boot with elevated privileges for the setting to be permanent. Windows Task Scheduler can do this for you. If the Windows install is inside a virtual machine and you change the amount of RAM allocated to that VM, you should also run it after the change.

You can run this script at any time on a running system even while in production use without having to reboot the system or shut down any services.

# Filename: setfc.ps1
$version = 1.1

#########################
# Settings
#########################

# The percentage of physical ram that will be used for SetSystemFileCache Maximum
$MaxPercent = 12.5

#########################
# Init multipliers
#########################
$OSBits = ([System.IntPtr]::Size) * 8
switch ( $OSBits)
{
    32 { $KiB = [int]1024 }
    64 { $KiB = [long]1024 }
    default {
        # not 32 or 64 bit OS. what are you doing??
        $KiB = 1024 # and hope it works anyway
        write-output "You have a weird OS which is $OSBits bit. Having a go anyway."
    }
}
# These values "inherit" the data type from $KiB
$MiB = 1024 * $KiB
$GiB = 1024 * $MiB
$TiB = 1024 * $GiB
$PiB = 1024 * $TiB
$EiB = 1024 * $PiB


#########################
# Calculated Settings
#########################

# Note that because we are using signed integers instead of unsigned
# these values are "limited" to 2 GiB or 8 EiB for 32/64 bit OSes respectively

$PhysicalRam = 0
$PhysicalRam = [long](invoke-expression (((get-wmiobject -class "win32_physicalmemory").Capacity) -join '+'))
if ( -not $? ) {
    write-output "Trying another method of detecting amount of installed RAM."
 }
if ($PhysicalRam -eq 0) {
    $PhysicalRam = [long]((Get-WmiObject -Class Win32_ComputerSystem).TotalPhysicalMemory) # gives value a bit less than actual
}
if ($PhysicalRam -eq 0) {
    write-error "Cannot Detect Physical Ram Installed. Assuming 4 GiB."
    $PhysicalRam = 4 * $GiB
}
$NewMax = [long]($PhysicalRam * 0.01 * $MaxPercent)
# The default value
# $NewMax = 1 * $TiB


#########################
# constants
#########################

# Flags bits
$FILE_CACHE_MAX_HARD_ENABLE     = 1
$FILE_CACHE_MAX_HARD_DISABLE    = 2
$FILE_CACHE_MIN_HARD_ENABLE     = 4
$FILE_CACHE_MIN_HARD_DISABLE    = 8


################################
# C# code
# for interface to kernel32.dll
################################
$source = @"
using System;
using System.Runtime.InteropServices;

namespace MyTools
{
    public static class cache
    {
        [DllImport("kernel32", SetLastError = true, CharSet = CharSet.Unicode)]
        public static extern bool GetSystemFileCacheSize(
            ref IntPtr lpMinimumFileCacheSize,
            ref IntPtr lpMaximumFileCacheSize,
            ref IntPtr lpFlags
            );

        [DllImport("kernel32", SetLastError = true, CharSet = CharSet.Unicode)]
        public static extern bool SetSystemFileCacheSize(
          IntPtr MinimumFileCacheSize,
          IntPtr MaximumFileCacheSize,
          Int32 Flags
        );

        [DllImport("kernel32", CharSet = CharSet.Unicode)]
        public static extern int GetLastError();

        public static bool Get( ref IntPtr a, ref IntPtr c, ref IntPtr d )
        {
            IntPtr lpMinimumFileCacheSize = IntPtr.Zero;
            IntPtr lpMaximumFileCacheSize = IntPtr.Zero;
            IntPtr lpFlags = IntPtr.Zero;

            bool b = GetSystemFileCacheSize(ref lpMinimumFileCacheSize, ref lpMaximumFileCacheSize, ref lpFlags);

            a = lpMinimumFileCacheSize;
            c = lpMaximumFileCacheSize;
            d = lpFlags;
            return b;
        }

        
        public static bool Set( IntPtr MinimumFileCacheSize, IntPtr MaximumFileCacheSize, Int32 Flags )
        {
            bool b = SetSystemFileCacheSize( MinimumFileCacheSize, MaximumFileCacheSize, Flags );
            if ( !b ) {
                Console.Write("SetSystemFileCacheSize returned Error with GetLastError = ");
                Console.WriteLine( GetLastError() );
            }
            return b;
        }
    }

    public class AdjPriv
    {
        [DllImport("advapi32.dll", ExactSpelling = true, SetLastError = true)]
        internal static extern bool AdjustTokenPrivileges(IntPtr htok, bool disall, ref TokPriv1Luid newst, int len, IntPtr prev, IntPtr relen);

        [DllImport("advapi32.dll", ExactSpelling = true, SetLastError = true)]
        internal static extern bool OpenProcessToken(IntPtr h, int acc, ref IntPtr phtok);

        [DllImport("advapi32.dll", SetLastError = true)]
        internal static extern bool LookupPrivilegeValue(string host, string name, ref long pluid);

        [StructLayout(LayoutKind.Sequential, Pack = 1)]
        internal struct TokPriv1Luid
        {
            public int Count;
            public long Luid;
            public int Attr;
        }
        internal const int SE_PRIVILEGE_ENABLED = 0x00000002;
        internal const int SE_PRIVILEGE_DISABLED = 0x00000000;
        internal const int TOKEN_QUERY = 0x00000008;
        internal const int TOKEN_ADJUST_PRIVILEGES = 0x00000020;

        public static bool EnablePrivilege(long processHandle, string privilege, bool disable)
        {
            bool retVal;
            TokPriv1Luid tp;
            IntPtr hproc = new IntPtr(processHandle);
            IntPtr htok = IntPtr.Zero;
            retVal = OpenProcessToken(hproc, TOKEN_ADJUST_PRIVILEGES | TOKEN_QUERY, ref htok);
            tp.Count = 1;
            tp.Luid = 0;
            if(disable)
            {
                tp.Attr = SE_PRIVILEGE_DISABLED;
            } else {
                tp.Attr = SE_PRIVILEGE_ENABLED;
            }
            retVal = LookupPrivilegeValue(null, privilege, ref tp.Luid);
            retVal = AdjustTokenPrivileges(htok, false, ref tp, 0, IntPtr.Zero, IntPtr.Zero);
            return retVal;
        }
    }
}
"@
# Add the c# code to the powershell type definitions
Add-Type -TypeDefinition $source -Language CSharp

#########################
# Powershell Functions
#########################
function output-flags ($flags)
{
    Write-output ("FILE_CACHE_MAX_HARD_ENABLE  : " + (($flags -band $FILE_CACHE_MAX_HARD_ENABLE) -gt 0) )
    Write-output ("FILE_CACHE_MAX_HARD_DISABLE : " + (($flags -band $FILE_CACHE_MAX_HARD_DISABLE) -gt 0) )
    Write-output ("FILE_CACHE_MIN_HARD_ENABLE  : " + (($flags -band $FILE_CACHE_MIN_HARD_ENABLE) -gt 0) )
    Write-output ("FILE_CACHE_MIN_HARD_DISABLE : " + (($flags -band $FILE_CACHE_MIN_HARD_DISABLE) -gt 0) )
    write-output ""
}

#########################
# Main program
#########################

write-output ""

#########################
# Get and set privilege info
$ProcessId = $pid
$processHandle = (Get-Process -id $ProcessId).Handle
$Privilege = "SeIncreaseQuotaPrivilege"
$Disable = $false
Write-output ("Enabling SE_INCREASE_QUOTA_NAME status: " + [MyTools.AdjPriv]::EnablePrivilege($processHandle, $Privilege, $Disable) )

write-output ("Program has elevated privledges: " + ([Security.Principal.WindowsPrincipal] [Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole([Security.Principal.WindowsBuiltInRole] "Administrator") )
write-output ""
whoami /PRIV | findstr /I "SeIncreaseQuotaPrivilege" | findstr /I "Enabled"
if ( -not $? )  {
    write-error "user Security Token SE_INCREASE_QUOTA_NAME: Disabled`r`n"
}
write-output "`r`n"


#########################
# Get Current Settings
# Init variables
$SFCMin = 0
$SFCMax = 0
$SFCFlags = 0
#Get Current values from kernel
$status = [MyTools.cache]::Get( [ref]$SFCMin, [ref]$SFCMax, [ref]$SFCFlags )
#typecast values so we can do some math with them
$SFCMin = [long]$SFCMin
$SFCMax = [long]$SFCMax
$SFCFlags = [long]$SFCFlags
write-output "Return values from GetSystemFileCacheSize are: "
write-output "Function Result : $status"
write-output "            Min : $SFCMin"
write-output ("            Max : $SFCMax ( " + $SFCMax / 1024 / 1024 / 1024 + " GiB )")
write-output "          Flags : $SFCFlags"
output-flags $SFCFlags


#########################
# Output our intentions
write-output ("Physical Memory Detected : $PhysicalRam ( " + $PhysicalRam / $GiB + " GiB )")
write-output ("Setting Max to " + $MaxPercent + "% : $NewMax ( " + $NewMax / $MiB + " MiB )`r`n")

#########################
# Set new settings
$SFCFlags = $SFCFlags -bor $FILE_CACHE_MAX_HARD_ENABLE # set max enabled
$SFCFlags = $SFCFlags -band (-bnot $FILE_CACHE_MAX_HARD_DISABLE) # unset max dissabled if set
# or if you want to override this calculated value
# $SFCFlags = 0
$status = [MyTools.cache]::Set( $SFCMin, $NewMax, $SFCFlags ) # calls the c# routine that makes the kernel API call
write-output "Set function returned: $status`r`n"
# if it was successfull the new SystemFileCache maximum will be NewMax
if ( $status ) {
    $SFCMax = $NewMax
}


#########################
# After setting the new values, get them back from the system to confirm
# Re-Init variables
$SFCMin = 0
$SFCMax = 0
$SFCFlags = 0
#Get Current values from kernel
$status = [MyTools.cache]::Get( [ref]$SFCMin, [ref]$SFCMax, [ref]$SFCFlags )
#typecast values so we can do some math with them
$SFCMin = [long]$SFCMin
$SFCMax = [long]$SFCMax
$SFCFlags = [long]$SFCFlags
write-output "Return values from GetSystemFileCacheSize are: "
write-output "Function Result : $status"
write-output "            Min : $SFCMin"
write-output ("            Max : $SFCMax ( " + $SFCMax / 1024 / 1024 / 1024 + " GiB )")
write-output "          Flags : $SFCFlags"
output-flags $SFCFlags

There is line near the top that says $MaxPercent = 12.5 that sets the new maximum working set (active memory) to 12.5% of the total physical RAM. Windows will dynamically size the amount of metafile data in active memory based on system demands, so you don't need to dynamically adjust this maximum.

This will not fix any issues you have with the mapped file cache getting too big.

I've also made a GetSystemFileCacheSize Powershell script and posted it on StackOverflow.


Edit: I should also point out that you should not run either of these 2 scripts from the same Powershell instance more than once, or you will receive the error that the Add-Type call has already been made.

Edit: updated SetSystemFileCacheSize script to version 1.1 that calculates an appropriate max cache value for you and has a nicer status output layout.

Edit: Now I've upgraded my Windows 7 laptop, I can tell you that the script runs successfully in Windows 10, though I haven't tested if it is still needed. But my system is still stable even when moving virtual machine HDD files around.

6
  • The DynCache tool is freely available for download from microsoft.com/en-us/download/details.aspx?id=9258 and supports 2008 R2. Nov 7, 2014 at 11:03
  • It is now. There was a long time between the R2 windows release and the DynCache release. See blogs.technet.com/b/yongrhee/archive/2010/02/16/… for the ms blog post update. I still prefer my solution as it does not require extra resources to run yet another service. Our servers have become very stable with my script, so I'm not changing them over to DynCache. Nov 8, 2014 at 2:49
  • @BeowulfNode42 - We are having problems with the mapped file cache getting to big. Do you have any pointers on how to resolve that? I was under the assumption that setting the system file cache size would also solve that problem?! Do you have any idea if the (bloated) DynCache tool would solve that problem? Feb 29, 2016 at 9:06
  • fwiw - I just tried on a testserver and the Mapped File (rammap) got from 12GB Active, 0GB Standby to 8GB Active, 4GB Standby. For all intents and purposes, this does seem to work for Mapped Files also?! Feb 29, 2016 at 9:15
  • @LievenKeersmaekers that is weird. Perhaps it is some sort of flow on effect. I haven't come across a good way to solve the mapped file cache issue experienced when copying files from fast storage to slow, though I haven't tried the DynCache tool, because for us the mapped file cache issue is mainly just an annoying temporary slow down on our servers that manage our backups. For future reference how much ram does that test system have and were you running the script as posted with the 12.5% setting, and if you remember or have records the other types of memory sizes? Mar 1, 2016 at 23:52
4

I don't claim to be an expert regarding the internal workings of memory or disk caching in a Windows OS, but I have two observations:

  1. If the OS didn't cache the data in memory it would have to read it from disk, which is an exponentially slower storage media then memory, so the performance problem you're seeing now would almost certainly be worse.

  2. You're trying to solve the problem by treating a symptom of the problem instead of the cause of the problem. The cause of the problem is almost certainly a lack of sufficient physical RAM and my suggestion would be to address that.

In addition, while the cache may be using 1.5GB of RAM I would wonder what the memory usage is for other processes and services and might the solution be to investigate that usage for potential problems.

6
  • Exactly. If the OP reduced the RAM usage of the metafile, the system would have to load more metadata from disk since less of it would be in memory, making things worse. Oct 27, 2011 at 12:07
  • 1
    Thanks for the feedback. Few things, the server is primarily a web server with a MySQL database and does not read files very often so the slight impact of the metadata of not being in the cache is minimal, the performance increases dramatically when cleared. It is the number of different files it reads over time is why the cache size gets higher and higher. I am well aware that more RAM would solve it, but isn't the idea of a "cache" is to free up memory when applications or scripts on the server really need it to avoid paging? Why this certain cache is always marked as active confuses me.
    – al2k4
    Oct 27, 2011 at 13:14
  • 1
    You've obviously never experienced this issue yourself. Many people with 32, 64, and 128 GB of RAM have this problem where far too much of RAM is taken up by the metafile data and windows does not release it as it is marked as active and not standby (aka cache) memory. Using the SetSystemFileCacheSize API as I've described in my answer forces Windows to flag much of the metafile data as standby memory and the cache management system is then able to prioritise what to keep in RAM and what to discard. Jan 7, 2014 at 7:37
  • Troll much? This question is more than two years old.
    – joeqwerty
    Jan 7, 2014 at 15:30
  • @joeqwerty I see new posts all over the net about this problem all the time. Many of the related searches arrive at this question. Since I was updating my own answer and I believe your answer is "not useful" I marked it as such and commented why. If that makes me a troll, so be it. Jan 8, 2014 at 2:59
3

To the people who gave the obvious but ineffective solution of just adding more RAM, you clearly haven't dealt with this issue first hand.

As stated by an earlier poster, it doesn't matter how much RAM you throw at the problem...it will all fill up. I am running an Atlassian tool set on our app server that was migrated from 32 bit (2003) to 64 bit (2008). It was immediately apparent that there was a performance loss.

When looking at the task manager, nearly all of the memory was used up; even though the processes that are running do not reflect this. When we increased memory from 8 GB to 16 GB, the problem consumed the additional memory as well.

The only way to treat the problem was to restart the server, which brought down the memory usage equal to the processes (about 3.5 GB). This started climbing again within a day or so.

I knew this was a new Microsoft bug/feature and was happy to find this article. I love how Microsoft leaves this all-important detail for the users to figure out. I downloaded RamMap, which you would think would be a native utility, and now I can see the Metafile usage. We will set the cache to be cleared every few days and hopefully this will solve the issue.

Its interesting that I've only seen this problem on one out of several of our migrated servers, so I'm wondering if the metafile is only fed from certain types of applications.

2
  • 1
    In my experience, metafile memory usage won't grow much beyond the size of the filesystem metadata (that's what it's caching, after all), so upgrading RAM to allow filesystem metadata to fit in memory is a viable solution in at least some cases. I've also recommended clients reduce the size of the filesystem metadata by removing cruft such as millions of temporary files that haven't been touched in months. It is rather awkward that Windows effectively prefers NTFS metadata in memory over application memory, but looking up clusters for a file can be horribly slow without the MFT in memory. May 26, 2013 at 20:21
  • 2
    I concur - adding more ram does not correct the issue, it'll just consume more and all other processes will eventually grind to a halt. I recently upgraded to 24 GB, only to have SQL take up 8 (fine), and have 12 in the metafile.. James N - which tool are you using to periodically clear it?
    – sirthomas
    Aug 23, 2013 at 15:09
2

This issue can be addressed quickly and for free using the SysInternals CacheSet tool. Simply set the working set maximum to a suitable value less than the amount of system RAM, and apply.

1

Sorry to be so direct but what about you upgrade the server to an amount of ram that is a little higher than a what workstations have these days? 16gb memroy are freaking cheap. Less expensive than even half a day of your time.

4
  • 2
    That would solve it for good but our servers are hosted remotely by a third party. Our hosts will charge a large sum just to increase the RAM on a monthly basis. So we would like to avoid that if possible.
    – al2k4
    Oct 27, 2011 at 11:09
  • 2
    Yeah. Guess what ;) This is why I buy my hardware. Hosts are crazy - you can buy the RAM in 3 months. Well, lessons to learn: an unprofessional setup comes back to bite you.
    – TomTom
    Oct 27, 2011 at 11:34
  • All 64bit windows systems have a setting of 1TB for the maximum amount of metafile data to have in active memory (not treated as cached standby memory ready to be used when other things need more memory). I don't know about you but I haven't seen any windows box with that much memory in it yet. You should not have to install more RAM to make use of more files stored on a HDD. NTFS is supposed to support up to 4,294,967,295 files per volume. A single 4TB NTFS drive should then be able to support over 900 million files. Try and run defrag or backup on that and it will fail or crawl. Jan 7, 2014 at 7:50
  • 1
    Yes, you should have if your "Server" has less ram than a powerfull laptop. This is not about "get ridiculously high". It is about "get enough that the server deserves that name".
    – TomTom
    Jan 7, 2014 at 9:01
1

Here is a link to download the Microsoft DynCache tool - no need to create a ticket or pay. http://www.microsoft.com/en-us/download/details.aspx?displaylang=en&id=9258

(apologies - only noticing now that this is not for R2 version)

The known issue for continued cache growth is described here on Microsoft blog: http://blogs.msdn.com/b/ntdebugging/archive/2007/11/27/too-much-cache.aspx

[update] working fix for Windows Server 2008 R2.

I found sample C# code on Codeplex, quickly created a console C# project with Visual Studio and compiled, worked.

https://asstoredprocedures.svn.codeplex.com/svn/ASSP/FileSystemCache.cs

Note, you'll need to add a reference to Microsoft.AnalysisServices.AdomdClient which can be found here:

C:\Program Files (x86)\Microsoft.NET\ADOMD.NET

and comment out the ClearAllCaches() method with (in my case) unneeded references to XMLaDiscover. Throw this into TaskScheduler.

0

You can get DynCache tool from MS that will allow to restrict the RAM usage by metafile.

Click here to get the tool from MS.

1
  • That link provides nothing for 2008 R2 systems. M$ still want your credit card so you can pay to fix this known issue. Dec 25, 2013 at 5:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .