up vote 7 down vote favorite
3
share [g+] share [fb]

I have a rather old server that has 4GB of RAM and it is pretty much serving the same files all day, but it is doing so from the hard drive while 3GBs of RAM are "free".

Anyone who has ever tried running a ram-drive can witness that It's awesome in terms of speed. The memory usage of this system is usually never higher than 1GB/4GB so I want to know if there is a way to use that extra memory for something good.

  • Is it possible to tell the filesystem to always serve certain files out of RAM?
  • Are there any other methods I can use to improve file reading capabilities by use of RAM?

More specifically, I am not looking for a 'hack' here. I want file system calls to serve the files from RAM without needing to create a ram-drive and copy the files there manually. Or at least a script that does this for me.

Possible applications here are:

  • Web servers with static files that get read alot
  • Application servers with large libraries
  • Desktop computers with too much RAM

Any ideas?

Edit:

  • Found this very informative: The Linux Page Cache and pdflush
  • As Zan pointed out, the memory isn't actually free. What I mean is that it's not being used by applications and I want to control what should be cached in memory.
link|improve this question
feedback

14 Answers

up vote 8 down vote

Linux will cache as much disk IO in memory as it can. This is what the cache and buffer memory stats are. It'll probably do a better job than you will at storing the right things.

However, if you insist in storing your data in memory, you can create a ram drive using either tmpfs or ramfs. The difference is that ramfs will allocate all the memory you ask for, were as tmpfs will only use the memory that your block device is using. My memory is a little rusty, but you should be able to do:

 # mount -t ramfs ram /mnt/ram

or

 # mount -t tmpfs tmp /mnt/tmp

and then copy your data to the directory. Obviously, when you turn the machine off or unmount that partition, your data will be lost.

link|improve this answer
Thanks for your answer, but this is obviously what I want to avoid. Otherwise I'd just script it so the computer would create the ramdrive, copy the files and symbolically link to the ramdrive. But then my data is inconsistent. I was hoping for a filesystem where I can 'tag' certain files to be cached in memory. But maybe I'm a bit too optimistic. – Andrioid Jul 21 '09 at 7:21
2  
You "tag" files to be cached by accessing them. – womble Jul 21 '09 at 7:35
2  
If only there was some way to automatically tag the most commonly used files. – David Pashley Jul 21 '09 at 7:53
depending your memory available, it's done automatically – asdmin Jul 21 '09 at 8:16
@David, it is done automatically. To test it, run: time cat some_large_file > /dev/null. Then run that again. You'll notice that on the second run the time is far less because the file has been cached. – Josh Jul 21 '09 at 12:29
show 3 more comments
feedback
up vote 6 down vote accepted

After some extensive reading on the 2.6 kernel swapping and page-caching features I found 'fcoretools'. Which consists of two tools;

  • fincore: Will reveal how many pages the application has stored in core memory
  • fadvise: Allows you to manipulate the core memory (page-cache).

(In case someone else finds this interesting I'm posting this here)

link|improve this answer
1  
I figured there was a program to do that somewhere. +1 – Brad Gilbert Jul 22 '09 at 3:29
feedback
up vote 3 down vote

A poor man's trick for getting stuff into the filesystem cache is to simply cat it and redirect that to /dev/null.

link|improve this answer
Agree. And if you want to ensure certain files are cached, make a cron job which cats the file to /dev/null periodically – Josh Jul 21 '09 at 12:31
feedback
up vote 1 down vote

I very much doubt that it is actually serving files from the disk with 3 GB RAM free. Linux file caching is very good.

If you are seeing disk IO, I would look into your logging configurations. Many logs get set as unbuffered, in order to guarantee that the latest log information is available in the event of a crash. In systems that have to be fast regardless, use buffered log IO or use a remote log server.

link|improve this answer
Right you are, I just want to control what is being cached. – Andrioid Jul 21 '09 at 7:12
feedback
up vote 1 down vote

If you have plenty of memory you can simply read in the files you want to cache with cat or similar. Linux will then do a good job of keeping it around.

link|improve this answer
feedback
up vote 1 down vote

You may be able to have a program that just mmaps your files then stays running.

link|improve this answer
1  
That is pretty much what 'fadvise' (fcoretools) does, as far as I can tell. – Andrioid Jul 21 '09 at 18:09
feedback
up vote 0 down vote

There are various ramfs systems you can use (eg, ramfs, tmpfs), but in general if files are actually being read that often, they sit in your filesystem cache. If your working set of files is larger than your free ram, then files will be cleared out of it - but if your working set is larger than your free ram, there's no way you'll fit it all into a ramdisk either.

Check the output of the "free" command in a shell - the value in the last column, under "Cached", is how much of your free ram is being used for filesystem cache.

link|improve this answer
feedback
up vote 0 down vote

As for your latter question, ensure that your RAM is sitting on different memory channels so that the processor can fetch the data in parallel.

link|improve this answer
feedback
up vote 0 down vote

I think this might be better solved at the application level. For instance, there are probably specialized web servers for this, or you might consider mod_cache with Apache. If you have a specific goal, such as serving web content faster, then you can get improvements form this sort of thing I think.

But your question is general in nature, the Linux memory subsystem is designed to provide the best general use of RAM. If you want to target certain types of performance, consider looking up everything in /proc/sys/vm .

The fcoretools package is interesting, I'd be interested in any articles about its application... This link talks about the actual system calls used in an application.

link|improve this answer
1  
find /var/lib/mysql | xargs fadvise -willneed (dirty, but it should provide faster access to the database files; as an example) – Andrioid Jul 21 '09 at 18:03
Very good hack, but such hack doesn't disable a lot of waiting fsyncs from mysql :( fsyncs are needed to ensure ACID (Atomicity, Consistency, Isolation, Durability). – osgx Feb 7 '10 at 1:48
feedback
up vote 0 down vote

I too am seeking something along these lines. I don't think that general filesystem disk block caching is the answer.

Suppose that I want disk block X to always be cached. Something accesses it, and the kernel caches it. So far so good, but the next process wants block Y, so the kernel discards my block X and caches Y instead. The next process that wants X will have to wait for it to come off the disk; that's what I want to avoid.

What I would like (and what I think the original poster is after too) is to overlay a write-through cache onto a filesystem that will guarantee the files are always in memory for reading, while making sure that writes are committed to disk.

link|improve this answer
feedback
up vote 0 down vote

Desktop computers (eg. ubuntu) already uses preloading files (at least, popular shared libraries) to memory on boot. It is used to speed up booting and startup time of different bloarware like FF, OO, KDE and GNOME (with evolution bloat-mailer).

The tool is named readahead http://packages.ubuntu.com/dapper/admin/readahead

There is also corresponding syscall: readahead(2) http://linux.die.net/man/2/readahead

There is also project of preloading daemon: http://linux.die.net/man/8/preload

link|improve this answer
feedback
up vote 0 down vote

http://www.coker.com.au/memlockd/ does this

though you really don't need it, linux will do a pretty good job of caching the files you are using on its own.

link|improve this answer
feedback
up vote 0 down vote

i just tried dd if=/dev/yourrootpartition of=/dev/null \ bs=1Mcount=howmuchmemoryyouwanttofill

it does not give me the control that you desire but it at least tries to use wasted memory

link|improve this answer
feedback
up vote 0 down vote

i use find / -name stringofrandomcharacter it helps alot

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.