2

I need to backup data from a web server with 10GB of RAM (Ubuntu). To backup data I use Rsync.

My first test was to use one rsync for all website, but it uses 5GB. So I decided to split rsync but it still uses 5GB. In both of theses cases, it consume to much memory and are not always completely free after the process

If I manualy drop the cache between rsync, It only uses between 150Mb and 500Mb completely free the memory after the process. (And free 50%-70% memory of the server)

for CURDIR in "$ROOTDIR"*
do
    echo "Start $CURDIR"    
    rsync -aHh --stats --compress --delete "$CURDIR" --link-dest="saveofyesterday" "saveoftoday"    
    echo "Clear"
    sync
    echo 3 > /proc/sys/vm/drop_caches
    echo "Finish $CURDIR."
done

It's seems to be not recommend. What is the proper way to achieve this ?

  • 2
    What is the real problem? By playing devil's advocate, I can say that you still have 5 GB free. As Hrvoje Špoljar wrote, In reality, that memory you consider used, it is used but it is used for disk cache and will be released if needed by programs. So it looks like you are worried by the RAM usage but if this is only a cache it won't affect your server, all your programs will be able to run. – A.L Mar 8 '16 at 15:05
  • Yes, you're right. I didn't noticed the difference between free and available. – Inglebard Mar 8 '16 at 16:40
  • obligatory: linuxatemyram.com – Corey Goldberg Oct 14 '17 at 21:21
5

I'd suggest looking into fadvise patch for rsync. Purpose is to preserve disk cache while backing up. Effect you should see this memory you consider as 'used' will not be 'used'. In reality, that memory you consider used, it is used but it is used for disk cache and will be released if needed by programs. If system is low on memory eventually all disk cache will be evicted from memory in order to try and provide memory to needy apps. This is bad, especially on busy systems which don't have extremely fast disk sub system because once disk cache is evicted from memory all that once cached data will need to be read from disk again which will cause huge IO load and wait times.

Dropping cache manually is just as bad and will get your server in tough spot. This is not advisable route to go. Memory management is not something your script should worry about, kernel takes care of that part.

1

It sounds like what you want to do is run a process and limit how much effect it can have on the file system cache, so that it doesn't create a "noisy neighbor" problem.

You can put rsync in a container and limit the memory size. This is basically like fadvise on steroids.

Creating such a container is complex, luckily there is a tool that will do the work for you called LMCTFY: https://github.com/google/lmctfy

With lmctfy, you can create a container with a memory limit like this:

lmctfy create mycontainer "memory:{limit:100000000}"
lmctfy run mycontainer "rsync ..."
lmctfy destroy mycontainer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.