We have a particular sub-platform with around 900 web-facing RHEL VMs (on ESXi 4U2 if that helps). Each VM runs a custom application that logs to a per-VM file, we don't want these logs to get very big as we want the VMs to be small. What we've been doing to far is to get each VM to put it's file in a shared NFS mount, each named after the VM.
This has worked a treat, until yesterday when we lost half a shelf in the NetApp box running the NFS server - although it's all RAID'ed and has dual controllers we needed to switch the whole system off to swap the shelf (one aggregate has multiple members per shelf). At which point the VMs can't write their logs and we lost all this logging data.
So to the actual question, what I'd like to do is have the VMs each write to a file local to themselves (i.e. within the .vmdk), then have a process that updates the central NFS copy with any updates to this local file and clears the local file, if the NFS is down this doesn't happen and the local file gets bigger until the NFS is back, at which point it updates the central copy and then clears the local copy. Does that make sense?
Either way I've not been able to find anything like this, I'm pretty sure that rsync can't, have you any ideas?
Oh and we're having the application rewritten to use syslog but it'll take 3-6 months apparently :(