13

I have a virtual machine (Debian) running on a physical machine host. The virtual machine acts as a buffer for data that it frequently receives over the local network (the period for this data is 0.5s, so a fairly high throughput). Any data received is stored on the virtual machine and repeatedly forwarded to an external server over UDP. Once the external server acknowledges (over UDP) that it has received a data packet, the original data is deleted from the virtual machine and not sent to the external server again. The internet connection that connects the VM and the external server is unreliable, meaning it could be down for days at a time.

The physical machine that hosts the VM gets its power cut several times per day at random. There is no way to tell when this is about to happen and it is not possible to add a UPS, a battery, or a similar solution to the system.

Originally, the data was stored on a file-based HSQLDB database on the virtual machine. However, the frequent power cuts eventually cause the database script file to become corrupted (not at the file system level, i.e. it is readable, but HSQLDB can't make sense of it), which leads to my question:

How should data be stored in an environment where power cuts can and do happen frequently?

One option I can think of is using flat files, saving each packet of data as a file on the file system. This way if a file is corrupted due to loss of power, it can be ignored and the rest of the data remains intact. This poses a few issues however, mainly related to the amount of data likely being stored on the virtual machine. At 0.5s between each piece of data, 1,728,000 files will be generated in 10 days. This at least means using a file system with an increased number of inodes to store this data (the current file system setup ran out of inodes at ~250,000 messages and 30% disk space used). Also, it is hard (not impossible) to manage.

Are there any other options? Are there database engines that run on Debian that would not get corrupted by power cuts? Also, what file system should be used for this? ext3 is what is used at the moment.

The software that runs on the virtual machine is written using Java 6, so hopefully the solution would not be incompatible.

Improve this question
13
  • 17
    "The physical machine that hosts the VM gets its power cut several times per day at random. There is no way to tell when this is about to happen and it is not possible to add a UPS, a battery, or a similar solution to the system." I really want to know how that's possible. Is it in the International Space Station so it requires $20 million to send a UPS up or something?
    – ceejayoz
    Nov 7, 2012 at 17:23
  • 3
    Does the machine at least have a RAID controller with battery backed cache?
    – Zoredache
    Nov 7, 2012 at 17:40
  • 4
    We could recommend very interesting, creative and perhaps theoretically correct solutions to this problem. However, we don't know what hypervisor and hardware is running on the host, so there would be no guarantee that disk writes are really written, or written in the correct order…
    – pino42
    Nov 7, 2012 at 19:39
  • 5
    @Sevas Sounds like it's not your call, but I'd suggest that it's worthwhile to point out that 50 basic, cheap UPSs would cost $2500, and don't need maintenance (you replace them after a couple years when the batteries start to go). The cost of trying to solve this in software is going to be much higher than that, unless you know a bunch of coders who work for free. Might be helpful to getting management to solve this for $50/unit, instead of dozens or hundreds of skilled man-hours @ 3-figures an hour.
    – HopelessN00b
    Nov 7, 2012 at 22:14
  • 11
    This actually sounds like a malicious program. The user does not know the "VM" is running on their computer. It is stealing data from across the whole network - then funneling it out through one connection to hide itself. The user "turns the computer off and on" randomly - so you cant just add a UPS.
    – Laurence
    Nov 8, 2012 at 0:51

4 Answers 4

Reset to default
23

Honestly your best approach here is to either fix the power-cuts, or deploy a different system in a better location.

Yes there are systems such as redis which will store data in an append-only-log for replay, but you risk corruption at lower levels - e.g. if your filesystem is scrambled then the data on disk is potentially at risk.

I appreciate any improvement would be useful to you, but really the problem is not one that can be solved given the scenario you've outlined.

Improve this answer
3
  • 8
    +1 The correct answer is "Don't do that"
    – Chris S
    Nov 7, 2012 at 17:36
  • 6
    +1 Eventually random power cuts will corrupt your filesystem. Electronics do weird unpredictable things as their power fails.
    – Grant
    Nov 7, 2012 at 17:45
  • -1 (virtual -1). I think that such a system must be build on the assumption that power cuts happens from time to time. This assumption is a real world fact that you have to deal with.
    – Igal Serban
    Nov 13, 2012 at 13:12
11

Your approach can work. Let me suggest some enhancements to it. There was a question in stack overflow on atomic writing to file. Essentially you save each packet of data to a temporary file and then you rename it to it's final name. Renaming is an atomic operation that will be safe from power failures. That way you are guaranteed that all your files in your final destination have been saved correctly with no corruption.

Then what you can do to deal with the issue of having millions of files. Is cron a job that runs maybe every hour that takes all the files older then an hour and combines them into one big file using again atomic file operations so that this job runs safely even during a power failures, and then deletes the old files. Kind of like log rotation. An hours worth of files would be around 7,200 files. So at any point in times you shouldn't have more then 20,000 files on disk.

Improve this answer
6
  • 1
    Not a bad answer, but the problem with it is in assuming that the write itself is an atomic operation, which it's not. So a power failure at the wrong time could still create data or FS corruption. Probably about the best option short of fixing the power, or plugging the thing into a UPS, though, so +1.
    – HopelessN00b
    Nov 7, 2012 at 17:52
  • @HopelessN00b renaming will be an atomic operation this is a POSIX requirement
    – Marwan Alsabbagh
    Nov 7, 2012 at 18:15
  • Yes, renaming the file once written is an atomic operation. Writing the file in the first place, is not.
    – HopelessN00b
    Nov 7, 2012 at 19:01
  • 3
    @HopelessN00b It doesn't matter that the new file is half-written or corrupt. You have the old file which is in a good state. When you recover the system you destroy the half-written file.
    – DJClayworth
    Nov 7, 2012 at 21:23
  • 2
    @HopelessN00b Exactly! only temporary files in a temporary directory lets say could ever be half written. All the files in your final destination directory will always be non-corrupt and safely on disk
    – Marwan Alsabbagh
    Nov 8, 2012 at 4:14
7

You install a UPS or a RAID card with a battery-backed write cache to the system, and for as little as $49.95, you accomplish what is simply impossible to accomplish in software alone.

Your claim that it's somehow not possible to hook this server up to a UPS or battery... is simply not believable.

Improve this answer
7
  • 9
    Bureaucratic stupidity is always believable.
    – Dan Is Fiddling By Firelight
    Nov 7, 2012 at 21:16
  • 3
    @DanNeely My PHB won't let me hook this up to a UPS/battery is a very different thing from it is not possible to add a UPS, a battery, or a similar solution to the system. Not to get too pedantic, but it's an important distinction because it changes the approach and solutions available.
    – HopelessN00b
    Nov 7, 2012 at 21:36
  • Or, as mentioned elsewhere, the hijacked computer's user would be surprised if I asked to install a UPS. Situation is a bit unbelievable otherwise. Anyone would, within reason, accept a UPS over corrupted data given the proper business case.
    – WernerCD
    Nov 8, 2012 at 15:03
  • @WernerCD I'd like you to meet our CIO. While I agree that hijacking someone's computer is a possible use-case for this, I can think of legitimate ones as well, so I'll give the guy the benefit of the doubt. Think about embedded systems and controllers, or like a Raspberry Pi - it can definitely be the case that the "computer" you're using is worth less than the $50 it would take to attach it to a UPS.
    – HopelessN00b
    Nov 8, 2012 at 15:08
  • Even if the computer is worth less than the $50 UPS - it's the data on the computer that is actually worth something. Google was built on "worthless" computers. More important than the cost of the CPU is the cost of lost data, lost man-power (This programming adventure, data corruption chasing, bug tracking in the old system as well as this new part), lost customers value (Lost my data? Next company please.), etc.
    – WernerCD
    Nov 8, 2012 at 16:29
5

Mount the entire system read-only, except for a block device that stores all your data. Use that block device directly and implement your own data storage mechanism using that raw block device.

Improve this answer
3
  • 3
    ...and invest in a battery-backed disk controller card, and make sure there's no write-cache on the disk, or you're still screwed.
    – voretaq7
    Nov 7, 2012 at 17:43
  • Came here to say they should be booting off of a Live-CD or equivalent ROM system, with some solid state storage used with the flat file solutions.
    – Mark Allen
    Nov 7, 2012 at 23:55
  • The write cache can be disabled. This approach is viable. Append only Storage mechanism is advised. Blocks are written atomically (i assume) so you can have two "pointer" blocks which point to the start and end of the section with new/todo data. The pointers are updated after writing/finishing data. NCQ should be disabled too.
    – sleeplessnerd
    Mar 19, 2013 at 17:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .