5

I have a Ubuntu 10.04 server with several disks in it. The disks are setup with a union filesystem, which presents them all as one logical /home.

A few days ago, one of the disks appears to have suddenly 'become empty', for lack of better explanation. The amount of data on the /home mount almost halved within minutes - the disk appears to have had just over 400 GB of data prior to 'becoming empty'.

I have absolutely no idea what happened. I was not using the server at the other time, but there are half a dozen other users who may have been (without root access and without the ability to hose a whole disk).

I've ran SMART tests on the disk and it comes back clean. The filesystem checks fine (it has 12 GB used now, as some user software continued downloading after the incident).

All I know is that around around midnight on October 19, the disk usage changed dramatically: http://i.imgur.com/8R6ia.png (sorry, spam filter won't embed it)

The data points are every 15 minutes, and the full loss occured between captures: 2012-10-18 23:58:03.399647 - has 953.97/2059.07 GB [46.33 percent] 2012-10-19 00:13:15.909010 - has 515.18/2059.07 GB [25.02 percent]

Other than that, I have not much to go off :-(

I know that:

  • There's nothing interesting in log files at that time
  • Nobody appeared to be logged in via SSH at the time it occured (most users do not even use SSH)
  • The server was online through whatever occured (3 months uptime)
  • None of the other disks were affected and everything else on the server looks completely normal
  • I have tried using "extundelete" on the disk and it didn't really find anything (some temporary files, but they looked new anyway)

I am completely at a loss to what could have caused this. I was initially thinking maybe root escalation exploit, but even if someone did maliciously "rm" the disk contents, it would take more than 15 minutes for 400 GB?

7
  • I assume that "union filesystem" means unionfs? What are the relevant mount commands?
    – Andrew
    Oct 22, 2012 at 0:09
  • Hi Andrew, it is actually 'mhddfs' using stock mount commands (just allow others and large writes).I am pretty certain the overlying filesystem is not at fault, I have other servers with this same setup that have been online for more than a year without any issues.
    – Ohnomydisk
    Oct 22, 2012 at 1:35
  • Does mhddfs mask SMART? Sounds like the drive just shat its pants and you don't have any kind of redundancy.. (this sounds like JBOD^WSPAN)
    – Grizly
    Oct 30, 2012 at 5:14
  • 1
    Has any user reported a loss? Could it be one user was a hog of space, then decided not to be?
    – Skaperen
    Nov 6, 2012 at 6:19
  • Have you run an fsck on the said disk? SMART may check out OK but you can still have a filesystem error.
    – hookenz
    Jan 6, 2013 at 20:21

2 Answers 2

1

I cannot guess how that happen.

But to "wipe" a disk "clean", the fastest way is deleting the partition and recreating the partition. You can get a "clean" "formatted" EXT4 partition that way in just a few seconds.

If somehow root escalation is used, that is one way to do it.

1
  • The problem description suggests that NOT all data has been deleted, nor has the filesystem been wrecked. I'd du --max-depth=1 /home to get an impression on how space is NOW used, and probably do the same du on a version of the /home tree from a backup prior to that event. Recreating a partition with the same geometry will, btw, usually not purge the filesystem - unless it is the wrong geometry in which case you will not have an empty filesystem but something corrupted or no longer mountable. mkfs with large blocksize/low inode count/... would be fast though :) Dec 6, 2012 at 16:56
1

Restore your backup from the time before the data-loss, and restore last night's backup. Compare the two.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .