20

In mid-November, a VPS that I am renting from a hosting company stopped responding. When I contacted support, they explained that a power outage in the datacenter caused a forced reboot and fsck. Eventually, I asked why it was taking so long, and was told that the size of the volume is 30 TB. The last time I received an update was in February, and they have not responded to my most recent inquiry.

I understand that fsck can be very slow for some file systems, but is it possible for fsck to take 6 months on a 30 TB volume, or should I assume that this hosting company is lying to me so that I continue to pay my bill every month?

Improve this question
8
  • 39
    They were probably lying to you from the start. I would expect that to take hours. You should have stopped paying in December.
    – Michael Hampton
    May 7, 2019 at 19:05
  • 15
    Even if they're not lying, choosing a HW + software setup that could require an FSCK that long shows they're incompetent. And whatever the reason, they're not providing the service you're paying for.
    – Peter Cordes
    May 8, 2019 at 9:45
  • 35
    Sounds like a real cluster fsck!
    – JMK
    May 8, 2019 at 11:40
  • 2
    @JMK Now I wish there was a way to flag comments for extra merit, maybe add to a hall-of-fame.
    – pipe
    May 8, 2019 at 11:49
  • 2
    What @PeterCordes says is the key point. You're paying for a service. You're really sorry to hear that they're having problems but you're calling about the service that you're paying for and not getting.
    – Rob Moir
    May 8, 2019 at 14:19

3 Answers 3

Reset to default
33

fsck speed mainly depends on the number of files and how they are spread in the respective directory. That said, 6 month for a fsck is absolutely absurd: it should had completed in some hours at most, especially if using xfs which has the speedy xfs_repair utility. Here you can find some fsck run at a scale - all completed under one hour (3600s). So, it is not possible that your fsck is still running.

Anyway, an unexpected power loss will not cause a full-blow fsck, rather only a very fast (some seconds) journal replay. However, if some key files was damaged, the OS can be unbootable.

But they probably just lied to you. You should stop paying immediately, ask for an explanation and apply for a total refund.

Improve this answer
3
  • 8
    If they're using ext2, then a power failure will require a full fsck, and I wouldn't be surprised if it takes days on a heavily-used 30TB volume. On the other hand, if they're using ext2 on a 30TB volume, that in and of itself is a reason to look elsewhere for hosting services.
    – Mark
    May 7, 2019 at 21:54
  • 14
    ext2 uses 32-bit block counter, with a maximum block size of 4096 bytes (ie: a page) on x86 and x86_64. This means ext2 (and ext3) are limited to 8TB volumes so no, the OP can not be using ext2/3. Anyway, using any non-journaled filesystem on a 30 TB volume would be absolutely insane.
    – shodanshok
    May 7, 2019 at 22:05
  • I think ext4 fsck might be little better if one has a 30Tb FS containing vast numbers of tiny files. Lunacy to create that, so still a reason to look elsewhere.
    – nigel222
    May 9, 2019 at 10:51
8

Conjecture: Their system uses a BBU/FBWC-less RAID (or even software RAID) with all possible write caches (including these in the hard drives themselves) set at their most aggressive settings, in order to get maximum performance for minimal cost. A hard power outage on such a setup can leave a journaling filesystem in a condition where the journal cannot be trusted and cannot be used for recovery. The problem is that such a system aggressively reorders and postpones writes, which means that a journal entry can be written with the effect of the data action being lost ... or the journal entry being lost on a data action that was consequential.

Recovering such a system from a worst case outage can mean that you have to do a "slow" fsck/repair that actually examines all the filesystem structures as they are, which could indeed take a day or two for 30TB.... and it is not unlikely that you will have to run multiple repair cycles. Add to that that personnel might not be always available to monitor this, you could easily be down to one fsck being done per week. They probably gave up and forgot.

Improve this answer
1

For most filesystems it will be much faster, even when there are errors, as normally only the metadata is checked.

In the worst case, it may read the whole disk, (e.g. something like fsck.ext4 -cc /dev/sda, which does a non-destructive write test on every block), that could take a few days for 30 TB. If you know the speed of the drives, you can calculate size/speed. For a consumer hard drive with about 100 MB/s copying a few TB can take more hours than most people would expect.

If it were your server, you could have the problem that it boots then hangs when fsck asks you if you want to fix an error. But the datacenter admin won't leave a fsck hanging for 6 month while all VPS are offline.

So they are either lying to you, or there is a huge misunderstanding. Or they were running fsck some time ago and did not update you about the new problem after it finished.

Improve this answer
5
  • 4
    fsck traverses all the filesystem structures, which mostly means executing random i/o. So the above calculation, based on sequential transfer rate, is not very useful.
    – shodanshok
    May 8, 2019 at 12:40
  • @shodanshok indeed file structure is irrelevant in a general drive check, as I've just explained in my answer.
    – Overmind
    May 8, 2019 at 12:55
  • @shodanshok my worst-case assumption was based on a very extensive fsck. For example the typical xfs fsck doesn't do much. ext2 has a long running extensive check and old MS-DOS scandisk had a read-write test on each hard drive block when running it in full mode. So you have an upper bound at the size of the disk.
    – allo
    May 8, 2019 at 13:20
  • 1
    @Overmind And you answer is irrelevant to the question which is about fsck and not a general drive check.
    – BlackJack
    May 8, 2019 at 13:54
  • Please be aware that taking typical disk throughput as an indicator can be misleading. I've done the math when once re-syncing an array, which should (in my opinion) have taken less than a day, and it took over two weeks! Seeks are the one dominating factor for total time and even when you think that you're doing a strictly sequential operation, it sometimes isn't one. Now fsck is strictly non-sequential, so... no way can you judge from usual disk throughput to the length of the operation (still, months is ridiculous... it's an obvious lie).
    – Damon
    May 9, 2019 at 8:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .