-3

I've recently faced a huge data loss (everything from last 5 months) after one of my disk crashed, even if the RAID array is still fine.

This disk was a SSD with my root partition (ext4), mirrored in a mdadm RAID 1 with a fake hardware raid RAID 1 of two HDD. (see below)

  [RAID_1]
      |
  +---+---+
  |       | 
[SSD]  [RAID_1]
          |
       +--+--+
       |     |
     [HDD] [HDD]

(write-mostly was used to speed up read IO)

It's a strange configuration but I don't think that the problem comes from there.

When the SSD failed I noticed it because the server didn't answer anymore on ssh, so I've plug a monitor and there was a lot of disk failure. I've tried to reboot the server and it didn't start again because the boot partition was only on the SSD (my bad..). So I've used a live USB to install a boot directory on the root partition, mounting the RAID 1 array wasn't a problem, reboot with an USB GRUB key, and everything seems to work.

But, it didn't take me long to notice that some files was on a really old state (~ 30 oct 2018, when it was 10 apr 2019). And it seems that all the root partition have been reverted to the end of October.

The log files confirm it :

# journalctl --list-boots
-19 8ee3d21dfdd8447b9b13d02b939f0a57 Tue 2018-07-24 15:45:02 CEST—Fri 2018-09-14 12:45:38 CEST
...
-10 8738ed87a849441dbeffd8571d9ebae5 Sun 2018-10-28 19:32:09 CET—Tue 2018-10-30 16:52:39 CET
 -9 b21291a3607b4b4ba42e8d99ec4b2b40 Wed 2019-04-10 17:27:55 CEST—Thu 2019-04-11 14:19:19 CEST
 -8 fc347339334a465c91d3807d2ca06ee0 Thu 2019-04-11 14:23:41 CEST—Thu 2019-04-11 14:39:21 CEST
 -7 a59cf07431844cecaefb58de81737957 Thu 2019-04-11 14:41:45 CEST—Thu 2019-04-11 15:36:56 CEST
...
  0 e29340c8edc44b79863634a790968a93 Thu 2019-05-02 18:05:29 CEST—Mon 2019-05-13 16:57:19 CEST

# journalctl -n 500000
...
oct. 30 16:52:31 kxkm-dev systemd[1]: Hardware watchdog 'INTCAMT', version 0
oct. 30 16:52:31 kxkm-dev systemd[1]: Set hardware watchdog to 10min.
oct. 30 16:52:39 kxkm-dev systemd-shutdown[1]: Sending SIGTERM to remaining processes...
oct. 30 16:52:39 kxkm-dev systemd-journal[323]: Journal stopped
-- Reboot --
avril 10 17:27:55 kxkm-dev systemd-journald[16674]: Missed 22960 kernel messages
avril 10 17:27:55 kxkm-dev kernel: usb 3-1: new low-speed USB device number 21 using xhci_hcd
avril 10 17:27:55 kxkm-dev kernel: usb 3-1: New USB device found, idVendor=0461, idProduct=4e22
avril 10 17:27:55 kxkm-dev kernel: usb 3-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
...
# journalctl -b -9 --dmesg 
-- Logs begin at Tue 2018-07-24 15:45:02 CEST, end at Mon 2019-05-13 18:55:21 CEST. --
avril 10 17:27:55 kxkm-dev systemd-journald[16674]: Missed 22960 kernel messages
avril 10 17:27:55 kxkm-dev kernel: usb 3-1: new low-speed USB device number 21 using xhci_hcd
avril 10 17:27:55 kxkm-dev kernel: usb 3-1: New USB device found, idVendor=0461, idProduct=4e22
avril 10 17:27:55 kxkm-dev kernel: usb 3-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
avril 10 17:27:55 kxkm-dev kernel: md: md0 stopped.
avril 10 17:27:55 kxkm-dev kernel:  md126:
avril 10 17:27:55 kxkm-dev kernel: md: md126 does not have a valid v1.2 superblock, not importing!
avril 10 17:27:55 kxkm-dev kernel:  md126:
avril 10 17:27:55 kxkm-dev kernel: md: md_import_device returned -22
avril 10 17:27:55 kxkm-dev kernel: md: md0 stopped.
avril 10 17:27:55 kxkm-dev kernel: md: md126 does not have a valid v1.2 superblock, not importing!
avril 10 17:27:55 kxkm-dev kernel: md: md_import_device returned -22
avril 10 17:27:55 kxkm-dev kernel: md: md0 stopped.
avril 10 17:27:55 kxkm-dev kernel: md: md126 does not have a valid v1.2 superblock, not importing!
avril 10 17:27:55 kxkm-dev kernel: md: md_import_device returned -22
avril 10 17:27:55 kxkm-dev kernel: md: md0 stopped.
avril 10 17:27:55 kxkm-dev kernel: md: md126 does not have a valid v1.2 superblock, not importing!
avril 10 17:27:55 kxkm-dev kernel: md: md_import_device returned -22
avril 10 17:27:55 kxkm-dev kernel: md: md0 stopped.
avril 10 17:27:55 kxkm-dev kernel: md: md126 does not have a valid v1.2 superblock, not importing!
avril 10 17:27:55 kxkm-dev kernel: md: md_import_device returned -22
avril 10 17:27:55 kxkm-dev kernel: md: md0 stopped.
avril 10 17:27:55 kxkm-dev kernel: md: md0 stopped.
avril 10 17:27:55 kxkm-dev kernel:  md126:
avril 10 17:27:55 kxkm-dev kernel: md: bind<md126>
avril 10 17:27:55 kxkm-dev kernel:  md126:
avril 10 17:27:55 kxkm-dev kernel: md/raid1:md0: active with 1 out of 1 mirrors
avril 10 17:27:55 kxkm-dev kernel: created bitmap (2 pages) for device md0
avril 10 17:27:55 kxkm-dev kernel: md0: bitmap initialized from disk: read 1 pages, set 81 of 3319 bits
avril 10 17:27:55 kxkm-dev kernel: md0: detected capacity change from 0 to 222722785280
avril 10 17:27:55 kxkm-dev kernel: EXT4-fs (md0): mounted filesystem with ordered data mode. Opts: (null)
avril 10 17:27:55 kxkm-dev kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
avril 10 17:27:55 kxkm-dev systemd[1]: systemd 232 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
avril 10 17:27:55 kxkm-dev systemd[1]: Detected architecture x86-64.
avril 10 17:27:55 kxkm-dev systemd[1]: Set hostname to <kxkm-dev>.
avril 10 17:27:55 kxkm-dev systemd[1]: [/etc/systemd/system/docker.service.d/execWithDeviceMapper.conf:4] Missing '='.
avril 10 17:27:55 kxkm-dev systemd[1]: Listening on Journal Socket.
avril 10 17:27:55 kxkm-dev systemd[1]: Listening on LVM2 poll daemon socket.
avril 10 17:27:55 kxkm-dev systemd[1]: Listening on Device-mapper event daemon FIFOs.
avril 10 17:27:55 kxkm-dev systemd[1]: Created slice User and Session Slice.
avril 10 17:27:55 kxkm-dev systemd[1]: Listening on fsck to fsckd communication Socket.
avril 10 17:27:55 kxkm-dev systemd[1]: Listening on RPCbind Server Activation Socket.
avril 10 17:27:55 kxkm-dev kernel: EXT4-fs (md0): re-mounted. Opts: errors=remount-ro
avril 10 17:27:56 kxkm-dev kernel: RPC: Registered named UNIX socket transport module.
avril 10 17:27:56 kxkm-dev kernel: RPC: Registered udp transport module.
avril 10 17:27:56 kxkm-dev kernel: RPC: Registered tcp transport module.
avril 10 17:27:56 kxkm-dev kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
avril 10 17:27:56 kxkm-dev kernel: lp: driver loaded but no devices found
avril 10 17:28:06 kxkm-dev kernel: ppdev: user-space parallel port driver
avril 10 17:28:06 kxkm-dev kernel: parport_pc 00:05: reported by Plug and Play ACPI
avril 10 17:28:06 kxkm-dev kernel: parport0: PC-style at 0x378 (0x778), irq 5 [PCSPP,TRISTATE,EPP]
avril 10 17:28:06 kxkm-dev kernel: lp0: using parport0 (interrupt-driven).
avril 10 17:28:06 kxkm-dev kernel: Loading iSCSI transport class v2.0-870.
avril 10 17:28:06 kxkm-dev kernel: iscsi: registered transport (tcp)
avril 10 17:28:06 kxkm-dev kernel: iscsi: registered transport (iser)
avril 10 17:28:06 kxkm-dev kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
avril 10 17:28:06 kxkm-dev kernel: sd 0:0:0:0: Attached scsi generic sg0 type 0
avril 10 17:28:06 kxkm-dev kernel: sd 1:0:0:0: Attached scsi generic sg1 type 0
avril 10 17:28:06 kxkm-dev kernel: sd 6:0:0:0: Attached scsi generic sg2 type 0
avril 10 17:28:06 kxkm-dev kernel: sd 7:0:0:0: Attached scsi generic sg3 type 0
avril 10 17:28:06 kxkm-dev kernel: RAPL PMU: API unit is 2^-32 Joules, 4 fixed counters, 655360 ms ovfl timer
avril 10 17:28:06 kxkm-dev kernel: RAPL PMU: hw unit of domain pp0-core 2^-14 Joules
avril 10 17:28:06 kxkm-dev kernel: RAPL PMU: hw unit of domain package 2^-14 Joules
avril 10 17:28:06 kxkm-dev kernel: RAPL PMU: hw unit of domain dram 2^-14 Joules
avril 10 17:28:06 kxkm-dev kernel: RAPL PMU: hw unit of domain pp1-gpu 2^-14 Joules
avril 10 17:28:06 kxkm-dev kernel: snd_hda_intel 0000:00:03.0: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915])
avril 10 17:28:06 kxkm-dev kernel: snd_hda_codec_realtek hdaudioC1D0: autoconfig for ALC892: line_outs=3 (0x14/0x15/0x16/0x0/0x0) type:line
avril 10 17:28:06 kxkm-dev kernel: snd_hda_codec_realtek hdaudioC1D0:    speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)
avril 10 17:28:06 kxkm-dev kernel: snd_hda_codec_realtek hdaudioC1D0:    hp_outs=1 (0x1b/0x0/0x0/0x0/0x0)
avril 10 17:28:06 kxkm-dev kernel: snd_hda_codec_realtek hdaudioC1D0:    mono: mono_out=0x0
avril 10 17:28:06 kxkm-dev kernel: snd_hda_codec_realtek hdaudioC1D0:    dig-out=0x1e/0x0
avril 10 17:28:06 kxkm-dev kernel: snd_hda_codec_realtek hdaudioC1D0:    inputs:
avril 10 17:28:06 kxkm-dev kernel: snd_hda_codec_realtek hdaudioC1D0:      Front Mic=0x19
avril 10 17:28:06 kxkm-dev kernel: snd_hda_codec_realtek hdaudioC1D0:      Rear Mic=0x18
avril 10 17:28:06 kxkm-dev kernel: snd_hda_codec_realtek hdaudioC1D0:      Line=0x1a
avril 10 17:28:06 kxkm-dev kernel: input: HDA Intel HDMI HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:03.0/sound/card0/input2816
avril 10 17:28:06 kxkm-dev kernel: input: HDA Intel HDMI HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:03.0/sound/card0/input2817
avril 10 17:28:06 kxkm-dev kernel: input: HDA Intel HDMI HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:03.0/sound/card0/input2818
avril 10 17:28:06 kxkm-dev kernel: input: HDA Digital PCBeep as /devices/pci0000:00/0000:00:1b.0/sound/card1/input2815
avril 10 17:28:06 kxkm-dev kernel: input: HDA Intel PCH Rear Mic as /devices/pci0000:00/0000:00:1b.0/sound/card1/input2819
avril 10 17:28:06 kxkm-dev kernel: input: HDA Intel PCH Line as /devices/pci0000:00/0000:00:1b.0/sound/card1/input2820
avril 10 17:28:06 kxkm-dev kernel: input: HDA Intel PCH Line Out Front as /devices/pci0000:00/0000:00:1b.0/sound/card1/input2821
avril 10 17:28:06 kxkm-dev kernel: input: HDA Intel PCH Line Out Surround as /devices/pci0000:00/0000:00:1b.0/sound/card1/input2822
avril 10 17:28:06 kxkm-dev kernel: input: HDA Intel PCH Line Out CLFE as /devices/pci0000:00/0000:00:1b.0/sound/card1/input2823
avril 10 17:28:06 kxkm-dev kernel: input: PC Speaker as /devices/platform/pcspkr/input/input2824
avril 10 17:28:06 kxkm-dev kernel: intel_rapl: Found RAPL domain package
avril 10 17:28:06 kxkm-dev kernel: intel_rapl: Found RAPL domain core
avril 10 17:28:06 kxkm-dev kernel: intel_rapl: Found RAPL domain uncore
avril 10 17:28:06 kxkm-dev kernel: intel_rapl: Found RAPL domain dram
avril 10 17:28:06 kxkm-dev kernel: iTCO_vendor_support: vendor-support=0
avril 10 17:28:06 kxkm-dev kernel: device-mapper: table: 253:0: mirror: Device lookup failure
avril 10 17:28:06 kxkm-dev kernel: device-mapper: ioctl: error adding target to table
avril 10 17:28:06 kxkm-dev kernel: iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11
avril 10 17:28:06 kxkm-dev kernel: iTCO_wdt: Found a 9 Series TCO device (Version=2, TCOBASE=0x1860)
avril 10 17:28:06 kxkm-dev kernel: iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
avril 10 17:28:06 kxkm-dev kernel: device-mapper: table: 253:0: mirror: Device lookup failure
avril 10 17:28:06 kxkm-dev kernel: device-mapper: ioctl: error adding target to table
avril 10 17:28:06 kxkm-dev kernel: device-mapper: table: 253:1: mirror: Device lookup failure
avril 10 17:28:06 kxkm-dev kernel: device-mapper: ioctl: error adding target to table
avril 10 17:28:06 kxkm-dev kernel: device-mapper: table: 253:0: mirror: Device lookup failure
avril 10 17:28:06 kxkm-dev kernel: device-mapper: ioctl: error adding target to table
avril 10 17:28:06 kxkm-dev kernel: EXT4-fs (md125): mounted filesystem with ordered data mode. Opts: (null)
avril 10 17:28:06 kxkm-dev kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
avril 10 17:28:06 kxkm-dev kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
avril 10 17:28:07 kxkm-dev kernel: e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
avril 10 17:28:07 kxkm-dev kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
avril 10 17:28:14 kxkm-dev kernel: fuse init (API version 7.26)
avril 10 17:28:14 kxkm-dev kernel: FS-Cache: Loaded
avril 10 17:28:14 kxkm-dev kernel: Key type dns_resolver registered
avril 10 17:28:14 kxkm-dev kernel: FS-Cache: Netfs 'cifs' registered for caching
avril 10 17:28:14 kxkm-dev kernel: Key type cifs.spnego registered
avril 10 17:28:14 kxkm-dev kernel: Key type cifs.idmap registered
avril 10 17:28:20 kxkm-dev kernel: CIFS VFS: Error connecting to socket. Aborting operation.
avril 10 17:28:20 kxkm-dev kernel: CIFS VFS: Error connecting to socket. Aborting operation.
avril 10 17:28:20 kxkm-dev kernel: CIFS VFS: cifs_mount failed w/return code = -113
avril 10 17:28:20 kxkm-dev kernel: CIFS VFS: cifs_mount failed w/return code = -113
avril 10 17:29:27 kxkm-dev kernel: Netfilter messages via NETLINK v0.30.
avril 10 17:29:27 kxkm-dev kernel: nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
avril 10 17:29:27 kxkm-dev kernel: ctnetlink v0.93: registering with nfnetlink.
avril 10 17:29:28 kxkm-dev kernel: tun: Universal TUN/TAP device driver, 1.6
avril 10 17:29:28 kxkm-dev kernel: tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
avril 10 17:29:31 kxkm-dev kernel: aufs: loading out-of-tree module taints kernel.
avril 10 17:29:31 kxkm-dev kernel: aufs 4.9-20161219
avril 10 17:29:32 kxkm-dev kernel: e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
avril 10 17:29:32 kxkm-dev kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
avril 10 17:29:37 kxkm-dev kernel: ip6_tables: (C) 2000-2006 Netfilter Core Team
avril 10 17:29:37 kxkm-dev kernel: Ebtables v2.0 registered
avril 10 17:29:51 kxkm-dev kernel: bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
avril 10 17:29:51 kxkm-dev kernel: Bridge firewalling registered
avril 10 17:29:51 kxkm-dev kernel: Initializing XFRM netlink socket

Same thing with my nginx logs

# zmore error.log.3.gz
2018/10/28 19:27:46 [error] 2570#0: *169 connect() failed (111: Connection refused) while connecting to upstream, client: 10.2.0.242, server: , request: "GET /node/metrics HTTP/1.0", upstream: "http://[::1]:8000/metrics", host: "kxkm-dev"
2018/10/30 16:46:31 [notice] 30592#30592: using inherited sockets from "8;9;10;11;12;13;"
2019/04/10 17:30:02 [emerg] 19588#19588: BIO_new_file("/var/lib/acme/live/***/fullchain") failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/var/lib/acme/live/***/fullchain','r') error:2006D080:BIO routines:BIO_new_file:no such file)

Here is the state of the root partition manage by mdadm :

# mdadm -D /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Mon Apr 18 14:22:03 2016
     Raid Level : raid1
     Array Size : 217502720 (207.43 GiB 222.72 GB)
  Used Dev Size : 217502720 (207.43 GiB 222.72 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Mon May 13 18:42:23 2019
          State : clean 
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           Name : debian:0
           UUID : 57cd1ad8:b3979a16:2900c1b1:7dfa7649
         Events : 16011

    Number   Major   Minor   RaidDevice State
       1       9      126        0      active sync writemostly   /dev/md/System
       3       8       34        1      active sync   /dev/sdc2

# mdadm -D /dev/md/System
/dev/md/System:
      Container : /dev/md/imsm0, member 1
     Raid Level : raid1
     Array Size : 254816256 (243.01 GiB 260.93 GB)
  Used Dev Size : 254816388 (243.01 GiB 260.93 GB)
   Raid Devices : 2
  Total Devices : 2

          State : active 
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0


           UUID : 7bf5b497:78188413:e395ae40:aa31e7b9
    Number   Major   Minor   RaidDevice State
       1       8        0        0      active sync   /dev/sda
       0       8       16        1      active sync   /dev/sdb

Okay, so it could be a filesystem problem where ext4 trying to recover a usable state wipe a lot of data, but it seems too big to be that simple, isn't it ? But in an other hand I can't see any other explanation, RAID 1 array reconstruct gone well, and I've haven't noticed any major issue with the RAID configuration..

Any ideas ?

PS : It's mainly a development server, so there wasn't much backup, and I known that RAID isn't a backup, but I still try to understand what could have happened.

closed as off-topic by Chopper3, Wesley, Greg Askew, Ward May 14 at 3:51

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions should demonstrate reasonable business information technology management practices. Questions that relate to unsupported hardware or software platforms or unmaintained environments may not be suitable for Server Fault - see the help center." – Chopper3, Wesley, Greg Askew, Ward
If this question can be reworded to fit the rules in the help center, please edit the question.

  • The log files you posted do not show drive failures nor the status of the md array. Serious forensics would be required to get to root cause, SMART diagnostics or dmesg log lines or an image of the failed drive may help. – John Mahowald May 13 at 15:48
  • I've add mdadm status of the RAID, but I can't get SMART diagnostics because the disk can't be read anymore and it doesn't appear in /dev/ if I try to plug it in an other computer. dmesg of the first session after RAID outrage seems to be partial because of : systemd-journald[16674]: Missed 22960 kernel messages – Olivier May 13 at 17:00
  • Most likely your strange RAID broke five months ago, and you have only noticed today. – Michael Hampton May 13 at 18:12
2

If your top level pair was actually a mirror then ALL disks would be identical, meaning the boot sector would exist on both the SSD and the two HDDs. The clue that your boot sector was missing is a huge one. The proper way to set up 3 disks in a mirror would be all at the top level. There is nothing gained by making a set and then using the resultant disk as a member in another set unless you want to use a different RAID level like 1+0 or 0+1. You've added two levels of RAID for no reason. I must assume that a mistake was made during your attempt to nest RAID levels. The HDDs hold the old data that existed before you broke the array. After you broke the array you must have been working off of the SSD.

You should start over and use ZFS if you want to use software RAID.

1

Without getting too deep into your post, I see "fake raid" and "ssd raid" both of which I take to be mistakes. Fake raid is invariably a failure waiting to happen. SSD raid doesn't allow trim as far as I know.

Not the answer you're looking for? Browse other questions tagged or ask your own question.