I have a file server with 48 2TB HDDs split across 2x 3ware 9650SE-24M8 RAID cards. Both cards have 24 drives in RAID 1+0. MD softraid is setup in raid 0 with the two RAID 1+0s making RAID 100. LVM2 is running on top of the RAID 100. I have a couple logical volumes formatted with ext4 and XFS being shared by samba. Also the whole volume/volume group is being shared to another server over iSCSI for virtual machine disk images.
Servers been running great for over a year. Sometimes when I would reboot, I would get:
"The disk drive for /mnt/NOVELL is not ready yet or not present Continue to wait: or Press S to skip mounting or M for manual recovery"
But if I skipped it and rebooted the server it would come up fine. I restarted the server a few days ago but now I can't get the partitions to mount. Every time I restart I get the above message, and if I run "mount -a" I get: mount: /dev/mapper/FORTRESS.MD0-NOVELL already mounted or /mnt/NOVELL busy
Servers running Ubuntu 10.04 LTS 64 bit Kernel version: 3.0.0-17-server
LVM looks "OK"
vgdisplay --- Volume group --- VG Name FORTRESS.MD0 System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 15 VG Access read/write VG Status resizable MAX LV 0 Cur LV 5 Open LV 0 Max PV 0 Cur PV 1 Act PV 1 VG Size 43.66 TiB PE Size 4.00 MiB Total PE 11444027 Alloc PE / Size 3895296 / 14.86 TiB Free PE / Size 7548731 / 28.80 TiB VG UUID ehsmLU-Euax-PzQK-0BoH-GCHe-siqA-9QGO0t pvdisplay --- Physical volume --- PV Name /dev/md0 VG Name FORTRESS.MD0 PV Size 43.66 TiB / not usable 0 Allocatable yes PE Size 4.00 MiB Total PE 11444027 Free PE 7548731 Allocated PE 3895296 PV UUID kyJt0B-7vlh-KZJI-pAyX-ATQ6-1TgS-cnn48Y lvdisplay --- Logical volume --- LV Name /dev/FORTRESS.MD0/NOVELL VG Name FORTRESS.MD0 LV UUID PKGyKb-4MNN-jIpL-sA02-xZle-OVtL-oFn3s8 LV Write Access read/write LV Status suspended # open 0 LV Size 9.77 TiB Current LE 2560000 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 252:0
MD also looks "OK"
mdadm --misc -D /dev/md0 /dev/md0: Version : 00.90 Creation Time : Tue Feb 22 14:33:49 2011 Raid Level : raid0 Array Size : 8589933568 (8192.00 GiB 8796.09 GB) Raid Devices : 2 Total Devices : 2 Preferred Minor : 0 Persistence : Superblock is persistent Update Time : Tue Feb 22 14:33:49 2011 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Chunk Size : 512K UUID : 5640da17:2b94f7b6:25fcb762:85538530 (local to host FORTRESS) Events : 0.1 Number Major Minor RaidDevice State 0 8 17 0 active sync /dev/sdb1 1 8 33 1 active sync /dev/sdc1 cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid0 sdb1 sdc1 8589933568 blocks 512k chunks unused devices: <none> mdadm.conf: # mdadm.conf # # Please refer to mdadm.conf(5) for information about this file. # # by default, scan all partitions (/proc/partitions) for MD superblocks. # alternatively, specify devices to scan, using wildcards if desired. DEVICE partitions # auto-create devices with Debian standard permissions CREATE owner=root group=disk mode=0660 auto=yes # automatically tag new arrays as belonging to the local system HOMEHOST <system> # instruct the monitoring daemon where to send mail alerts MAILADDR root # definitions of existing MD arrays # This file was auto-generated on Sat, 08 Jan 2011 21:54:08 -0800 # by mkconf $Id$ ARRAY /dev/md0 level=raid0 num-devices=2 UUID=5640da17:2b94f7b6:25fcb762:85538530
I have looked through the log files and this is the most interesting thing I have found so far (but this might be "normal" and nothing new):
Apr 25 15:17:43 FORTRESS kernel: [ 5.443935] device-mapper: table: 252:0: md0 too small for target: start=2048, len=20971520000, dev_size=17179867136 Apr 25 15:17:43 FORTRESS kernel: [ 5.459123] device-mapper: table: 252:1: md0 too small for target: start=25266489344, len=67108864, dev_size=17179867136 Apr 25 15:17:43 FORTRESS kernel: [ 5.475125] device-mapper: table: 252:2: md0 too small for target: start=25333598208, len=67108864, dev_size=17179867136 Apr 25 15:17:43 FORTRESS kernel: [ 5.491027] device-mapper: table: 252:3: md0 too small for target: start=25400707072, len=10737418240, dev_size=17179867136 Apr 25 15:17:43 FORTRESS kernel: [ 5.506932] device-mapper: table: 252:4: md0 too small for target: start=36138125312, len=67108864, dev_size=17179867136
I am having trouble identifying what layer the problem is on. MD/LVM/FS etc.