2

Please bare in mind I am very noob and have not much idea what I am doing. We have a Dell PowerEdge 420 with a 716iS iSCSI raid. The network connect dropped in the middle of disk usage and we are no longer able to mount. I tried xfs_check /dev/sdf

Phase 1 - find and verify superblock...
bad primary superblock - bad magic number !!!

attempting to find secondary superblock...
.....<> .....
found candidate secondary superblock
unable to verify superblock continuing...
.....<> .....

But it is never able to find a valid superblock. Similarly:

xfs_repair -n /dev/sdf
Phase 1 - find and verify superblock...
bad primary superblock - bad magic number !!!

attempting to find secondary superblock...

This does not find a valid superblock.

What can I do to recover this file system?

edit: The util fdisk doesn't support GPT. Use GNU Parted. Here is the parted: parted -l /dev/sdf

  Model: DELL PERC H310 (scsi)
  Disk /dev/sda: 500GB
  Sector size (logical/physical): 512B/512B
  Partition Table: msdos

  Number  Start   End    Size   Type     File system  Flags
   1      1049kB  525MB  524MB  primary  ext4         boot
   2      525MB   500GB  499GB  primary               lvm


  Model: JetStor JetStor -VOL1 - (scsi)
  Disk /dev/sdb: 39.0TB
  Sector size (logical/physical): 512B/512B
  Partition Table: gpt

  Number  Start   End     Size    File system  Name     Flags
   1      1049kB  39.0TB  39.0TB               primary


  Model: JetStor JetStor -VOL2 (scsi)
  Disk /dev/sdc: 52.0TB
  Sector size (logical/physical): 512B/512B
  Partition Table: gpt

  Number  Start   End     Size    File system  Name     Flags
   1      1049kB  52.0TB  52.0TB               primary


  Model: JetStor JetStor -VOL1 - (scsi)
  Disk /dev/sdd: 39.0TB
  Sector size (logical/physical): 512B/512B
  Partition Table: gpt

  Number  Start   End     Size    File system  Name     Flags
   1      1049kB  39.0TB  39.0TB               primary


  Model: JetStor JetStor -VOL2 (scsi)
  Disk /dev/sde: 52.0TB
  Sector size (logical/physical): 512B/512B
  Partition Table: gpt

  Number  Start   End     Size    File system  Name     Flags
   1      1049kB  52.0TB  52.0TB               primary


  Model: AC&Ncorp JetStor (scsi)
  Disk /dev/sdf: 52.0TB
  Sector size (logical/physical): 512B/512B
  Partition Table: gpt

  Number  Start   End     Size    File system  Name     Flags
   1      1049kB  52.0TB  52.0TB               primary


  Error: /dev/mapper/mpathcp1: unrecognised disk label

  Model: Linux device-mapper (multipath) (dm)
  Disk /dev/mapper/mpathc: 52.0TB
  Sector size (logical/physical): 512B/512B
  Partition Table: gpt

  Number  Start   End     Size    File system  Name     Flags
   1      1049kB  52.0TB  52.0TB               primary


  Model: Linux device-mapper (linear) (dm)
  Disk /dev/mapper/vg_cfile-lv_var: 105GB
  Sector size (logical/physical): 512B/512B
  Partition Table: loop

  Number  Start  End    Size   File system  Flags
   1      0.00B  105GB  105GB  ext4


  Model: Linux device-mapper (linear) (dm)
  Disk /dev/mapper/vg_jet716s_vol2-lv_jet716sV2a: 37.7TB
  Sector size (logical/physical): 512B/512B
  Partition Table: loop

  Number  Start  End     Size    File system  Flags
   1      0.00B  37.7TB  37.7TB  xfs


  Model: Linux device-mapper (linear) (dm)
  Disk /dev/mapper/vg_jet716s-lv_jet716s_jet: 39.0TB
  Sector size (logical/physical): 512B/512B
  Partition Table: loop

  Number  Start  End     Size    File system  Flags
   1      0.00B  39.0TB  39.0TB  xfs


  Error: /dev/mapper/mpathbp1: unrecognised disk label

  Error: /dev/mapper/mpathdp1: unrecognised disk label

  Model: Linux device-mapper (multipath) (dm)
  Disk /dev/mapper/mpathb: 39.0TB
  Sector size (logical/physical): 512B/512B
  Partition Table: gpt

  Number  Start   End     Size    File system  Name     Flags
   1      1049kB  39.0TB  39.0TB               primary


  Model: Linux device-mapper (multipath) (dm)
  Disk /dev/mapper/mpathd: 52.0TB
  Sector size (logical/physical): 512B/512B
  Partition Table: gpt

  Number  Start   End     Size    File system  Name     Flags
   1      1049kB  52.0TB  52.0TB               primary


  Model: Linux device-mapper (linear) (dm)
  Disk /dev/mapper/vg_cfile-lv_swap: 33.8GB
  Sector size (logical/physical): 512B/512B
  Partition Table: loop

  Number  Start  End     Size    File system     Flags
   1      0.00B  33.8GB  33.8GB  linux-swap(v1)


  Model: Linux device-mapper (linear) (dm)
  Disk /dev/mapper/vg_cfile-lv_root: 53.7GB
  Sector size (logical/physical): 512B/512B
  Partition Table: loop

  Number  Start  End     Size    File system  Flags
   1      0.00B  53.7GB  53.7GB  ext4

edit2: Here is the file -s.

file -s /dev/sdf{,1,2,3,4,5,6}
    /dev/sdf:  x86 boot sector; partition 1: ID=0xee, starthead 0, startsector 1, 4294967295 sectors, extended partition table (last)\011, code offset 0x0
    /dev/sdf1: cannot open `/dev/sdf1' (No such file or directory)
    /dev/sdf2: cannot open `/dev/sdf2' (No such file or directory)
    /dev/sdf3: cannot open `/dev/sdf3' (No such file or directory)
    /dev/sdf4: cannot open `/dev/sdf4' (No such file or directory)
    /dev/sdf5: cannot open `/dev/sdf5' (No such file or directory)
    /dev/sdf6: cannot open `/dev/sdf6' (No such file or directory)

edit 3: Here's the output of multipath -l

      mpathd (35001b4d0189b4801) dm-2 JetStor,JetStor -VOL2
      size=47T features='0' hwhandler='0' wp=rw
      `-+- policy='round-robin 0' prio=0 status=active
        |- 7:0:0:1 sdc 8:32 active undef running
        `- 7:0:1:1 sde 8:64 active undef running
      mpathc (32029001378917600) dm-9 AC&Ncorp,JetStor
      size=47T features='0' hwhandler='0' wp=rw
      `-+- policy='round-robin 0' prio=0 status=active
        `- 8:0:0:0 sdf 8:80 active undef running
      mpathb (35001b4d0189b4800) dm-3 JetStor,JetStor -VOL1 -
      size=35T features='0' hwhandler='0' wp=rw
      `-+- policy='round-robin 0' prio=0 status=active
        |- 7:0:0:0 sdb 8:16 active undef running
        `- 7:0:1:0 sdd 8:48 active undef running 

And the content of /etc/multipath.conf

      # This is a basic configuration file with some examples, for device mapper
      # multipath.
      # For a complete list of the default configuration values, see
      # /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.defaults
      # For a list of configuration options with descriptions, see
      # /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.annotated
      #
      # REMEMBER: After updating multipath.conf, you must run
      #
      # service multipathd reload
      #
      # for the changes to take effect in multipathd

      ## By default, devices with vendor = "IBM" and product = "S/390.*" are
      ## blacklisted. To enable mulitpathing on these devies, uncomment the
      ## following lines.
      #blacklist_exceptions {
      #       device {
      #               vendor  "IBM"
      #               product "S/390.*"
      #       }
      #}

      ## Use user friendly names, instead of using WWIDs as names.
      #defaults {
      #       user_friendly_names yes
      #}

      ##
      ## Here is an example of how to configure some standard options.
      ##
      # 5/21/14: uncommented this
      defaults {
              udev_dir                /dev
              polling_interval        10
              path_selector           "round-robin 0"
              path_grouping_policy    multibus
              getuid_callout          "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
              prio                    alua
              path_checker            readsector0
              rr_min_io               100
              max_fds                 8192
              rr_weight               priorities
              failback                immediate
              no_path_retry           fail
              user_friendly_names     yes
      }
      ##
      ## The wwid line in the following blacklist section is shown as an example
      ## of how to blacklist devices by wwid.  The 2 devnode lines are the
      ## compiled in default blacklist. If you want to blacklist entire types
      ## of devices, such as all scsi devices, you should use a devnode line.
      ## However, if you want to blacklist specific devices, you should use
      ## a wwid line.  Since there is no guarantee that a specific device will
      ## not change names on reboot (from /dev/sda to /dev/sdb for example)
      ## devnode lines are not recommended for blacklisting specific devices.
      ##
      #blacklist {
      #       wwid 26353900f02796769
      #       devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
      #       devnode "^hd[a-z]"
      #}
      #multipaths {
      #       multipath {
      #               wwid                    3600508b4000156d700012000000b0000
      #               alias                   yellow
      #               path_grouping_policy    multibus
      #               path_checker            readsector0
      #               path_selector           "round-robin 0"
      #               failback                manual
      #               rr_weight               priorities
      #               no_path_retry           5
      #       }
      #       multipath {
      #               wwid                    1DEC_____321816758474
      #               alias                   red
      #       }
      #}
      #devices {
      #       device {
      #               vendor                  "COMPAQ  "
      #               product                 "HSV110 (C)COMPAQ"
      #               path_grouping_policy    multibus
      #               getuid_callout          "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
      #               path_checker            readsector0
      #               path_selector           "round-robin 0"
      #               hardware_handler        "0"
      #               failback                15
      #               rr_weight               priorities
      #               no_path_retry           queue
      #       }
      #       device {
      #               vendor                  "COMPAQ  "
      #               product                 "MSA1000         "
      #               path_grouping_policy    multibus
      #       }
      #}

      blacklist {
      }

There is another SCSI array on this, that may be why it changes devices.

2
  • You're running xfs_check on the whole disk. filesystems like xfs reside in partitions. something like file -s /dev/sdf{1,5,6} might reveal which partition has xfs filesystem. Sep 9, 2015 at 16:09
  • I have added the file -s info to my original post. It doesn't seem very informative. What else will can I use to get some info about this file system? Sep 9, 2015 at 17:31

2 Answers 2

2

It's hard to say without more information, but there are a couple of possibilities. I would comment if I could.

Worst case: You're hosed. Restore from backup. You have backups, right?

Best case: /dev/sdf is no longer the device name. Check the iSCSI LUN IDs and see if it got reassigned to another device name. Are you using a hardware iSCSI adapter or software? It might also be helpful to try fdisk -l /dev/sdf and post the output here.

EDIT: Thanks for that. I see several issues in your latest output:

Model: AC&Ncorp JetStor (scsi) Disk /dev/sdf: 52.0TB Sector size (logical/physical): 512B/512B Partition Table: gpt Number Start End Size File system Name Flags 1 1049kB 52.0TB 52.0TB primary Error: /dev/mapper/mpathcp1: unrecognised disk label

First of all, it's weird that the array changes model across devices. Is /dev/sdf provided by a different storage array?

Second, and probably more important, it looks like you have multipathing configured. Can you run multipath -l and paste the output? Also, can you paste the contents of your /etc/multipath.conf?

5
  • There are back ups but they are a little out of date so some data will be lost if we are force to use them. Here is the I'll post the fdisk -l /dev/sdf Sep 9, 2015 at 16:41
  • Start by running xfs_check /dev/sdf1. This probably won't fix it, but let's see what kind of output that gives you.
    – Jesse K
    Sep 9, 2015 at 17:50
  • xfs_check was the first thing I tried. It has a bad primary superblock - bad magic number and is not able to find a secondary superblock to use. Basically it give no information beyond that. Sep 9, 2015 at 18:15
  • My mistake, focusing on the wrong detail. See above for edits.
    – Jesse K
    Sep 9, 2015 at 20:39
  • I have added the multipath -l and /etc/multipath.conf to my original post. Sep 11, 2015 at 18:24
2

OK, post the result from this command. Change the device as necessary: sudo xfsdump -f /dev/sdb1 | grep -i superblock

I want to note my personal experience, that Ubuntu hasn’t had good XFS support for a while, and you might find it very helpful to download an old copy of warty warthog, and use a livecd with that.

You might get a response that looks like:

Couldn't find valid filesystem superblock.

if so, don't give up yet. You can try sudo mkfs.xfs -N /dev/sdb1 which will list the details about the filesystem, without writeing any changes. Hopefully you will see something at the end of that list like: ... superblock backups: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208

That is a list (examples: yours may vary) of alternative superblocks. you can then use in a trial and error type method, the list of superblocks, to reconstruct your XFS filesystem using mkfs, fsck, and mknode.

Before I go writing a long answer, post back details from xfsdump, and mkfs.xfs

1
  • What do you mean by "reconstructing the XFS" using the superblock numbers. How would one do that? Thanks
    – ChrisR
    Feb 25, 2017 at 0:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.