2

I am upgrading a server with 2 SAS cards from Debian 11 -> Debian 12

I have come upon an interesting problem. At first it didn't load any SAS cards and I got to initramfs.

Increase disk detection timeout at boot with Linux/Systemd

Then I used the fix in here and used rootdelay=180 and then 1 SAS card loaded itself, but the second one didnt come online.

I have added a picture aswell: 1_sas_card_detected

I have added the output of dmesg -T | grep sas aswell

[Wed Aug  9 12:50:24 2023] mpt3sas version 43.100.00.00 loaded
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: 63 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (3988540 kB)
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: MSI-X vectors supported: 96
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0:  0 16 16
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: High IOPs queues : disabled
[Wed Aug  9 12:50:24 2023] mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 250
[Wed Aug  9 12:50:24 2023] mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 252
[Wed Aug  9 12:50:24 2023] mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 254
[Wed Aug  9 12:50:24 2023] mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 256
[Wed Aug  9 12:50:24 2023] mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 258
[Wed Aug  9 12:50:24 2023] mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 260
[Wed Aug  9 12:50:24 2023] mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 262
[Wed Aug  9 12:50:24 2023] mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 264
[Wed Aug  9 12:50:24 2023] mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 266
[Wed Aug  9 12:50:24 2023] mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 268
[Wed Aug  9 12:50:24 2023] mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 270
[Wed Aug  9 12:50:24 2023] mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 272
[Wed Aug  9 12:50:24 2023] mpt3sas0-msix12: PCI-MSI-X enabled: IRQ 274
[Wed Aug  9 12:50:24 2023] mpt3sas0-msix13: PCI-MSI-X enabled: IRQ 276
[Wed Aug  9 12:50:24 2023] mpt3sas0-msix14: PCI-MSI-X enabled: IRQ 278
[Wed Aug  9 12:50:24 2023] mpt3sas0-msix15: PCI-MSI-X enabled: IRQ 280
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: iomem(0x00000000e0e40000), mapped(0x000000004222aca5), size(65536)
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: ioport(0x000000000000b000), size(256)
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: sending message unit reset !!
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: message unit reset: SUCCESS
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: scatter gather: sge_in_main_msg(1), sge_per_chain(7), sge_per_io(128), chains_per_io(19)
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: request pool(0x0000000051d56977) - dma(0x938a3a000): depth(3200), frame_size(128), pool_size(400 kB)
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: sense pool(0x000000009c61e8a2) - dma(0x93f7ba000): depth(2939), element_size(96), pool_size (275 kB)
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: reply pool(0x00000000c769fa7c) - dma(0x93f83a000): depth(3264), frame_size(128), pool_size(408 kB)
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: config page(0x000000008a8c65b6) - dma(0x93f72a000): size(512)
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: Allocated physical memory: size(8887 kB)
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: Current Controller Queue Depth(2936),Max Controller Queue Depth(3072)
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: Scatter Gather Elements per IO(128)
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: _base_display_fwpkg_version: complete
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: LSISAS3008: FWVersion(16.00.10.00), ChipRevision(0x02), BiosVersion(08.37.00.00)
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: sending port enable !!
[Wed Aug  9 12:50:24 2023] mpt3sas_cm1: 63 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (3988540 kB)
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: hba_port entry: 000000006a2e3bf6, port: 255 is added to hba_port list
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x500304801bd55b02), phys(8)
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: handle(0x9) sas_address(0x4433221100000000) port_type(0x1)
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: handle(0xa) sas_address(0x4433221101000000) port_type(0x1)
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: handle(0xb) sas_address(0x4433221102000000) port_type(0x1)
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: handle(0xc) sas_address(0x4433221103000000) port_type(0x1)
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: handle(0xd) sas_address(0x4433221104000000) port_type(0x1)
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: handle(0xe) sas_address(0x4433221105000000) port_type(0x1)
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: handle(0xf) sas_address(0x4433221106000000) port_type(0x1)
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: handle(0x10) sas_address(0x4433221107000000) port_type(0x1)
[Wed Aug  9 12:50:24 2023] mpt3sas_cm0: port enable: SUCCESS
[Wed Aug  9 12:50:24 2023] scsi 0:0:0:0: SATA: handle(0x0009), sas_addr(0x4433221100000000), phy(0), device_name(0x5002538f024576d9)
[Wed Aug  9 12:50:24 2023]  end_device-0:0: add: handle(0x0009), sas_addr(0x4433221100000000)
[Wed Aug  9 12:50:24 2023] scsi 0:0:1:0: SATA: handle(0x000a), sas_addr(0x4433221101000000), phy(1), device_name(0x5002538f024576dc)
[Wed Aug  9 12:50:24 2023]  end_device-0:1: add: handle(0x000a), sas_addr(0x4433221101000000)
[Wed Aug  9 12:50:24 2023] scsi 0:0:2:0: SATA: handle(0x000b), sas_addr(0x4433221102000000), phy(2), device_name(0x5002538f024576d8)
[Wed Aug  9 12:50:24 2023]  end_device-0:2: add: handle(0x000b), sas_addr(0x4433221102000000)
[Wed Aug  9 12:50:24 2023] scsi 0:0:3:0: SATA: handle(0x000c), sas_addr(0x4433221103000000), phy(3), device_name(0x5002538f024576d7)
[Wed Aug  9 12:50:24 2023]  end_device-0:3: add: handle(0x000c), sas_addr(0x4433221103000000)
[Wed Aug  9 12:50:24 2023] scsi 0:0:4:0: SATA: handle(0x000d), sas_addr(0x4433221104000000), phy(4), device_name(0x5002538f024576e0)
[Wed Aug  9 12:50:24 2023]  end_device-0:4: add: handle(0x000d), sas_addr(0x4433221104000000)
[Wed Aug  9 12:50:24 2023] scsi 0:0:5:0: SATA: handle(0x000e), sas_addr(0x4433221105000000), phy(5), device_name(0x5002538f024576e1)
[Wed Aug  9 12:50:24 2023]  end_device-0:5: add: handle(0x000e), sas_addr(0x4433221105000000)
[Wed Aug  9 12:50:24 2023] scsi 0:0:6:0: SATA: handle(0x000f), sas_addr(0x4433221106000000), phy(6), device_name(0x5002538f0245768d)
[Wed Aug  9 12:50:24 2023]  end_device-0:6: add: handle(0x000f), sas_addr(0x4433221106000000)
[Wed Aug  9 12:50:24 2023] scsi 0:0:7:0: SATA: handle(0x0010), sas_addr(0x4433221107000000), phy(7), device_name(0x5002538f02457689)
[Wed Aug  9 12:50:24 2023] mpt3sas_cm1: CurrentHostPageSize is 0: Setting default host page size to 4k
[Wed Aug  9 12:50:24 2023] mpt3sas_cm1: MSI-X vectors supported: 96
[Wed Aug  9 12:50:24 2023] mpt3sas_cm1:  0 16 16
[Wed Aug  9 12:50:24 2023] mpt3sas 0000:b3:00.0: xen map irq failed -22 for 32752 domain
[Wed Aug  9 12:50:24 2023]  end_device-0:7: add: handle(0x0010), sas_addr(0x4433221107000000)
[Wed Aug  9 12:50:24 2023]  mpt3sas_base_map_resources.cold+0x296/0x718 [mpt3sas]
[Wed Aug  9 12:50:24 2023]  mpt3sas_base_attach+0xeb/0x4c0 [mpt3sas]
[Wed Aug  9 12:50:24 2023]  _scsih_probe+0x6e9/0x8f0 [mpt3sas]
[Wed Aug  9 12:50:24 2023]  _mpt3sas_init+0x198/0x1000 [mpt3sas]
[Wed Aug  9 12:50:24 2023] Modules linked in: ahci libahci xhci_pci i40e mpt3sas(+) libata xhci_hcd crct10dif_pclmul crct10dif_common crc32_pclmul raid_class scsi_transport_sas crc32c_intel usbcore scsi_mod ptp i2c_i801 pps_core i2c_smbus lpc_ich usb_common scsi_common

EDIT:

I have added normal dmesg -T logs it seems to be a kernel error.

[Wed Aug  9 16:13:21 2023] mpt3sas_cm1: MSI-X vectors supported: 96
[Wed Aug  9 16:13:21 2023]   no of cores: 16, max_msix_vectors: -1
[Wed Aug  9 16:13:21 2023] mpt3sas_cm1:  0 16 16
[Wed Aug  9 16:13:21 2023] mpt3sas 0000:b3:00.0: xen map irq failed -22 for 32752 domain
[Wed Aug  9 16:13:21 2023] BUG: kernel NULL pointer dereference, address: 000000000000004d
[Wed Aug  9 16:13:21 2023] #PF: supervisor read access in kernel mode
[Wed Aug  9 16:13:21 2023] #PF: error_code(0x0000) - not-present page
[Wed Aug  9 16:13:21 2023] PGD 0 P4D 0 
[Wed Aug  9 16:13:21 2023] Oops: 0000 [#1] PREEMPT SMP NOPTI
[Wed Aug  9 16:13:21 2023] CPU: 0 PID: 230 Comm: (udev-worker) Not tainted 6.1.0-10-amd64 #1  Debian 6.1.38-2
[Wed Aug  9 16:13:21 2023] Hardware name: Supermicro SYS-1019P-WTR/X11SPW-TF, BIOS 3.9 03/15/2023
[Wed Aug  9 16:13:21 2023] RIP: e030:xen_destroy_irq+0x98/0x110
[Wed Aug  9 16:13:21 2023] Code: c7 80 ff ba 82 e8 f8 e8 3d 00 48 8b 44 24 08 65 48 2b 04 25 28 00 00 00 75 7d 48 83 c4 10 44 89 e0 5b 5d 41 5c c3 cc cc cc cc <f6> 45 4d 04 75 c1 0f b7 45 48 bf 0e 00 00 00 48 89 e6 89 44 24 04
[Wed Aug  9 16:13:21 2023] RSP: e02b:ffffc90040c2b8d0 EFLAGS: 00010202
[Wed Aug  9 16:13:21 2023] RAX: 0000000000000001 RBX: 000000000000013a RCX: 0000000000000000
[Wed Aug  9 16:13:21 2023] RDX: ffff888118460000 RSI: ffff888100520490 RDI: ffffffff82baff80
[Wed Aug  9 16:13:21 2023] RBP: 0000000000000000 R08: ffff888100520688 R09: ffffffff82ad47c0
[Wed Aug  9 16:13:21 2023]  end_device-0:7: add: handle(0x0010), sas_addr(0x4433221107000000)
[Wed Aug  9 16:13:21 2023] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88810276d0d0
[Wed Aug  9 16:13:21 2023] R13: 0000000000000010 R14: ffffc90040239000 R15: ffff88810276d000
[Wed Aug  9 16:13:21 2023] FS:  00007f8c107fa8c0(0000) GS:ffff888195000000(0000) knlGS:0000000000000000
[Wed Aug  9 16:13:21 2023] CS:  e030 DS: 0000 ES: 0000 CR0: 0000000080050033
[Wed Aug  9 16:13:21 2023] CR2: 000000000000004d CR3: 0000000118104000 CR4: 0000000000050660
[Wed Aug  9 16:13:21 2023] Call Trace:
[Wed Aug  9 16:13:21 2023]  <TASK>
[Wed Aug  9 16:13:21 2023]  ? __die_body.cold+0x1a/0x1f
[Wed Aug  9 16:13:21 2023]  ? page_fault_oops+0xd2/0x2b0
[Wed Aug  9 16:13:21 2023]  ? exc_page_fault+0x70/0x170
[Wed Aug  9 16:13:21 2023]  ? asm_exc_page_fault+0x22/0x30
[Wed Aug  9 16:13:21 2023]  ? xen_destroy_irq+0x98/0x110
[Wed Aug  9 16:13:21 2023]  xen_teardown_msi_irqs+0x3b/0x70
[Wed Aug  9 16:13:21 2023]  msi_domain_free_irqs_descs_locked+0x18/0x40
[Wed Aug  9 16:13:21 2023]  pci_msi_teardown_msi_irqs+0x3e/0x40
[Wed Aug  9 16:13:21 2023]  __pci_enable_msix_range+0x33f/0x4e0
[Wed Aug  9 16:13:21 2023]  pci_alloc_irq_vectors_affinity+0xa9/0x110
[Wed Aug  9 16:13:21 2023]  mpt3sas_base_map_resources.cold+0x296/0x718 [mpt3sas]
[Wed Aug  9 16:13:21 2023]  ? ncpus_cmp_func+0x10/0x10
[Wed Aug  9 16:13:21 2023]  mpt3sas_base_attach+0xeb/0x4c0 [mpt3sas]
[Wed Aug  9 16:13:21 2023]  _scsih_probe+0x6e9/0x8f0 [mpt3sas]
[Wed Aug  9 16:13:21 2023]  local_pci_probe+0x3e/0x80
[Wed Aug  9 16:13:21 2023]  pci_device_probe+0xc3/0x230
[Wed Aug  9 16:13:21 2023]  really_probe+0xdb/0x380
[Wed Aug  9 16:13:21 2023]  ? pm_runtime_barrier+0x50/0x90
[Wed Aug  9 16:13:21 2023]  __driver_probe_device+0x78/0x120
[Wed Aug  9 16:13:21 2023]  driver_probe_device+0x1f/0x90
[Wed Aug  9 16:13:21 2023]  __driver_attach+0xce/0x1c0
[Wed Aug  9 16:13:21 2023]  ? __device_attach_driver+0x110/0x110
[Wed Aug  9 16:13:21 2023]  bus_for_each_dev+0x84/0xd0
[Wed Aug  9 16:13:21 2023]  bus_add_driver+0x1ae/0x200
[Wed Aug  9 16:13:21 2023]  driver_register+0x89/0xe0
[Wed Aug  9 16:13:21 2023]  ? 0xffffffffc06a2000
[Wed Aug  9 16:13:21 2023]  _mpt3sas_init+0x198/0x1000 [mpt3sas]
[Wed Aug  9 16:13:21 2023]  ? 0xffffffffc06a2000
[Wed Aug  9 16:13:21 2023]  do_one_initcall+0x56/0x220
[Wed Aug  9 16:13:21 2023]  do_init_module+0x4a/0x200
[Wed Aug  9 16:13:21 2023]  __do_sys_finit_module+0xac/0x120
[Wed Aug  9 16:13:21 2023]  do_syscall_64+0x58/0xc0
[Wed Aug  9 16:13:21 2023]  ? fpregs_assert_state_consistent+0x22/0x50
[Wed Aug  9 16:13:21 2023]  ? exit_to_user_mode_prepare+0x40/0x1d0
[Wed Aug  9 16:13:21 2023]  ? syscall_exit_to_user_mode+0x17/0x40
[Wed Aug  9 16:13:21 2023]  ? do_syscall_64+0x67/0xc0
[Wed Aug  9 16:13:21 2023]  ? check_preemption_disabled+0x31/0xf0
[Wed Aug  9 16:13:21 2023]  ? fpregs_assert_state_consistent+0x22/0x50
[Wed Aug  9 16:13:21 2023] usb 1-7: New USB device found, idVendor=0557, idProduct=7000, bcdDevice= 0.00
[Wed Aug  9 16:13:21 2023]  ? exit_to_user_mode_prepare+0x40/0x1d0
[Wed Aug  9 16:13:21 2023] usb 1-7: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[Wed Aug  9 16:13:21 2023]  ? syscall_exit_to_user_mode+0x17/0x40
[Wed Aug  9 16:13:21 2023]  ? do_syscall_64+0x67/0xc0
[Wed Aug  9 16:13:21 2023]  ? do_syscall_64+0x67/0xc0
[Wed Aug  9 16:13:21 2023]  ? do_syscall_64+0x67/0xc0
[Wed Aug  9 16:13:21 2023]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[Wed Aug  9 16:13:21 2023] RIP: 0033:0x7f8c10f06539
[Wed Aug  9 16:13:21 2023] Code: 08 89 e8 5b 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 97 08 0d 00 f7 d8 64 89 01 48
5
  • A quick Google returned this: suse.com/support/kb/doc/?id=000021158 Is this a VM with passthru?
    – symcbean
    Commented Aug 9, 2023 at 10:37
  • I tried it. No it is not a VM it is a supermicro server. Commented Aug 9, 2023 at 12:06
  • I think symcbean meant: are you seeing this problem in dom0 or domU? (I refer to the xen tag you added.) [Wed Aug 9 12:50:24 2023] mpt3sas 0000:b3:00.0: xen map irq failed -22 for 32752 domain is suspicious. What is relevant xl dmesg (or whatever equivalent you have)? Commented Aug 9, 2023 at 12:23
  • xl dmesg just keeps loading... that means no output is given. Usually xl commands respond very fast. xendomains service gets a timeout aswel.. Commented Aug 9, 2023 at 13:21
  • I used xen 4.14 (old-stable version) and kernel 6.1 (stable kernel) and it still only loads the one card. When I use the oldstable kernel and the old(4.14) or new (4.17) then it works. Commented Aug 9, 2023 at 13:39

1 Answer 1

0

This problem is still present even with the latest Debian 12 kernels.

Using 5.10 Kernel with xen 4.17 is still the working workaround for now.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .