2

I have notified LSI support, twice, but so far they are unable to reproduce the problem. I wanted to post here to get some unbiased expert thoughts about it and see if anyone else has seen a similar problem.

We manage a number of servers that supply Internet services with very heavy disk IO. All run Debian testing (Sid)-amd64 and use 3ware RAID cards from the 85xx - 96xx series. With the Debian kernel updates to 3.9.x-amd64 we started getting a segfault with tw_cli. We tested tdm2 and it also segfaults.

To reproduce the problem: (You don't need a RAID card in your system to do this) 1. Fresh install of Debian testing (Sid). ISO is http://cdimage.debian.org/cdimage/weekly-builds/amd64/iso-cd/ 2. Install tw_cli and try to run it.

We ran tw_cli as root with strace under 3.2 and 3.9.6/3.9.8-amd64 and the segfault is happening right after tw_cli calls uname as you can see below.

Good Run:

execve("/usr/local/sbin/tw_cli", ["tw_cli", "/c0", "show", "all"], ["TERM=xterm", "SHELL=/bin/bash", "SSH_CLIENT=71.207.183.174 60609 "..., "SSH_TTY=/dev/pts/0", "USER=root", "MAIL=/var/mail/root", "PATH=/usr/local/sbin:/usr/local/"..., "PWD=/root", "LANG=C", "PS1=\\h:\\w\\$ ", "SHLVL=1", "HOME=/root", "LOGNAME=root", "SSH_CONNECTION=71.207.183.174 60"..., "_=/usr/bin/strace"]) = 0
uname({sysname="Linux", nodename="yorick.ironicdesign.com", release="3.2.0-4-amd64", version="#1 SMP Debian 3.2.46-1", machine="x86_64"}) = 0
brk(0)                                  = 0x2664000
brk(0x2685000)                          = 0x2685000
uname({sysname="Linux", nodename="yorick.ironicdesign.com", release="3.2.0-4-amd64", version="#1 SMP Debian 3.2.46-1", machine="x86_64"}) = 0
open("/proc/devices", O_RDONLY)         = 3
...

Bad run:

execve("/usr/local/sbin/tw_cli", ["tw_cli", "/c0", "show", "all"], ["SHELL=/bin/bash", "TERM=screen", "SSH_CLIENT=98.26.9.112 58271 22", "SSH_TTY=/dev/pts/0", "USER=root", "SSH_AUTH_SOCK=/tmp/ssh-595iwzIik"..., "TERMCAP=SC|screen|VT 100/ANSI X3"..., "PATH=/usr/local/sbin:/usr/local/"..., "MAIL=/var/mail/root", "STY=17473.mdorman", "PWD=/root", "LANG=C", "PS1=\\h:\\w\\$ ", "HOME=/root", "SHLVL=2", "LOGNAME=root", "WINDOW=0", "SSH_CONNECTION=98.26.9.112 58271"..., "_=/usr/bin/strace"]) = 0
uname({sysname="Linux", nodename="yorick.ironicdesign.com", release="3.10-1-amd64", version="#1 SMP Debian 3.10.1-1 (2013-07-16)", machine="x86_64"}) = 0
brk(0)                                  = 0x26ef000
brk(0x2710000)                          = 0x2710000
uname({sysname="Linux", nodename="yorick.ironicdesign.com", release="3.10-1-amd64", version="#1 SMP Debian 3.10.1-1 (2013-07-16)", machine="x86_64"}) = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---

In the good run above, the next call after uname is to open /proc/devices which DOES exist and should not be a problem. Something else we think is notable and you can see it in the bad run above, uname in the 3.9/3.10 kernel adds a date to the string.

We think these two strace runs may indicate tw_cli is crashing because it is getting an unexpected response from the uname call. LSI support says:

"3dm2 and tw_cli work fine even with Ubuntu latest kernels 3.10.x and Ubuntu usually pulls unstable kernels from Debian and use it for their releases."

FWIW, I am not sure what LSI support is talking about. We just tested with a fresh, up-to-date install of Ubuntu 1304 (Raring Ringtail) and uname -a shows "Linux mac-workstation 3.8.0-26-generic #38-Ubuntu SMP Mon Jun 17 21:43:33 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux". So Ubuntu 1304 is using the 3.8 kernel, not 3.10. And tw_cli & tdm2 both work fine.

So any helpful thoughts? At the moment our options appear to be: - pin our kernel version to 3.2 and hope whatever the problem is gets fixed soon - stop monitoring our RAIDs (not really an option) - compile custom kernels for all our servers because apparently the stock Debian Testing kernel has this problem - switch to Ubuntu for all our servers (not feasible) - switch our RAID cards to someone like Areca (also not feasible for existing servers, but is being considered for our next server generation)

=================== followup ============================

I just received the following response from LSI/3ware support. I am afraid my response to them was not very nice, though I believe it summed up the situation accurately.

LSI/3ware said: We are able to reproduce the issue with Debian unstable kernel 3.9-1-amd64 but engineering does not release software for un-stable or un-released kernels. If possible, please wait until Debian officially releases the kernel. 3dm2 and tw_cli should work with Ubuntu official release 13.04 including updated kernels 3.8.x to 3.10.

My response:

So the end result is:

  • You will not do a fresh install of Debian Testing which will reproduce the problem. I even gave you the link to the "Official" Testing ISO which DOES have the problem.

Instead you first compile a custom kernel which somehow avoids the problem. Then you jump OVER Testing to Unstable to reproduce the problem. Except "engineering does not release software for un-stable or un-released kernels"...so once again you avoid having to take any action.

  • Then you have the nerve to suggest we are not using the Debian official release (we ARE) OR that we can just shut down our services running on all our servers and swap to a new distribution???

The good news for us is we are in the Debian community and will let everyone know how this has been handled by LSI. This is going to send a STRONG signal to the rest of the Linux community about the long term viability of your products.

Thank you

============= my conclusion =============

Yes, we DO use the official Debian Testing release in production and some think that is not wise.

Debating that does not address the problem here though, that eventually the kernel in Testing makes it way into Stable. And the time for any manufacturer to fix their proprietary software that is essential to the use of their product is with the Testing distribution...BEFORE it gets to Stable.

So while we wait for LSI/3ware to decide to load the official Debian Testing and fix their software, we will probably pin our kernel to 3.2. We may also find the time to compile a 3.10 kernel that does not output a date with uname -r to see if that is indeed the cause. If it is we may be able to get that changed in the uname call for the kernel.

Improve this question
5
  • 2
    Why do you try to run a testing kernel on a production system?
    – Sven
    Jul 24, 2013 at 13:51
  • Has 3ware even updated their stuff for Linux kernels > 2.6?
    – Tim
    Jul 24, 2013 at 19:00
  • 3ware doesn't exist anymore for a long time. It is now owned by LSI and they actively update the admin software and firmware of these controllers, as well as release new hardware using the 3ware name. en.wikipedia.org/wiki/3ware
    – aseq
    Jul 24, 2013 at 23:12
  • 1
    We run Debian testing (and have for over 10 years now) in production because we run very heavy disk IO loads and have needed every improvement as soon as we could get them, modulo the obvious risks of using leading edge software. But that is beside the point. Eventually the testing kernel WILL be the stable kernel. Heck, Ubuntu is already up to 3.8.
    – Andy Dorman
    Jul 25, 2013 at 12:54
  • In the meantime the kernel versions that have different uname output are stable, but LSI still hasn't released any updated software. The latest version of tw_cli was released in 2012. Since LSI also doesn't update the 3Ware product line I think we can safely assume there wont be any update. We are already making the switch to LSI cards.
    – Rudy Broersma
    Nov 26, 2015 at 12:59

4 Answers 4

Reset to default
5

I had the same problem here on Debian Testing with Kernel 3.12.XXX. For me the command setarch (or linux64) worked:

web3:~# setarch x86_64 --uname-2.6 tw_cli /c0/u0 show all

or

web3:~# linux64 --uname-2.6 tw_cli /c0/u0 show all
Improve this answer
3

The problem isn't the date, it's that tw_cli is looking for X.Y.Z(-R-arch) in the release and it's only getting X.Y(-R-arch) - 3.2.0-4-amd64 vs 3.10-2-amd64. When the release is set to 3.10.0-2-amd64 it runs fine. They may be doing a sscanf() with limited formats, and little or no error checking.

jam:~# uname -r
3.10-2-amd64
jam:~# tw_cli /c0 show firmware
Segmentation fault
jam:~# echo 3.10.0-2-amd64 > /sys/module/utsname/parameters/release
jam:~# uname -r
3.10.0-2-amd64
jam:~# tw_cli /c0 show firmware
/c0 Firmware Version = FE9X 4.10.00.027

If the binary was dynamic you could see about a uname() replacement with LD_PRELOAD but it's static. There's no source code so our options are limited:

  • LSI/3ware fixes tw_cli, hopefully removing all the uname() nonsense
  • Get Debian to use X.Y.Z-R-arch in release
  • Someone good with assembly comes up with a binary patch or something similar
  • Run a custom kernel
  • Run an older kernel
  • Ditch 3ware

I like my 9650 but this is crap.

Improve this answer
5
  • Where does your utsname module come from?
    – derobert
    Oct 4, 2013 at 19:33
  • I made it. It's my first kernel module. I'm sure someone who knows what they're doing would have many complaints about what I've done. pastebin.com/QJdYhs40
    – logan
    Oct 10, 2013 at 0:58
  • Thank you, you saved me having to figure out how to write kernel modules (and remember enough C) to write my own. Suggest you might want to put this somewhere more permanent, to help out other 3ware sufferers...
    – derobert
    Oct 10, 2013 at 5:35
  • If I was more confident with the code I might put it up somewhere better but even beyond the code quality the whole thing is pretty sketchy if you ask me. If you plan to use this to run the tw_cli binary then I would minimize the impact by making a small shell script wrapper that sets a working value, runs the binary, and resets it after. I'm just not sure what would happen with kernel module loading or initrd rebuilds if you run for long periods of time with a different release. pastebin.com/YLMet1y0
    – logan
    Oct 12, 2013 at 4:10
  • Indeed, I'd expect it to break update-initramfs, and possibly all kinds of things (e.g., modprobe). I've thought of a better solution—ptrace—writing a program for it now... Hopefully I can get this to work.
    – derobert
    Oct 12, 2013 at 8:02
0

You should not run Debian testing unless you want to test it. Especially not on a server. I would say try to reproduce it in Debian stable.

In addition the LSI 3ware cards come with an excellent web based admin interface that allows you to configure it to send alerts. In that case you don't need to use tw_cli in a script to email such alerts and thus avoids the problem you're having.

Actually come to think of it, if tdm2 segfaults then the admin interface isn't working either...

Improve this answer
2
  • We run Debian testing (and have for over 10 years now) in production because we run very heavy disk IO loads and have needed every improvement as soon as we could get them, modulo the obvious risks of using leading edge software.
    – Andy Dorman
    Jul 25, 2013 at 12:51
  • Debain stable uses 3.2, which we DID test and tw_cli does not segfault...For the moment we have pinned our kernel version to that (3.2) And yeah, the reason we tested tdm2 was with the intent of dropping tw_cli and letting tdm2 notify us of problems...although that carries the security risk of a proprietary, closed source web browser...who knows what back door access may be built in?
    – Andy Dorman
    Jul 25, 2013 at 12:59
0

Have the same issue on Debian testing with 3.10 kernel(including tw_cli segfault). Need to mention, Debian testing is much more stable than other distros called 'stable releases' ;) However seems like problem there is related to kernel version more than testing distro. Switched back to 3.2 and it works fine, also waiting for LSI software update, but we have 3ware 9500 series, so no big hope for that.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.