1

We're experiencing a mysterious hang on shutdown of a newly-imaged Oracle/Sun Solaris 10 SPARC box. It is repeatable (in the same spot ... from what we can tell). We let it try to work itself out multiple times for 5-10 minutes and it never progressed.

I've never seen this happen before.

The last thing displayed on the console is that syslogd was sent signal 15. Prior to us disabling snmpdx on the box, the last thing on the console was that snmpdx was sent signal 15 (after syslogd was sent signal 15).

While very rare to find, in Solaris days past, I'd have a better idea from experience where the problem might be, and could then narrow things down further with silly (but effective) debugging echo statments in /etc/*.d scripts. With SMF in the picture, I'm not really quite sure where to start.

We forced a crash dump via sync at the {ok} prompt for later analysis, and then let the box come up because it's a production server and our scheduled outage window was closing.

/var/adm/messages shows nothing of use.

How would you debug this situation? Is there a way to stop hiding what is happening at shutdown, to show, for example, each thing that is being done? (I never did like that a lot is hidden since Solaris 10 at boot-time too)

mdb ps of the savecore shows the following processes were running at hang time (afsd is the OpenAFS client and that many are expected):

> > ::ps
S    PID   PPID   PGID    SID    UID      FLAGS             ADDR NAME
R      0      0      0      0      0 0x00000001 00000000018387c0 sched
R    108      0      0      0      0 0x00020001 00000600110fe010 zpool-silmaril-p
R      3      0      0      0      0 0x00020001 0000060010b29848 fsflush
R      2      0      0      0      0 0x00020001 0000060010b2a468 pageout
R      1      0      0      0      0 0x4a024000 0000060010b2b088 init
R   1327      1   1327    329      0 0x4a024002 00000600176ab0c0 reboot
R    747      1      7      7      0 0x42020001 0000060017f9d0e0 afsd
R    749      1      7      7      0 0x42020001 00000600180104d0 afsd
R    752      1      7      7      0 0x42020001 0000060017cb44b8 afsd
R    754      1      7      7      0 0x42020001 0000060017fc8068 afsd
R    756      1      7      7      0 0x42020001 0000060017fcb0e8 afsd
R    760      1      7      7      0 0x42020001 00000600177f4048 afsd
R    762      1      7      7      0 0x42020001 000006001800f8b0 afsd
R    764      1      7      7      0 0x42020001 000006001800ec90 afsd
R    378      1    378    378      0 0x42020000 0000060013aee480 inetd
R      7      1      7      7      0 0x42020000 0000060010b28008 svc.startd
R    329      7    329    329      0 0x4a024000 00000600110ff850 sh
Z    317      7    317    317      0 0x4a014002 0000060013b3a490 sac 

5 Answers 5

0

/var/svc/log contains log files for all services that fall under SMF control. That's at least a starting point for debugging issues with SMF processes.

I agree with David, check there are no NFS mounts from this server that might be held open and that this server does not mount other NFS filesystems that may not be available.

3
  • Thank you - /var/svc/log! Duh. My lack of knowledge of that shows how seldom we've had to debug svcs.
    – jblaine
    Feb 23, 2011 at 16:52
  • But is there a way to get Solaris 10 to stop hiding everything from me? I want to see its shutdown process as it happens, which I think is an incredibly reasonable request, no?
    – jblaine
    Feb 23, 2011 at 17:03
  • I dug around on this one a little more. It seems the shutdown process is relatively rudimentary - the system calls the 'init' command to switch to a lower runlevel. This will then call the rc.# kill scripts. I assume in Solaris 10 it does SMF-controlled shutdowns (which are scripts anyway). I have not found a way to make this process more verbose though. Perhaps 'man init' can assist. Mar 5, 2011 at 12:57
0

You can still do the old fashioned putting echos in the scripts when using SMF. Just go into /lib/svc/method and edit away. Just from that process list I'd say it was AFS related, but I haven't use that.

0
0

Two things come to my mind immediately. Firstly, I've had NFS do this when a client was shutting down and there was no server available.

Alternately, in debugging I'd look at using the -x option to sh or ksh in the startup, assuming you can get at the scripts still.

0

It's been a while, but from what I recall, if you boot the box with the kernel arguments -v -m verbose, you'll get kernel messages (-v) and SVCS messages (-m verbose) displayed to the console. At least that way, you could get a better idea what's it trying to do...

1
  • 1
    That handles boot only from what I know. Do you have any ideas re: shutdown verbosity?
    – jblaine
    Feb 23, 2011 at 19:15
0

tail /var/svc/log/rc6.log helps if you performed an init 6. However, any instance of fmd problems can cause it to hang.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .