Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Server suddenly became unresponsive to SSH, I reported the previous time it happened here on another server I have about 2 months ago. The server setup is CentOS6 64bit, it runs a very basic apache2 server that serves up static files and should be under very small loads, as it behind a heavily cached Amazon CloudFront.

I was told to look through /var/log/messages but when I open it for the time of the problem ( around 18:00 on 2/27/2012 ) there a lot of messages there that I have no idea what they mean, and Googling is not helping much either so I was wondering if someone more experienced could possible take a look at this log and tell me if they see anything weird.

Feb 26 03:06:02 173 kernel: imklog 4.6.2, log source = /proc/kmsg started.
Feb 26 03:06:02 173 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="1439" x-info="http://www.rsyslog.com"] (re)start
Feb 26 04:20:01 173 auditd[1414]: Audit daemon rotating log files
Feb 26 13:37:09 173 auditd[1414]: Audit daemon rotating log files
Feb 27 08:03:03 173 auditd[1414]: Audit daemon rotating log files
Feb 27 18:35:13 173 init: tty (/dev/tty2) main process (1958) killed by TERM signal
Feb 27 18:35:13 173 init: tty (/dev/tty3) main process (1960) killed by TERM signal
Feb 27 18:35:13 173 init: tty (/dev/tty4) main process (1962) killed by TERM signal
Feb 27 18:35:13 173 init: tty (/dev/tty5) main process (1964) killed by TERM signal
Feb 27 18:35:13 173 init: tty (/dev/tty6) main process (1966) killed by TERM signal
Feb 27 18:35:15 173 qpidd[1893]: 2012-02-27 18:35:15 notice Shut down
Feb 27 18:35:16 173 abrtd: Got signal 15, exiting
Feb 27 18:35:20 173 acpid: exiting
Feb 27 18:35:20 173 init: Disconnected from system bus
Feb 27 18:35:20 173 rpcbind: rpcbind terminating on signal. Restart with "rpcbind -w"
Feb 27 18:35:20 173 console-kit-daemon[1982]: WARNING: no sender#012
Feb 27 18:35:20 173 auditd[1414]: The audit daemon is exiting.
Feb 27 18:35:20 173 kernel: type=1305 audit(1330389320.836:995029): audit_pid=0 old=1414 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
Feb 27 18:35:20 173 kernel: type=1305 audit(1330389320.936:995030): audit_enabled=0 old=1 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditctl_t:s0 res=1
Feb 27 18:35:20 173 kernel: Kernel logging (proc) stopped.
Feb 27 18:35:20 173 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="1439" x-info="http://www.rsyslog.com"] exiting on signal 15.
Feb 27 18:38:07 173 kernel: imklog 4.6.2, log source = /proc/kmsg started.
Feb 27 18:38:07 173 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="1437" x-info="http://www.rsyslog.com"] (re)start
Feb 27 18:38:07 173 kernel: Initializing cgroup subsys cpuset
Feb 27 18:38:07 173 kernel: Initializing cgroup subsys cpu
Feb 27 18:38:07 173 kernel: Linux version 2.6.32-220.2.1.el6.x86_64 (mockbuild@c6-x8664-build.centos.org) (gcc version 4.4.6 20110731 (Red Hat 4.4.6-3) (GCC) ) #1 SMP Fri Dec 23 02:21:33 CST 2011
Feb 27 18:38:07 173 kernel: Command line: ro root=/dev/mapper/vg_173-lv_root rd_LVM_LV=vg_173/lv_swap rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_173/lv_root rd_NO_MD quiet SYSFONT=latarcyrheb-sun16 rhgb crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM
Feb 27 18:38:07 173 kernel: KERNEL supported cpus:
Feb 27 18:38:07 173 kernel:  Intel GenuineIntel
Feb 27 18:38:07 173 kernel:  AMD AuthenticAMD
Feb 27 18:38:07 173 kernel:  Centaur CentaurHauls
Feb 27 18:38:07 173 kernel: BIOS-provided physical RAM map:
Feb 27 18:38:07 173 kernel: BIOS-e820: 0000000000000000 - 000000000009b800 (usable)
Feb 27 18:38:07 173 kernel: BIOS-e820: 000000000009b800 - 00000000000a0000 (reserved)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
Feb 27 18:38:07 173 kernel: BIOS-e820: 0000000000100000 - 00000000be7a8000 (usable)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000be7a8000 - 00000000be7f4000 (ACPI NVS)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000be7f4000 - 00000000be7fc000 (ACPI data)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000be7fc000 - 00000000bf47a000 (reserved)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000bf47a000 - 00000000bf47b000 (ACPI NVS)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000bf47b000 - 00000000bf48c000 (reserved)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000bf48c000 - 00000000bf48f000 (ACPI NVS)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000bf48f000 - 00000000bf4b0000 (reserved)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000bf4b0000 - 00000000bf4b2000 (usable)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000bf4b2000 - 00000000bf503000 (reserved)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000bf503000 - 00000000bf50d000 (ACPI NVS)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000bf50d000 - 00000000bf533000 (reserved)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000bf533000 - 00000000bf576000 (ACPI NVS)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000bf576000 - 00000000bf800000 (usable)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000fed1c000 - 00000000fed40000 (reserved)
Feb 27 18:38:07 173 kernel: BIOS-e820: 00000000ff000000 - 0000000100000000 (reserved)
Feb 27 18:38:07 173 kernel: BIOS-e820: 0000000100000000 - 0000000240000000 (usable)
Feb 27 18:38:07 173 kernel: DMI 2.7 present.
Feb 27 18:38:07 173 kernel: SMBIOS version 2.7 @ 0xF0480
Feb 27 18:38:07 173 kernel: AMI BIOS detected: BIOS may corrupt low RAM, working around it.
Feb 27 18:38:07 173 kernel: last_pfn = 0x240000 max_arch_pfn = 0x400000000
Feb 27 18:38:07 173 kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
Feb 27 18:38:07 173 kernel: total RAM covered: 8192M
Feb 27 18:38:07 173 kernel: Found optimal setting for mtrr clean up
Feb 27 18:38:07 173 kernel: gran_size: 64K  chunk_size: 64K     num_reg: 4      lose cover RAM: 0G
Feb 27 18:38:07 173 kernel: last_pfn = 0xbf800 max_arch_pfn = 0x400000000
Feb 27 18:38:07 173 kernel: init_memory_mapping: 0000000000000000-00000000bf800000
Feb 27 18:38:07 173 kernel: init_memory_mapping: 0000000100000000-0000000240000000
Feb 27 18:38:07 173 kernel: RAMDISK: 3711a000 - 37fef46f
Feb 27 18:38:07 173 kernel: ACPI: RSDP 00000000000f0450 00024 (v02 SUPERM)
Feb 27 18:38:07 173 kernel: ACPI: XSDT 00000000be7f4078 00074 (v01 SUPERM SMCI--MB 00000001 AMI  00010013)
Feb 27 18:38:07 173 kernel: ACPI: FACP 00000000be7faf40 000F4 (v04 SUPERM SMCI--MB 00000001 AMI  00010013)
Feb 27 18:38:07 173 kernel: ACPI: DSDT 00000000be7f4180 06DC0 (v02 SUPERM SMCI--MB 00000000 INTL 20051117)
Feb 27 18:38:07 173 kernel: ACPI: FACS 00000000bf50af80 00040
Feb 27 18:38:07 173 kernel: ACPI: APIC 00000000be7fb038 00092 (v03 SUPERM SMCI--MB 00000001 AMI  00010013)
Feb 27 18:38:07 173 kernel: ACPI: SSDT 00000000be7fb0d0 001D6 (v01 AMICPU     PROC 00000001 MSFT 03000001)
Feb 27 18:38:07 173 kernel: ACPI: MCFG 00000000be7fb2a8 0003C (v01 SUPERM SMCI--MB 00000001 MSFT 00000097)
Feb 27 18:38:07 173 kernel: ACPI: HPET 00000000be7fb2e8 00038 (v01 SUPERM SMCI--MB 00000001 AMI. 00000004)
Feb 27 18:38:07 173 kernel: ACPI: SPMI 00000000be7fb320 00040 (v05 A M I   OEMSPMI 00000000 AMI. 00000000)
Feb 27 18:38:07 173 kernel: ACPI: EINJ 00000000be7fb360 00130 (v01    AMI AMI EINJ 00000000      00000000)
Feb 27 18:38:07 173 kernel: ACPI: ERST 00000000be7fb490 00210 (v01  AMIER AMI ERST 00000000      00000000)
Feb 27 18:38:07 173 kernel: ACPI: HEST 00000000be7fb6a0 000A8 (v01    AMI AMI HEST 00000000      00000000)
Feb 27 18:38:07 173 kernel: ACPI: BERT 00000000be7fb748 00030 (v01    AMI AMI BERT 00000000      00000000)
Feb 27 18:38:07 173 kernel: No NUMA configuration found
Feb 27 18:38:07 173 kernel: Faking a node at 0000000000000000-0000000240000000
Feb 27 18:38:07 173 kernel: Bootmem setup node 0 0000000000000000-0000000240000000
Feb 27 18:38:07 173 kernel:  NODE_DATA [0000000000017000 - 000000000004afff]
Feb 27 18:38:07 173 kernel:  bootmap [000000000004b000 -  0000000000092fff] pages 48
Feb 27 18:38:07 173 kernel: (8 early reservations) ==> bootmem [0000000000 - 0240000000]
Feb 27 18:38:07 173 kernel:  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
Feb 27 18:38:07 173 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
Feb 27 18:38:07 173 kernel:  #2 [0001000000 - 000200c7e4]    TEXT DATA BSS ==> [0001000000 - 000200c7e4]
Feb 27 18:38:07 173 kernel:  #3 [003711a000 - 0037fef46f]          RAMDISK ==> [003711a000 - 0037fef46f]
Feb 27 18:38:07 173 kernel:  #4 [000009b800 - 0000100000]    BIOS reserved ==> [000009b800 - 0000100000]
Feb 27 18:38:07 173 kernel:  #5 [000200d000 - 000200d3a8]              BRK ==> [000200d000 - 000200d3a8]
Feb 27 18:38:07 173 kernel:  #6 [0000010000 - 0000012000]          PGTABLE ==> [0000010000 - 0000012000]
Feb 27 18:38:07 173 kernel:  #7 [0000012000 - 0000017000]          PGTABLE ==> [0000012000 - 0000017000]
Feb 27 18:38:07 173 kernel: found SMP MP-table at [ffff8800000fcdc0] fcdc0
Feb 27 18:38:07 173 kernel: Reserving 129MB of memory at 48MB for crashkernel (System RAM: 9216MB)
Feb 27 18:38:07 173 kernel: Zone PFN ranges:
Feb 27 18:38:07 173 kernel:  DMA      0x00000010 -> 0x00001000
Feb 27 18:38:07 173 kernel:  DMA32    0x00001000 -> 0x00100000
Feb 27 18:38:07 173 kernel:  Normal   0x00100000 -> 0x00240000
Feb 27 18:38:07 173 kernel: Movable zone start PFN for each node
Feb 27 18:38:07 173 kernel: early_node_map[5] active PFN ranges
Feb 27 18:38:07 173 kernel:    0: 0x00000010 -> 0x0000009b
Feb 27 18:38:07 173 kernel:    0: 0x00000100 -> 0x000be7a8
Feb 27 18:38:07 173 kernel:    0: 0x000bf4b0 -> 0x000bf4b2
Feb 27 18:38:07 173 kernel:    0: 0x000bf576 -> 0x000bf800
Feb 27 18:38:07 173 kernel:    0: 0x00100000 -> 0x00240000
Feb 27 18:38:07 173 kernel: ACPI: PM-Timer IO Port: 0x408
Feb 27 18:38:07 173 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
Feb 27 18:38:07 173 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
Feb 27 18:38:07 173 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
Feb 27 18:38:07 173 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
Feb 27 18:38:07 173 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x01] enabled)
Feb 27 18:38:07 173 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x03] enabled)
Feb 27 18:38:07 173 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x05] enabled)
Feb 27 18:38:07 173 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled)
Feb 27 18:38:07 173 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
Feb 27 18:38:07 173 kernel: ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
Feb 27 18:38:07 173 kernel: IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
Feb 27 18:38:07 173 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Feb 27 18:38:07 173 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Feb 27 18:38:07 173 kernel: Using ACPI (MADT) for SMP configuration information
Feb 27 18:38:07 173 kernel: ACPI: HPET id: 0x8086a701 base: 0xfed00000
Feb 27 18:38:07 173 kernel: SMP: Allowing 8 CPUs, 0 hotplug CPUs
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 000000000009b000 - 000000000009c000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 000000000009c000 - 00000000000a0000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000000e0000 - 0000000000100000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000be7a8000 - 00000000be7f4000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000be7f4000 - 00000000be7fc000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000be7fc000 - 00000000bf47a000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000bf47a000 - 00000000bf47b000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000bf47b000 - 00000000bf48c000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000bf48c000 - 00000000bf48f000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000bf48f000 - 00000000bf4b0000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000bf4b2000 - 00000000bf503000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000bf503000 - 00000000bf50d000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000bf50d000 - 00000000bf533000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000bf533000 - 00000000bf576000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000bf800000 - 00000000fed1c000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000fed1c000 - 00000000fed40000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000fed40000 - 00000000ff000000
Feb 27 18:38:07 173 kernel: PM: Registered nosave memory: 00000000ff000000 - 0000000100000000
Feb 27 18:38:07 173 kernel: Allocating PCI resources starting at bf800000 (gap: bf800000:3f51c000)

There's more but I can not share it here because there is a 30,000 character limit for a post.

EDIT : LIKE LAST TIME, my multipackets received number spiked at around the time of the incident.

04:20:01 PM      eth1     82.02     84.18     13.40     93.86      0.00      0.00      0.00
04:30:01 PM        lo      0.03      0.03      0.00      0.00      0.00      0.00      0.00
04:30:01 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:30:01 PM      eth1     72.94     74.44      7.29     86.59      0.00      0.00      0.00
04:40:01 PM        lo      0.01      0.01      0.00      0.00      0.00      0.00      0.00
04:40:01 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:40:01 PM      eth1     77.25     83.17      6.75     99.82      0.00      0.00      0.00
04:50:01 PM        lo      0.01      0.01      0.00      0.00      0.00      0.00      0.00
04:50:01 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
04:50:01 PM      eth1     71.76     77.19      6.24     91.93      0.00      0.00      0.00
05:00:01 PM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:00:01 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:00:01 PM      eth1     64.39     63.84      8.28     71.20      0.00      0.00      0.00
05:10:01 PM        lo      0.03      0.03      0.00      0.00      0.00      0.00      0.00
05:10:01 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:10:01 PM      eth1     65.18     66.56      5.77     76.39      0.00      0.00      0.00
05:20:01 PM        lo      0.03      0.03      0.00      0.00      0.00      0.00      0.00
05:20:01 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:20:01 PM      eth1     92.23     89.68     33.78     88.57      0.00      0.00      0.00
05:30:01 PM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:30:01 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:30:01 PM      eth1     73.42     74.93      8.05     87.59      0.00      0.00      0.00
05:40:01 PM        lo      0.03      0.03      0.00      0.00      0.00      0.00      0.00
05:40:01 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:40:01 PM      eth1     60.54     58.69     19.79     59.61      0.00      0.00 507865361.98
05:50:01 PM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:50:01 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
05:50:01 PM      eth1      0.00      0.00      0.00      0.00      0.00      0.00 2146553474.33
06:00:01 PM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:00:01 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:00:01 PM      eth1      0.00      0.00      0.00      0.00      0.00      0.00 2150279010.21

06:00:01 PM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
06:10:01 PM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:10:01 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:10:01 PM      eth1      0.00      0.00      0.00      0.00      0.00      0.00 2145874241.82
06:20:01 PM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:20:01 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:20:01 PM      eth1      0.00      0.00      0.00      0.00      0.00      0.00 2150063723.97
06:30:01 PM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:30:01 PM      eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
06:30:01 PM      eth1      0.00      0.00      0.00      0.00      0.00      0.00 2146303180.75
Average:           lo      0.01      0.01      0.00      0.00      0.00      0.00      0.00
Average:         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:         eth1    122.77    119.70     24.14     84.60      0.00      0.00 101305280.31

06:38:02 PM       LINUX RESTART

06:40:01 PM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
06:50:01 PM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00

EDIT: /var/log/secure from around the time of the incident:

Feb 27 18:35:15 173 runuser: pam_unix(runuser:session): session opened for user qpidd by (uid=0)
Feb 27 18:35:15 173 runuser: pam_unix(runuser:session): session closed for user qpidd
Feb 27 18:35:16 173 sshd[1774]: Received signal 15; terminating.
Feb 27 18:35:16 173 sshd[5490]: Exiting on signal 15
Feb 27 18:35:16 173 sshd[5500]: Exiting on signal 15
Feb 27 18:35:16 173 sshd[5500]: pam_unix(sshd:session): session closed for user root
Feb 27 18:35:16 173 sshd[5490]: pam_unix(sshd:session): session closed for user root
Feb 27 18:35:16 173 sshd[5510]: Exiting on signal 15
Feb 27 18:35:16 173 sshd[5520]: Exiting on signal 15
Feb 27 18:35:16 173 sshd[5510]: pam_unix(sshd:session): session closed for user root
Feb 27 18:35:16 173 sshd[5520]: pam_unix(sshd:session): session closed for user root
Feb 27 18:35:16 173 sshd[23046]: Exiting on signal 15
Feb 27 18:35:16 173 sshd[23056]: Exiting on signal 15
Feb 27 18:35:16 173 sshd[23066]: Exiting on signal 15
Feb 27 18:35:16 173 sshd[23046]: pam_unix(sshd:session): session closed for user root
Feb 27 18:35:16 173 sshd[23056]: pam_unix(sshd:session): session closed for user root
Feb 27 18:35:16 173 sshd[23066]: pam_unix(sshd:session): session closed for user root
Feb 27 18:35:16 173 sshd[23076]: Exiting on signal 15
Feb 27 18:35:16 173 sshd[23076]: pam_unix(sshd:session): session closed for user root
Feb 27 18:38:19 173 sshd[5112]: Server listening on 0.0.0.0 port 22.
Feb 27 18:38:19 173 sshd[5112]: Server listening on :: port 22.
Feb 27 18:38:20 173 runuser: pam_unix(runuser:session): session opened for user qpidd by (uid=0)
Feb 27 18:38:20 173 runuser: pam_unix(runuser:session): session closed for user qpidd
Feb 27 18:38:20 173 runuser: pam_unix(runuser:session): session opened for user qpidd by (uid=0)
Feb 27 18:38:20 173 runuser: pam_unix(runuser:session): session closed for user qpidd
Feb 27 18:38:20 173 runuser: pam_unix(runuser-l:session): session opened for user qpidd by (uid=0)
Feb 27 18:38:20 173 runuser: pam_unix(runuser-l:session): session closed for user qpidd
Feb 27 18:38:34 173 sshd[5349]: Accepted publickey for root from MY IP port 40702 ssh2
Feb 27 18:38:34 173 sshd[5349]: pam_unix(sshd:session): session opened for user root by (uid=0)
Feb 27 18:40:57 173 sshd[5349]: Received disconnect from MY IP: 11: disconnected by user
Feb 27 18:40:57 173 sshd[5349]: pam_unix(sshd:session): session closed for user root

So when I first try to connect after the incident it seems that it would accept my publickey but it would not respond.

share|improve this question
    
oops I meant to ask this on serverfault. –  Emil Feb 28 '12 at 16:35
    
This seems to be a log of a server rebooting. Did you reboot it at 18:35? If so, there are no logs from around the time of the problem. (Did the node respond to pings? How did it respond to SSH connections? Did it ignore them? Reject them? Accept them but then not send any data? Did other services remain responsive? A few data points like that will rapidly divide the problem space.) –  David Schwartz Feb 28 '12 at 16:39
    
No I did not reboot it at 18:30, but I did have to use the remote control to power it on/off at around 20:00. I did not try pinging at the time of unresponsiveness. SSH connections would simply not connect, I would have to press CTRL+C to stop it from trying to connect. Other services like the apache2 server had also become unresponsive. –  Emil Feb 28 '12 at 19:45
add comment

migrated from superuser.com Feb 28 '12 at 18:43

This question came from our site for computer enthusiasts and power users.

1 Answer

Check /var/log/secure - if there are many Failed attempts, the sshd might be getting hung

If this is the case you probably want to move to key based auth or install something like Deny Hosts

share|improve this answer
    
I added /var/log/secure from around the incident, do you see anything suspicious? –  Emil Feb 28 '12 at 20:12
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.