1

My instance was running fine and this morning I was able to access a Ruby on Rails app hosted on it. An hour later I suddenly wasn't able to access my site, my SSH connection attempts were refused and the server wasn't even responding to ping. I didn't change anything on my system during that hour and reboots aren't fixing it.

I've never had any problems connecting or pinging the system before.

Can someone please help? This is on my production system!

  • OS: CentOS 5
  • AMI ID: ami-10b55379
  • Type: m1.small


[] ~% ssh -v *****@meeteor.com 
OpenSSH_5.2p1, OpenSSL 0.9.8l 5 Nov 2009 
debug1: Reading configuration data /etc/ssh_config 
debug1: Connecting to meeteor.com [184.73.235.191] port 22. 
debug1: connect to address 184.73.235.191 port 22: Connection refused 
ssh: connect to host meeteor.com port 22: Connection refused 
[] ~% ping meeteor.com 
PING meeteor.com (184.73.235.191): 56 data bytes 
Request timeout for icmp_seq 0 
Request timeout for icmp_seq 1 
Request timeout for icmp_seq 2 
^C 
--- meeteor.com ping statistics --- 
4 packets transmitted, 0 packets received, 100.0% packet loss 
[] ~% 

========= System Log =========

Restarting system. 
Linux version 2.6.16-xenU ([email protected]) (gcc version 4.0.1 20050727 (Red Hat 4.0.1-5)) #1 SMP Mon May 28 03:41:49 SAST 2007 
BIOS-provided physical RAM map: 
Xen: 0000000000000000 - 000000006a400000 (usable) 
980MB HIGHMEM available. 
727MB LOWMEM available. 
NX (Execute Disable) protection: active 
IRQ lockup detection disabled 
Built 1 zonelists 
Kernel command line: root=/dev/sda1 ro 4 
Enabling fast FPU save and restore... done. 
Enabling unmasked SIMD FPU exception support... done. 
Initializing CPU#0 
PID hash table entries: 4096 (order: 12, 65536 bytes) 
Xen reported: 2599.998 MHz processor. 
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes) 
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes) 
Software IO TLB disabled 
vmalloc area: ee000000-f53fe000, maxmem 2d7fe000 
Memory: 1718700k/1748992k available (1958k kernel code, 20948k reserved, 620k data, 144k init, 1003528k highmem) 
Checking if this processor honours the WP bit even in supervisor mode... Ok. 
Calibrating delay using timer specific routine.. 5202.30 BogoMIPS (lpj=26011526) 
Mount-cache hash table entries: 512 
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line) 
CPU: L2 Cache: 1024K (64 bytes/line) 
Checking 'hlt' instruction... OK. 
Brought up 1 CPUs 
migration_cost=0 
Grant table initialized 
NET: Registered protocol family 16 
Brought up 1 CPUs 
xen_mem: Initialising balloon driver. 
highmem bounce pool size: 64 pages 
VFS: Disk quotas dquot_6.5.1 
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes) 
Initializing Cryptographic API 
io scheduler noop registered 
io scheduler anticipatory registered (default) 
io scheduler deadline registered 
io scheduler cfq registered 
i8042.c: No controller found. 
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize 
Xen virtual console successfully installed as tty1 
Event-channel device installed. 
netfront: Initialising virtual ethernet driver. 
mice: PS/2 mouse device common for all mice 
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27 
md: bitmap version 4.39 
NET: Registered protocol family 2 
Registering block device major 8 
IP route cache hash table entries: 65536 (order: 6, 262144 bytes) 
TCP established hash table entries: 262144 (order: 9, 2097152 bytes) 
TCP bind hash table entries: 65536 (order: 7, 524288 bytes) 
TCP: Hash tables configured (established 262144 bind 65536) 
TCP reno registered 
TCP bic registered 
NET: Registered protocol family 1 
NET: Registered protocol family 17 
NET: Registered protocol family 15 
Using IPI No-Shortcut mode 
md: Autodetecting RAID arrays. 
md: autorun ... 
md: ... autorun DONE. 
kjournald starting. Commit interval 5 seconds 
EXT3-fs: mounted filesystem with ordered data mode. 
VFS: Mounted root (ext3 filesystem) readonly. 
Freeing unused kernel memory: 144k freed 
*************************************************************** 
*************************************************************** 
** WARNING: Currently emulating unsupported memory accesses ** 
** in /lib/tls glibc libraries. The emulation is ** 
** slow. To ensure full performance you should ** 
** install a 'xen-friendly' (nosegneg) version of ** 
** the library, or disable tls support by executing ** 
** the following as root: ** 
** mv /lib/tls /lib/tls.disabled ** 
** Offending process: init (pid=1) ** 
*************************************************************** 
*************************************************************** 
Pausing... 5Pausing... 4Pausing... 3Pausing... 2Pausing... 1Continuing... 
INIT: version 2.86 booting 
Welcome to CentOS release 5.4 (Final) 
Press 'I' to enter interactive startup. 
Setting clock : Fri Oct 1 14:35:26 EDT 2010 [ OK ] 
Starting udev: [ OK ] 
Setting hostname localhost.localdomain: [ OK ] 
No devices found 
Setting up Logical Volume Management: [ OK ] 
Checking filesystems 
Checking all file systems. 
[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/sda1 
/dev/sda1: clean, 275424/1310720 files, 1161123/2621440 blocks 
[ OK ] 
Remounting root filesystem in read-write mode: [ OK ] 
Mounting local filesystems: [ OK ] 
Enabling local filesystem quotas: [ OK ] 
Enabling /etc/fstab swaps: [ OK ] 
INIT: Entering runlevel: 4 
Entering non-interactive startup 
Starting background readahead: [ OK ] 
Applying ip6tables firewall rules: modprobe: FATAL: Module ip6_tables not found. 
ip6tables-restore v1.3.5: ip6tables-restore: unable to initializetable 'filter' 
Error occurred at line: 3 
Try `ip6tables-restore -h' or 'ip6tables-restore --help' for more information. 
[FAILED] 
Applying iptables firewall rules: [ OK ] 
Loading additional iptables modules: ip_conntrack_netbios_ns [ OK ] 
Bringing up loopback interface: [ OK ] 
Bringing up interface eth0: 
Determining IP information for eth0... done. 
[ OK ] 
Starting auditd: [FAILED] 
Starting irqbalance: [ OK ] 
Starting portmap: [ OK ] 
FATAL: Module lockd not found. 
Starting NFS statd: [ OK ] 
Starting RPC idmapd: FATAL: Module sunrpc not found. 
FATAL: Error running install command for sunrpc 
Error: RPC MTAB does not exist. 
Starting system message bus: [ OK ] 
Starting Bluetooth services:[ OK ] 
[ OK ] 
Can't open RFCOMM control socket: Address family not supported by protocol 
Mounting other filesystems: [ OK ] 
Starting PC/SC smart card daemon (pcscd): [ OK ] 
Starting hidd: Can't open HIDP control socket: Address family not supported by protocol 
[FAILED] 
Starting autofs: Starting automount: automount: test mount forbidden or incorrect kernel protocol version, kernel protocol version 5.00 or above required. 
[FAILED] 
[FAILED] 
Starting sshd: [ OK ] 
Starting cups: [ OK ] 
Starting sendmail: [ OK ] 
Starting sm-client: [ OK ] 
Starting console mouse services: no console device found[FAILED] 
Starting crond: [ OK ] 
Starting xfs: [ OK ] 
Starting anacron: [ OK ] 
Starting atd: [ OK ] 
% Total % Received % Xferd Average Speed Time Time Time Current 
Dload Upload Total Spent Left Speed 
100 390 100 390 0 0 58130 0 --:--:-- --:--:-- --:--:-- 58130 
100 390 100 390 0 0 56984 0 --:--:-- --:--:-- --:--:-- 0 
Starting yum-updatesd: [ OK ] 
Starting Avahi daemon... [ OK ] 
Starting HAL daemon: [ OK ] 
Starting OSSEC: [ OK ] 
Starting smartd: [ OK ] 
c 
CentOS release 5.4 (Final) 
Kernel 2.6.16-xenU on an i686 
domU-12-31-39-00-C4-97 login: 
INIT: Id "2" respawning too fast: disabled for 5 minutes 
INIT: Id "3" respawning too fast: disabled for 5 minutes 
INIT: Id "4" respawning too fast: disabled for 5 minutes 
INIT: Id "5" respawning too fast: disabled for 5 minutes 
INIT: Id "6" respawning too fast: disabled for 5 minutes 

3 Answers 3

1

It's hard to figure out what went wrong exactly. From the console output (note: not the system log) it looks like ssh started and despite the errors, the server finished the boot process - I see the login prompt.

If it's still down, I'd recover the EBS volume with your data and move on to a new instance. Get used to it. In the cloud you don't get attached to servers. Throw away and relaunch!

And you should definitely post about these kind of issues on the (official) AWS forums.

1
  • I really wish we had setup an EBS Volume...
    – Chris
    Oct 2, 2010 at 18:31
0

For whatever it's worth, your site seems to be accessible to me right now.

0

The SSH connection was actively refused either by your server or a firewall in between. Your pings aren't returning and others say your site is up. I would check the firewall rules in your server or in your local machine (or gateway). Not enough information.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .