Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

So last week, an instance on EC2 stopped responding, I still don't know exactly why because I can no longer SSH in, I suspect the /tmp/ directory which was mounted to another drive is no longer accessible for some unknown reason.

I have some very important files I need to get off this server...

I am still able to pull the logs in the AWS console, here are some very relevant lines (I am still able to reboot the server):

        Welcome to  CentOS release 5.4 (Final)
        Press 'I' to enter interactive startup.
Cannot access the Hardware Clock via any known method.
Use the --debug option to see the details of our search for an access method.
Setting clock : Thu Dec 29 13:52:43 EST 2011 [  OK  ]

Starting udev: [  OK  ]

Setting hostname localhost.localdomain:  [  OK  ]

No devices found
Setting up Logical Volume Management: File descriptor 7 (/sys/kernel/hotplug) leaked on lvm.static invocation. Parent PID 232: /bin/bash
[  OK  ]

Checking filesystems
Checking all file systems.
[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/sda1 
/dev/sda1: clean, 202786/1310720 files, 1428718/2621440 blocks
[  OK  ]

Remounting root filesystem in read-write mode:  [  OK  ]

Mounting local filesystems:  [  OK  ]

Enabling local filesystem quotas:  [  OK  ]

chown: cannot access `/tmp/.ICE-unix': No such file or directory
Enabling /etc/fstab swaps:  [  OK  ]

INIT: Entering runlevel: 4

Entering non-interactive startup
Starting background readahead: [  OK  ]

Bringing up loopback interface:  [  OK  ]

Bringing up interface eth0:  
Determining IP information for eth0...mktemp: cannot create temp file /tmp/wnt890: No such file or directory
/sbin/dhclient-script: line 57: $rscf: ambiguous redirect
/sbin/dhclient-script: line 62: $rscf: ambiguous redirect
/sbin/dhclient-script: line 69: $rscf: ambiguous redirect
[  OK  ]

Starting getsshkey:  /etc/rc4.d/S11getsshkey: line 12: /tmp/my-key: No such file or directory
getting ssh-key...
/etc/rc4.d/S11getsshkey: line 17: /tmp/my-key: No such file or directory
getting ssh-key...

I'm certain its not a firewall issue. Here is the output of nmap

[root@ip-xxxxxxxxx ~]# nmap -sS -P0 xxxxxxxxxxx

Starting Nmap 4.11 ( ) at 2011-12-29 16:32 EST
Interesting ports on xxxxxx (xxxxxxxxx):
Not shown: 1675 filtered ports
22/tcp   closed ssh
25/tcp   closed smtp
80/tcp   closed http
443/tcp  closed https
8000/tcp closed http-alt
share|improve this question

migrated from Dec 29 '11 at 22:47

This question came from our site for professional and enthusiast programmers.

It's mentioning /tmp/my-key because that's the location used by the startup script to temporarily store your public SSH key (the one associated with the ssh keypair for the instance) before it copies it to ~/.ssh/authorized_keys. So you have no ssh access. What happened to the drive that was mounted at /tmp ? Was it an EBS device? If so, create a new one and give it the relevant device name so it is mounted at /tmp. – James Little Dec 29 '11 at 22:03

I don't think asking anyone on here to help you "hack into a server" is particularly conducive to answers.

  1. Create a snapshot of your running EC2 instance
  2. Create a new instance.
  3. Mount the snapshot as a new EBS volume on the instance.
  4. Copy the data off the snapshot
  5. Kill the previous and new virtual machine instances.

Ta Dah! You've just recovered the data, no hacking involved.

Some tools here might help.

share|improve this answer
hacking doesn't have to be malicious – Joel K Dec 29 '11 at 23:09
It's all about context. – Tom O'Connor Dec 29 '11 at 23:24
Actually the data is stored on the instance, so I will be paying Amazon Premium Support to hack into the server, ethically. – Eduardo Dec 30 '11 at 20:06
@Dan I don't think that's necessary -- taking a snapshot of the instance storage, as Tom suggested, should work around the need for paid support. – Skyhawk Dec 30 '11 at 22:44
@TomO'Connor - Let's not get sidetracked over Dan's choice of wording and instead focus on solving the problem. Will your suggested plan of action work for an Instance Store backed instance? Will the Instance Store be included in the snapshot? – Mark Johnson Jan 2 '12 at 21:23

The basics for accessing and fixing an EBS root volume (e.g., edit /etc/fstab) when you can't access the instance are:

  1. Stop the original instance A and detach the volume.
  2. Start a temporary instance B, attach the volume to it, and mount the volume.
  3. Access instance B and fix the files on the attached/mounted volume.
  4. umount the volume, detach it from instance B, and terminate the temporary instance B.
  5. Attach the volume back to the original instance A and start instance A.

Here's an article I wrote that has more detail including sample command lines on how to get out of situations like this:

Fixing Files on the Root EBS Volume of an EC2 Instance

This only works for EBS boot instances. I don't recommend running instance-store as it can get you into situations like this with no way to recover your data.

share|improve this answer
up vote 0 down vote accepted

So my data is actually stored on the instance itself, so neither of these really works. The actual solution is basically to sign up for AWS support and maybe get your data. Here's an excerpt from the EC2 Instance FAQ

"Data Recovery Data recovery of the instance store is usually not possible, although AWS Premium Support may be able to recover some portion of the data if the instance has not been terminated and no underlying hardware issues exist. Data recovery is not a guaranteed process though and can take days to complete, so do not rely on the possibility of data recovery by AWS Premium Support as your sole backup strategy."

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.