Linux - SSH Connection refused due to non-existence of /dev/urandom

Question

Today one of our production machine (Amazon EC2) was down and I couldn't bring-up the instance since I can't get into SSH, connection was refused and I had no clue what to do, after sometime I was able to bring-up the instance - (ok short and sweet).

I somehow figured out the root-cause of the problem which was /dev/urandom file missing and hence, SSH was not able to start. I had to create this file upon reboot and I created it by adding few lines of code in one of the existing start-up (init) script and could get the server up and running means I can SSH into the box.

I need an expert advice on the following and please don't hesitate to give me more information:

1. should I leave those lines of code that I had written in one of the init-script file?
2. what could be the reason of missing /dev/urandom file?
3. what should I do to avoid such situation in future?

Thanks.

UPDATE:

For those who want to know what I wrote:

#!/bin/bash
cd /dev ; /sbin/MAKEDEV urandom ; /etc/init.d/ssh start

Answer 1

I'd recommend leaving the code in place. I know it sounds silly to even suggest it, but the fact that your /dev/urandom disappeared at all is definitely strange, and it might happen again. Modify your code to emit log messages that you cannot overlook if it needs to re-create device files in the future.

There are no good reasons for your /dev/urandom to be deleted. The absolute best that could be hoped for would be a tool such as puppet or chef being configured to write a directory of files from a tarball, and first clearing the directory entirely as part of expanding the tarball. (I would consider such a use a serious misconfiguration of the tool.) But any process running as root will have the privileges to remove the file, so it could be nearly anything.

You could configure auditd to watch file creates, deletes, and renames, in the /dev/ directory. Place rules into /etc/audit/audit.rules to configure persistent watches:

-w /dev/ -p wa

See the auditctl(8) for full details on configuring the audit watch lists; it is pretty configurable and you might need to tune the configuration to your system so "standard" events don't clutter your audit logs.

Another option is to set the immutable attribute on the file using the chattr(1) program. It's possible that whatever program or tool that might have removed the file might not go to the effort of removing the immutable attribute before deleting files.

Answer 2

Generally similar problems are noticed when you upgrade OS and due to some reason the upgrade breaks. Within AWS (EC2) you are not supposed to upgrade the kernel provided by them. Did you try to upgrade your OS lately? Its better you find the actual cause of the missing /dev/urandom and fix it. Till then leave the code there in init script. let us know how it goes.