Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Today one of our production machine (Amazon EC2) was down and I couldn't bring-up the instance since I can't get into SSH, connection was refused and I had no clue what to do, after sometime I was able to bring-up the instance - (ok short and sweet).

I somehow figured out the root-cause of the problem which was /dev/urandom file missing and hence, SSH was not able to start. I had to create this file upon reboot and I created it by adding few lines of code in one of the existing start-up (init) script and could get the server up and running means I can SSH into the box.

I need an expert advice on the following and please don't hesitate to give me more information:

  1. should I leave those lines of code that I had written in one of the init-script file?
  2. what could be the reason of missing /dev/urandom file?
  3. what should I do to avoid such situation in future?

Thanks.

UPDATE:

For those who want to know what I wrote:

#!/bin/bash
cd /dev ; /sbin/MAKEDEV urandom ; /etc/init.d/ssh start
share|improve this question

migrated from stackoverflow.com Dec 12 '11 at 14:35

This question came from our site for professional and enthusiast programmers.

2 Answers 2

up vote 4 down vote accepted

I'd recommend leaving the code in place. I know it sounds silly to even suggest it, but the fact that your /dev/urandom disappeared at all is definitely strange, and it might happen again. Modify your code to emit log messages that you cannot overlook if it needs to re-create device files in the future.

There are no good reasons for your /dev/urandom to be deleted. The absolute best that could be hoped for would be a tool such as puppet or chef being configured to write a directory of files from a tarball, and first clearing the directory entirely as part of expanding the tarball. (I would consider such a use a serious misconfiguration of the tool.) But any process running as root will have the privileges to remove the file, so it could be nearly anything.

You could configure auditd to watch file creates, deletes, and renames, in the /dev/ directory. Place rules into /etc/audit/audit.rules to configure persistent watches:

-w /dev/ -p wa

See the auditctl(8) for full details on configuring the audit watch lists; it is pretty configurable and you might need to tune the configuration to your system so "standard" events don't clutter your audit logs.

Another option is to set the immutable attribute on the file using the chattr(1) program. It's possible that whatever program or tool that might have removed the file might not go to the effort of removing the immutable attribute before deleting files.

share|improve this answer
    
good practices I have configured the auditd on /dev/, if you don't mind can you take a look at the above code and see if that makes sense to leave it in the init.d? –  RakeshS Dec 12 '11 at 9:28
    
It's probably fine to leave in, but I really think you'd be better off knowing whether or not the device node needed to be created. –  sarnold Dec 12 '11 at 9:32

Generally similar problems are noticed when you upgrade OS and due to some reason the upgrade breaks. Within AWS (EC2) you are not supposed to upgrade the kernel provided by them. Did you try to upgrade your OS lately? Its better you find the actual cause of the missing /dev/urandom and fix it. Till then leave the code there in init script. let us know how it goes.

share|improve this answer
    
well I had upgraded in the past and I am sure I had rebooted this machine after the upgrade, I didn't face this problem. –  RakeshS Dec 12 '11 at 8:39

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.