Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

In our production server suddenly /dev/null became a regular file and due to this sshd service got stopped and not able to login the server. And also we tried to the below steps to configure back to character device file,

rm -rf /dev/null
mknod /dev/null c 1 3

As soon as we run the rm command /dev/null is being re-created as a regular file before mknod can run. We can't figure out how this happening and which component is creating this file. So until we solve this issue we are unable to create /dev/null as character device file.

share|improve this question
    
What OS and release are you using on the server? udev might be creating the file. –  ptman Nov 8 '13 at 12:36
    
Centos 5.2 and can you please elaborate how udev is creating this file. –  user197719 Nov 8 '13 at 12:40
    
man fuser, you can find a process accessing a file and kill it. You can maybe put an attribute on such file - man chattr. –  Jiri Xichtkniha Nov 8 '13 at 12:40
    
I have no centos machines handy, but ubuntu 12.04 has a rule in /lib/udev/rules.d/50-udev-default.rules for creating /dev/null –  ptman Nov 8 '13 at 12:42
2  
lsof /dev/null is your friend. –  Andrew B Nov 8 '13 at 20:22

3 Answers 3

Any programs/scripts that are used to ">/dev/null" or equivalent will re-create a new file. And those can spawn anytime (and some may also continously write to it)

To beat them:

you create a new /dev/null special file (under a different name)

mknod /dev/newnull c 1 3
chmod 777 /dev/newnull

and you move it over (as root) the continuously created ones:

mv -f /dev/newnull /dev/null

And only then you can reboot (don't reboot without a proper /dev/null file in place... it's usually not easy) [I forgot that step, which is of course necessary. Thanks @Random832 for the reminder!]

You need to reboot in the end, to get rid of existing program who will still have a "/dev/null" opened and still will write to the filesystem even though you replaced it afterwards, filling up that filesystem little by little) (Indeed, like when deleting a file, any program who still have that file descriptor opened will still be able to write to the former inode, even though the filename is now pointing to the new one)

share|improve this answer
1  
He should still restart the server afterward - anything that did start writing to the ordinary file will continue writing to the deleted file and using up disk space. –  Random832 Nov 8 '13 at 18:07
    
@Random832: very true, but at least rebooting AFTER having a new correct /dev/null file in place is much easier... (many programs and scripts depend on it to work correctly) –  Olivier Dulac Nov 8 '13 at 18:17

The reason why you can't recreate /dev/null is likely that something is contiuously writing to it like this:

echo "foo" > /dev/null

Examining the file's content will tell you what process it might be.

To fix your system for now, follow these instructions:

  1. shutdown the system
  2. boot with init=/bin/bash
  3. remount / writeable
  4. create the char device
  5. reboot

I'd strongly suggest to do an intense examination of the system to determine how /dev/null got deleted. Make sure your system isn't compromised, check your system's log thoroughly.

share|improve this answer

You could run lsof /dev/null and see if there is a process that has it open but it would not show you what is happening in real time.

Another option would be to make the device and move it in place.

mknod /dev/null.tmp c 1 3 && mv /dev/null.tmp /dev/null

But I would want to know what is breaking the system first. Have you changed anything recently that may be causing this?

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.