Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

In our production server suddenly /dev/null became a regular file and due to this sshd service got stopped and not able to login the server. And also we tried to the below steps to configure back to character device file,

rm -rf /dev/null
mknod /dev/null c 1 3

As soon as we run the rm command /dev/null is being re-created as a regular file before mknod can run. We can't figure out how this happening and which component is creating this file. So until we solve this issue we are unable to create /dev/null as character device file.

share|improve this question
What OS and release are you using on the server? udev might be creating the file. –  ptman Nov 8 '13 at 12:36
Centos 5.2 and can you please elaborate how udev is creating this file. –  user197719 Nov 8 '13 at 12:40
man fuser, you can find a process accessing a file and kill it. You can maybe put an attribute on such file - man chattr. –  Jiri Xichtkniha Nov 8 '13 at 12:40
I have no centos machines handy, but ubuntu 12.04 has a rule in /lib/udev/rules.d/50-udev-default.rules for creating /dev/null –  ptman Nov 8 '13 at 12:42
lsof /dev/null is your friend. –  Andrew B Nov 8 '13 at 20:22

4 Answers 4

Any programs/scripts that are used to ">/dev/null" or equivalent will re-create a new file. And those can spawn anytime (and some may also continously write to it)

To beat them:

you create a new /dev/null special file (under a different name)

mknod /dev/newnull c 1 3
chmod 777 /dev/newnull

and you move it over (as root) the continuously created ones:

mv -f /dev/newnull /dev/null

And only then you can reboot (don't reboot without a proper /dev/null file in place... it's usually not easy) [I forgot that step, which is of course necessary. Thanks @Random832 for the reminder!]

You need to reboot in the end, to get rid of existing program who will still have a "/dev/null" opened and still will write to the filesystem even though you replaced it afterwards, filling up that filesystem little by little) (Indeed, like when deleting a file, any program who still have that file descriptor opened will still be able to write to the former inode, even though the filename is now pointing to the new one)

share|improve this answer
He should still restart the server afterward - anything that did start writing to the ordinary file will continue writing to the deleted file and using up disk space. –  Random832 Nov 8 '13 at 18:07
@Random832: very true, but at least rebooting AFTER having a new correct /dev/null file in place is much easier... (many programs and scripts depend on it to work correctly) –  Olivier Dulac Nov 8 '13 at 18:17

The reason why you can't recreate /dev/null is likely that something is contiuously writing to it like this:

echo "foo" > /dev/null

Examining the file's content will tell you what process it might be.

To fix your system for now, follow these instructions:

  1. shutdown the system
  2. boot with init=/bin/bash
  3. remount / writeable
  4. create the char device
  5. reboot

I'd strongly suggest to do an intense examination of the system to determine how /dev/null got deleted. Make sure your system isn't compromised, check your system's log thoroughly.

share|improve this answer

You could run lsof /dev/null and see if there is a process that has it open but it would not show you what is happening in real time.

Another option would be to make the device and move it in place.

mknod /dev/null.tmp c 1 3 && mv /dev/null.tmp /dev/null

But I would want to know what is breaking the system first. Have you changed anything recently that may be causing this?

share|improve this answer

I found the cause and the fix on my archlinux system.

If you use bash and HISTFILE=/dev/null is in environment, you shouldn’t execute more commands than $HISTFILESIZE or $HISTSIZE. If you executed more commands than $HISTFILESIZE on bash while HISTFILE is /dev/null and you exited bash, bash moves /dev/null somewhere else and recreates /dev/null as a regular file with permission 600.

If you use tramp on emacs 24.4, tramp-sh.el sets HISTFILE to /dev/null. Thus, if bash is the shell for root and if you do a lot of root operations with tramp on emacs 24.4, when you kill emacs, tramp makes bash delete /dev/null.

Please check if HISTFILE is set to /dev/null in .bashrc or in programs like emacs 24.4.

In my case, changing the shell to zsh works around the fact that tramp makes bash delete /dev/null on emacs 24.4.

share|improve this answer
There is no reason writing HISTORY to /dev/null. You should set HISTSIZE to "0" to disable HISTORY entirely. –  Tim Haegele Jan 30 at 9:15
You can also unset HISTFILE for disabling history, without doing anything to /dev/null. –  Michael Hampton Jan 30 at 14:17
However, tramp-sh.el on emacs 24.4 sets HISTFILE to /dev/null, and there is nothing I can do about it for now. I symlinked /bin/dash to /bin/sh to work around the problem. –  Minsky Feb 1 at 4:50

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.