/dev/null file became regular file

Question

In our production server suddenly /dev/null became a regular file and due to this sshd service got stopped and not able to login the server. And also we tried to the below steps to configure back to character device file,

rm -rf /dev/null
mknod /dev/null c 1 3

As soon as we run the rm command /dev/null is being re-created as a regular file before mknod can run. We can't figure out how this happening and which component is creating this file. So until we solve this issue we are unable to create /dev/null as character device file.

Answer 1

When you delete (rm) /dev/null, any programs/scripts that are running and that needs to ">/dev/null" or equivalent will re-create a new (regular) file with that name. And those can spawn anytime (and some may also continously write to it)

To beat them:

you create a new /dev/null special file (under a different name)

mknod /dev/newnull c 1 3
chmod 777 /dev/newnull

and you move it over (as root) the continuously created ones:

mv -f /dev/newnull /dev/null

And only then you can reboot (don't reboot without a proper /dev/null file in place... it's usually not easy) [I forgot that step, which is of course necessary. Thanks @Random832 for the reminder!]

You need to reboot in the end, to get rid of existing program who will still have a "/dev/null" opened and still will write to the filesystem even though you replaced it afterwards, filling up that filesystem little by little) (Indeed, like when deleting a file, any program who still have that file descriptor opened will still be able to write to the former inode, even though the filename is now pointing to the new one)

Answer 2

You could run lsof /dev/null and see if there is a process that has it open but it would not show you what is happening in real time.

Another option would be to make the device and move it in place.

mknod /dev/null.tmp c 1 3 && mv /dev/null.tmp /dev/null

But I would want to know what is breaking the system first. Have you changed anything recently that may be causing this?

Answer 3

The reason why you can't recreate /dev/null is likely that something is contiuously writing to it like this:

echo "foo" > /dev/null

Examining the file's content will tell you what process it might be.

To fix your system for now, follow these instructions:

1. shutdown the system
2. boot with init=/bin/bash
3. remount / writeable
4. create the char device
5. reboot

I'd strongly suggest to do an intense examination of the system to determine how /dev/null got deleted. Make sure your system isn't compromised, check your system's log thoroughly.

Answer 4

I found the cause and the fix on my archlinux system.

If you use bash and HISTFILE=/dev/null is in environment, you shouldn’t execute more commands than $HISTFILESIZE or $HISTSIZE. If you executed more commands than $HISTFILESIZE on bash while HISTFILE is /dev/null and you exited bash, bash moves /dev/null somewhere else and recreates /dev/null as a regular file with permission 600.

If you use tramp on emacs 24.4, tramp-sh.el sets HISTFILE to /dev/null. Thus, if bash is the shell for root and if you do a lot of root operations with tramp on emacs 24.4, when you kill emacs, tramp makes bash delete /dev/null.

Please check if HISTFILE is set to /dev/null in .bashrc or in programs like emacs 24.4.

In my case, changing the shell to zsh works around the fact that tramp makes bash delete /dev/null on emacs 24.4.