19

In our production server suddenly /dev/null became a regular file and due to this sshd service got stopped and not able to login the server. And also we tried to the below steps to configure back to character device file,

rm -rf /dev/null
mknod /dev/null c 1 3

As soon as we run the rm command /dev/null is being re-created as a regular file before mknod can run. We can't figure out how this happening and which component is creating this file. So until we solve this issue we are unable to create /dev/null as character device file.

6
  • What OS and release are you using on the server? udev might be creating the file.
    – ptman
    Nov 8, 2013 at 12:36
  • Centos 5.2 and can you please elaborate how udev is creating this file.
    – user197719
    Nov 8, 2013 at 12:40
  • man fuser, you can find a process accessing a file and kill it. You can maybe put an attribute on such file - man chattr.
    – jirib
    Nov 8, 2013 at 12:40
  • I have no centos machines handy, but ubuntu 12.04 has a rule in /lib/udev/rules.d/50-udev-default.rules for creating /dev/null
    – ptman
    Nov 8, 2013 at 12:42
  • 3
    lsof /dev/null is your friend.
    – Andrew B
    Nov 8, 2013 at 20:22

4 Answers 4

29

When you delete (rm) /dev/null, any programs/scripts that are running and that needs to ">/dev/null" or equivalent will re-create a new (regular) file with that name. And those can spawn anytime (and some may also continously write to it)

To beat them:

you create a new /dev/null special file (under a different name)

mknod /dev/newnull c 1 3
chmod 777 /dev/newnull

and you move it over (as root) the continuously created ones:

mv -f /dev/newnull /dev/null

And only then you can reboot (don't reboot without a proper /dev/null file in place... it's usually not easy) [I forgot that step, which is of course necessary. Thanks @Random832 for the reminder!]

You need to reboot in the end, to get rid of existing program who will still have a "/dev/null" opened and still will write to the filesystem even though you replaced it afterwards, filling up that filesystem little by little) (Indeed, like when deleting a file, any program who still have that file descriptor opened will still be able to write to the former inode, even though the filename is now pointing to the new one)

2
  • 2
    He should still restart the server afterward - anything that did start writing to the ordinary file will continue writing to the deleted file and using up disk space.
    – Random832
    Nov 8, 2013 at 18:07
  • @Random832: very true, but at least rebooting AFTER having a new correct /dev/null file in place is much easier... (many programs and scripts depend on it to work correctly) Nov 8, 2013 at 18:17
8

You could run lsof /dev/null and see if there is a process that has it open but it would not show you what is happening in real time.

Another option would be to make the device and move it in place.

mknod /dev/null.tmp c 1 3 && mv /dev/null.tmp /dev/null

But I would want to know what is breaking the system first. Have you changed anything recently that may be causing this?

0
7

The reason why you can't recreate /dev/null is likely that something is contiuously writing to it like this:

echo "foo" > /dev/null

Examining the file's content will tell you what process it might be.

To fix your system for now, follow these instructions:

  1. shutdown the system
  2. boot with init=/bin/bash
  3. remount / writeable
  4. create the char device
  5. reboot

I'd strongly suggest to do an intense examination of the system to determine how /dev/null got deleted. Make sure your system isn't compromised, check your system's log thoroughly.

4

I found the cause and the fix on my archlinux system.

If you use bash and HISTFILE=/dev/null is in environment, you shouldn’t execute more commands than $HISTFILESIZE or $HISTSIZE. If you executed more commands than $HISTFILESIZE on bash while HISTFILE is /dev/null and you exited bash, bash moves /dev/null somewhere else and recreates /dev/null as a regular file with permission 600.

If you use tramp on emacs 24.4, tramp-sh.el sets HISTFILE to /dev/null. Thus, if bash is the shell for root and if you do a lot of root operations with tramp on emacs 24.4, when you kill emacs, tramp makes bash delete /dev/null.

Please check if HISTFILE is set to /dev/null in .bashrc or in programs like emacs 24.4.

In my case, changing the shell to zsh works around the fact that tramp makes bash delete /dev/null on emacs 24.4.

3
  • There is no reason writing HISTORY to /dev/null. You should set HISTSIZE to "0" to disable HISTORY entirely. Jan 30, 2015 at 9:15
  • You can also unset HISTFILE for disabling history, without doing anything to /dev/null. Jan 30, 2015 at 14:17
  • However, tramp-sh.el on emacs 24.4 sets HISTFILE to /dev/null, and there is nothing I can do about it for now. I symlinked /bin/dash to /bin/sh to work around the problem.
    – crocket
    Feb 1, 2015 at 4:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.