Error: No space left on device when starting/stopping services only

Question

I know there are multiple questions like this already, but none of their answers applies to my problem.

So I tried to upgrade from Debian 7.7 to Debian 8.1. Every works fine, but certain operations reports Error: No space left on device.

df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       443G   28G  393G   7% /
udev             10M     0   10M   0% /dev
tmpfs           1.6G  8.7M  1.6G   1% /run
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs           795M     0  795M   0% /run/user/0

df -i

Filesystem       Inodes  IUsed    IFree IUse% Mounted on
/dev/sda1      29450240 419083 29031157    2% /
udev            1015256    338  1014918    1% /dev
tmpfs           1017372    502  1016870    1% /run
tmpfs           1017372      1  1017371    1% /dev/shm
tmpfs           1017372      7  1017365    1% /run/lock
tmpfs           1017372     13  1017359    1% /sys/fs/cgroup
tmpfs           1017372      4  1017368    1% /run/user/0

Example of error messages:

$: service apache2 restart
Error: No space left on device

$: /etc/init.d/mysql restart
[....] Restarting mysql (via systemctl): mysql.serviceError: No space left on device

I can also touch and add/delete files in /tmp. ls -al:

drwxrwxrwt   8 root root  4096 Jul 23 18:47 tmp

How do I go forward trying to debug or figure out what is going on here? I suspect there is something wrong with the /tmp directory, but I have no idea what.

Answer 1

After much debugging I found the answer in another SE answer over at unix.SE.

It turns out that this error was triggered because there were too few watchers for inotify. In my case, this problem was not logged in any error log anywhere in the system.

Simply editing the file /etc/sysctl.conf adding:

fs.inotify.max_user_watches = 262144

...and then running sysctl -p solved the problem.