sudo: unable to create sockets: Cannot allocate memory

Question

On a VPS run by OVH (apparently OpenVZ-based, given /proc/user_beancounters exists), with relatively few processes running, trying to sudo gives me the error in the title.

Here is a sample transcript:

ekleog@ekleog:~$ sudo echo a
[sudo] password for ekleog:
sudo: unable to create sockets: Cannot allocate memory
ekleog@ekleog:~$ free -h
             total       used       free     shared    buffers     cached
Mem:          8.0G       212M       7.8G         0B         0B        43M
-/+ buffers/cache:       168M       7.8G
Swap:         128M         0B       128M
ekleog@ekleog:~$ sudo echo a
sudo: unable to create sockets: Cannot allocate memory

As you can see, there is no problem forking, as the shell forks to run free, but sudo seems to be unable to open a socket. In the same domain, thunderbird is unable to open an SMTP connexion, but ssh keeps tunneling new requests without any issue.

The fact that the issue originates from too many sockets open seems confirmed by the fact that, when closing Thunderbird (which keeps something like 50 connexions to monitor all my IMAP folders) the problem vanishes. Besides, when reopening it, the issue does not prop back, so there must be a resource leak somewhere?

I am currently having a single user (me), so I hope OVH's restrictions are not that serious.

Finally, during the "crisis", I tried running netstat (not really used to its use, so I may be wrong):

ekleog@ekleog:~$ netstat -a | wc -l
608
ekleog@ekleog:~$ cat /proc/sys/fs/file-max
1627524

It seems strange to me that sudo would block.

Do you have any idea how to stop having this? It comes up from time to time (approx. once every other day), and is getting pretty annoying.

Apparently the problem comes from OpenVZ settings, as in /proc/user_beancounters I have numothersock with a huge failcnt.

Trying to reduce the number of open sockets being dependent on each individual program, I will ask separate questions.

Answer 1

I am going to share my findings in this answer, hoping that will help someone in the future. These findings would be impossible without the prompt observation in @EEAA's comment.

Indeed the restriction comes from the OpenVZ software. The numothersock restriction can be seen in /proc/user_beancounters and according to the documentation: 'UBC resources in Parallels Virtuozzo Containers for Linux':

numothersock - maximum number of non-TCP sockets (local sockets, UDP, and other types of sockets).

You can check the amount of sockets with ss:

ss is used to dump socket statistics.

ss -xa | wc -l

To identify which process binds each socket:

sudo ss -xap

For my specific case it turns out that more than 25% of my restriction in sockets was due to postfix, so I cut down the default_process_limit parameter in /etc/postfix/main.cf (relevant question here).

Answer 2

(Synonymous answer but adding some diagnostic info and search keywords.)

Symptoms

The issue at hand is that no new non-TCP sockets can be created, which means mostly Unix domain sockets used for inter-process communication in the system. It may not be reproducible always, as sometimes a few sockets are deleted, opening up a few options to create new sockets below the limit.

Symptoms are not restricted to sudo use but will affect most or all websites running on your server and using MySQL via a socket connection. Error messages would be like these:

PDOException: SQLSTATE[HY000] [2001] Can't create UNIX socket (12) in […]
PDOException: SQLSTATE[HY000] [2002] Can't connect to local MySQL server 
    through socket '/var/run/mysqld/mysqld.sock' (12) in […]
PHP Warning:  mysqli_connect(): (HY000/2001): Can't create UNIX socket 
    (12) in […]
PHP Warning:  mysqli_connect(): (HY000/2002): Can't connect to local MySQL 
    server through socket '/var/run/mysqld/mysqld.sock' (12) in […]

In addition, sending and fetching e-mails via a postfix resp. dovecot server will probably also fail, at the login stage. Error messages include:

mailq: fatal: inet_addr_local[getifaddrs]: getifaddrs: Cannot allocate memory

The common element in these errors is error code 12, which means "OS error code  12:  Cannot allocate memory" as per perror 12 [source].

Solution

Look at the output of cat /proc/user_beancounters, which shows the OpenVZ virtualization environment limits of your VPS. In all probability, you will see a huge number of fails for the numothersock resource limit there (>10,000 for me). This means software tried to create a socket but was prohibited by OpenVZ from doing so because the maximum number of non-TCP sockets already existed.

To see the number and use of all sockets counted into this numothersock resource limit, look at the output file generated by:

ss --processes --all --socket=udp,unix,unix_dgram,unix_stream > results.txt 

In my case, it showed that about 80% of sockets were related to postfix and dovecot processes. Postfix related sockets can be reduced to about 10% by limiting the number of concurrent connections from default 100 to a reasonable 10 – see the instructions. After applying that solution to postfix (default_process_limit = 10) and restarting it, my sockets in use went from ~1020 to ~520 immediately. Problem solved for now.