0

Recently I have rent a VPS as private playground and for learning docker. I want to host a few webservices behind nginx-proxy together with docker-letsencrypt-nginx-proxy-companion.

Setup

The Server runs Ubuntu 18.04.4 LTS, Docker version 19.03.6, build 369ce74a3c, docker-compose version 1.17.1, build unknown.

4GB vRAM, 4vCores (The hardware runs on Intel Xeon)

The Problem

As soon as I want to run more than 8 containers I get strage errors. There are a few different errors, and I can't especially say when which one of these appears:

  • OCI runtime create failed e.g. when trying to start joomla as compose-project with mysql:
ERROR: for joomla_joomladb_1  Cannot start service joomladb: OCI runtime create failed:
unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v1.linux/moby/741c[...]6e3/log.json:
no such file or directory): fork/exec /usr/sbin/runc: resource temporarily unavailable: unknown
  • socket reset
ERROR: for joomla_joomladb_1  Cannot start service joomladb: read unix @->@/containerd-shim/24f42b8[...]32fa1.sock:
read: connection reset by peer: unknown
  • pthread_create failed: Resource temporarily unavailable I don't have any corresponding logs right now. I will post them here if the error occurs once more. But one line always seems to be parent id not found
  • fork failed. When this error appears, I can't do anything anymore. See next point:

Additionally an in relation to that, sometimes I get some strange errors on my shell when typing any command. When the server is in this stage, I neither can su to root or reboot per command line. Yout can see me trying to type sudo in this picture: Errors with any command. Even shutdown or reboot

What I have tried

  • I have seen this report on Github and a few others and the refer to a maximum task limit on the machine. I already set that to a highter valume as well as to infinity but all of that doesn't seem to help. (I once changed it in /etc/systemd/system.conf as well as per systemctl edit docker.service).
$ systemctl show --property=DefaultTasksMax
 DefaultTasksMax=4659
$ systemctl status docker
...
  Tasks: 52
...
  • I installed exactly the same system on my local laptop (except the virtualization software - the VPS runs on Virtuozzo, locally I used Quemu with virtmanager) - and all works as expected!

  • The problems apper with any container I try to start. hello-world, an other compose project or an already built project...

  • I contacted my hosting provider, also the referred to the above issue and said they wouldn't limit any usage.

0

Now I found the problem: The hosting provider limits the number of tasks of the virtual machines. That technique is called user beancounters.

There is a file where you can inspect these limits that are set by Virtuozzo at /proc/user_beancounters

So, when you know that your VPS is hosted on Virtuozzo, you can cat this file or use of of the scripts from here to read this ressource limits.

The file mainly describes 3 fields:

  • held: the current usage of a resource
  • failcnt: number of all refused resource allocations
  • barrier/limit: a control parameter which generally defines the normal resource allocation border

(copied that from here)

If the displayed value in any field is insanely high, it means that these ressources aren't forced to a limit.

The crucial line in my case was numprocs. It tells the maximum number of tasks the machine is allowed to run. And as processes in a docker container also run on the same kernel as the VPS, the all count in.

So, the only solution here was to switch to another hoster, that doesn't use Virtuozzo (like that)!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.