I have VPS. It has Burstable memory service. My guaranteed memory limit is 1GB. I want linux not to take more that 1GB to programs on OS level. How to do it? It is ubuntu server.
I don't believe this is possible. If this were possible, it would require kernel tweaks, which isn't an option in your case due to the fact that you're running an OpenVZ and as such, don't have your own kernel.
This is one of the many reasons why OpenVZ-based VPSes are looked down upon here. There are a few use cases where they will work just fine, but in general, the limitations imposed by the platform create more issues than are worth dealing with.
I'd recommend ditching the OpenVZ VPS and instead using a VPS providers that uses Xen, KVM, etc., where this isn't an issue.