You could do it by user by using the built in
limits.conf. You could add all the Git users to an arbitrary group then limit the memory for that group.
But I'm not really condoning this, as if it needs memory - it needs memory. I would just imagine the Git process would terminate early as its out-of-memory - saving the VPS from crashing, but not really achieving anything.
A way to do it ...
Eg. Your Git group is called gitgroup and you've got 3 existing users, bob, jane and tim
Create the group
Then add the users to that group
usermod -a -G gitgroup bob
usermod -a -G gitgroup jane
usermod -a -G gitgroup tim
#groupname type item limit
@gitgroup hard rss 512000
This will limit that group (thus any user in it) to a hard memory limit of 512000KB. This may have other side effects too on other running processes owned by those users.
It may be more preferable just to add a memory limit to the particular users shell when they log in.
You could do this by add this to the user's respective
ulimit -v 512000
The solution ...
You need more RAM. If you are trying to do something and run out of memory in the process - then either don't do what your doing, or add more memory. Clearly your hardware is insufficient for what your trying to do.