Ulimit man page suggests -r option will set this but it only appears to be valid for the root user and there is no equivalent for setting it via /etc/security/limits?
You said AIX, but in the case of linux I think it would just be the
The man page for setrlimit states:
So maybe it is the same for AIX? This link states: "AIX does not define RLIMIT_NPROC or RLIMIT_MEMLOCK resources.", but you might be able to find a more definitive answer on developerworks.
For the permissions issue, the solution may be to set the limit with root, and then su to the user you need to increase the limit for. I have had to do this for other resource limits in the past, for example, max open files. The limits.conf was not applied so I had do use sudo or su from root, and then the limit gets inherited.