Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

/var/log/secure:

su: pam_keyinit(su-l:session): Unable to change UID to 500 temporarily
su: pam_keyinit(su-l:session): Unable to change UID to 500 temporarily
su: pam_unix(su-l:session): session opened for user adtech by root(uid=0)
su: pam_unix(su-l:session): session closed for user adtech

I guess this is caused by per-user limit, but there is no different when comparing with another user.

Here're ulimit -n for adtech:

[adtech@hmaster87 root]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 192025
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 655360
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

and this one for quanta:

[quanta@hmaster87 ~]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 192025
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 655360
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

the number of processes running by adtech:

[root@hmaster87 ~]# ps -U adtech | wc -l
25

Any other thing to check?


UPDATE Sat Jul 21 09:21:26 ICT 2012:

# getent passwd adtech
adtech:x:500:502::/home/adtech:/bin/bash

As I said in the below comment, my co-worker had found out the process which maybe the culprit:

adtech 12901 1 0 08:58 ? 00:00:00 /home/adtech/nexus/bin/../bin/jsw/linux-x86-64/wrapper /home/adtech/nexus/bin/../bin/jsw/conf/wrapper.conf wrapper.syslog.ident=nexus wrapper.pidfile=/home/adtech/nexus/bin/../bin/jsw/linux-x86-64/nexus.pid wrapper.daemonize=TRUE

adtech 12903 12901 1 08:58 ? 00:00:24 java -Dsun.net.inetaddr.ttl=3600 -DbundleBasedir=. -Djava.io.tmpdir=./tmp -DjettyContext=nexus.properties -DjettyContextIncludeKeys=bundleBasedir -DjettyPlexusCompatibility=true -Djava.library.path=bin/jsw/lib -classpath bin/jsw/lib/wrapper-3.2.3.jar:./lib/plexus-classworlds-2.4.jar:./conf/ -Dwrapper.key=ejxHaBJASiFkAB8w -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.pid=12901 -Dwrapper.version=3.2.3 -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 org.codehaus.plexus.classworlds.launcher.Launcher ./conf/jetty.xml

By killing this process the problem go away but we still don't know which limit was exceeded.


UPDATE Sat Dec 15 00:56:13 ICT 2012:

The @favadi's answer is right, but I update here in case someone google's this thread.

The log file said that:

jvm 1    | Server daemon died!
jvm 1    | java.lang.OutOfMemoryError: unable to create new native thread
jvm 1    |      at java.lang.Thread.start0(Native Method)
jvm 1    |      at java.lang.Thread.start(Thread.java:640)
jvm 1    |      at org.tanukisoftware.wrapper.WrapperManager.privilegedStopInner(WrapperManager.java:3152)
jvm 1    |      at org.tanukisoftware.wrapper.WrapperManager.handleSocket(WrapperManager.java:3797)
jvm 1    |      at org.tanukisoftware.wrapper.WrapperManager.run(WrapperManager.java:4084)
jvm 1    |      at java.lang.Thread.run(Thread.java:662)
share|improve this question

migrated from unix.stackexchange.com Jul 28 '12 at 16:13

This question came from our site for users of Linux, FreeBSD and other Un*x-like operating systems..

    
Apologies if this is too obvious, but is there a userID 500 on your system? Does it relate to a userName that would be using? Good luck. –  shellter Jul 20 '12 at 17:00
    
Sure, adtech user has UID 500. See my updated. My co-worker had found out the process which is culprit. By killing this process the problem go away but we still don't know which limit was exceeded: open files does not, number of processes does not, maybe memory or anything else. Any thoughts? –  quanta Jul 21 '12 at 2:28
    
Try attaching strace -f -p to that process and looking for obviously failing syscalls and what they are trying to do... –  rackandboneman Dec 10 '12 at 11:58

4 Answers 4

up vote 5 down vote accepted
+100

May be max user processes (-u) 1024 is too low.

Remember that processes and threads are counting together. You can use ps -eLF | grep adtech | wc -l to show your current value.

share|improve this answer
4  
More precisely, it should be ps -eLF -U adtech | wc -l. –  quanta Dec 14 '12 at 18:03
1  
If you're wondering where that's set, look in /etc/security/limits.d/90-nproc.conf (assuming you're on a RH system). –  mricon Dec 14 '12 at 18:04

Look in the jvm log for evidence it is hitting resource limits. Stack size might be the problem, depending how many java threads the killed process was running.

Searching on your error message finds bug reports for pam_keyinit: check with your vendor's repository whether an updated version is available.

share|improve this answer
    
+1. I forgot the lesson: take a look at the log when getting the problem. Updated my question. –  quanta Dec 14 '12 at 18:09

The error was reported by pam_keyinit. Since I am not familiar with this module, I searched for documentation and found this manpage. Based on the description, I wonder if perhaps the process that you killed prevented the necessary access to some files that pam_keyinit needs to modify? Hopefully this gives you some direction.

share|improve this answer

This issue can happen if the user's process-run-limit is reached. The process limit can be increased by editing: /etc/security/limits.conf file with a user having root permission. The entry to check will be similar to:

*          hard     nproc         100

No need to restart any services.

share|improve this answer
    
In my case raising the hard limit had no immediate effect; I had to change the soft one. –  Nicola Musatti May 15 '13 at 10:54

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.