We have a Java application which is significantly slower in Solaris 10 server than in Windows PC.
We have profiled (-Xprof) the application and observed that UnixFileSystem.getBooleanAttributes0 method consumes about 40% CPU time with native calls.
How can we follow our search to identify which is the cause of the slow behaviour?
Update:
We have executed dtrace
and we observe the CPU time is 0.489 sec, while the time
results are
real 0m40.759s
user 0m47.669s
sys 0m2.287s
It seems time is not spent on CPU but in another place. How can we proceed with further investigation having this dtrace
results? Can we use another dtrace
options to show more info?
Thank you very much.
# /dtrace-toolkit/DTraceToolkit-0.99/procsystime -aTp 5695
Tracing... Hit Ctrl-C to end...
dtrace: 1032 dynamic variable drops with non-empty dirty list
^C
Elapsed Times for PID 5695,
SYSCALL TIME (ns)
gtime 7799
sysconfig 8018
lwp_kill 10105
getsockname 10215
systeminfo 10984
getrlimit 11093
lwp_self 20210
schedctl 26361
lwp_continue 27349
priocntlsys 33390
unlink 53050
lwp_sigmask 136303
setcontext 143447
so_socketpair 202427
lwp_create 223626
brk 227141
so_socket 263276
memcntl 461091
ioctl 1094955
lwp_cond_broadcast 1946728
lwp_mutex_wakeup 2579492
lwp_cond_signal 2903277
mmap 3082973
access 4446250
mprotect 4534229
getdents64 5713309
mmap64 6272480
fsat 6925239
stat 6926115
open 8324983
fcntl 8840868
close 11264404
llseek 11678791
munmap 11867401
fstat64 17852129
lwp_park 22098570
open64 23549922
stat64 39673214
read 41876097
resolvepath 81905859
yield 175946072
write 225507878
lwp_mutex_timedlock 887174392
pollsys 20393318412
lwp_cond_wait 345099369278
TOTAL: 367108549202
CPU Times for PID 5695,
SYSCALL TIME (ns)
gtime 1208
sysconfig 1647
lwp_self 2198
systeminfo 2856
lwp_kill 3186
getsockname 3625
getrlimit 4613
lwp_continue 13839
schedctl 14059
priocntlsys 14498
lwp_sigmask 20541
unlink 45801
setcontext 96874
brk 121804
lwp_park 158601
so_socketpair 195178
lwp_create 209017
so_socket 241199
memcntl 365972
ioctl 606397
lwp_cond_broadcast 1367556
lwp_mutex_wakeup 1561103
lwp_cond_signal 1803831
access 1885436
mmap 2655710
llseek 3266410
open 3614795
fsat 4043369
getdents64 4253373
mprotect 4345864
close 4547395
fcntl 4871095
stat 5183012
mmap64 5308789
fstat64 8141116
munmap 10774529
pollsys 11928811
lwp_mutex_timedlock 19212980
open64 19443958
lwp_cond_wait 23144761
read 28043647
stat64 31710269
resolvepath 70641662
yield 77731698
write 137678654
TOTAL: 489282936
Syscall Counts for PID 5695,
SYSCALL COUNT
getrlimit 1
getsockname 1
gtime 1
lwp_kill 1
rexit 1
so_socketpair 1
sysconfig 1
systeminfo 1
unlink 1
lwp_continue 2
lwp_create 2
schedctl 2
lwp_self 3
priocntlsys 3
so_socket 3
lwp_exit 4
setcontext 7
lwp_park 11
memcntl 14
brk 16
lwp_sigmask 19
mprotect 25
mmap 56
access 67
lwp_cond_broadcast 86
ioctl 87
open 100
fsat 129
mmap64 133
lwp_mutex_wakeup 148
munmap 153
lwp_cond_signal 162
getdents64 224
stat 283
pollsys 348
lwp_cond_wait 537
lwp_mutex_timedlock 558
open64 590
fcntl 625
close 777
stat64 1146
llseek 1368
read 1404
fstat64 1559
resolvepath 1644
yield 10049
write 13416
TOTAL: 35769
Update 2:
Here is the source of the dtraced Java application. It just loads several thousand classes from one single .jar file, reproducing the same behaviour observed in a bigger app.
import java.io.IOException;
import java.io.InputStream;
import java.util.Enumeration;
import java.util.Properties;
public class TestCL {
public static void main(String[] args) {
Properties properties = new Properties();
InputStream in = TestCL.class.getClassLoader().getResourceAsStream("TestCL.properties");
if (in == null) {
System.err.println("TestCL.properties not found");
System.exit(-1);
}
try {
properties.load(in);
} catch (IOException e) {
System.err.println("TestCL.properties loading error");
e.printStackTrace();
System.exit(-2);
} finally {
try {
in.close();
} catch (IOException e) {
e.printStackTrace();
}
}
Enumeration e = properties.keys();
while (e.hasMoreElements()) {
String className = (String) e.nextElement();
try {
Class.forName(className);
System.out.println("OK " + className);
} catch (Throwable t) {
System.err.println("KO " + className + " " + t.getClass().getName() + " " + t.getMessage());
}
}
}
}
Update 3:
All classes are packed in a single .jar file. We have also run the test with all classes unpacked and there's no significant improvement on performance.
java -cp library.jar:. TestCL