vmstat output incompatible

Question

as you can see when run a vmstat like below.

bash-3.00# vmstat 1 5
kthr      memory            page                     disk              faults        cpu
r  b  w   swap   free    re   mf   pi   po   fr   de sr m1 m1 m1 m2   in   sy   cs    us sy id
2  0  0 12812792 7094520 668  1023 213  939  939  0  0  1  1  1  0  30519 66084 35693 12 7  81
10 19 0 5469784  797392  265  325  583  1119 1119 0  0  1  1  0  0  35404 75227 44916 14 8  78
34 10 0 5577672  846056  570  554  3802 1233 1233 0  0  1  0  0  0  35013 72253 46505 10 10 79
19 0  0 5576352  843112  462  2825 1200 963  963  0  0  0  0  0  0  35070 73182 44379 13 8  78
2  0  0 5582624  854664  1698 1885 134  1198 1198 0  0  2  1  1  0  35688 71013 43705 11 7  82

what is wrong with this ? because the memory usage is about %95.

***---****---****----*****

Since the system is an HA system here is the output for standby machine.

bash-3.00# vmstat 1 5
kthr      memory            page          disk                faults            cpu
r b w   swap   free    re   mf   pi   po fr de sr m1 m1 m1 m2 in    sy    cs    us sy id
1 0 0 13949104 8516736 553  1047 121  21 20 0  0  1  1  1  9  30691 69172 34352 12 7  81
0 0 0 12456136 7171232 1122 2554 4117 0  0  0  0  3  1  2  0  31505 85240 35958 11 8  81
0 0 0 12353888 7147488 1401 664  0    0  0  0  0  0  0  0  0  31647 73507 35881 13 6  81
0 0 0 12345536 7129280 13   40   0    0  0  0  0  0  0  0  0  30787 69357 34695 14 6  80
0 0 0 12345440 7125040 217  1646 0    24 24 0  0  0  0  0  0  31390 73710 38393 12 10 78

Answer 1

Solaris? Which version (presumably 8 or 9 from the idle column)? Sun cluster or vcs?

Anyway, ignore the first line since that is an average since boot.

Is a backup running on the passive node, by the way, and causing the minor faults and page-ins on a machine ostensibly doing nothing? And look at the context switches.

On the active node, memory usage at 90 per cent probably is just memory used by whatever you are running: Oracle's SGA, perhaps; and the rest is in use for cacheing. This is normal. The system does seem to be io-bound, at least for the few seconds you show. There is paging in and out but no serious memory pressure, so this probably reflects memory-mapped io. Some swap space is in use but that may reflect pre-allocation rather than any shortfall.

Run: vmstat -s; vmstat -p 10 5. If it is Solaris, also: prstat -a; prstat -t; prstat -a -s rss; and on Solaris 9, echo ::memstat | mdb -k (which can take a while so do not panic). These will show what the system is using memory for, and which applications are using it. If you think your application software is leaking memory, then compare measurements from prstat over time.

If you are consistently io-bound, then run iostat -xn 60 and speak to your storage administrators if appropriate. But that is a separate matter.

Answer 2

I'm not sure what's being asked here but you do remember that the first line of the vmstat output shows the average values for the items since the last reboot ? Subsequent lines show the values for the last sampling period, so in general ignore the first line.