Definition of these metrics recorded in /proc/vmstat on RHEL 5.3 (paging/swapping?)

Question

This question relates to the definition of specific metrics recorded in /proc/vmstat on RHEL 5.3.

I'm using nmon to monitor a load test which consists of simulating 2500 users carrying out a days workload in one hour. Recently, I've seen bad performance and am in the process of diagnosing and excluding various considerations.

We are running Red Hat Enterprise Linux Server release 5.3 (Tikanga) on vmware ESX. The physical server I'm focussing on runs and Oracle Application Server (this comprises and Apache HTTP server and OC4J J2EE container).

The nmon charts I am viewing are showing consistent allocation to the pswpin metric. Summarised as; min = 4312; max = 245352; avg = 86734. Nmon shows these values measured in "kBytes per second"

The following metrics are zero throughout the test;

• pswpout
• pgpgin
• pgpgout

I'm confused as to what this combination of metrics means, given my understanding of paging and swapping.

My Question(s):

1. Can someone please confirm what these metrics represent?
2. Any idea what system behaviour might cause this type of VM behaviour?

At the moment I'm trying to exclude virtual memory issues as a cause of poor performance.

EDIT: I have found evidence of a large number of fork() calls throughout the test. I suspect the Apache daemons. But could process creation be the cause of these metrics?

EDIT: I've added a typical sample of the VM output from nmon. Apologies for the poor formatting.

Thanks in advance for any responses.

T0001   -1	22	-1	-1	-1	150	-1	-1	-1	5196046163	-1	0	30	100199751	3060	-1	0	-1	885	-1	-1	-1	46163	-1	-1	18	-1	828189171	-1	-1	3838	-1	-1	-1	-1	-1	165231	
03:07:23    Paging and Virtual Memory	nr_dirty	nr_writeback	nr_unstable	nr_page_table_pages	nr_mapped	nr_slab	pgpgin	pgpgout	pswpin	pswpout	pgfree	pgactivate	pgdeactivate	pgfault	pgmajfault	pginodesteal	slabs_scanned	kswapd_steal	kswapd_inodesteal	pageoutrun	allocstall	pgrotated	pgalloc_high	pgalloc_normal	pgalloc_dma	pgrefill_high	pgrefill_normal	pgrefill_dma	pgsteal_high	pgsteal_normal	pgsteal_dma	pgscan_kswapd_high	pgscan_kswapd_normal	pgscan_kswapd_dma	pgscan_direct_high	pgscan_direct_normal	pgscan_direct_dma
03:07:33    -1	99	-1	-1	-1	241	0	0	0	77526	0	0	0	824	0	0	0	0	0	0	0	0	77526	0	0	0	0	0	0	0	78216	0	0	0	0	0	0	
03:07:43    -1	10	-1	-1	-1	262	0	0	0	21653	0	0	8	500	2	0	0	0	0	0	0	0	21653	0	0	0	0	0	0	0	17675	0	0	0	0	0	0	
03:07:53    -1	69	-1	-1	-1	257	0	0	0	115744	0	0	0	724	0	0	0	0	0	0	0	0	115744	0	0	0	0	0	0	0	-79544	0	0	0	0	0	0	
03:08:03    -1	69	-1	-1	-1	196	0	0	0	81202	0	0	0	628	0	0	0	0	0	0	0	0	81202	0	0	0	0	0	0	0	-18335	0	0	0	0	0	0	
03:08:13    -1	81	-1	-1	-1	205	0	0	0	29051	0	0	0	352	0	0	0	0	0	0	0	0	29051	0	0	0	0	0	0	0	24449	0	0	0	0	0	0	
03:08:24    -1	91	-1	-1	-1	131	0	0	0	122795	0	0	0	1172	0	0	0	0	0	0	0	0	122795	0	0	0	0	0	0	0	9640	0	0	0	0	0	0	
03:08:34    -1	6	-1	-1	-1	182	0	0	0	74914	0	0	4	372	1	0	0	0	0	0	0	0	74914	0	0	0	0	0	0	0	-24477	0	0	0	0	0	0	
03:08:44    -1	38	-1	-1	-1	200	0	0	0	42957	0	0	4	464	1	0	0	0	0	0	0	0	42957	0	0	0	0	0	0	0	42778	0	0	0	0	0	0	
03:08:54    -1	6	-1	-1	-1	141	0	0	0	89751	0	0	36	1000	9	0	0	0	0	0	0	0	89751	0	0	0	0	0	0	0	-9665	0	0	0	0	0	0	
03:09:04    -1	6	-1	-1	-1	171	0	0	0	74740	0	0	4	516	1	0	0	0	0	0	0	0	74740	0	0	0	0	0	0	0	-24583	0	0	0	0	0	0	
03:09:14    -1	10	-1	-1	-1	179	0	0	0	56063	0	0	0	500	0	0	0	0	0	0	0	0	56063	0	0	0	0	0	0	0	56384	0	0	0	0	0	0	
03:09:24    -1	6	-1	-1	-1	74	0	0	0	75623	0	0	0	696	0	0	0	0	0	0	0	0	75623	0	0	0	0	0	0	0	-23994	0	0	0	0	0	0	
03:09:34    -1	6	-1	-1	-1	137	0	0	0	75466	0	0	8	972	2	0	0	0	0	0	0	0	75466	0	0	0	0	0	0	0	-23837	0	0	0	0	0	0	
03:09:44    -1	3	-1	-1	-1	153	0	0	0	72535	0	0	4	460	1	0	0	0	0	0	0	0	-927465	0	0	0	0	0	0	0	-26880	0	0	0	0	0	0	
03:09:54    -1	6	-1	-1	-1	170	0	0	0	56775	0	0	0	284	0	0	0	0	0	0	0	0	56775	0	0	0	0	0	0	0	56895	0	0	0	0	0	0
03:10:04    -1	6	-1	-1	-1	166	0	0	0	74756	0	0	0	1116	0	0	0	0	0	0	0	0	74756	0	0	0	0	0	0	0	-24568	0	0	0	0	0	0
03:10:14    -1	6	-1	-1	-1	148	0	0	0	78043	0	0	0	432	0	0	0	0	0	0	0	0	78043	0	0	0	0	0	0	0	-21241	0	0	0	0	0	0
03:10:24    -1	64	-1	-1	-1	189	0	0	0	64057	0	0	0	412	0	0	0	0	0	0	0	0	64057	0	0	0	0	0	0	0	60788	0	0	0	0	0	0

Answer 1

pgpgin      -  Number of kilobytes the system has paged in from disk per second.
pgpgout     -  Number of kilobytes the system has paged out to disk per second.
pswpin      -  Number of kilobytes the system has swapped in from disk per second.
pswpout     -  Number of kilobytes the system has swapped out to disk per second.

I am 87% sure, that each page that increased pswpin counter should also increase pgpgin. You say it isn't . Hmmm.

This may be too simplistic thing to check (sorry!) but... Are you 200% sure, that the metric you observe is pswpin, not pgpgin? The later would translate to: process is reading some files.

Other explanation is that application has been heavily swapped out before the test, then the system obtained a lot of free memory. And during the test you are observing as it is "coming back to life" (constantly swapping itself in - as the code execution progresses), without reading/writing any files. But why in such scenario isn't the pgpgin increased along pswpin is beyond my comprehension.

Maybe your charts are tweaked, so pswpin is substracted from pgpgin? One point to back this up is that both metrics are typically in pages (in /proc/vmstat), and you have them converted to KB/s.

EDIT: This might be ESX-related. My wild guess that it is a side effect of either balooning or transparent page sharing (TPS). Are you able to analyze via esxtop on the ESX? Here is another esxtop guide.

EDIT: Your nmon stats seem broken. First of all, there are more column names than actual metrics (i.e. you don't have data for last column pgscan_direct_dma). There are a lot -1 or 0 values on metrics that should be there on a busy system, not only pgpgin is missing. Pgsteal and pgrotated are there, but sometimes negative, which is not possible.

So, look at /proc/vmstat, what's going on there? And use other tools to confirm nmon stats.