1

I am experiencing a massive amount of latency between 2 servers boxes. (900 milliseconds) The servers have a physical lan line connecting them - so there is virtually no network latency.

1 application on 1 server connects to 1 application on the other server . Nothing else.

I am running windows server 2008 R2 - on a dell box with 12 cores & 32gb ram, intel 5680

The OS was not tweaked or configured - just turned on out of the box so it is highly likely that I need to enable/disable some features.

I have attached the entire perfmon report.

Can you please glance over the report and tell me if anything unusual sticks out?

Computer: BGTWEXSEC07
Collected: Wednesday, January 26, 2011 8:30:00 AM
Duration: 23400 Seconds
Process
Total CPU%: 0
Top Process Group: logon.scr
Group CPU%: 0
Disk
Top Disk by IO Rate: 0
IO/sec: 2
Disk Queue Length: 0.701
Memory
Utilization: 8 %
Memory: 32757 MB
Top Process: mmc
Private Working Set: 279,844 KB
Network
Utilization: 0 %
Top Outbound Client: 10.51.2.19
Sent: 268 bytes
Top Inbound Client: 192.168.139.138
Received: 587 bytes
Resouce Overview
CPU Idle 4 % Low CPU load.
Network Idle 0 % Busiest network adapter is less than 15%.
Disk Idle 2 /sec Disk I/O is less than 100 (read/write) per second on disk 0.
Memory Idle 8 % 30016 MB Available.
Component Status Utilization Details
Image Statistics Top: 3 of 122
Idle 0 12 12 95.7 0.0 95.7
logon.scr 3752 1 1 0.2 0.0 0.2
svchost.exe 324 3304 1923 0.0 0.1 0.1
10162 2146 96.2
Threads
Image Name Process ID Launched Used
Kernel
CPU%
User
CPU% CPU%
Process Top: 21 of 21
Virtual Bytes 8,278,145,165 8,042,393,600 8,864,038,912
Private Bytes 2,118,715,078 1,997,709,312 2,265,440,256
Page File Bytes 2,106,033,862 1,985,028,096 2,252,759,040
Working Set 1,320,179,694 1,183,739,904 1,474,002,944
Working Set - Private 840,454,234 728,547,328 981,385,216
Pool Paged Bytes 13,459,627 13,105,288 13,637,208
Pool Nonpaged Bytes 928,471 885,548 949,036
IO Data Bytes/sec 49,133 0 2,286,673
IO Write Bytes/sec 40,506 0 2,167,365
IO Other Bytes/sec 11,389 2,175 468,750
IO Read Bytes/sec 8,627 0 1,564,463
% Processor Time 1,200 1,195 1,200
% Privileged Time 1,198 1,115 1,200
Page Faults/sec 823 0 47,344
Thread Count 813 790 841
Performance Counter Mean Minimum Maximum
Page 1 of 11
file://C:\PerfLogs\Admin\TassoneWex\20110126-0001\report.html 1/26/2011
IO Other Operations/sec 233 40 11,522
IO Data Operations/sec 23 0 1,014
IO Write Operations/sec 13 0 779
IO Read Operations/sec 10 0 425
% User Time 2 0 85
Priority Base 0 0 0
Performance Counter Mean Minimum Maximum
Processor Top: 9 of 9
Interrupts/sec 1,355 851 22,780
DPCs Queued/sec 765 727 1,365
% Idle Time 100 91 100
DPC Rate 11 8 305
% Processor Time 0.455 0 9
% Privileged Time 0.312 0 7
% User Time 0.143 0 7
% DPC Time 0.001 0 0.391
% Interrupt Time 0.001 0 0.26
Performance Counter Mean Minimum Maximum
Processor User Time by CPU Top: 13 of 13
2 2 0 62
4 1.047 0 70
3 0.587 0 55
5 0.57 0 86
_Total 0.455 0 9
0 0.361 0 55
1 0.344 0 55
6 0.203 0 12
10 0.021 0 5
9 0.011 0 5
11 0.009 0 8
7 0.005 0 27
8 0.002 0 2
Instance Mean Minimum Maximum
Processor Interrupts by CPU Top: 13 of 13
_Total 1,355 851 22,780
3 149 67 5,103
0 140 65 8,876
1 140 65 5,018
2 129 67 5,395
6 115 64 2,011
7 105 65 2,036
9 101 67 2,026
5 98 65 2,063
8 95 66 2,026
10 95 65 2,013
11 95 65 2,013
4 94 65 2,093
Instance Mean Minimum Maximum
Services Top: 27 of 27
Page 2 of 11
file://C:\PerfLogs\Admin\TassoneWex\20110126-0001\report.html 1/26/2011
svchost.exe 324 0.1
dsm_sa_datamgr32.exe 1488 0.0
svchost.exe 840 0.0
svchost.exe 620 0.0
svchost.exe 976 0.0
dsm_om_connsvc32.exe 1808 0.0
dsm_sa_eventmgr32.exe 1456 0.0
svchost.exe 900 0.0
mysqld.exe 1608 0.0
4484 0.0
dsm_om_shrsvc32.exe 1636 0.0
FreeSSHDService.exe 1508 0.0
lsass.exe 640 0.0
mr2kserv.exe 1596 0.0
msdtc.exe 1632 0.0
SLsvc.exe 356 0.0
snmp.exe 1820 0.0
snmptrap.exe 1832 0.0
spoolsv.exe 1392 0.0
Stopped Services - 0.0
svchost.exe 268 0.0
svchost.exe 3928 0.0
svchost.exe 1124 0.0
svchost.exe 1900 0.0
svchost.exe 884 0.0
svchost.exe 1784 0.0
svchost.exe 508 0.0
Process processId CPU%
System Top: 15 of 15
% Registry Quota In Use 5 5 5
Context Switches/sec 952 394 15,518
Exception Dispatches/sec 4 0 30,245
File Control Bytes/sec 11,395 3,637 468,750
File Control Operations/sec 234 40 11,522
File Data Operations/sec 23 0 1,020
File Read Bytes/sec 8,634 0 1,564,463
File Read Operations/sec 10 0 425
File Write Bytes/sec 40,511 0 2,183,776
File Write Operations/sec 13 0 785
Processes 69 67 70
Processor Queue Length 0.009 0 1
System Calls/sec 14,622 536 232,944
System Up Time 3,109,447 3,097,747 3,121,147
Threads 813 790 841
Performance Counter Mean Minimum Maximum
Server Top: 26 of 26
Blocking Requests Rejected 0 0 0
Bytes Received/sec 0.231 0 587
Bytes Total/sec 8 0 19,988
Bytes Transmitted/sec 8 0 19,410
Context Blocks Queued/sec 0.09 0 189
Performance Counter Mean Minimum Maximum
Page 3 of 11
file://C:\PerfLogs\Admin\TassoneWex\20110126-0001\report.html 1/26/2011
Errors Access Permissions 0 0 0
Errors Granted Access 0 0 0
Errors Logon 513 513 513
Errors System 0 0 0
File Directory Searches 0 0 0
Files Open 2 1 11
Files Opened Total 3,900 3,754 4,012
Logon Total 1,277 1,273 1,281
Logon/sec 0 0 2
Pool Nonpaged Bytes 5,531,164 5,500,387 5,558,085
Pool Nonpaged Failures 0 0 0
Pool Nonpaged Peak 6,471,047 6,471,047 6,471,047
Pool Paged Bytes 95,106 93,773 97,922
Pool Paged Failures 0 0 0
Pool Paged Peak 101,731 101,731 101,731
Server Sessions 3 2 4
Sessions Errored Out 0 0 0
Sessions Forced Off 0 0 0
Sessions Logged Off 85 83 86
Sessions Timed Out 0 0 0
Work Item Shortages 0 0 0
Performance Counter Mean Minimum Maximum
TCP Outbound Traffic Top: 24 of 24
10.51.2.19 884 3389 3.5 268
192.168.139.138 3284 52919 1.5 246
10.51.2.19 4 445 0.0 7
192.168.139.135 3284 52941 0.0 4
192.168.139.135 3284 52943 0.0 4
127.0.0.1 3284 52940 3.0 3
192.168.139.130 3284 52944 0.0
10.51.2.19 1508 22 0.0
127.0.0.1 3284 52942 0.0
127.0.0.1 3284 52938 0.0
127.0.0.1 3284 52945 0.0
127.0.0.1 3284 52918 0.0
127.0.0.1 3284 52916 0.0
127.0.0.1 3284 52914 0.0
127.0.0.1 3284 52939 0.0
127.0.0.1 3284 52932 0.0
127.0.0.1 3284 52926 0.0
127.0.0.1 3284 52922 0.0
127.0.0.1 3284 52933 0.0
127.0.0.1 3284 52930 0.0
127.0.0.1 3284 52929 0.0
127.0.0.1 3284 52925 0.0
127.0.0.1 3284 52928 0.0
127.0.0.1 3284 52924 0.0
To Machine
Process
ID Port Requests/sec
Outbound
Bytes/Sec
TCP Inbound Traffic Top: 12 of 12
Page 4 of 11
file://C:\PerfLogs\Admin\TassoneWex\20110126-0001\report.html 1/26/2011
192.168.139.138 3284 52919 2.6 587
192.168.139.130 3284 52944 3.0 246
10.51.2.19 884 3389 0.4 28
10.51.2.19 4 445 0.0 8
192.168.139.135 3284 52941 0.0 5
192.168.139.135 3284 52943 0.0 4
127.0.0.1 3284 9001 3.0 3
10.51.2.19 1508 22 0.0
127.0.0.1 3284 52917 0.0
127.0.0.1 3284 52915 0.0
127.0.0.1 3284 52913 0.0
10.51.2.19 1808 1311 0.0
From Machine
Process
ID Port Requests/sec
Inbound
Bytes/Sec
TCP Version 4 Top: 9 of 9
Connection Failures 2,397 2,397 2,397
Connections Active 2,802 2,784 2,808
Connections Established 18 2 24
Connections Passive 59,644 59,406 59,873
Connections Reset 754 753 754
Segments Received/sec 15 0 384
Segments Retransmitted/sec 0.015 0 3
Segments Sent/sec 15 0 327
Segments/sec 30 0 710
Performance Counter Mean Minimum Maximum
TCP Version 6 Top: 9 of 9
Connection Failures 0 0 0
Connections Active 36 34 37
Connections Established 0.003 0 2
Connections Passive 36 34 37
Connections Reset 38 36 39
Segments Received/sec 0.021 0 147
Segments Retransmitted/sec 0 0 0
Segments Sent/sec 0.021 0 147
Segments/sec 0.042 0 294
Performance Counter Mean Minimum Maximum
Network Interface Top: 56 of 56
Bytes
Received/sec
Broadcom BCM5709C
NetXtreme II GigE [NDIS VBD
Client] _2
0 0 0
Bytes Sent/sec
Broadcom BCM5709C
NetXtreme II GigE [NDIS VBD
Client] _2
0 0 0
Bytes Total/sec
Broadcom BCM5709C
NetXtreme II GigE [NDIS VBD
Client] _2
0 0 0
Current Bandwidth
Broadcom BCM5709C
NetXtreme II GigE [NDIS VBD
Client] _2
9,223,372,036,854,776,000 9,223,372,036,854,776,000 9,223,372,036,854,776,000
Output Queue
Broadcom BCM5709C
NetXtreme II GigE [NDIS VBD
0 0 0
Performance
Counter Instance Mean Minimum Maximum
Page 5 of 11
file://C:\PerfLogs\Admin\TassoneWex\20110126-0001\report.html 1/26/2011
Length Client] _2
Packets Outbound
Discarded
Broadcom BCM5709C
NetXtreme II GigE [NDIS VBD
Client] _2
0 0 0
Packets Outbound
Errors
Broadcom BCM5709C
NetXtreme II GigE [NDIS VBD
Client] _2
0 0 0
Packets/sec
Broadcom BCM5709C
NetXtreme II GigE [NDIS VBD
Client] _2
0 0 0
Bytes
Received/sec
Broadcom BCM5709C
NetXtreme II GigE [NDIS VBD
Client] _3
0 0 0
Bytes Sent/sec
Broadcom BCM5709C
NetXtreme II GigE [NDIS VBD
Client] _3
0 0 0
Bytes Total/sec
Broadcom BCM5709C
NetXtreme II GigE [NDIS VBD
Client] _3
0 0 0
Current Bandwidth
Broadcom BCM5709C
NetXtreme II GigE [NDIS VBD
Client] _3
9,223,372,036,854,776,000 9,223,372,036,854,776,000 9,223,372,036,854,776,000
Output Queue
Length
Broadcom BCM5709C
NetXtreme II GigE [NDIS VBD
Client] _3
0 0 0
Packets Outbound
Discarded
Broadcom BCM5709C
NetXtreme II GigE [NDIS VBD
Client] _3
0 0 0
Packets Outbound
Errors
Broadcom BCM5709C
NetXtreme II GigE [NDIS VBD
Client] _3
0 0 0
Packets/sec
Broadcom BCM5709C
NetXtreme II GigE [NDIS VBD
Client] _3
0 0 0
Bytes
Received/sec
Broadcom BCM5709C
NetXtreme II GigE [NDIS VBD
Client] _4
0 0 0
Bytes Sent/sec
Broadcom BCM5709C
NetXtreme II GigE [NDIS VBD
Client] _4
0 0 0
Bytes Total/sec
Broadcom BCM5709C
NetXtreme II GigE [NDIS VBD
Client] _4
0 0 0
Current Bandwidth
Broadcom BCM5709C
NetXtreme II GigE [NDIS VBD
Client] _4
9,223,372,036,854,776,000 9,223,372,036,854,776,000 9,223,372,036,854,776,000
Output Queue
Length
Broadcom BCM5709C
NetXtreme II GigE [NDIS VBD
Client] _4
0 0 0
Packets Outbound
Discarded
Broadcom BCM5709C
NetXtreme II GigE [NDIS VBD
Client] _4
0 0 0
Packets Outbound
Errors
Broadcom BCM5709C
NetXtreme II GigE [NDIS VBD
Client] _4
0 0 0
Packets/sec
Broadcom BCM5709C
NetXtreme II GigE [NDIS VBD
Client] _4
0 0 0
Bytes
Received/sec
Intel[R] Gigabit ET Quad Port
Server Adapter
251 0 19,336
Bytes Sent/sec
Intel[R] Gigabit ET Quad Port
Server Adapter
517 0 57,586
Bytes Total/sec
Intel[R] Gigabit ET Quad Port
Server Adapter
769 0 60,435
Performance
Counter Instance Mean Minimum Maximum
Page 6 of 11
file://C:\PerfLogs\Admin\TassoneWex\20110126-0001\report.html 1/26/2011
Current Bandwidth
Intel[R] Gigabit ET Quad Port
Server Adapter
1,000,000,000 1,000,000,000 1,000,000,000
Output Queue
Length
Intel[R] Gigabit ET Quad Port
Server Adapter
0 0 0
Packets Outbound
Discarded
Intel[R] Gigabit ET Quad Port
Server Adapter
0 0 0
Packets Outbound
Errors
Intel[R] Gigabit ET Quad Port
Server Adapter
0 0 0
Packets/sec
Intel[R] Gigabit ET Quad Port
Server Adapter
7 0 177
Bytes
Received/sec
Intel[R] Gigabit ET Quad Port
Server Adapter _2
1,196 0 105,381
Bytes Sent/sec
Intel[R] Gigabit ET Quad Port
Server Adapter _2
567 0 41,817
Bytes Total/sec
Intel[R] Gigabit ET Quad Port
Server Adapter _2
1,763 0 146,667
Current Bandwidth
Intel[R] Gigabit ET Quad Port
Server Adapter _2
1,000,000,000 1,000,000,000 1,000,000,000
Output Queue
Length
Intel[R] Gigabit ET Quad Port
Server Adapter _2
0 0 0
Packets Outbound
Discarded
Intel[R] Gigabit ET Quad Port
Server Adapter _2
0 0 0
Packets Outbound
Errors
Intel[R] Gigabit ET Quad Port
Server Adapter _2
0 0 0
Packets/sec
Intel[R] Gigabit ET Quad Port
Server Adapter _2
11 0 596
Bytes
Received/sec
Intel[R] Gigabit ET Quad Port
Server Adapter _3
0 0 0
Bytes Sent/sec
Intel[R] Gigabit ET Quad Port
Server Adapter _3
0 0 0
Bytes Total/sec
Intel[R] Gigabit ET Quad Port
Server Adapter _3
0 0 0
Current Bandwidth
Intel[R] Gigabit ET Quad Port
Server Adapter _3
9,223,372,036,854,776,000 9,223,372,036,854,776,000 9,223,372,036,854,776,000
Output Queue
Length
Intel[R] Gigabit ET Quad Port
Server Adapter _3
0 0 0
Packets Outbound
Discarded
Intel[R] Gigabit ET Quad Port
Server Adapter _3
0 0 0
Packets Outbound
Errors
Intel[R] Gigabit ET Quad Port
Server Adapter _3
0 0 0
Packets/sec
Intel[R] Gigabit ET Quad Port
Server Adapter _3
0 0 0
Bytes
Received/sec
Intel[R] Gigabit ET Quad Port
Server Adapter _4
0 0 0
Bytes Sent/sec
Intel[R] Gigabit ET Quad Port
Server Adapter _4
0 0 0
Bytes Total/sec
Intel[R] Gigabit ET Quad Port
Server Adapter _4
0 0 0
Current Bandwidth
Intel[R] Gigabit ET Quad Port
Server Adapter _4
9,223,372,036,854,776,000 9,223,372,036,854,776,000 9,223,372,036,854,776,000
Output Queue
Length
Intel[R] Gigabit ET Quad Port
Server Adapter _4
0 0 0
Packets Outbound
Discarded
Intel[R] Gigabit ET Quad Port
Server Adapter _4
0 0 0
Packets Outbound
Errors
Intel[R] Gigabit ET Quad Port
Server Adapter _4
0 0 0
Packets/sec
Intel[R] Gigabit ET Quad Port
Server Adapter _4
0 0 0
Performance
Counter Instance Mean Minimum Maximum
IP Version 4 Top: 17 of 17
Page 7 of 11
file://C:\PerfLogs\Admin\TassoneWex\20110126-0001\report.html 1/26/2011
Datagrams Forwarded/sec 0 0 0
Datagrams Outbound Discarded 3 3 3
Datagrams Outbound No Route 0 0 0
Datagrams Received Address Errors 0 0 0
Datagrams Received Delivered/sec 16 0 384
Datagrams Received Discarded 112 110 113
Datagrams Received Header Errors 0 0 0
Datagrams Received Unknown Protocol 20 20 20
Datagrams Received/sec 10 0 334
Datagrams Sent/sec 15 0 327
Datagrams/sec 24 0 660
Fragment Re-assembly Failures 0 0 0
Fragmentation Failures 0 0 0
Fragmented Datagrams/sec 0 0 0
Fragments Created/sec 0 0 0
Fragments Re-assembled/sec 0 0 0
Fragments Received/sec 0 0 0
Performance Counter Mean Minimum Maximum
IP Version 6 Top: 17 of 17
Datagrams Forwarded/sec 0 0 0
Datagrams Outbound Discarded 0 0 0
Datagrams Outbound No Route 4 4 4
Datagrams Received Address Errors 0 0 0
Datagrams Received Delivered/sec 0.022 0 147
Datagrams Received Discarded 0 0 0
Datagrams Received Header Errors 0 0 0
Datagrams Received Unknown Protocol 0 0 0
Datagrams Received/sec 0 0 0
Datagrams Sent/sec 0.055 0 147
Datagrams/sec 0.055 0 147
Fragment Re-assembly Failures 0 0 0
Fragmentation Failures 0 0 0
Fragmented Datagrams/sec 0 0 0
Fragments Created/sec 0 0 0
Fragments Re-assembled/sec 0 0 0
Fragments Received/sec 0 0 0
Performance Counter Mean Minimum Maximum
UDP Version 4 Top: 5 of 5
Datagrams No Port/sec 0 0 1.002
Datagrams Received Errors 0 0 0
Datagrams Received/sec 0.003 0 2
Datagrams Sent/sec 0.052 0 5
Datagrams/sec 0.055 0 6
Performance Counter Mean Minimum Maximum
UDP Version 6 Top: 5 of 5
Datagrams No Port/sec 0 0 0
Datagrams Received Errors 0 0 0
Datagrams Received/sec 0 0 0
Datagrams Sent/sec 0.034 0 4
Performance Counter Mean Minimum Maximum
Page 8 of 11
file://C:\PerfLogs\Admin\TassoneWex\20110126-0001\report.html 1/26/2011
Datagrams/sec 0.034 0 4
Performance Counter Mean Minimum Maximum
Files Causing Most Disk IOs Top: 3 of 20
0 C:\$LogFile 0.0 0 0.6 10
0 C:\PerfLogs\Admin\TassoneWex\20110126-0001\Performance Counter.blg 0.0 0 0.6 63
0 C:\$Mft 0.0 0 0.2 4
Disk File Reads/sec Kb/Read Writes/sec Kb/Write
Disk Totals Top: 1 of 1
0 0.0 6 1.9 25
Disk Number Reads/sec Kb/Read Writes/sec Kb/Write
Physical Disk Counters Top: 17 of 17
% Disk Read Time _Total 0.003 0 70
% Disk Time _Total 0.027 0 70
% Disk Write Time _Total 0.024 0 24
Avg. Disk Bytes/Read _Total 6,559 0 32,768
Avg. Disk Bytes/Transfer _Total 25,916 0 183,501
Avg. Disk Bytes/Write _Total 26,026 0 183,501
Avg. Disk Queue Length _Total 0 0 0.701
Avg. Disk Read Queue Length _Total 0 0 0.699
Avg. Disk Write Queue Length _Total 0 0 0.239
Current Disk Queue Length _Total 0.005 0 4
Disk Bytes/sec _Total 48,262 0 2,318,959
Disk Read Bytes/sec _Total 70 0 1,563,065
Disk Reads/sec _Total 0.011 0 244
Disk Transfers/sec _Total 2 0 486
Disk Write Bytes/sec _Total 48,192 0 2,318,959
Disk Writes/sec _Total 2 0 486
Split IO/Sec _Total 0.044 0 9
Performance Counter Instance Mean Minimum Maximum
Physical Disk Percent Idle Time Top: 2 of 2
0 C: D: 100 0.227 300
_Total 100 0.227 300
Instance Mean Minimum Maximum
Physical Disk Average Second Counters Top: 3 of 3
Avg. Disk sec/Read 0 C: D: 0.003 0 0.011
Avg. Disk sec/Transfer 0 C: D: 0 0 0.028
Avg. Disk sec/Write 0 C: D: 0 0 0.028
Performance Counter Instance Mean Minimum Maximum
Page 9 of 11
file://C:\PerfLogs\Admin\TassoneWex\20110126-0001\report.html 1/26/2011
Memory Top: 10 of 70
mmc 4032 759,172 303,068 23,224 279,844
mysqld 1608 1,238,492 158,828 5,072 153,756
WTP 3284 239,160 82,752 31,664 51,088
svchost##4 324 723,072 57,284 19,688 37,596
explorer 2920 246,404 64,020 32,504 31,516
dsm_om_connsvc32 1808 491,556 35,792 7,952 27,840
dsm_sa_datamgr32 1488 148,932 35,032 7,720 27,312
explorer##1 1572 216,768 48,172 24,952 23,220
svchost##6 620 130,920 30,644 9,256 21,388
explorer##2 176 210,752 44,024 22,808 21,216
Process Process ID Commit (KB)
Working
Set (KB) Shareable (KB) Private (KB)
Memory Top: 28 of 28
% Committed Bytes In Use 4 4 5
Available Bytes 31,473,648,067 31,324,831,744 31,604,862,976
Cache Bytes 308,574,976 307,150,848 309,473,280
Cache Faults/sec 6 0 1,154
Commit Limit 67,972,259,840 67,972,259,840 67,972,259,840
Committed Bytes 2,919,334,580 2,796,412,928 3,068,207,104
Demand Zero Faults/sec 483 0 48,391
Free & Zero Page List Bytes 30,548,318,032 30,399,991,808 30,648,487,936
Free System Page Table Entries 33,559,793 33,559,498 33,560,003
Modified Page List Bytes 37,567,172 37,412,864 37,736,448
Page Faults/sec 894 0 48,528
Page Reads/sec 0.076 0 11
Page Writes/sec 0 0 0
Pages Input/sec 0.083 0 18
Pages Output/sec 0 0 0
Pages/sec 0.083 0 18
Pool Nonpaged Allocs 348,920 347,335 349,877
Pool Nonpaged Bytes 176,115,460 176,103,424 176,308,224
Pool Paged Allocs 115,893 114,710 117,091
Pool Paged Bytes 163,979,738 163,135,488 164,085,760
Pool Paged Resident Bytes 162,643,676 161,280,000 162,746,368
Standby Cache Core Bytes 0 0 0
Standby Cache Normal Priority Bytes 792,436,374 776,654,848 826,740,736
Standby Cache Reserve Bytes 132,893,561 130,535,424 135,876,608
System Cache Resident Bytes 141,134,883 140,615,680 142,036,992
Transition Faults/sec 411 0 14,710
Transition Pages RePurposed/sec 0 0 0
Write Copies/sec 3 0 286
Performance Counter Mean Minimum Maximum
Handle Count Top: 10 of 71
_Total 223,751 223,188 224,085
mysqld 206,300 206,299 206,314
svchost##4 1,159 1,142 1,208
lsass 863 854 879
explorer 768 767 794
Instance Mean Minimum Maximum
Page 10 of 11
file://C:\PerfLogs\Admin\TassoneWex\20110126-0001\report.html 1/26/2011
System 663 659 728
csrss 656 645 671
mmc 646 635 655
svchost##7 587 584 593
explorer##1 528 528 531
Instance Mean Minimum Maximum
Computer Information
Computer: BGTWEXSEC07
Windows Build: 6002
Processors: 12
Processor Speed: 3325 MHz
Memory: 32757 MB
Platform: 64 Bit
Collection Information
Start Time: Wednesday, January 26, 2011 8:30:00 AM
End Time: Wednesday, January 26, 2011 3:00:00 PM
Duration: 23400 Seconds
Buffers: 423
Processed Events: 475581
Lost Events: 0
Skipped Events: 22
Use Timing Window: Yes
Files
File: C:\PerfLogs\Admin\TassoneWex\20110126-0001\NtKernel.etl
Logger Name: NT Kernel Logger
Start Time: Wednesday, January 26, 2011 8:30:00 AM
End Time: Wednesday, January 26, 2011 3:00:00 PM
File Duration: 23400 Seconds
File: Performance Counter.blg
Start Time: Wednesday, January 26, 2011 8:30:00 AM
End Time: Wednesday, January 26, 2011 3:00:00 PM
File Duration: 23399 Seconds
Processed Events Top: 3 of 40
TcpIp RecvIPV4 0 {9a280ac0-c8e0-11d1-84e2-00c04fb998a2} 11 2 211076
TcpIp SendIPV4 0 {9a280ac0-c8e0-11d1-84e2-00c04fb998a2} 10 2 190909
DiskIo Write 0 {3d6fa8d4-fe05-11d0-9dda-00c04fd7ba7c} 11 2 43348
475581
Event Opcode Task Payload Guid Payload ID Version Count
Page 11 of 11
file://
1
  • sorry about the report -- could not figure out how to paste it and keep the formatting -- i dont see anywhere I can upload the pdf report?
    – user68495
    Jan 27, 2011 at 17:23

2 Answers 2

3

The very first thing I'd do is attempt to update the drivers for those Broadcom NICs. I've seen weird issues much like this one due to driver-whackery on Broadcoms.

1
  • * The Broadcom NICs are disabled so that should not be an issue * There 4 on-board Broadcom Gig NICS (disabled) and 4 Intel Pro NICS (quad port). One port for our LAN and the second facing the other server * No teaming (fault tolerant pairing) * The switch is an Arista 10Gig and this server is connected via a copper SFP at 1 Gig speed * NICs are set to auto negotiate speed and duplex
    – user68495
    Jan 27, 2011 at 20:42
0

Sysadmin1138 is correct in trying to update the firmware and drivers of your NIC's first. After that...

Have a look at the following MS KB article Q146005 to get some specific counters that will help you diagnose the problem. You're missing the counters related to "Suspect Bottleneck: Network Components"

Can you be clear about the setup please? How many NIC's on each server? Are you connecting them to a switch? And are they in some kind of special setup (teaming)? What is the switch, and is there anything special configured for the ports? What speed/duplex is set on the switch and Servers? Simplest test is a single crossover cable. From there you add one complexity at a time...

4
  • * The Broadcom NICs are disabled so that should not be an issue * There 4 on-board Broadcom Gig NICS (disabled) and 4 Intel Pro NICS (quad port). One port for our LAN and the second facing the other server * No teaming (fault tolerant pairing) * The switch is an Arista 10Gig and this server is connected via a copper SFP at 1 Gig speed * NICs are set to auto negotiate speed and duplex
    – user68495
    Jan 27, 2011 at 20:40
  • Only use one port for right now. What happens? You may also try to adjust the Binding Order of the cards placing your server-server card firts. If that works, you may need to add manual persistent route entries on the server. I assume the two ports are on different subnets?
    – cwheeler33
    Jan 28, 2011 at 14:06
  • no change in latency. any more ideas? I am specifically worried about the configuration of the OS. Wondering if there is anything I should configure differently - or enable/disable.
    – user68495
    Jan 30, 2011 at 15:11
  • What's the application? You may want to call up the vendor and ask them about it. Also, do you have an AntiVirus on the server? Does the application have issues with AV software? This latency, where are you getting the data from? And did you read through that article I mentioned before for the missing counters?
    – cwheeler33
    Jan 31, 2011 at 15:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .