We're seeing latencies of ~100-300ms for services accessing a remote resource within the same subnet, yet when we use ping (ping size was of a similar size as of packets sent normally), we see responses of 5ms or less. Anything else we should check?
What does "latency" mean ?
Are you comparing ICMP echo response times with application processing round-trip times ?
If you are, that would explain it - and also tell you not to compare apples and oranges :)
can it be you have packet losses [that cause reteransmissions] for instance for larger packets?
ping -s 1400 address under linux or
ping -l 1400 address under windows.
if that does not help try looking at the traffic with wireshark - maybe there are some strange re-transmissions, corrupted packets?
A discrepancy like that usually shows an issue in the application / server. While it is possible that your ping is different due to QoS / network prioritization, that's not the first place I'd look if you're talking about systems on the same local network.
What's your latency time accessing the service from the same host it is served by? Does that host provide other services faster? Is it under a heavy IO load?
As you're suspecting here, ping latency != a TCP round trip time, particularly if there's data being transferred. It's possible that all of this this is due to delayed ACKs (which can impose ~200ms, the common default setting) or the bad interaction between Nagle/and DACK which can push it up to ~500ms. A packet capture is in order here.
Also note that the delayed ack timer is independent of a TCP flow, and is instantiated when the system boots. So it can fire any time between 1-200ms inside of a flow, and commonly causes a 'range' style delay as you describe.