I usually run uptime/latency tests against (and from) two servers that we own at different sites and until recently I've used the google dns servers as a control group.

However, I've realized there is a potential problem with monitoring latency over extended periods of time.

Almost all of the major service providers are using ANYCAST.

For short tests this doesn't matter, but I need to run a set of tests for at least a week to try and catch an intermittent problem, and a change in the anycast priority while trying to test latency will cause the latency values for that server to change accordingly.

Since I'm submitting graphs of this data to the ISP, I need to avoid/account for as many variables as possible. Spikes in the data for only one of the tested servers will only cause headaches.

So can anyone recommend servers that:

  1. are not using anycast
  2. are owned by an entity that has a good uptime reputation (so they can't claim that the problem is server-side)
  3. will respond to ICMP requests
  4. Have an available service that runs on TCP/UDP (http or dns preferably)
  5. Wont consider an automated request every 10 minutes to be abuse
  6. Are accessible from anywhere in the world
  7. Are not local to the isp ( consider this an investigation of a hostile party )

Thanks in advance.

Edit: added #6 and #7 above.

More info: I am attempting to demonstrate a network problem for an entire node of our local ISP's network. They are actively blaming the issue on the equipment installed at the customer sites (our backup site is one of these), and refuse to escalate the problem. (even though 2 of these businesses have ISP provided modems, and all of us have completely different routers/services running)

I am already quite familiar with the need to test an isp controlled IP, but they are actively dropping all packets targeted at gateway ip addresses and are only passing traffic addressed beyond the gateways. So to demonstrate the issue, I am sending packets to other systems in the same node, systems one hop away from the affected node, and systems completely outside the network.

Unfortunately, all of the systems I have currently are either administered directly by myself, or by people who are biased enough to assist me.

I need to have several systems included in the trace/log/graphs that are 100% not in the control of either myself or the isp so that the graphs have a stable/unbiased control group.

These requirements are straight from legal, I'm just trying to make sure that everything that could be argued to invalidate the data is already covered.

In Summary: I need to be able to show tcp/udp/icmp as 3 separate data points, and I need to be able to show the connections inside the local node, from local node to another nearby node, from those 2 nodes to the internet, and through the internet to both verifiable servers and a control group that I have no control over whatsoever.

Again, Google/opendns/yahoo/msn/facebook/etc all use anycast, which throws the numbers off every time the anycast caches expire, so I need suggestions of an IP or server that is available for this type of testing.

I was hoping someone knew of a system run by someone such as ISC or ICANN, or perhaps even a .gov server (fcc or nsa maybe?) setup for this type of testing.

Thanks again.

  • (since someone will ask) The actual issue is packet loss. It happens the same way every time. First we start seeing about 5% tcp packet loss, which increases steadily for about 45 minutes. At the 45 minute mark we start losing udp packets and the % dropped increases over time. After about 2 hours we hit 100% lost. Packet captures show a flood of ARP and dhcp/bootp traffic coming from the local nodes gateway 24/7, but no other odd traffic. This happens no matter what equipment is connected to the modem. – eldorel Mar 7 '12 at 12:53

As part of one of their desktop dashboards, Symantec set up ping.symantec.com. I'm not sure if it meets your #4, but the rest seem OK.

  • Not perfect, but definetely a good start. Thank you! – eldorel Mar 30 '12 at 18:03

My suggestion would be to test against a node at the ISP, preferrably the remote end of your circuit/connection. Anything upstream from the ISP is beyond their control and irrelevant to the problem (meaning there's nothing they can do about upstream packet loss or packet loss that isn't occurring on their network).

In addition, ping isn't a good tool for testing and reporting packet loss, as far as I'm concerned. Responding to ICMP traffic may be given low priority by upstream nodes or it may be dropped entirely. Routers are concerned with routing "real" traffic, not responding to your ping test. I would suggest you use something like Ixia QCheck or iperf to test for packet loss using real traffic.

  • The ISP's gateways don't respond to icmp packets at all actually, and they've got no ports listening. I also attempted to use their web server for testing, but it's hosted 3 states away. As for the ping issue, The script I've written uses a few different things to test both tcp and udp packets as well as ping/traceroute. Even when the connection hits 100% dropped on both tcp and udp, icmp is passed just fine for another 10 minutes. – eldorel Mar 7 '12 at 13:11
  • 1
    My point is that pinging/testing against a node that doesn't exist on the ISP's network only proves that that node has a problem, it doesn't prove anything about the ISP's network. You need to test the ISP's network somehow. – joeqwerty Mar 7 '12 at 13:24
  • I'm open to suggestions. We are currently running outgoing tests against their upstream, which is closest we could get. I'm hoping that showing tests to about 5 different sites for both tcp and udp on top of the icmp requests to the upstream will at least force them to acknowledge that there is a real problem. – eldorel Mar 7 '12 at 13:29
  • added more info to the initial question to address this more fully. – eldorel Mar 12 '12 at 18:08

The symptoms you state in your follow-up sound a lot like over subscription with the ISP's peer(s). This is very hard to prove. Most SLAs only cover connectivity to the ISP not the ISP's connectivity to the rest of the Internet. I would get another Internet provider, negotiate down your level of service with your current (if possible) for the remainder of the contract and drop them at the end of the contract. You can use them as a backup provider until the contract runs out so you don't have to write it completely off.

Post the details of your SLA and tests can be presented to measure those SLAs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.