TTL Changes During Continuous Ping

Question

We use Viasat and have the modem plugged directly into a linux based load balancer.

I SSH'ed into the loadbalancer using putty and pinged google

When I pinged google this is the result:

- PING 8.8.8.8 (8.8.8.8) from x.x.x.x : 64(92) bytes of data.

- 72 bytes from 8.8.8.8: icmp_seq=1 ttl=123 time=4.02 ms
- 72 bytes from 8.8.8.8: icmp_seq=2 ttl=123 time=8.40 ms
- 72 bytes from 8.8.8.8: icmp_seq=3 ttl=123 time=11.1 ms
- 72 bytes from 8.8.8.8: icmp_seq=4 ttl=123 time=16.2 ms
- 72 bytes from 8.8.8.8: icmp_seq=5 ttl=123 time=7.37 ms
- 72 bytes from 8.8.8.8: icmp_seq=6 ttl=123 time=10.0 ms
- 72 bytes from 8.8.8.8: icmp_seq=7 ttl=123 time=12.9 ms
- 72 bytes from 8.8.8.8: icmp_seq=8 ttl=123 time=7.04 ms
- 72 bytes from 8.8.8.8: icmp_seq=9 ttl=123 time=11.4 ms
- 72 bytes from 8.8.8.8: icmp_seq=10 ttl=123 time=10.5 ms
- 72 bytes from 8.8.8.8: icmp_seq=11 ttl=123 time=7.30 ms
- 72 bytes from 8.8.8.8: icmp_seq=12 ttl=123 time=4.42 ms
- 72 bytes from 8.8.8.8: icmp_seq=13 ttl=123 time=12.9 ms
- 72 bytes from 8.8.8.8: icmp_seq=14 ttl=55 time=573 ms
- 72 bytes from 8.8.8.8: icmp_seq=15 ttl=123 time=9.64 ms
- 72 bytes from 8.8.8.8: icmp_seq=16 ttl=123 time=12.0 ms
- 72 bytes from 8.8.8.8: icmp_seq=17 ttl=123 time=6.09 ms
- 72 bytes from 8.8.8.8: icmp_seq=18 ttl=123 time=9.14 ms
- 72 bytes from 8.8.8.8: icmp_seq=19 ttl=55 time=567 ms
- 72 bytes from 8.8.8.8: icmp_seq=20 ttl=55 time=556 ms
- 72 bytes from 8.8.8.8: icmp_seq=21 ttl=55 time=567 ms
- 72 bytes from 8.8.8.8: icmp_seq=22 ttl=123 time=7.83 ms
- 72 bytes from 8.8.8.8: icmp_seq=23 ttl=55 time=557 ms
- 72 bytes from 8.8.8.8: icmp_seq=24 ttl=55 time=555 ms
- 72 bytes from 8.8.8.8: icmp_seq=25 ttl=55 time=564 ms
- 72 bytes from 8.8.8.8: icmp_seq=26 ttl=55 time=565 ms
- 72 bytes from 8.8.8.8: icmp_seq=27 ttl=55 time=562 ms
- 72 bytes from 8.8.8.8: icmp_seq=28 ttl=123 time=10.0 ms
- 72 bytes from 8.8.8.8: icmp_seq=29 ttl=123 time=13.0 ms
- 72 bytes from 8.8.8.8: icmp_seq=30 ttl=123 time=12.5 ms
- 72 bytes from 8.8.8.8: icmp_seq=31 ttl=123 time=5.36 ms
- 72 bytes from 8.8.8.8: icmp_seq=32 ttl=123 time=8.44 ms
- 72 bytes from 8.8.8.8: icmp_seq=33 ttl=123 time=10.8 ms
- ^C
- --- 8.8.8.8 ping statistics ---
- 33 packets transmitted, 33 received, 0% packet loss, time 47026ms
- rtt min/avg/max/mdev = 4.020/160.648/573.236/246.800 ms

These entries:
72 bytes from 8.8.8.8: icmp_seq=19 ttl=55 time=567 ms

with that latency are the only ones resembling satellite latency and the rest well, certainly not sat. latency and the TTL changes from 123 to 55!

Also this ping is performed only out the load balancer's Sat. Modem interface. When you try to ping just the modem(Gateway) from the balancer or from an outside host it doesn't respond, like ICMP is blocked.

Same when we traceroute out the interface, the next hop(the gateway) is just * * *

Anyone with any experience with Viasat connections or any idea of whats going please help.

Answer 1

I believe your answer is going to come from some form of traceroute. But, as discussed in the comments, you have one path that is clearly not going through the satellite. Some systems have this utility named tracert or tracerout. tracepath offers similar functionality.

Note that you may need to run traceroute multiple times for it to actually catch the unexpected path. Also note that firewalls can and frequently do impair traceroute's effectiveness. While it's clear that icmp is not being blocked outright, security devices are fairly commonly configured to not generate the 'destination unreachable' icmp messages on which traceroute depends in its default mode of operation. Just because traceroute answers * * * for one TTL does not mean the next one won't have answers. Your ping test has verified that google.com will at least respond.

Another option could be ping -R. However, playing with this option myself, it appears Google doesn't support this usage, so you'll need to find a different target that will. This option works by including a header option that asks the routers to all record who they are on the packet as they pass it to the target, and then pass the response back to the source. I haven't used this option much, but I would guess that security devices may also mess with its output - either by just not mentioning their existence, or by dropping the request on the floor.

Answer 2

It could be a variety of things. First, the manual page for ping on my system says the following regarding TTL:

In normal operation ping prints the TTL value from the packet it receives. When a remote system receives a ping packet, it can do one of three things with the TTL field in its response:

• Not change it; this is what Berkeley Unix systems did before the 4.3BSD Tahoe release. In this case the TTL value in the received packet will be 255 minus the number of routers in the round-trip path.

• Set it to 255; this is what current Berkeley Unix systems do. In this case the TTL value in the received packet will be 255 minus the number of routers in the path from the remote system to the pinging host.

• Set it to some other value. Some machines use the same value for ICMP packets that they use for TCP packets, for example either 30 or 60. Others may use completely wild values.

Second, there is nothing technically preventing a router or security device imposing a maximum TTL on traffic passing through it, which could easily have the impact you are seeing.

So, my first suggestion is that your traffic might be load balanced to a destination host that uses a TTL of (e.g.) 60 (option 3 above)

Alternatively, there could be a router or other device doing something unexpected with the TTL that is on one of the possible routes between your computer and the destination. Remember, those routes may all be via your satellite connection; the Internet does not promise to send your packets using the same path every time.

The suggestion of using traceroute is a reasonable one. If repeated iterations of that command don't show any variation in the path from your host to the destination, then it may well be the destination that is modifying the TTL.

Note that when you see a long series of "* * *" responses in traceroute are a sign that a device between you and your destination is doing something unexpected with the TTL, at least for the 'destination unreachable' ICMP messages used by traceroute (which are handled a little differently from 'echo request' packets used by ping).

If your ping command offers you the ability to set the TTL, you might try a few different values to see what TTLs you get in response.

Answer 3

I found out that 8.8.8.8 is default balanced. I would have to use 4.2.2.2 to have pings only go out WAN2. Thanks everyone for your responses and I hope we all flexed our brains trying to figure this out. So we all win right? heh...😅😬