32

Is there a way to measure/report the overall latency in a tunneled SSH session?

My particular setup is:

  • Client (OS X + wifi router + ADSL modem)
  • Gateway SSH server exposed to Internet
  • Internal SSH target to which I'm tunneling

I'm interested in seeing the latency between the console on my local machine and the final machine on which I have the session open.

1
  • Why not SSH tunnel to the first server, and then SSH console to the second server.
    – Bert
    Mar 14, 2017 at 20:27

6 Answers 6

21

See the sshping utility: https://github.com/spook/sshping

Example:

# sshping 172.16.47.143
--- Login: 1725 msec
--- Minimum Latency: 4046 nsec
---  Median Latency: 11026 nsec  +/- 0 std dev
--- Average Latency: 178105 nsec
--- Maximum Latency: 8584886 nsec
---      Echo count: 1000 Bytes
---  Transfer Speed: 11694919 Bytes/second

# sshping --help
Usage: sshping [options] [user@]addr[:port]

  SSH-based ping that measures interactive character echo latency
  and file transfer throughput.  Pronounced "shipping".

Options:
  -c  --count NCHARS   Number of characters to echo, default 1000
  -e  --echocmd CMD    Use CMD for echo command; default: cat > /dev/null
  -h  --help           Print usage and exit
  -i  --identity FILE  Identity file, ie ssh private keyfile
  -p  --password PWD   Use password PWD (can be seen, use with care)
  -r  --runtime SECS   Run for SECS seconds, instead of count limit
  -t  --tests e|s      Run tests e=echo s=speed; default es=both
  -v  --verbose        Show more output, use twice for more: -vv
3
  • 3
    Not to confuse with ssh-ping from ssh-tools.
    – ks1322
    Feb 7, 2022 at 19:14
  • FYI I was able to get this to build on macOS with: brew install libssh, cd sshping-master/src, g++ -I ~/.homebrew/Cellar/libssh/0.10.4/include -I ../ext -L ~/.homebrew/Cellar/libssh/0.10.4/lib -lssh sshping.cxx
    – jacobsa
    Nov 22, 2022 at 23:03
  • I was able to compile this on an Apple Silicon MPB with brew install. libssh and g++ -Wall -I /opt/homebrew/include -I ext -L /opt/homebrew/lib -o bin/sshping src/sshping.cxx -lssh
    – David Alsh
    Sep 1, 2023 at 5:02
10

I skipped some steps suggested by @nicht-verstehen:

python -m timeit --setup 'import subprocess; p = subprocess.Popen(["ssh", "user@host", "cat"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, bufsize=0)' 'p.stdin.write(b"z"); assert p.stdout.read(1) == b"z"'

Where

python -m timeit executes the timeit Python module.

The -s/--setup option tells timeit which statement(s) to execute before each repeat.

subprocess.Popen(["ssh", "user@host", "cat"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, bufsize=0) launches ssh - executing cat on your host - as a child/subprocess, redirecting its IO streams to Python file-like objects. bufsize=0 makes sure no IO is buffered, which may cause IO waits.

And for each loop:
p.stdin.write(b"z") writes a single byte to the child (in turn through ssh to cat).
p.stdout.read(1) reads a single byte from the child. The assertion around it tests whether that byte is the same as the one you wrote to it.

Boils down to the same thing, but skips creating the named pipes (mkfifo). I noticed that the more loops you run, the faster each loop is. Control it using -n/--number: python -m timeit --number 50 ...

4
  • 1
    If you swap the string delimiters (i.e., ' and ") then you can replace user@host with a bash variable.
    – Leo
    Jun 24, 2020 at 11:18
  • Good point @Leo! thanks. Jun 25, 2020 at 14:47
  • 1
    This is a great snippet, but there is one issue. You noticed that the more loops you run, the faster each loop is. That's because when you send the first "z", the connection is not initialized yet. You can add ; p.stdin.write(b"z"); assert p.stdout.read(1) == b"z" to the end of the --setup to wait for the initialization, and then the resulting time will be correct. It won't be faster when you increase the loops.
    – Ian Lin
    Feb 9, 2023 at 6:25
  • The modified command: python -m timeit --setup 'import subprocess; p = subprocess.Popen(["ssh", "user@host", "cat"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, bufsize=0); p.stdin.write(b"z"); assert p.stdout.read(1) == b"z"' 'p.stdin.write(b"z"); assert p.stdout.read(1) == b"z"'
    – Ian Lin
    Feb 9, 2023 at 6:28
8

Was trying to do this myself and came up with this. Probably there is a simpler way, but this is what I came up with.

First, prepare pipes which will be used to make the benchmarking program communicate through the SSH connection.

$ mkfifo /tmp/up /tmp/down

Then establish a connection in ControlMaster mode without executing any remote command. This allows us to authenticate with the host interactively. After the connection is established, SSH will just "hang" here in foreground.

$ ssh $HOST -N -M -S /tmp/control

In a parallel terminal, execute remote cat in background. It will be our echo server whose latency we will measure. Inputs and outputs are connected to FIFOs:

$ ssh $HOST -S /tmp/control cat </tmp/up >/tmp/down &

And then benchmark a small program (send a byte to up FIFO, receive a byte from down FIFO):

$ python -m timeit -s 'import os' \
    'os.write(3, "z"); z=os.read(4, 1); assert z=="z", "got %s" % z' \
    3>/tmp/up 4</tmp/down
10 loops, best of 3: 24.6 msec per loop

The measure obviously shows the round-trip latency. If you need to repeat the experiment, run the last two commands (ssh and python) again.

If something seems to go wrong, use SSH -v flag to get more debugging output.

6

My idea was to use terminal query sequences for this; the advantage is that this can simply be run on the server, the disadvantage is that it measures terminal latency, not only the latency of the connection (but i guess, usually, your terminal's response time will be negligible compared to network delays)—maybe this is even what you mean with overall latency

#!/usr/bin/env python3
# Measure terminal latency (round-trip time) using "Query device code" command
from sys import stdin, stdout
import tty, termios, time

oldtty = termios.tcgetattr(stdin)
try:
    tty.setcbreak(stdout)

    runs = 10
    results = []
    for _ in range(runs):
        stdout.write("\x1b[c")
        stdout.flush()
        t1 = time.time()
        ch = stdin.read(1)
        assert(ch == '\x1b')
        t2 = time.time()
        while stdin.read(1) != 'c': # swallow rest of report
            continue
        latency = (t2 - t1) * 1000
        print('%.1fms' % (latency))
        results.append(latency)

    print()
    print('avg: %.1fms min: %.1fms max: %.1fms' % (
        sum(results) / runs,
        min(results),
        max(results)))
finally:
    termios.tcsetattr(stdin, termios.TCSADRAIN, oldtty)

(This uses "Device Code Query", all terminals that I've tried respond to this: xterm, alacritty, gnome-terminal. I cannot myself try this on MacOS. So YMMV, if this one doesn't, another of the requests that interrogate some information about the terminal might work, see http://web.archive.org/web/20190624214929/http://www.termsys.demon.co.uk/vtansi.htm)

5
  • 1
    This answer could still use some explanation for those not as familiar with tty internals (e.g., why \x1b[c would be favorable)
    – anx
    Jul 29, 2019 at 8:04
  • not sure it's favorable, i think it's a valid discussion: any from the "Device Status" termsys.demon.co.uk/vtansi.htm would work, given the terminal supports it, and i have no idea what's well-supported, but my guess is that the basic status query would be (even works in bare-bones Alacritty)
    – laanwj
    Jul 29, 2019 at 8:19
  • i think \x1b[c is Query Device Code: <ESC>[c
    – NorthIsUp
    Oct 7, 2021 at 2:33
  • 2
    I loved this solution because it doesn't require any setup or installation, just a working python3 environment on the destination. Very helpful if you have multiple SSH layers or other weird configurations.
    – kgutwin
    Nov 6, 2021 at 18:01
  • 1
    Wow, I just found out that my VTE based terminals show a 10ms lag whereas my xterm only 0.1. BTW, the link to vtansi.htm is dead but there is a copy on archive.org.
    – xebeche
    Jul 4, 2022 at 19:14
2

ssh_ping is a small Go program for this that is easier to build than the C++ one in another answer if you already have Go installed (and if you don't it's as easy as brew install go on macOS):

> go install github.com/jacobsa/ssh_ping
> ssh_ping --host some.host.com
100 samples so far...
200 samples so far...
Collected 294 samples.

Min:      13.0 ms
p05:      13.6 ms
p50:      13.0 ms
p95:      21.1 ms
Max:      26.0 ms

Mean:     17.0 ms
Std. dev:  2.6 ms
2
  • Does this work through a jump host as specified in the question? It seems like it does if because if I use a target specified in my ssh_config with a JumpHost but not sure how to confirm
    – pufferfish
    Jan 12, 2023 at 11:33
  • It should; it just runs the ssh binary, so it’s realistic in doing whatever that does. Certainly it works for the proxy case in my question (although it’s not a JumpHost in particular).
    – jacobsa
    Jan 14, 2023 at 22:09
1

I wrote a Bash script that repeatedly calculates ping-pong RTT/latency. The basic concepts are similar to what others have suggested. The main part does something along these lines:

fifo="fifo"; mkfifo "$fifo"; sleep=1; now=(date +%s%N)

( echo 'o' >"$fifo"; ) & # to trigger the following loop

cat 0<> "$fifo" | ssh $sshargs $ssh cat \
| while read -r R ; do
   echo $(( $(date +%s%N)-now ))
   sleep "$sleep"
   echo 'o' >"$fifo"
   now=$(date +%s%N)
done

I have to admit it works better than I expected :]

Example output for my sshpingpong:

Logfile is spp_example_net.log
2022-07-24 14:41:00  seq=0  13.349
2022-07-24 14:41:05  seq=1  14.640
2022-07-24 14:41:10  seq=2  13.990
2022-07-24 14:41:15  seq=3  12.802
2022-07-24 14:41:20  seq=4  12.070
2022-07-24 14:41:25  seq=5  11.869
...

Example output of compact mode using sshpingpong -L -c [email protected] where "_" means the RTT was <100ms and e.g. "2" means the RTT was >=200 and < 300ms:

0 _ 100 1 200 2 300 3 400 4 500 5 600 6 700 7 800 8 900 9 999 # ...
__________________________1_________________________1__________
______________________________________1____________#_2_________
____1____________________________________#______1______________
____1_____________________________________________1_____1___1__
_____________________1____1____________________________________
__________1__________________1__________________1______________
___2___________________1_______________________________________
_____________________22____1____11____59####4444454523444543444
12222#21112______#________________...

The full script is available as Github gist.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .