Nested RDP and VMWare and ILO console sessions: keystroke repetition and latency

Question

I'm working on a remote server installation entirely through ILO (but this also applies to IPMI and VMWare console sessions). Due to the software application and environment, my access is restricted to a Windows server that I must access through RDP. Going from that system to the target server is accomplished via HP ILO2 or ILO3.

I'm trying to run a CentOS installation in an environment where I can't use a fully-automated deployment system. I'm doing this via text mode, but the keystrokes are repeating randomly and it's difficult to select the proper installation options. For example:

ks=http://all.yourbase.org/kickstart/ks.cfg

ends up looking like:

ks====httttttp://allll..yourbaseee.....org/kicksstart/ks.cccfg

I'm doing this using Microsoft's RDP client (on Mac and Windows). I've also noticed this before when running installations or doing remote work in nested sessions.

Is there a nice fix for this, or it it simply a function of the protocol(s)?

Answer 1

Whereas an SSH connection transmits keystrokes, an HP ILO connection transmits key states. Each time you press a key, the server receives separate KeyDown and KeyUp events. The repeated keystrokes result when the KeyUp event is received late.

The two most likely reasons for the KeyUp event to be received late are:

1. Network congestion/performance issues.
2. Poor performance of the client system initiating the ILO connection. If the client is a virtual machine, is the underlying host system overloaded, or does the VM have inadequate memory/CPU resources allocated?

If the root cause cannot be addressed:

1. The key repeat problem can be worked around by disabling an ILO2 setting called "Key Up/Down". This will cause ILO2 to transmit keystrokes instead of key states. Unfortunately, this setting was removed from ILO3.
2. If the target operating system is Linux, you may be able to work around the issue by redirecting the console to ttyS0 and using a Virtual Serial Port (VSP) session instead of a virtual console. This will eliminate the Key Up/Down issue, because serial connections transmit keystrokes instead of key up/down events.
3. It may be helpful to adjust the key repeat rate and/or disable autorepeat entirely on the target system. I acknowledge that this may not be easy to accomplish, depending on the severity of the key repeat problem.
4. Given that you're using a Mac as your local workstation, it might be worthwhile to try pasting complete commands into your Mac RDP client using Command-V. I do not know whether this is a viable workaround, but it might have an interesting effect. I've often appreciated working on remote Windows machines from a Mac workstation specifically because local command-hotkey combinations continue to work predictably.

References:

• Advisory: HP Integrated Lights-Out 3 (iLO 3) - Using iLO 3 Integrated Remote Console (IRC) Over a Slow or Congested Network May Cause Keystrokes to Repeat
• HP Communities - ILO3 typing / mouse lag problems?

Answer 2

This looks likes it's just a problem with the protocol. I've reduced the issue somewhat by using Ericom Blaze as the RDP transport for the central server I connect from; e.g. "jump box".

Other things:

I'm trying to avoid multiple nested sessions.

I'm running VMWare Fusion with Windows 7 on my Mac to allow me to use the native RDP from Windows in certain cases.

That's about all I can see for now.

Answer 3

you need to edit the .vmx file to add the following line:

keyboard.typematicMinDelay = "2000000"

it takes out the "bounce".

With my version of vmware, I have to make this change when the VM is down. I understand that it can be made from an edit window, but I have not been able to find that place.

Answer 4

Is the problem happening with your connection to the rdp (can you type in notepad correctly?) or between the RDP and the iLO)?

If between RDP and iLO (I know you have done this already)

1. Using the Java remote console was near impossible. I found if I used the "remote console" (it may be called .Net), resulted in a massive improvement. The latency was less, the latency was not jittery, and repeated and lost keystrokes didn't happen.

2. Boot off live cd, install openssh server, and use ssh to connect. Do our install over ssh (if connection is bad use screen also.

If between you and RDP:

Use freenx or vnc tuned to low bandwith to your windows box. This should at least clean up the keystrokes. Is the connection to the RDP ok (is that where the keystroke problems are happening?

If both: Write out commands in a notepad, then copy and paste if you can, hopefully it will work better than typing.

Answer 5

The first critical thing to remember is to disable key repeat on everything processing your keystrokes, including on the virtual machine or RDP session you're connecting through, as well as the top-level host machine. This does not fix the final target machine but it does a lot to improve the situation.

As for the target machine:

There are reports that using ssh to connect to HP iLO's SSH port avoids key repeat issues, but I could not use this method because my host (online.net) did not let port 22 through their iLO firewall. But if you have access to iLO's SSH port (likely 22), that seems like the easiest approach.

I tried using a systemd unit to set the keyboard repeat rate and delay time on boot:

# Note that kbdrate only affects existing keyboards, and HP iLO attaches a new
# USB keyboard when you connect, so you may have to reboot (with the iLO console
# attached) to get the keyboard delay and repeat rate to take effect.

[Unit]
Description=Set longer delay time for key repeat

[Service]
Type=oneshot
RemainAfterExit=yes
StandardInput=tty
StandardOutput=tty
ExecStart=/sbin/kbdrate -d 1000 -r 2

[Install]
WantedBy=multi-user.target
WantedBy=rescue.target

(Make sure /sbin/kbdrate is where you have kbdrate. Write to /etc/systemd/systemd/slower-keyboard-repeat.service and systemctl daemon-reload && systemctl enable slower-keyboard-repeat.service)

but as mentioned in the comment, this was only a partial success because it required a reboot to set the repeat rate on the new keyboard that iLO attaches. But it is good enough if you are OK with rebooting the machine.

Ultimately, I ended up patching the Linux kernel to change the default repeat rate and delay time on all keyboards:

From 78c32f539b89bf385985bea47a7058a540d31da0 Mon Sep 17 00:00:00 2001
From: Ivan Kozik <[email protected]>
Date: Thu, 30 Mar 2017 13:31:17 +0000
Subject: [PATCH] Increase the default keyboard repeat delay from 250ms to
 1000ms and repeat rate from 1000/33 Hz to 1000/500 Hz to avoid unintentional
 repeated keystrokes when using remote consoles such as HP iLO over
 high-latency links.  These consoles (HP iLO included) often transmit key
 states (up/down) instead of keystrokes, making it impossible to even enter a
 password and log in.

Fixing this in the kernel avoids problems with kbdrate where the parameters
passed to kbdrate don't apply to the new keyboards attached by HP iLO.
---
 drivers/input/input.c          | 2 +-
 drivers/input/keyboard/atkbd.c | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/input/input.c b/drivers/input/input.c
index 880605959aa6..a195af2d062a 100644
--- a/drivers/input/input.c
+++ b/drivers/input/input.c
@@ -2126,7 +2126,7 @@ int input_register_device(struct input_dev *dev)
     * is handled by the driver itself and we don't do it in input.c.
     */
    if (!dev->rep[REP_DELAY] && !dev->rep[REP_PERIOD])
-       input_enable_softrepeat(dev, 250, 33);
+       input_enable_softrepeat(dev, 1000, 500);

    if (!dev->getkeycode)
        dev->getkeycode = input_default_getkeycode;
diff --git a/drivers/input/keyboard/atkbd.c b/drivers/input/keyboard/atkbd.c
index ec876b5b1382..9dd04c2215b3 100644
--- a/drivers/input/keyboard/atkbd.c
+++ b/drivers/input/keyboard/atkbd.c
@@ -1096,8 +1096,8 @@ static void atkbd_set_device_attrs(struct atkbd *atkbd)
            BIT_MASK(LED_MUTE) | BIT_MASK(LED_MISC);

    if (!atkbd->softrepeat) {
-       input_dev->rep[REP_DELAY] = 250;
-       input_dev->rep[REP_PERIOD] = 33;
+       input_dev->rep[REP_DELAY] = 1000;
+       input_dev->rep[REP_PERIOD] = 500;
    }

    input_dev->mscbit[0] = atkbd->softraw ? BIT_MASK(MSC_SCAN) :
-- 
2.11.0

and that solved the problem for me.

Answer 6

I know you said you're restricted, but I can't think of anything better than: install VNC or TeamViewer, at least just to do the critical part of your install.

Second solution is to use a Media Center type forwarding proxy for input messages, so you would hook up a second keyboard to your computer, and using HID, forward only that keyboard over TCP/SOAP to the server. But as that involves installing software daemons on the server, you might as well start with VNC.

I've never experienced repeated keystrokes, but I do get major mouse lag when working with VMware over RDP, when the Guest OS doesn't have VMware Tools loaded.

The final option I have, if none above are suitable, is to contact Microsoft Support, and report the resolution they give you here.. like an opensource ticket.

Answer 7

In my experience it has helped me if I try to forget all I have learned about touch typing and try to punch the keys one by one and very, very quickly. Preferably use only one finger so that you don't get too comfortable and start typing too fast. It also lets you concentrate on trying to press the key quickly. This whole thing may sound like a joke, but I have found that my right middle finger (I am right handed) is by far most capable of pushing keys quickly.

And of course, I try to get SSH up and running as quickly as possible after that. If are too restricted to be allowed to do that... ouch.

Also try to use the different consoles. Normally the Java version would be the worst, but if you are having problems with the .NET version, then you may want to give the Java a try. Just be prepared that the java plugin may crash your browser (this is only a problem with iLO 2; iLO 3 moved from a plugin to a web start app).