Whenever you login via SSH, your locale (the one on the machine your are connecting from) is sent to the server. I assume, you have de_DE.UTF-8 set locally. Whenever you execute the code snippet from Hetzner while you are logged in on the machine you temporarily switch to a different LC and everything works (for a while).
First off look in /etc/default/locale. You should find a LANG="en_US.UTF-8" in there (if not, add it).
I would recommend to install your locale on the server. Try choosing any de_DE.* in the dpkg-reconfigure locales dialog..
Or you could append the export commands (not the other commands!) into your ~/.bashrc on the server, so they will be set automatically whenever you login.
Or you could disable accepting LC_* and LANG variables on you SSH server (and your server should fallback to it's default). Therefore remove them from - or uncomment the whole - AcceptEnv in /etc/ssh/sshd_config.