ssh reads ~/.ssh/environment, and adds lines of the format
“VARNAME=value” to the environment if the file exists and users are
allowed to change their environment. For more information, see the
PermitUserEnvironment option in sshd_config(5).
Rats, I hoped it could be on the local side, still, if there's a will, there's a way. man ssh_conf:
Specifies what variables from the local environ(7) should be sent
to the server. Note that environment passing is only supported
for protocol 2. The server must also support it, and the server
must be configured to accept these environment variables. Refer
to AcceptEnv in sshd_config(5) for how to configure the server.
Variables are specified by name, which may contain wildcard char-
acters. Multiple environment variables may be separated by
whitespace or spread across multiple SendEnv directives. The
default is not to send any environment variables.
Depending on the configuration of sshd on the receiving end this may or may not fulfil the requirement of "no remote file modification".