For complex reasons rsync/scp/sftp runs .bashrc when connecting to another host. You must have any of these commands at the top of your .bashrc:
either
[[ $- != *i* ]] && return
or
[ -z "$PS1" ] && return
Any of the above commands will only allow the execution of the rest of .bashrc commands for interactive sessions. As far as I know you don't need them for any other type of session (and indeed I have seen default bashrc from Arch and Debian using this technique in their bashrc).
If however you want to be extra paranoid about letting your bashrc commands run even for non interactive sessions you should at least wrap the commands of your bashrc that do produce output like this (reference) so that they only run in interactive sessions:
if shopt -q login_shell; then
# this is an interactive session, we _can_ display output
...code that produces output goes here...
fi
Note that others suggest moving commands that output text to your bash_profile but I have my doubts about whether this is always good (for reasons explained here)