Setting up a section in ~/.ssh/config is a fine solution, but it may be useful to know about another method.
The common scp-like syntax of user@host:path does not have a place for a port, but Git also supports an ssh: URL scheme that can be used to specify the port:
ssh://git@github.com:22/asdf/asdf.git
While an ssh: URL supports port specification, it does not support relative paths (e.g. there is no direct equivalent to the scp-like syntax of user@host:path where path does not start with a slash).
GitHub treats relative and absolute paths identically, so it works for them, but it may not work for all SSH-based Git repositories. For simple SSH-based hosting, you may need to insert /home/username/ or /Users/username/ when switching from relative to absolute paths. Some hosting systems may not handle absolute paths at all (though I would tend to call such lack of support a bug).