I know I can whitelist a client IP for connecting to SSH on server and then Deny All other IPs. However, seeing as how the packet arriving at the server network interface must include the IP, what is stopping someone just pretending to be the whitelisted IP?
To be clear: the packets that return from your server will go to the IP address that the packets claim to be from, and not to where they actually came from.
IP traffic flows are not "pipes". :)
If the imposter sends a packet which says "My source address is x.x.x.x", your server will send its reply to x.x.x.x, and not to the imposter. So even if they controlled one router along the path, that wouldn't be enough to ensure the return packets arrive.