It depends on the layer 4 protocol involved.
For UDP and other connectionless transport protocols (UDP-Lite, RUDP, DCCP, and some others), it just doesn't matter. Because there's no connection, you don't have to worry about a socket being persistently associated with a particular remote host, and therefore don't have to worry about the fact that port numbers are a 16-bit integer. You just send messages to the target backend, and track where things went. Depending on the load balancing software and how it's configured, there may however be a functional limit of 65536 outstanding requests to backend servers.
For SCTP and other inherently multiplexed connection oriented transport protocols (TIPC for example, though just about nobody uses that), it really doesn't matter either. You make exactly one connection from the load balancer to each backend server, and just multiplex however many streams you need over that one connection, because the transport protocol supports doing that. The same thing can actually be done with some application layer protocols too, like HTTP/2 (but not HTTP 1.1 or earlier) or the SSH protocol.
For TCP and other single-stream connection oriented transport protocols, it gets a bit trickier. Some configurations multiplex requests over a set number of persistent connections (by keeping the connections persistent, you save time, since setting up a TCP connection is insanely slow), while others use proprietary extensions at the application layer to multiplex things over a single connection.
For any of the above, there is also the option of using multiple backend-networks, either via multiple physical networks, via VLAN's, or via some other network multiplexing technology. This approach is most common when the backend servers are virtual machines, but is still widely used in other situations too. By having a separate subnet for each backend system, your 64k port limit goes from being per-frontend to per-backend (that is, instead of your load balancer only allowing 64k active connections to all backend servers, it can handle 64k to each back-end server), which pushes the bottleneck to the number of backends and how powerful each of them are.