There are three common ways of doing this:
First you can change your load balancer forwarding logic (either keep track of the number of connections to each host & try to distribute load evenly, do a simple round-robin, etc.). Either of the options I mention eliminate the deterministic nature of your current setup (clients from IP X no longer go to server Y), which also eliminates (or reduces) your problem.
Note that you want to implement "sticky sessions" or their equivalent so that once a client is randomly assigned to a back-end server they keep going to the same one as long as their connection is active.
Second you can decrypt the information, read some server identifier from it and then pass it off (either re-encrypting it or passing it in the clear over your back-end network). Note that this isn't really practical at large scale unless your load-balancing hardware is SSL-accelerated (e.g. a Cisco content switch with SSL modules) since the device you're funneling all the traffic through has to do ALL the SSL work.
Per note in the original question, #2 is probably not an option since the traffic needs to be kept encrypted end-to-end (sounds like decrypting on the load balancer would be a policy violation?)
The third method I don't recommend: Setting up split-horizon or round-robin DNS for your target server (either directly pointing to a back-end server or pointing to separate IPs on the load balancer which are statically tied to a back-end, have different balancing pools, etc.) -- This is pretty common in smaller operations as "ghetto load balancing", but in your situation (where you already have a load-balancing equipment) it adds needless complexity compared to the other solutions.