You'll need to use DNS round robin or NLB to distribute the incoming connections. An incoming connection will be routed to one of the servers (based on whatever mechanism you choose), the server receiving the incoming connection will query the Session Broker server to find out if the user has a disconnected session and will direct the connection to that server, or if no disconnected session exists the connection will be directed to the least loaded server.
Session Broker load balances the sessions, but it doesn't load balance the incoming connections. That's what DNS round robin or NLB is for. If you use DNS round robin then you'd create a dns A record for every TS server, using the Session Broker farm name for the A record.
So if you use DNS round robin, for instance, and your Session Broker farm name is "Farm", then you'd create two A records for farm with the ip address of each RDS server:
Users would connect to "farm" and the ensuing connection would be directed to one of the RDS servers based on the DNS round robin, which would then query the Session Broker server, which would then direct the connection to one of the servers (as described above).