Generally the response to the the load balanced request will go straight to the gateway, not back to the load balancer.
So the load balancer is a SPOF for the inbound requests, but because the outbound request goes straight to the gateway, it can bypass that SPOF - although because the ACK would need to come back in through the load balancer, it is still kind of pointless.
Generally you would have two load balancers. Only one of them owns the public IP address at any given time, and they run a heartbeat between them. As soon as the live load balancer loses its heartbeat (i.e. crashes, goes offline, etc), then the 2nd load balancer takes over the IP address.