Yes. I have a set of external VIPs "paired" with a set of internal VIPs that have this same requirement. I also have failover working in both directions.
Here's a look at my config and some explanation. I've redacted the public IPs.
So, there are two internal floating VIPs, 10.0.9.1/24 and 10.0.9.4/24. 10.0.9.1/24 is primary on server1 and 10.0.9.4/24 is primary on server2.
For external VIPs, x.y.z.2/27 is primary on server1 and x.y.z.25/27 is primary on server2.
I just realized this but I am using the eth0 interface for all these IPs without issue. In reality, the x.y.z.0/27 network is physically connecting to eth1. Give it a shot.
See also keepalived/vrrpd and vrrp_sync_groups: