1

I have 2 VPC's

  • VPC1 - default google subnets
  • VPC2 - single aditionl subnet not in VPC1

Server (instdual) setup with nic0 from VPC1 and nic1 from VPC2

Static public IP on both nic's

  • Ping from outside to Public IP -> VPC1 working
  • Ping from outside to Public IP -> VPC2 not working

Setup 2 more instances insta and instb one only on VPC1 and other only on VPC2

  • Ping from outside to Public IP -> VPC1 working (insta)
  • Ping from outside to Public IP -> VPC2 working (instb)

  • from insta I can ping instdual nic0

  • from instb I can ping instdual nic1

  • from insta I CAN NOT ping private IP of nic1

  • from instb I CAN NOT ping private IP of nic0

VPC's are network peered - routes shows correct

Firewall set a default allow all rule to negate firewall issues.

Basically on instdual I can only access it on the nic0 public IP. not the nic1 public IP.

Any ideas ? I am 12 coffees behind and seeing double at the moment.

0

2 Answers 2

1

This is what was missing (Now working 100%):

Follow these steps to configure policy routing for a Linux-based instance with multiple interfaces:

  1. Connect to an instance configured with multiple network interfaces:

gcloud compute ssh multinic-vm

  1. Configure policy routing with ifconfig for nic1. The example below assumes that GCP has assigned the internal IP address 192.168.0.2 to nic1 and the gateway is 192.168.0.1.

sudo ifconfig eth1 192.168.0.2 netmask 255.255.255.255 broadcast 192.168.0.2 mtu 1430
sudo echo "1 rt1" | sudo tee -a /etc/iproute2/rt_tables # (sudo su - first if permission denied)
sudo ip route add 192.168.0.1 src 192.168.0.2 dev eth1
sudo ip route add default via 192.168.0.1 dev eth1 table rt1
sudo ip rule add from 192.168.0.2/32 table rt1
sudo ip rule add to 192.168.0.2/32 table rt1

  1. Repeat the commands in step 2 for additional interfaces on the instance (nic2, nic3.... nic7).
0

Google cloud has some documentation about how to do this here: https://cloud.google.com/vpc/docs/create-use-multiple-interfaces But this lacks information about how to make sure the additional interfaces will also work again after a reboot.

In order for the additional external IP's to be persistant, you need to make sure the commands to activate the policy routing are executed after a reboot, but only once the additional interfaces have been activated. This is done by the dhcp-client. So the best way I could find to do is is by putting a script inside /etc/dhcp/dhclient-exit-hooks.d/ with this in it:

#!/bin/sh
#

if [[ $reason == "REBOOT" || $reason == "BOUND" ]] ; then
    sudo ip route add 192.168.0.1 src 192.168.0.2 dev eth1
    sudo ip route add default via 192.168.0.1 dev eth1 table rt1
    sudo ip rule add from 192.168.0.2/32 table rt1
    sudo ip rule add to 192.168.0.2/32 table rt1
fi

(change the IP's to the IP's of your internal network)

If you also want to be able to bind a server (like nginx or httpd) to one of those IP's at boot time, then you'll notice this fails because the server is started before dhcp-client has completed its task. One way to overcome this is allowing software to bind to IP's that aren't actually active yet. For this put

net.ipv4.ip_nonlocal_bind=1

into /etc/sysctl.d/10-policyrouting.conf

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .