I have a VPC ( with 3 VMs. VM2 and VM3 are the same except that VM2 has an external IP address (ephemeral) and VM3 does not. They're both running a Docker image with a web server exposing port 80.

I have an "allow all internal" firewall rule:

dev-internal    dev       INGRESS    1000  icmp,tcp:0-65535,udp:0-65535

I can shell into VM1, and from it I can ping both VM2 and VM3.

When I nmap VM2, I can see 998 ports closed and both ssh and http open - as expected.

When I nmap VM3, I get 999 ports filtered - implying the firewall is masking them? - and just the ssh port exposed.

I can curl from VM2, but VM3 times out.

I expect to be able to communicate across the internal network without assigning external IPs. What am I doing wrong?

(The only slight complication is that the VPC is defined in 1 project, and VPC-Shared into a second project. Firewall rules and routes are defined in the VPC host project. VMs live in the second project but on the shared VPC.)

Edit: The result of netstat -antp from VM2:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0    *               LISTEN      1207/nginx: master  
tcp        0      0    *               LISTEN      311/sshd            
tcp        0      0  *               LISTEN      299/systemd-resolve 
tcp        0      0      ESTABLISHED 381/python2.7       
tcp        0    272       ESTABLISHED 43364/sshd: jamie_h 
tcp        0      0      ESTABLISHED 401/device_policy_m 
tcp        0      0      CLOSE_WAIT  385/python2.7       
tcp        0      0      ESTABLISHED 385/python2.7       
tcp        0      0      ESTABLISHED 384/python2.7       
tcp6       0      0 :::5355                 :::*                    LISTEN      299/systemd-resolve 

VM3 is identical - both container OS, both running the same Docker image - and because I don't have an external IP, I can't shell into it (without doing a complicated dance...).

EDIT: Steps to reproduce:

  1. Create a VPC
  2. Create firewall rules for external SSH and HTTP access to the subnet (I use ssh and http tags respectively)
  3. Create firewall rule for internal traffic on VPC (source = subnet, target = all instances on subnet)
  4. Create VM called source on VPC using Debian with ssh network tag and external IP (to support SSH connection)
  5. Create VM called target1 on VPC using COS and the nginxdemos/hello docker image ("hello world" exposed on port 80) with http network tag and external IP
  6. Create VM called target2 on VPC using COS and the nginxdemos/hello docker image with http network tag and no external IP
  7. SSH to source VM and ping both target1 and target2 - they respond
  8. curl target1 and it responds with the hello world page
  9. curl target2 and it times out
  10. nmap -Pn target1 shows ssh + http open, all others closed
  11. nmap -Pn target2 shows ssh open and all others filtered
  • 1
    What is the output of netstat -antp on VM2 and 3?
    – zymhan
    Nov 14 '18 at 20:46
  • Not an answer to your question, but just in case you need to look inside VM3 to debug, you can use the serial console to avoid the "complicated dance".
    – Dan
    Nov 14 '18 at 23:01
  • You got my hopes up! In looking into this, I've found that if you're using the serial console to troubleshoot an issue after boot, you first need to shell into the VM to set a password for a local user... which takes me back to the dance. :)
    – Jamie Hale
    Nov 15 '18 at 14:43

You do not need an external IP to connect between your GCE instances. You can communicate between your GCE instances using the internal IP of the VM. Make sure your firewall is configured properly, the necessary ports are open for your purpose and you have a service listening on port 80.

  • That's certainly what I expected, but not what I can demonstrate. I've added steps to reproduce above. Is there something obviously wrong in there?
    – Jamie Hale
    Nov 19 '18 at 18:56
  • Per my answer below, you are correct. My steps were failing where I wasn't expecting.
    – Jamie Hale
    Nov 20 '18 at 16:54

This is far simpler than I had expected...

Without an external IP address, the ContainerOS instance cannot reach the internet to pull the Docker image. If you add a Cloud NAT instance (https://cloud.google.com/nat/docs/using-nat), the instance pulls the image, fires it up, and everything works.

The reason nmap was showing no http and everything else "filtered" must be because the firewall rules don't get applied until the instance is up and running.

(To find this, I had to toggle the "Private Google access" flag on the subnet. This allowed the internal-only VM to push logs to Stackdriver, which clearly showed the Docker pull timing out.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.