I'm testing various GCP features and I've faced with the question in the title. After a little bit of experimentation I think the following should hold:
- 2 peered VPC can not share the same subnet ranges, while VPC sharing is sharing the same subnets: if we want instances to communicate and we adjust firewall (FW) rules, does this make any practical difference?
- Shared VPC creates a hierarchical relation where one end is the manager of the network and FW rules and therefore can dictate all service projects abilities and can revoke shares, this also means that the host part must have access to service projects to let pick them and allow them to use the host project VPCs. Anyway VPC peering require a certain level of access to projects if one wants to peer them, but the 2 projects are on-par (both ends must allow the peering): this is an administrative/auth difference
- Shared VPC allows for a simplified FW setup as you have only one central point to setup your FW rules: you have the same set of subnets shared; while peering - alike VPNs - requires to setup rules on both ends: this is a management simplification
- Shared VPC can exhaust its resources (IPv4 ranges) faster but this means you have a ton of instances connected...
- VPC peering can do passthrou (daisy chain) up to 1 level: I've 1 connection from VPC A to VPC B and one from VPC B to VPC C. VPC A and C can not communicate but VPC B can communicate with both. On the opposite in a share scenario a project can only be either a host or a service at the same time but I can create a scenario with multiple projects sharing the same subnet and talk to each other transitively... this is probably the most relevant difference I've seen
Let say we have N different projects with N different administrators, if all parts agree on having some sort of network connection among instances, is there any other pro/cons in choosing peering over shared?!
- maybe this is the really biggest difference: if a service project uses a shared VPC and you later want to remove it, you need first to create a new VPC (or use the service project's default), reassign a new nic to all the instances which are currently using the shared VPC, let this new nic use the project own VPC, redo the FW rules and check for correct connectivity of all instances before detaching them from the shared VPC.
- additionally VPC peering can be used within the same project to increase the isolation between workloads but letting few selected VM communicate.
- As of now (2021-01) shared VPC can be used on a second NIC but it is a beta feature
- also one can not add more than 25 peering to a network, therefore it is quite common to let project communicate via shared VPC in so-called hub-and-spoke designs.
- Just discovered that VPC peering can be applied even between different organizations!