Using GCP interconnect with Bucket (Cloud Storage API)

Question

I am trying to implement a hybrid workflow using local compute resource + Google Cloud Platform. The system works like a charm however we were getting terrible upload speeds and so that was the main hindrance in our pipeline (approx 300GB data / run) and we get about ( 5 - 10 / week ) and the usual upload speed using gsutil is roughly 5.5 mbps - terrible!

To upload a lot of data we did our research and have managed to get an interconnect connection which is attached to a specific project (projectIC). [Interconnect is basically a dedicated pipe to get to GCP quicker] - This was successful and if I spin up a VM in projectIC, I can ping the internal-ip from our internal-network. I could also do a simple scp and transfer at roughly 22mbps - Which is great news again!

However, we rely on gsutil to transfer our data to the cloud. It is part of the workflow, and it is the recommended method of storing data (in a bucket, rather than on a VM)

Another issue is that the interconnect is project specific, and would like to connect other projects to this project (research shown below)

I am a software developer with fundamental cloud experience, and am also willing to learn but just need someone to point me in the correct direction.

There are two apparent solutions: 1. Upload from LOCAL to VM to BUCKET 2. Connect the Interconnect to Cloud API - not sure if this is possible.

After some research, I believe these are the steps I need to take:

1. Create VPC Network in projectIC and projectOTHER
2. Update firewall rules for both
3. Create VPN for projectIC and projectOTHER
4. Create CloudRouter for projectIC and projectOTHER
5. Create (atleast 2) VPN Tunnels for projectIC and projectOTHER
6. BGP Routing (to route traffic)

And at this point, I should be able to communicate from projectOTHER to projectIC over the VPN Tunnel. If the above is correct, that solves one of my problems. If there is then a way for me to connect the cloud API so that when I use gsutil cp <local-file> gs://<some-bucket> it should use the dedicated pipe and give me (atleast 22.2mbps over the current 5.5mbps)

I hope I have managed to explain my problem fairly well. Any advice or direction would be greatly appreciated. If you require any more information, just let me know!

Seems like there has been no activity on this question and I can still not use my interconnect with GCP Storage Buckets. Is there no GCP professional over here?

Answer 1

You can use Shared VPC for this purpose:

Shared VPC allows an organization to connect resources from multiple projects to a common Virtual Private Cloud (VPC) network, so that they can communicate with each other securely and efficiently using internal IPs from that network.

Another approach would be to use VPC Network Peering:

VPC Network Peering enables you to connect VPC networks so that workloads in different VPC networks can communicate internally. Traffic stays within Google's network and doesn't traverse the public internet.

Which one will work better depends heavily on specifics of the project, you can find useful comparison of these two here.

Answer 2

Use VLAN attachments to connect multiple VPCs to your Interconnect.

• Creating VLAN attachments
• Enabling multiple VPC networks to access the same VLAN attachment

Use Private Google Access to access Google APIs and services such as Cloud Storage from on-premises. This will require updating your router on-premises. If you are using Partner Interconnect, they should be able to help you.

• Configuring Private Google Access for on-premises hosts