I am trying to implement a hybrid workflow using local compute resource + Google Cloud Platform. The system works like a charm however we were getting terrible upload speeds and so that was the main hindrance in our pipeline (approx 300GB data / run) and we get about ( 5 - 10 / week ) and the usual upload speed using gsutil
is roughly 5.5 mbps
- terrible!
To upload a lot of data we did our research and have managed to get an interconnect connection which is attached to a specific project (projectIC). [Interconnect is basically a dedicated pipe to get to GCP quicker] - This was successful and if I spin up a VM in projectIC, I can ping the internal-ip from our internal-network. I could also do a simple scp
and transfer at roughly 22mbps
- Which is great news again!
However, we rely on gsutil
to transfer our data to the cloud. It is part of the workflow, and it is the recommended method of storing data (in a bucket, rather than on a VM)
Another issue is that the interconnect is project specific, and would like to connect other projects to this project (research shown below)
I am a software developer with fundamental cloud experience, and am also willing to learn but just need someone to point me in the correct direction.
There are two apparent solutions: 1. Upload from LOCAL to VM to BUCKET 2. Connect the Interconnect to Cloud API - not sure if this is possible.
After some research, I believe these are the steps I need to take:
- Create VPC Network in projectIC and projectOTHER
- Update firewall rules for both
- Create VPN for projectIC and projectOTHER
- Create CloudRouter for projectIC and projectOTHER
- Create (atleast 2) VPN Tunnels for projectIC and projectOTHER
- BGP Routing (to route traffic)
And at this point, I should be able to communicate from projectOTHER to projectIC over the VPN Tunnel. If the above is correct, that solves one of my problems. If there is then a way for me to connect the cloud API so that when I use gsutil cp <local-file> gs://<some-bucket>
it should use the dedicated pipe and give me (atleast 22.2mbps over the current 5.5mbps)
I hope I have managed to explain my problem fairly well. Any advice or direction would be greatly appreciated. If you require any more information, just let me know!
Seems like there has been no activity on this question and I can still not use my interconnect with GCP Storage Buckets. Is there no GCP professional over here?