I've got a Google Cloud Function (GCF) that watches Google Cloud Storage (GCS). When a file drops into a particular bucket, the GCF fires up a LoadJob telling Big Query (BQ) to import the file.
I'm using the NodeJS LoadJob function that (from what I understand) is supposed to just instruct BQ to import the file directly. In other words, BQ is supposed to talk to GCS, rather than my GCF being a middleman and streaming the data in some fashion.
However, when my function runs on a lot of files at once, I get the following GCF quota error:
Outgoing socket traffic for region us-central1 : per 100 seconds
The default quota value is 10737418240
, which I assume is in bytes, making the quota ~ 10GB every 100 seconds.
Why should my GCF be making any outgoing socket traffic? The only thing I'm doing is setting up a BQ job to import. All of that should be internal, and further, it should all be directly between BQ and GCS, not using my GCF quotas/bandwidth. The files are pretty big, but isn't that exactly the point of loading into BQ directly from GCS?
Note: Upon reading my code a little closer, I'm also doing a query to move data from one BQ table to another. However, similar to the load job, this is a QueryJob that specifies a destination table to place the results in it. So, none of that data should be streaming through my GCF, and instead all data transfer should be occurring within BQ.