1

I've got a Google Cloud Function (GCF) that watches Google Cloud Storage (GCS). When a file drops into a particular bucket, the GCF fires up a LoadJob telling Big Query (BQ) to import the file.

I'm using the NodeJS LoadJob function that (from what I understand) is supposed to just instruct BQ to import the file directly. In other words, BQ is supposed to talk to GCS, rather than my GCF being a middleman and streaming the data in some fashion.

However, when my function runs on a lot of files at once, I get the following GCF quota error: Outgoing socket traffic for region us-central1 : per 100 seconds

The default quota value is 10737418240, which I assume is in bytes, making the quota ~ 10GB every 100 seconds.

Why should my GCF be making any outgoing socket traffic? The only thing I'm doing is setting up a BQ job to import. All of that should be internal, and further, it should all be directly between BQ and GCS, not using my GCF quotas/bandwidth. The files are pretty big, but isn't that exactly the point of loading into BQ directly from GCS?

Note: Upon reading my code a little closer, I'm also doing a query to move data from one BQ table to another. However, similar to the load job, this is a QueryJob that specifies a destination table to place the results in it. So, none of that data should be streaming through my GCF, and instead all data transfer should be occurring within BQ.

1
  • I believe this question would be better suited to Stack Overflow Aug 24, 2018 at 11:38

1 Answer 1

1

The data transfer actually comes from unzipping a file to GCS. I receive a large .tar.gz containing multiple files, so I unzip (as a stream) to GCS. This is where all the data transfer comes from.

It's a little strange strange to me that this is considered outgoing socket traffic, since it's internal to the Google datacenter.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .