Server Fault

Questions tagged [google-cloud-storage]

Ask Question

Google Cloud Storage, also called GCS, is a cloud service for storing and accessing arbitrarily large objects.

119 questions
Newest
Active
Bountied
Unanswered
Filter by
Sorted by
Tagged with
0
votes
1answer
82 views

I'm on GCS's free trial; how do I increase my storage from 5gb to 25 gb?

So I'm playing around in a notebook that lets me experiment with the T5 model. When I try to fine-tune the model (it has 3B parameters), I get: WARNING:tensorflow:The `3B` model is too large to use ...
0
votes
1answer
30 views

Google Cloud load balancer redirect if from a certain region

I have two static pages (English, Danish) which are currently in Google Cloud Storage Bucket. This bucket contains en and dk folders for those static pages. Also, I have a Load Balancer with Google ...
0
votes
1answer
104 views

possible to set up GCS bucket lifecycle rule to archive rarely used objects?

GCS offers lifecycle management where you can, for example, "Downgrade the storage class of objects older than 365 days to Coldline Storage". I'd like to archive rarely used objects but keep ...
1
vote
1answer
111 views

Cloud Run service account does not have permission to sign

I'm working on a program that will run on Google Cloud Run and has files stored in Google Cloud storage. The problem I'm experiencing happens when attempting to generate a signed URL to download a ...
0
votes
0answers
41 views

Download large amount of data from onto google cloud storage without local temporary copy

I am using Google Cloud Compute Engine and Google Cloud Storage. I want to download a huge amount of data using my instance, so I mounted my bucket onto a directory using GCS Fuse. However, since GCS ...
1
vote
2answers
134 views

Prevent Cloud CDN from serving cached elements?

I have some problem with Cloud CDN. I've added a CDN to a bucket connected on an instance. I need to get an element from CDN, but this element is updated each 1 sec from my server. What happen is ...
0
votes
1answer
1k views

How to get a download URL for files in Google Cloud Storage?

I have created a service using : storage.Client(project='project_name', credentials=credentials) https://googleapis.dev/python/storage/latest/client.html And I'm able to upload, delete and list the ...
0
votes
1answer
16 views

Search pdf content in google storage

could you help me with the following? How can I search pdf words hosted on google cloud storage? example: pdf1, pdf2, pdf3 hosted on storage. Look for the "father" word in all documents. Thanks
0
votes
1answer
93 views

How to Change Cloud API access scopes on GCP Compute Engine with out stopping the Instance

I'm using gcsfuse inside Google Compute Engine. I mounted a folder with cloud storage bucket but, unable to create any file inside of it: touch t1 with an error: touch: cannot touch 't1': Input/...
0
votes
1answer
261 views

Using Google Cloud Armor to Block Requests to Google Cloud Storage

Does Google Cloud Armor support blocking requests to public Google Cloud Storage buckets? I haven't been able to determine if GC Storage counts as a backend service. The scenario is I would like to ...
0
votes
0answers
106 views

Mounting GCS buckets in GKE in non-privileged mode

I'm using GKE and trying to mount GCS buckets in pods by running in privileged mode and it works fine. However, I don't think it is a good idea to run in privileged mode in production. Wanted to ...
0
votes
1answer
35 views

GCS GoogleCredential

My application has requirement to download file from GCS bucket.I am able to download files from GCS using StorageClient DownloadObjects method. But performace is really slow. for 50MB file size it ...
2
votes
0answers
181 views

gsutil rsync fails on Ubuntu with UnicodeDecodeError

I'm trying to rsync between google cloud storage and amazon s3. On my local machine (MacOS) this command works gsutil rsync -d -r gs://mira-internal/data/exports/20191108 s3://mira-temp/raw/20191108 ...
0
votes
2answers
40 views

GCS ResumableDownload Class

My business application has requirement to download file from GCS bucket. File size can be in GBs also. Is there any class like ResumableUpload for download? I want to use ResumableDownload, so in ...
2
votes
2answers
75 views

How do you assign storage permissions to a group of GCP service accounts?

How does one assign Google Cloud Storage bucket permissions to a group of users? There's no bucket-level permissions that can be specified in roles, and there's no way to create a group as far as I ...
0
votes
1answer
158 views

ETL to bigquery using airflow without have permission cloud storage/ cloud sql

i have done ETL from MySql to bigQuery with python, but because i haven't permission to connect google cloud storage/ cloud sql, i must dump data and partition that by last date, this way easy but ...
0
votes
1answer
46 views

Where can I find Google Cloud Account: Email ID?

One of my clients has a WordPress site. He has stored his "Media content" (i.e. images) in Google Cloud Storage by using the WP-Stateless plugin. While trying to retrieve it, the client didn't know ...
0
votes
0answers
40 views

Bulk data export from Google Cloud Storage?

I need to export a decent amount of data (~5TB) from Google Cloud Storage onto an external hard drive. For AWS users, Google Cloud Storage is equivalent to S3. Using my company's internet connection, ...
0
votes
1answer
54 views

GCS bucket migration across different organisations

We are currently planning to migrate some GCS buckets across different organisations. After having a look at the documentation, it seems that that's something one can do between projects and buckets, ...
0
votes
1answer
759 views

Slow upload speeds when using gcloud compute scp and gsutil cp

I tried uploading a single file from a local drive directly to a google cloud compute vm instance using gcloud compute scp and to a google bucket using gsutil cp but in both cases the transfer speeds ...
0
votes
0answers
148 views

Docker push to gcr fails

I have a jenkins agent running in gke which builds a docker image and pushes it to gcr. It works perfectly. But after some time I get this error when pushing to gcr: denied: Unable to write blob ...
1
vote
1answer
497 views

Bad credentials for bucket

I'm getting Bad credentials for bucket trying to mount a GCS bucket using gcsfuse. What I've done: Created a bucket "mac-gcs-fuse" Created a service account in the same project with all storage ...
0
votes
0answers
905 views

Duplicity to Google Cloud Storage

We are trying to utilize duplicity with a Google Cloud Storage bucket. It almost works, but the challenge is to utilize it with a service account. The HMAC option is way too coarse grained for us. ...
2
votes
1answer
207 views

Restrict Access to google cloud buckets by domain name

I want to host files that my users upload to my web application in google cloud buckets...the problem there is that I need to have the permissions open to the public because I also need to retrieve ...
1
vote
3answers
724 views

Upload folder onto virtual machine

I created a VM instance on the Compute Engine in a Google Cloud project. I want to upload a folder from my computer's local storage onto the boot disk of the virtual machine. I tried using the scp ...
0
votes
1answer
261 views

After resize disk , can't ssh to GCP Cloud

I can't ssh to my instance after resize disk . I search for information and see some issue like - The disk if full - The ip I create a snapshot , and the snapshot only has size 8G , even my disk is ...
3
votes
1answer
399 views

gsutil rsync “too many values to unpack” error

I'm trying to sync a folder with over 3k files to a bucket on Google Cloud with gsutil. The problem is that less than half of the files are uploaded and then I get this error: too many values to ...
0
votes
0answers
27 views

Can I get the CRC32C checksum of a range content when I use HTTP Range Get requests to get a range of content in google cloud storage

When I want to get a partial range of file content in the google cloud storage, I used XML API and use HTTP Range Get requests. From the google cloud response, I can find the header x-goog-hash, and ...
0
votes
1answer
121 views

Firewall rules for accessing Google-Cloud-Storage from the outside world

I'm deploying an application that relies on GCS at a customer's site. The customer's firewall allows whitelisting domains for outgoing connections, so I've whitelisted the known GCS domains (storage....
1
vote
1answer
131 views

How/why would ALL our Google Cloud Platform Service account keys suddenly disappear/be deleted?

We had a major issue with Google Cloud Platform some time between 2019-04-16T11:00:00Z and 2019-04-17T02:30:00Z, when all of our service accounts lost their access keys for an unknown reason, so our ...
0
votes
1answer
263 views

How can I set a Google Cloud Storage object to be publicly visible when uploading it via a signed URL?

I'm following this guide and have a solution which is uploading objects to Google Cloud storage using the signed URL and the XML REST API (e.g. curl -v -X POST $URL -F "file=@test.jpg"). ...
1
vote
1answer
593 views

Secure static web page hosted on Google Cloud Storage

I would like to add basic authentication to the website that hosts on Google Cloud Storage. The site is connected to a load balancer, has a working SSL certificate and CDN. Basic authentication ...
4
votes
1answer
2k views

Service account does not have storage.buckets.get access to bucket

I'm trying to get a list of buckets in a project, using python like this: from google.cloud import storage storage_client = storage.Client(project='[project-id]') bucket = storage_client.get_bucket([...
0
votes
1answer
49 views

Backend bucket versioning

We have bucket with some files behind the CDN, along with signed URLs. We are using query string with version to update content according to the best practices, e.g. file.ext?v=100. Everthing worked ...
1
vote
0answers
50 views

Access error when creating domain Storage Buckets via Deployment Manager

When I try to create storagebuckets via Deployment Manager I get the error message below. I have already tried adding 1058303916595@cloudservices.gserviceaccount.com to the list of Verified owners of ...
-2
votes
1answer
81 views

Google Cloud SQL database increase [closed]

My Google Cloud SQL automatic storage increased up to 31GB but still the database size is only 120Mb and all the above happens in 3 days?
2
votes
1answer
686 views

How to handle retention policies in GCP Container Registry

GCP Container Registry is built on top of GCP Cloud Storage and one can set quite complex retention policies there. However there's no such functionality in Container registry and as there's no ...
1
vote
2answers
2k views

NGINX proxy to GCS bucket with redirect all urls to index.html, getting 200 blank responses on nested routes

I am configuring a reverse-proxy from NGINX to a GCP Cloud Storage bucket containing static HTML, JS, image files, with a rewrite for all non-matching URLS to index.html since it is a single-page-...
0
votes
1answer
206 views

google-cloud-sdk - This account has no projects

When I accessed the project of a company that is a member from the Cloud SDK in the Gsuite account, it returned "This account has no projects." Logging in with the browser, I can log in to the company'...
1
vote
1answer
138 views

Is there a way to mount my Google Cloud storage account on Windows Server 2016?

I want to be able to host site images on Google Cloud Storage. One way I am thinking to get it does is by adding a drive on my windows server that point to the cloud. Then my site would read the ...
3
votes
1answer
743 views

How to use JSON keys with google cloud gsutil to manage multiple Keys

We have multiple GCP Service account keys from different environment such as DEV,STAGING,..PROD. I would like to run a command in my jenkins box which is there in Dev environment to create access DEV ...
0
votes
1answer
177 views

Enable scope permission

I am trying to create a signed URL following this documentation but I fail to understand how to enable the scope for my Cloud Function. I am sure that my Cloud Function has cloudfunctions....
0
votes
1answer
57 views

Cloud Storage access from PHP aplication

We configure PHP applications on 2 servers with databases in CloudSQL, they require read /write static content in a CloudStorage bucket from PHP applications. What I did was install GCS Fuse to mount ...
0
votes
1answer
615 views

Get “gs_secret_key” and “gs_access_key” values for Google Bucket

I need to create a directory in a Google Cloud Bucket using Ansible. I would like to know where I can find the values to complete "gs_secret_key" and "gs_access_key" to use "gc_storage" module in ...
0
votes
1answer
165 views

Slash added automaticaly before bucket name with fstab and gcsfuse

I use gcsfuse to mount bucket in Google Compute Engine. For this I use this syntax in my fstab: my_bucket /my_bucket gcsfuse rw,uid=1001,gid=1002 But gcsfuse process is launched like this: /usr/...
1
vote
1answer
654 views

Loading data into BigQuery via Cloud Functions and Google Cloud Storage - Exceeding socket traffic quota limit

I've got a Google Cloud Function (GCF) that watches Google Cloud Storage (GCS). When a file drops into a particular bucket, the GCF fires up a LoadJob telling Big Query (BQ) to import the file. I'm ...
0
votes
1answer
108 views

Dataproc cannot create cluster: no storage.objects.list permission to `hadoop-lib`

I cannot create Dataproc Spark cluster for some reason. I run: gcloud dataproc \ --region us-west1 clusters create my-test1 \ --project some_project \ --scopes 'https://www.googleapis.com/...
0
votes
1answer
50 views

Deny Creating Public Buckets / Objects in GCP Buckets

Is it possible to create a policy in GCP that will disallow the creation of public buckets or items in buckets? If yes, how that policy will look like?
0
votes
0answers
1k views

Accessing a file in Google Cloud Storage from a web browser

I am trying to set up a file distribution to a couple of registered users. I am experimenting with GCS. The files must not be publicly available without logging in even temporarily. I have created a ...
1
vote
2answers
507 views

Parallel downloads in Google Cloud Storage

Does anyone have experience with Google Cloud Storage? Is it made for a potential 10K of parallel requests to download 10-70mb files? Or would that require a CDN?

1
2 3