Questions tagged [gsutil]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
3 answers
305 views

gcloud auth login: Limit Oauth2 scope

How can I limit the Oauth2 scope when logging in with my user principal using (non service account): gcloud auth login Background: I created a Gsuite Takeout, got the Takeout Bucket. Now I want to ...
1 vote
0 answers
81 views

gsutil fails with "OSError: Operation not permitted" when running from cron

I have a php script that does backups. The backups have been failing, because one of the first commands throws an error and stops the script. When I run the script in the terminal, it works just fine. ...
0 votes
0 answers
505 views

What is the difference between -i and -u options in gsutil rsync command?

The gsutil rsync documentation describes two options which sound identical: Option Description -i This forces rsync to skip any files which exist on the destination and have a modified time that is ...
0 votes
2 answers
1k views

Using gsutil on Windows to download a Google Workspace Data Export

What is the easiest approach to automatically download all export files onto a Windows system? I need to download a full Google Workspace Data Export using Windows. The Google Workspace Data Export is ...
0 votes
1 answer
156 views

Where can I find the gsutil.tar.gz file for version 4.66 of gsutil?

The situation on my end is really very complicated1. The bottomline, however, is that I desperately need to download an version of the gsutil.tar.gz file that is older than the one currently ...
1 vote
2 answers
1k views

Specify account for single gsutil call

I want to be able to perform simultaneous gsutil calls against Google Cloud Storage, using different accounts. For that purpose, I'd like to know if there is a way to specify a Google Cloud account ...
1 vote
1 answer
963 views

Gcloud cannot detect network connection

I have internet connection on the ESXi server and I want to copy everything to GCP storage. But Gcloud couldn't authenticate. It looks like it can't reach the gloud login servers. I'm not if the esxi ...
1 vote
1 answer
763 views

Attempt to copy files from google cloud bucket to VM instance using gsutil is falling after reaching size limit

I have a bucket with a file whose size is larger than 11 GB. I note that I can access all bucket's data from the standard google cloud console. Unfortunately, this console has very limited ...
0 votes
1 answer
392 views

change storage class a folder gcp

change storage class in a folder into a bucket in GCP, I try with gsutil rewrite -s coldline gs://bucket-name/folder/ I see the gsutil stat gs://bucket-name/folder/ what am I doing wrong?
3 votes
1 answer
926 views

ServiceException: 401 Unauthorized WebHook callback channel

Can anyone help me with issue on setting up "Object Change Notification". I am trying to execute " gsutil notification watchbucket https://<my_project_id>.appspot.com/ gs://glrs_test_deepshikha ...
1 vote
1 answer
3k views

gsutil from gcloud/Google Cloud SDK failing with "SyntaxError: invalid syntax" error in bcrypt

Trying to use gsutil today I got the following error: + gsutil cp -nra public-read a gs://b /usr/local/lib/python2.7/dist-packages/cryptography/__init__.py:39: CryptographyDeprecationWarning: Python ...
1 vote
1 answer
796 views

start/stop a Google Cloud VM from desktop

I have a VM running on Google cloud that I would like to start and stop. I am able to start/stop it from another VM in the same project using the command sudo gcloud compute instances stop my-daily-...
0 votes
1 answer
123 views

remotely connect to files via gs path [closed]

I'm using an R package that requires a list of file locations to work. Normally, if I have files stored in remote servers I provide the ftp, http or ssh paths and it works. If I do the same providing ...
0 votes
1 answer
519 views

gsutil error /bootstrapping/gsutil.py

gsutil is giving me the following error. It is installed in its latest version on a linux suse 12. Can you help me? gsutil Traceback (most recent call last): File "/root/google-cloud-sdk/bin/...
2 votes
0 answers
396 views

gsutil rsync fails on Ubuntu with UnicodeDecodeError

I'm trying to rsync between google cloud storage and amazon s3. On my local machine (MacOS) this command works gsutil rsync -d -r gs://mira-internal/data/exports/20191108 s3://mira-temp/raw/20191108 ...
29 votes
2 answers
41k views

Using JSON keys with google cloud gsutil

I have a ssh private key in a key.json file, and I want to use this credential to access a storage bucket using gsutil. I can't seem to find anything about how to include json keys as a ...
3 votes
1 answer
654 views

gsutil rsync "too many values to unpack" error

I'm trying to sync a folder with over 3k files to a bucket on Google Cloud with gsutil. The problem is that less than half of the files are uploaded and then I get this error: too many values to ...
1 vote
1 answer
6k views

Access denied 403 when trying to download from the Google Cloud Platform, but user has all access

We are just trying to use gsutil from the command line to download some DCM data from the Google Cloud Platform: gsutil -m cp -R gs://dcdt_-dcm_account75701/dcm_account75701_activity_201803* C:\Users\...
1 vote
1 answer
369 views

How/why would ALL our Google Cloud Platform Service account keys suddenly disappear/be deleted?

We had a major issue with Google Cloud Platform some time between 2019-04-16T11:00:00Z and 2019-04-17T02:30:00Z, when all of our service accounts lost their access keys for an unknown reason, so our ...
0 votes
2 answers
567 views

Dedicated Interconnect to GCP Storage (gsutil)

We are have a dedicated interconnect setup from our data center to GCP using a cloud router. We have a RHEL linux server setup in the datacenter with a nic port connected to the public Internet and a ...
3 votes
1 answer
3k views

How to use JSON keys with google cloud gsutil to manage multiple Keys

We have multiple GCP Service account keys from different environment such as DEV,STAGING,..PROD. I would like to run a command in my jenkins box which is there in Dev environment to create access DEV ...
0 votes
0 answers
82 views

Why would gsutil come up short when transferring files into a bucket?

I have a very large directory tree sitting in the file system of a Google Compute instance. If I do a "find foo -type f | wc -l" on it, it comes back with 34577, and I have no reason to assume that ...
1 vote
1 answer
4k views

CommandException: arg [...] does not name a directory, bucket, or bucket subdir. for a directory on Compute Engine

I am trying to use gsutils command to deploy changes from CircleCI to a directory on my Compute Engine instance. The command I am using is: gsutil -m rsync -d -r dev/ [VM_INSTANCE_NAME]:/var/www/dev/ ...
0 votes
1 answer
2k views

gsutil "Copying mtime from src to dst" having moved source files?

I use gsutil to backup photos from a dedicated box to google nearline storage. Recently I moved all of my photos from that dedicated box to a new dedicated box. I made sure to use the relevant ...
0 votes
2 answers
1k views

How to run gsutil rsync without auth?

aws s3 sync provides a parameter named --no-sign-request, so user can sync a publicly-visible bucket without any auth in advance. gsutil help anon says that: gsutil users can access publicly ...
-2 votes
1 answer
853 views

CommandException: Caught non-retryable exception - aborting rsync

..... At destination listing 8360000... CommandException: Caught non-retryable exception - aborting rsync ..... I tried to locate the files with this sync problem but I am not able to do so. Is there ...
1 vote
2 answers
2k views

Convert filenames to lowercase [Google Cloud Storage] [gsutil]

We recently moved all product images from our website to Google Cloud Storage. Due to bad naming policies in the past the casing (lowercase, uppercase) in the file names does not always match the way ...
0 votes
0 answers
687 views

Why does gsutil rsync on CoreOS fail with 'CommandException: arg () does not name a directory, bucket, or bucket subdir'

On a recent Google Cloud Platform CoreOS VM, the command gsutil rsync -r gs://some-bucket-here dest fails with: CommandException: arg (dest) does not name a directory, bucket, or bucket subdir. $ ...
2 votes
1 answer
1k views

Google Storage ACL - write only for public, but don't allow read, list, delete or update operations

I need to setup Google Storage access for 3rd parties (public) to allow them to upload one or more files, but they shouldn't be allowed to read, list, delete or update an existing file. This structure ...
3 votes
1 answer
7k views

Copy list of files (-I flag) with gsutil preserving path

I am trying to copy all pictures and static files to a bucket of mine in Google Cloud Platform. I am attempting this command from the root dir of my app: find -regextype posix-extended -iregex ".*\.(...
2 votes
1 answer
970 views

gsutil ls -l fails when gsutil mb succeeded

I get the following when I attempt to list the buckets for the account I'm logged in with: $ gsutil ls -l gs://bucket AccessDeniedException: 403 Forbidden $ gcloud config list Your active ...
3 votes
1 answer
2k views

Setting up logging for buckets in Google Cloud storage

Trying to set up logging on a GCS bucket. every time i enter the command to set logging i get the error below. "Command exception: "logging" command does not support "file://" URLs. did you mean to ...
6 votes
1 answer
4k views

Slow synchronisation stage on gsutil rsync?

I've just started to use GCS as backup for my web servers. One server has 1.2 million JPEGS (3.5TB) and this all rsynced over flawlessly over 10 hours or so. The other has 2.5 million JPEGS (just ...
2 votes
1 answer
424 views

gsutil cp error: cannot have start index greater than total size

I'm trying to download a lot of images from a Google Bucket that my employer's web host (not so) helpfully set up. I run the command gsutil cp -r gs://bucket-link-info/images/ . and the download ...