1

We are performing backups for our Google Datastore documents through a cron as describere here. The database is about 55GB including indices and we do a full backup every 24 hours. The bucket we save into belongs to another project.

The backups succeed and we are able to restore them into our staging environment without problems!

Nonetheless, we see a lot of errors coming up while the backup is running. The errors are coming from the ah-builtin-python-bundle module and the error message is Request was aborted after waiting too long to attempt to service your request. This causes our monitoring to alert us (We have a threshold of 1,5 errors per second over a time of 3 minutes).

Obviously, we'd like to get rid of those errors to not have false positive alerts in our monitoring.

2 Answers 2

2

The backup kicks off a MapReduce job which fans work out into push tasks. It's normal for one of these tasks to occasionally time out while waiting in the pending queue and be retried, which would result in a "Request was aborted after waiting too long to attempt to service your request" error in the logs.

If seeing these errors is a problem, you can try to eliminate them by adjusting your queue or scaling parameters.

0

I know this post is somehow old, but I will leave some interesting documentation here which you might find helpful regarding the new suggested approach to schedule exports in Cloud Datastore using App Engine. I guess your issues are not still reproducible, so I would suggest anyone that comes to this post to better go with this new Cloud Platform feature.

Make sure to refer to the previous link for a detailed description (and updated, because as of writing, this feature is in beta and might change), but below I will leave a summary of the main steps to schedule exports from Cloud Datastore using chron and App Engine:

  1. Create a Cloud Storage bucket in the same location as Datastore.
  2. Assign the Cloud Datastore Import Export Admin role to your App Engine default service account (which should look like [email protected]).
  3. Assign write permissions to the bucket to the App Engine default service account.
  4. Create a new local folder with the files app.yaml (App Engine configuration), cloud_datastore_admin.py (App Engine application itself) and cron.yaml (cron job), and populate them with the content in the link I shared at the very beginning, making sure that you replace any required content in order to match the naming of your resources. Also edit the cron configuration file to your wish in order to set up an appropriate exporting policy as per your requirements.
  5. Once you have set up the full configuration for your application, deploy it (including the cron job), test it and see your exports in the Cloud Storage bucket.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .