1

On Google Compute Engine, I have a startup-script that uses gsutil and gcloud, which are part of the google-cloud-sdk snap that is preinstalled on the ubuntu-minimal-1804-lts public image, which is the base image for the image that I am using. But when I ran the startup script today, it failed 4 out of 8 times because snapd decided to auto-refresh at the same time that the startup-script is run.

Within the startup-script, how can I wait until snap’s auto-refresh has completed?

Here are the commands I ran to determine that snapd was updating at the same time as my startup-script:

$ snap changes --abs-time
ID   Status  Spawn                 Ready                 Summary
3    Done    2019-05-16T18:08:34Z  2019-05-16T18:08:49Z  Auto-refresh snaps "core", "google-cloud-sdk"

$ snap tasks 3 --abs-time
Status  Spawn                 Ready                 Summary
…
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:45Z  Ensure prerequisites for "google-cloud-sdk" are available
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:46Z  Download snap "google-cloud-sdk" (82) from channel "stable/ubuntu-18.04"
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:47Z  Fetch and check assertions for snap "google-cloud-sdk" (82)
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:48Z  Mount snap "google-cloud-sdk" (82)
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:48Z  Run pre-refresh hook of "google-cloud-sdk" snap if present
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:48Z  Stop snap "google-cloud-sdk" services
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:48Z  Remove aliases for snap "google-cloud-sdk"
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:48Z  Make current revision for snap "google-cloud-sdk" unavailable
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:49Z  Copy snap "google-cloud-sdk" data
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:49Z  Setup snap "google-cloud-sdk" (82) security profiles
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:49Z  Make snap "google-cloud-sdk" (82) available to the system
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:49Z  Automatically connect eligible plugs and slots of snap "google-cloud-sdk"
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:49Z  Set automatic aliases for snap "google-cloud-sdk"
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:49Z  Setup snap "google-cloud-sdk" aliases
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:49Z  Run post-refresh hook of "google-cloud-sdk" snap if present
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:49Z  Start snap "google-cloud-sdk" (82) services
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:49Z  Clean up "google-cloud-sdk" (82) install
Done    2019-05-16T18:08:34Z  2019-05-16T18:08:49Z  Run configure hook of "google-cloud-sdk" snap if present

$ journalctl -u google-startup-scripts
…
May 16 18:08:39 my-instance startup-script[1469]: INFO startup-script-url: + gsutil cp gs://my-bucket/app.tar.gz /opt/my-bucket/app.tar.gz
May 16 18:08:41 my-instance startup-script[1469]: INFO startup-script-url: /snap/google-cloud-sdk/41/usr/bin/python2: relocation error: /lib/x86_64-linux-gnu/libnss_dns.so.2: symbol 
__resolv_context_get, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

On another instance, gsutil failed with a different message:

ERROR: (gsutil) /snap/google-cloud-sdk/41/usr/bin/python2: command not found
1

I have been trying to reproduce your issue, by creating an instance with the 'ubuntu-minimal-1804-lts' image as you state

gcloud compute instances create \
--image-family ubuntu-minimal-1804-lts \
--image-project ubuntu-os-cloud \
--machine-type n1-standard-1 \
--boot-disk-size=15 \
--boot-disk-type=pd-ssd \
--zone=europe-west1-c \
--tags=allow-incoming-ssh \
--metadata=startup-script-url="gs://testinggrounds/startup.sh" \
ubuntu-test

and then reboot it several times

for i in {1..7}; \
do \
gcloud compute instances reset ubuntu-test --zone europe-west1-c; \
sleep 120s; \
done

but I did not manage to get this problem that you state.

Searching around for the error that you get, I found this thread where the user reporting the issue presents a highly similar problem which might be the same as you have found. The problem is supposed to be fixed, as reported by the team behind it. Can you check if the date for your Ubuntu 18.04 image is prior to the date opening of this thread I shared? Maybe by using a newer image your problem gets solved.

I saw that you said this suddenly appeared not too long ago, so probably it is not the version, but it does not hurt to check.

If this was not the case, then I would propose to just delay the execution of the startup script or try to make it run after other snapd services.

[Unit]
Description=Google Compute Engine Startup Scripts
After=local-fs.target network-online.target network.target rsyslog.service
After=google-instance-setup.service google-network-daemon.service
After=cloud-final.service multi-user.target
After=snapd.seeded.service
After=snapd.service
Wants=local-fs.target network-online.target network.target cloud-final.service
Wants=snapd.seeded.service

[Service]
ExecStartPre=/bin/sleep 20
ExecStart=/usr/bin/google_metadata_script_runner --script-type startup
KillMode=process
Type=oneshot
Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin

[Install]
WantedBy=multi-user.target

This file is located at /lib/systemd/system/google-startup-scripts.service. I added the After=snapd.service and ExecStartPre=/bin/sleep 20 to try and delay the execution of the service. Your file should look exactly like the one above without my additions.

If it still doesn't work, you can try to add other snap services with the After= directive, as there are a few more in the systemd services folder:

$ ls /lib/systemd/system/ | grep snap

snapd.autoimport.service
snapd.core-fixup.service
snapd.failure.service
snapd.seeded.service
snapd.service
snapd.snap-repair.service
snapd.snap-repair.timer
snapd.socket
snapd.system-shutdown.service

Hope it helps!

1
  • Thanks for all the research. In your linked issue, zmarano claimed that the issue is fixed, but I was already running an image that was the newest at the time he wrote that comment (--image-project ubuntu-os-cloud --image ubuntu-minimal-1804-bionic-v20180705). But today I am unable to reproduce the issue unfortunately (snap changes shows everything is Done already within the startup-script).
    – yonran
    May 29 '19 at 19:25
1

I ended up running snap changes until everything was Done by putting this at the top of my startup-script:

# Returns 0 if snapd is not updating google-cloud-sdk
# (nothing has happened, or all changes are Done, Hold, Error)
# Returns 1 if snapd is updating google-cloud-sdk
# (there is at least one chnage that is Doing, Abort, Undo, or Undoing)
# Echos the offending line
# https://serverfault.com/questions/967674/how-to-wait-for-snapd-auto-refresh-to-complete
function isSnapIdle {
  # For the format of snap changes, see:
  # see https://github.com/snapcore/snapd/blob/8ae45c207f3c488f0ab0dc3615661df9b0854b20/overlord/state/change.go
  # and https://github.com/snapcore/snapd/blob/8ae45c207f3c488f0ab0dc3615661df9b0854b20/cmd/snap/cmd_changes.go#L124
  local line
  local changes
  if changes=$(snap changes google-cloud-sdk); then
    :
  else
    # When snap is upgrading itself, snap changes can fail
    echo "snap changes gave error $? $changes"
    return 2
  fi
  while read line; do
    local state=$(echo "$line" | awk '{print $2}')
    if [ "$state" = Do ] || [ "$state" = Doing ] || [ "$state" = Abort ] || [ "$state" = Undo ] || [ "$state" = Undoing ]; then
      echo "$line"
      return 1
    fi
  done < <(echo "$changes" | tail -n +2)
  return 0
}
function waitUntilSnapIdle {
  local lastSnapChange=
  local snapChange
  while ! snapChange=$(isSnapIdle); do
    if [ "$snapChange" != "$lastSnapChange" ]; then
      echo >&2 "Waiting for snapd to finish installing: $snapChange"
    fi
    lastSnapChange="$snapChange"
    sleep 1
  done
  echo >&2 "Snapd does not appear to be installing google-cloud-sdk"
}

waitUntilSnapIdle
1

I was encountering the same issue and pin pointed the cause to snap's auto updates as well! In my case waiting for snap refresh takes way too much time and the number of jobs is still manageable for a single human so I just reduced the allowed refresh window to the calmer part of the work week.

Try for instance:

# Allows snap refreshes on Monday at 10AM and also on Friday at 3PM
sudo snap set system refresh.timer=mon,10:00,,fri,15:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.