2

It's been a about a month since I moved to GCE and I'm noticing that every now and then all my process or containers are down, volumes are unmounted and the systems registers a reboot recently.

Has anyone experience a problem with the Google Cloud Platform where compute instances reboot unexpectedly?

The restart happened last on Aug 16 22:25:27

The logs around the time of the restart indicate nothing, everything is normal then the machine starts booting again

Aug 16 20:22:36 dva kernel: [1612872.963240] init: google-clock-sync-manager main process (13004) terminated with status 1
Aug 16 20:22:36 dva kernel: [1612872.963258] init: google-clock-sync-manager main process ended, respawning
Aug 16 20:22:36 dva google-clock-sync: INFO Starting GCE clock sync
Aug 16 21:17:01 dva CRON[15754]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Aug 16 21:22:36 dva kernel: [1616473.015336] init: google-clock-sync-manager main process (14413) terminated with status 1
Aug 16 21:22:36 dva kernel: [1616473.015345] init: google-clock-sync-manager main process ended, respawning
Aug 16 21:22:37 dva google-clock-sync: INFO Starting GCE clock sync
Aug 16 22:17:01 dva CRON[17329]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Aug 16 22:25:27 dva rsyslogd: [origin software="rsyslogd" swVersion="7.4.4" x-pid="895" x-info="http://www.rsyslog.com"] start
Aug 16 22:25:27 dva rsyslogd-2307: warning: ~ action is deprecated, consider using the 'stop' statement instead [try http://www.rsyslog.com/e/2307 ]
Aug 16 22:25:27 dva rsyslogd: rsyslogd's groupid changed to 104
Aug 16 22:25:27 dva rsyslogd: rsyslogd's userid changed to 101
Aug 16 22:25:27 dva kernel: [    0.000000] Initializing cgroup subsys cpuset
Aug 16 22:25:27 dva kernel: [    0.000000] Initializing cgroup subsys cpu
Aug 16 22:25:27 dva kernel: [    0.000000] Initializing cgroup subsys cpuacct
Aug 16 22:25:27 dva kernel: [    0.000000] Linux version 3.19.0-66-generic (buildd@lgw01-40) (gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3) ) #74~14.04.1-Ubuntu SMP Tue Jul 19 19:56:11 UTC 2016 (Ub\
untu 3.19.0-66.74~14.04.1-generic 3.19.8-ckt22)
Aug 16 22:25:27 dva kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.19.0-66-generic root=UUID=5e5ef9d5-0969-4eaa-82ad-0234a67a2e9f ro console=ttyS0
Aug 16 22:25:27 dva kernel: [    0.000000] KERNEL supported cpus:
Aug 16 22:25:27 dva kernel: [    0.000000]   Intel GenuineIntel
Aug 16 22:25:27 dva kernel: [    0.000000]   AMD AuthenticAMD
Aug 16 22:25:27 dva kernel: [    0.000000]   Centaur CentaurHauls
Aug 16 22:25:27 dva kernel: [    0.000000] e820: BIOS-provided physical RAM map:
Aug 16 22:25:27 dva kernel: [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
Aug 16 22:25:27 dva kernel: [    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
  • Are there any relevant messages in your logs ? – user9517 Aug 18 '16 at 21:46
  • @lain nothing odd, I added a sample of my syslog, the machine suddently starts booting again – perrohunter Aug 18 '16 at 21:52
  • Are you still experiencing any issues? Did you have the chance to know what was causing it? If so, is it possible to post the answer so other people can benefit from it – George Nov 3 '16 at 20:33
1

It can be possible that some pods are consuming a lot of memory, so setting a memory limit can probably help in resolving this issue as mentioned in this Help Center article or sometimes increasing the instance resources can help as well. Another suggestion will be to monitor the node health which might help in future issues' debugging. The "kubectl describe node NODE-NAME" can give you some insight about the node's status and what might have caused it to reboot. Sometimes this can be caused by some maintenance events on the Google Cloud Platform which can be visible in the operations logs.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.