centosgoogle-compute-engineserver-administration

Google VM Instance Stopped Working after Upgrade

I have a google VM instance which was running flawlessly. It has Centos-7 and Plesk installed in it.

I just stopped it, upgraded machine type (to better CPU and RAM) and started it again. My server stopped responding at all. No websites are running, I can't connect to SSH & Google Cloud SDK Shell is unable to reach server. It says NETWORK ERROR, CONNECTION TIMED OUT. My all other instances works well.

I tried rebooting & resetting multiple times. Reading out stuff from internet since last 6 hours but for no luck. I also tried to clone disk of the instance and creating new instance with the cloned disk but for no luck. Same network connection issue. May be something in OS got corrupted? Please suggest. I have a number of websites hosted on the server which are down due to this. Thanks a lot in advance.

Solution

  • I took screenshot of my VM using Google Cloud Shell. It is as follows:

    VM Instance Screenshot

    I connected with serial console which is as follows:

    enter image description here

    While creating a ticket with google, I found that they've posted some information under "Known Issues". I am pasting the whole stuff as there is no direct link to reach there. The symptoms they told were exactly what was happening to me:

    Below is Known Issue Posted by Google:

    Description: We are experiencing an issue with Google Compute Engine instances running RHEL and CentOS 7 and 8. More details on this issue are available in the following article and bugs: https://access.redhat.com/solutions/5272311 https://bugzilla.redhat.com/show_bug.cgi?id=1861977 (RHEL 8) https://bugzilla.redhat.com/show_bug.cgi?id=1862045 (RHEL 7) Symptoms: Instances running RHEL and CentOS 7 and 8 that run yum update may fail to boot after restart with errors messages referring to a combination of: "X64 Exception Type - 0D(#GP - General Protection) CPU Apic ID", "FXSAVE_STATE" or "Find image based on IP". This issue affects instances with specific versions of the shim package installed. To find the currently installed shim version, use the following command: rpm -q shim-x64 Affected shim versions: CentOS 7: shim-x64-15-7.el7_9.x86_64 CentOS 8: shim-x64-15-13.el8.x86_64 RHEL 7: shim-x64-15-7.el7_8.x86_64 RHEL 8: shim-x64-15-14.el8_2.x86_64 Workaround: Do not update or reboot instances running RHEL or CentOS 7 and 8. If you are on an affected shim version, run yum downgrade shim\* grub2\* mokutil to downgrade to the correct version. This command may not work on CentOS 8. If you have already rebooted, you will need to attach the disk to another instance, chroot into the disk, then run the yum downgrade command. We will provide an update by Thursday, 2020-07-30 14:00 US/Pacific with current details.

    Start time: July 30, 2020 at 9:08:34 PM GMT+5

    How to diagnose: Instances running RHEL and CentOS 7 and 8 that run yum update may fail to boot after restart with errors messages referring to a combination of: "X64 Exception Type - 0D(#GP - General Protection) CPU Apic ID", "FXSAVE_STATE" or "Find image based on IP". This issue affects instances with specific versions of the shim package installed. To find the currently installed shim version, use the following command: rpm -q shim-x64 Affected shim versions: CentOS 7: shim-x64-15-7.el7_9.x86_64 CentOS 8: shim-x64-15-13.el8.x86_64 RHEL 7: shim-x64-15-7.el7_8.x86_64 RHEL 8: shim-x64-15-14.el8_2.x86_64

    Workaround: Do not update or reboot instances running RHEL or CentOS 7 and 8. If you are on an affected shim version, run yum downgrade shim\* grub2\* mokutil to downgrade to the correct version. This command may not work on CentOS 8. If you have already rebooted, you will need to attach the disk to another instance, chroot into the disk, then run the yum downgrade command.