Re: [ovirt-users] Some VMs in status "not responding" in oVirt interface

This is a multipart message in MIME format. ------=_NextPart_000_0662_01D0EB02.D258A0B0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hello, =20 unfortunately I still have this problem=E2=80=A6=20 Last week I checked all the hardware components. It=E2=80=99s a HP DL580 = Gen8 Server, 128GB RAM, 4TB storage. The firmware of all components is up to date. I ran a full check of all harddrives, CPUs etc., no problems detected. =20 This night 3 VMs stopped responding again, so I had to reboot the server = this morning to regain access. Some minutes ago 2 VMs stopped = responding=E2=80=A6 =20 The logs just show that the VMs aren=E2=80=99t responding anymore, = nothing else=E2=80=A6 does anybody have an idea how I can debug this = issue any further? =20 OS: CentOS Linux release 7.1.1503 =20 ovirt-iso-uploader-3.5.2-1.el7.centos.noarch ovirt-engine-setup-3.5.4.2-1.el7.centos.noarch ovirt-guest-tools-iso-3.5-7.noarch ovirt-log-collector-3.5.4-2.el7.centos.noarch ovirt-engine-userportal-3.5.4.2-1.el7.centos.noarch ovirt-engine-cli-3.5.0.6-1.el7.centos.noarch ovirt-engine-tools-3.5.4.2-1.el7.centos.noarch ovirt-release35-005-1.noarch ovirt-engine-lib-3.5.4.2-1.el7.centos.noarch ovirt-engine-setup-plugin-ovirt-engine-common-3.5.4.2-1.el7.centos.noarch= ovirt-host-deploy-java-1.3.2-1.el7.centos.noarch ovirt-engine-extensions-api-impl-3.5.4.2-1.el7.centos.noarch ovirt-engine-webadmin-portal-3.5.4.2-1.el7.centos.noarch ovirt-engine-restapi-3.5.4.2-1.el7.centos.noarch ovirt-engine-setup-base-3.5.4.2-1.el7.centos.noarch ovirt-engine-backend-3.5.4.2-1.el7.centos.noarch ovirt-engine-setup-plugin-websocket-proxy-3.5.4.2-1.el7.centos.noarch ovirt-host-deploy-1.3.2-1.el7.centos.noarch ovirt-engine-websocket-proxy-3.5.4.2-1.el7.centos.noarch ovirt-engine-dbscripts-3.5.4.2-1.el7.centos.noarch ovirt-engine-jboss-as-7.1.1-1.el7.x86_64 ovirt-engine-sdk-python-3.5.4.0-1.el7.centos.noarch ovirt-engine-setup-plugin-ovirt-engine-3.5.4.2-1.el7.centos.noarch ovirt-image-uploader-3.5.1-1.el7.centos.noarch ovirt-engine-3.5.4.2-1.el7.centos.noarch =20 vdsm-python-4.16.26-0.el7.centos.noarch vdsm-jsonrpc-java-1.0.15-1.el7.noarch vdsm-jsonrpc-4.16.26-0.el7.centos.noarch vdsm-yajsonrpc-4.16.26-0.el7.centos.noarch vdsm-xmlrpc-4.16.26-0.el7.centos.noarch vdsm-cli-4.16.26-0.el7.centos.noarch vdsm-4.16.26-0.el7.centos.x86_64 vdsm-python-zombiereaper-4.16.26-0.el7.centos.noarch =20 qemu-kvm-ev-2.1.2-23.el7_1.8.1.x86_64 qemu-kvm-common-ev-2.1.2-23.el7_1.8.1.x86_64 libvirt-daemon-kvm-1.2.8-16.el7_1.3.x86_64 qemu-kvm-tools-ev-2.1.2-23.el7_1.8.1.x86_64 =20 Linux ovirt 3.10.0-229.11.1.el7.x86_64 #1 SMP Thu Aug 6 01:06:18 UTC = 2015 x86_64 x86_64 x86_64 GNU/Linux =20 Any feedback is much appreciated!! =20 Best regards, Christian =20 Von: users-bounces(a)ovirt.org [mailto:users-bounces@ovirt.org] Im Auftrag = von Christian Hailer Gesendet: Samstag, 29. August 2015 22:48 An: users(a)ovirt.org Betreff: [ovirt-users] Some VMs in status "not responding" in oVirt = interface =20 Hello, =20 last Wednesday I wanted to update my oVirt 3.5 hypervisor. It is a = single Centos 7 server, so I started by suspending the VMs in order to = set the oVirt engine host to maintenance mode. During the process of = suspending the VMs the server crashed, kernel panic=E2=80=A6 After restarting the server I installed the updates via yum an restarted = the server again. Afterwards, all the VMs could be started again. Some = hours later my monitoring system registered some unresponsive hosts, I = had a look in the oVirt interface, 3 of the VMs were in the state = =E2=80=9Cnot responding=E2=80=9D, marked by a question mark.=20 I tried to shut down the VMs, but oVirt wasn=E2=80=99t able to do so. I = tried to reset the status in the database with the sql statement =20 update vm_dynamic set status =3D 0 where vm_guid =3D (select vm_guid = from vm_static where vm_name =3D 'MYVMNAME'); =20 but that didn=E2=80=99t help, either. Only rebooting the whole = hypervisor helped=E2=80=A6 afterwards everything worked again. But only = for a few hours, then one of the VMs entered the =E2=80=9Cnot = responding=E2=80=9D state again=E2=80=A6 again only a reboot helped. = Yesterday it happened again: =20 2015-08-28 17:44:22,664 INFO = [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] = (DefaultQuartzScheduler_Worker-60) [4ef90b12] VM DC = 0f3d1f06-e516-48ce-aa6f-7273c33d3491 moved from Up --> NotResponding 2015-08-28 17:44:22,692 WARN = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (DefaultQuartzScheduler_Worker-60) [4ef90b12] Correlation ID: null, Call = Stack: null, Custom Event ID: -1, Message: VM DC is not responding. =20 Does anybody know what I can do? Where should I have a look? Hints are = greatly appreciated! =20 Thanks, Christian ------=_NextPart_000_0662_01D0EB02.D258A0B0 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" = xmlns:o=3D"urn:schemas-microsoft-com:office:office" = xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" = xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta = http-equiv=3DContent-Type content=3D"text/html; charset=3Dutf-8"><meta = name=3DGenerator content=3D"Microsoft Word 15 (filtered = medium)"><style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0cm; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri",sans-serif; mso-fareast-language:EN-US;} a:link, span.MsoHyperlink {mso-style-priority:99; color:#0563C1; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:#954F72; text-decoration:underline;} span.E-MailFormatvorlage17 {mso-style-type:personal; font-family:"Calibri",sans-serif; color:windowtext;} span.E-MailFormatvorlage18 {mso-style-type:personal; font-family:"Calibri",sans-serif; color:#1F497D;} span.E-MailFormatvorlage19 {mso-style-type:personal-compose; font-family:"Calibri",sans-serif; color:windowtext;} .MsoChpDefault {mso-style-type:export-only; font-size:10.0pt;} @page WordSection1 {size:612.0pt 792.0pt; margin:70.85pt 70.85pt 2.0cm 70.85pt;} div.WordSection1 {page:WordSection1;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--></head><body lang=3DDE = link=3D"#0563C1" vlink=3D"#954F72"><div class=3DWordSection1><p = class=3DMsoNormal><span lang=3DEN-US = style=3D'color:#1F497D'>Hello,<o:p></o:p></span></p><p = class=3DMsoNormal><span lang=3DEN-US = style=3D'color:#1F497D'><o:p>&nbsp;</o:p></span></p><p = class=3DMsoNormal><span lang=3DEN-US = style=3D'color:#1F497D'>unfortunately I still have this problem=E2=80=A6 = <o:pstyle=3D'color:#1F497D'>Last week I checked all the hardware components. = It=E2=80=99s a HP DL580 Gen8 Server, 128GB RAM, 4TB = storage.<o:pstyle=3D'color:#1F497D'>The firmware of all components is up to = date.<o:pstyle=3D'color:#1F497D'>I ran a full check of all harddrives, CPUs etc., = no problems detected.<o:p></o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US style=3D'color:#1F497D'><o:p>&nbsp;</o:p></span></p><p = class=3DMsoNormal><span lang=3DEN-US style=3D'color:#1F497D'>This night = 3 VMs stopped responding again, so I had to reboot the server this = morning to regain access. Some minutes ago 2 VMs stopped = responding=E2=80=A6<o:p></o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US style=3D'color:#1F497D'><o:p>&nbsp;</o:p></span></p><p = class=3DMsoNormal><span lang=3DEN-US style=3D'color:#1F497D'>The logs = just show that the VMs aren=E2=80=99t responding anymore, nothing = else=E2=80=A6 does anybody have an idea how I can debug this issue any = further?<o:pstyle=3D'color:#1F497D'><o:p>&nbsp;</o:p></span></p><p = class=3DMsoNormal><span lang=3DEN-US style=3D'color:#1F497D'>OS: CentOS = Linux release 7.1.1503<o:p></o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US style=3D'color:#1F497D'><o:p>&nbsp;</o:p></span></p><p = class=3DMsoNormal><span lang=3DEN-US style=3D'color:#1F497D'>&gt;rpm = -qa|grep ovirt<o:p></o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US = style=3D'color:#1F497D'>ovirt-iso-uploader-3.5.2-1.el7.centos.noarch<o:p>= </o:p></span></pstyle=3D'color:#1F497D'>ovirt-engine-setup-3.5.4.2-1.el7.centos.noarch<o:= pstyle=3D'color:#1F497D'>ovirt-guest-tools-iso-3.5-7.noarch<o:p></o:p></sp= an></pstyle=3D'color:#1F497D'>ovirt-log-collector-3.5.4-2.el7.centos.noarch<o:p= style=3D'color:#1F497D'>ovirt-engine-userportal-3.5.4.2-1.el7.centos.noar= ch<o:pstyle=3D'color:#1F497D'>ovirt-engine-cli-3.5.0.6-1.el7.centos.noarch<o:p>= </o:p></span></pstyle=3D'color:#1F497D'>ovirt-engine-tools-3.5.4.2-1.el7.centos.noarch<o:= pstyle=3D'color:#1F497D'>ovirt-release35-005-1.noarch<o:p></o:p></span></p= style=3D'color:#1F497D'>ovirt-engine-lib-3.5.4.2-1.el7.centos.noarch<o:p>= </o:p></span></pstyle=3D'color:#1F497D'>ovirt-engine-setup-plugin-ovirt-engine-common-3.5= .4.2-1.el7.centos.noarch<o:p></o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US = style=3D'color:#1F497D'>ovirt-host-deploy-java-1.3.2-1.el7.centos.noarch<= o:pstyle=3D'color:#1F497D'>ovirt-engine-extensions-api-impl-3.5.4.2-1.el7.ce= ntos.noarch<o:pstyle=3D'color:#1F497D'>ovirt-engine-webadmin-portal-3.5.4.2-1.el7.centos= .noarch<o:pstyle=3D'color:#1F497D'>ovirt-engine-restapi-3.5.4.2-1.el7.centos.noarch<= o:pstyle=3D'color:#1F497D'>ovirt-engine-setup-base-3.5.4.2-1.el7.centos.noar= ch<o:pstyle=3D'color:#1F497D'>ovirt-engine-backend-3.5.4.2-1.el7.centos.noarch<= o:pstyle=3D'color:#1F497D'>ovirt-engine-setup-plugin-websocket-proxy-3.5.4.2= -1.el7.centos.noarch<o:p></o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US = style=3D'color:#1F497D'>ovirt-host-deploy-1.3.2-1.el7.centos.noarch<o:p><= /o:p></span></pstyle=3D'color:#1F497D'>ovirt-engine-websocket-proxy-3.5.4.2-1.el7.centos= .noarch<o:pstyle=3D'color:#1F497D'>ovirt-engine-dbscripts-3.5.4.2-1.el7.centos.noarc= h<o:pstyle=3D'color:#1F497D'>ovirt-engine-jboss-as-7.1.1-1.el7.x86_64<o:p></o:= p></span></pstyle=3D'color:#1F497D'>ovirt-engine-sdk-python-3.5.4.0-1.el7.centos.noar= ch<o:pstyle=3D'color:#1F497D'>ovirt-engine-setup-plugin-ovirt-engine-3.5.4.2-1.= el7.centos.noarch<o:p></o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US = style=3D'color:#1F497D'>ovirt-image-uploader-3.5.1-1.el7.centos.noarch<o:= pstyle=3D'color:#1F497D'>ovirt-engine-3.5.4.2-1.el7.centos.noarch<o:p></o:= p></span></pstyle=3D'color:#1F497D'><o:p>&nbsp;</o:p></span></p><p = class=3DMsoNormal><span lang=3DEN-US style=3D'color:#1F497D'>&gt;rpm = -qa|grep vdsm<o:p></o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US = style=3D'color:#1F497D'>vdsm-python-4.16.26-0.el7.centos.noarch<o:p></o:p= ></span></pstyle=3D'color:#1F497D'>vdsm-jsonrpc-java-1.0.15-1.el7.noarch<o:p></o:p><= /span></pstyle=3D'color:#1F497D'>vdsm-jsonrpc-4.16.26-0.el7.centos.noarch<o:p></o:= p></span></pstyle=3D'color:#1F497D'>vdsm-yajsonrpc-4.16.26-0.el7.centos.noarch<o:p></= o:p></span></pstyle=3D'color:#1F497D'>vdsm-xmlrpc-4.16.26-0.el7.centos.noarch<o:p></o:p= ></span></pstyle=3D'color:#1F497D'>vdsm-cli-4.16.26-0.el7.centos.noarch<o:p></o:p></= span></pstyle=3D'color:#1F497D'>vdsm-4.16.26-0.el7.centos.x86_64<o:p></o:p></span= ></pstyle=3D'color:#1F497D'>vdsm-python-zombiereaper-4.16.26-0.el7.centos.noa= rch<o:pstyle=3D'color:#1F497D'><o:p>&nbsp;</o:p></span></p><p = class=3DMsoNormal><span lang=3DEN-US style=3D'color:#1F497D'>&gt;rpm = -qa|grep kvm<o:p></o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US = style=3D'color:#1F497D'>qemu-kvm-ev-2.1.2-23.el7_1.8.1.x86_64<o:p></o:p><= /span></pstyle=3D'color:#1F497D'>qemu-kvm-common-ev-2.1.2-23.el7_1.8.1.x86_64<o:p>= </o:p></span></pstyle=3D'color:#1F497D'>libvirt-daemon-kvm-1.2.8-16.el7_1.3.x86_64<o:p></= o:p></span></pstyle=3D'color:#1F497D'>qemu-kvm-tools-ev-2.1.2-23.el7_1.8.1.x86_64<o:p><= /o:p></span></pstyle=3D'color:#1F497D'><o:p>&nbsp;</o:p></span></p><p = class=3DMsoNormal><span lang=3DEN-US style=3D'color:#1F497D'>&gt;uname = -a <o:pstyle=3D'color:#1F497D'>Linux ovirt 3.10.0-229.11.1.el7.x86_64 #1 SMP = Thu Aug 6 01:06:18 UTC 2015 x86_64 x86_64 x86_64 = GNU/Linux<o:pstyle=3D'color:#1F497D'><o:p>&nbsp;</o:p></span></p><p = class=3DMsoNormal><span lang=3DEN-US style=3D'color:#1F497D'>Any = feedback is much appreciated!!<o:p></o:p></span></p><p = class=3DMsoNormal><span lang=3DEN-US = style=3D'color:#1F497D'><o:p>&nbsp;</o:p></span></p><p = class=3DMsoNormal><span lang=3DEN-US style=3D'color:#1F497D'>Best = regards, Christian<o:p></o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US = style=3D'color:#1F497D'><o:p>&nbsp;</o:p></span></p><div><div = style=3D'border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm = 0cm 0cm'><p class=3DMsoNormal><b><span = style=3D'mso-fareast-language:DE'>Von:</span></b><span = style=3D'mso-fareast-language:DE'> users-bounces(a)ovirt.org = [mailto:users-bounces@ovirt.org] <b>Im Auftrag von </b>Christian = Hailer<br><b>Gesendet:</b> Samstag, 29. August 2015 22:48<br><b>An:</b> = users@ovirt.org<br><b>Betreff:</b> [ovirt-users] Some VMs in status = &quot;not responding&quot; in oVirt = interface<o:p></o:p></span></p></div></div><p = class=3DMsoNormal><o:p>&nbsp;</o:p></p><p class=3DMsoNormal><span = lang=3DEN-US>Hello,<o:p></o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US><o:p>&nbsp;</o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US>last Wednesday I wanted to update my oVirt 3.5 hypervisor. = It is a single Centos 7 server, so I started by suspending the VMs in = order to set the oVirt engine host to maintenance mode. During the = process of suspending the VMs the server crashed, kernel = panic=E2=80=A6<o:p></o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US>After restarting the server I installed the updates via yum = an restarted the server again. Afterwards, all the VMs could be started = again. Some hours later my monitoring system registered some = unresponsive hosts, I had a look in the oVirt interface, 3 of the VMs = were in the state =E2=80=9Cnot responding=E2=80=9D, marked by a question = mark. <o:p></o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US>I = tried to shut down the VMs, but oVirt wasn=E2=80=99t able to do so. I = tried to reset the status in the database with the sql = statement<o:p></o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US><o:p>&nbsp;</o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US>update vm_dynamic set status =3D 0 where vm_guid =3D = (select vm_guid from vm_static where vm_name =3D = 'MYVMNAME');<o:p></o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US><o:p>&nbsp;</o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US>but that didn=E2=80=99t help, either. Only rebooting the = whole hypervisor helped=E2=80=A6 afterwards everything worked again. But = only for a few hours, then one of the VMs entered the =E2=80=9Cnot = responding=E2=80=9D state again=E2=80=A6 again only a reboot helped. = Yesterday it happened again:<o:p></o:p></span></p><p = class=3DMsoNormal><span lang=3DEN-US><o:p>&nbsp;</o:p></span></p><p = class=3DMsoNormal><span lang=3DEN-US>2015-08-28 17:44:22,664 INFO&nbsp; = [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] = (DefaultQuartzScheduler_Worker-60) [4ef90b12] VM DC = 0f3d1f06-e516-48ce-aa6f-7273c33d3491 moved from Up --&gt; = NotResponding<o:p></o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US>2015-08-28 17:44:22,692 WARN&nbsp; = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (DefaultQuartzScheduler_Worker-60) [4ef90b12] Correlation ID: null, Call = Stack: null, Custom Event ID: -1, Message: VM DC is not = responding.<o:p></o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US><o:p>&nbsp;</o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US>Does anybody know what I can do? Where should I have a = look? Hints are greatly appreciated!<o:p></o:p></span></p><p = class=3DMsoNormal><span lang=3DEN-US><o:p>&nbsp;</o:p></span></p><p = class=3DMsoNormal><span lang=3DEN-US>Thanks,<o:p></o:p></span></p><p = class=3DMsoNormal><span = lang=3DEN-US>Christian<o:p></o:p></span></p></div></body></html> ------=_NextPart_000_0662_01D0EB02.D258A0B0--