--Apple-Mail=_AD24D2E9-79CF-4789-A356-B91097F38271 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Were you running gluster under you shared storage? If so, you probably = need to setup ganesha nfs yourself. If not, check your ha-agent logs and make sure it=E2=80=99s mounting the = storage properly and check for errors. Good luck!
From: Jayme <jaymef@gmail.com> Subject: Re: [ovirt-users] Some major problems after 4.2 upgrade, = could really use some assistance Date: January 11, 2018 at 12:28:32 PM CST To: Martin Sivak; users@ovirt.org =20 This is becoming critical for me, does anyone have any ideas or = recommendations on what I can do to recover access to hosted VM? As of = right now I have three hosts that are fully updated, they have the 4.2 = repo and a full yum update was performed on them, there are no new = updates to apply. The hosted engine had updates as well as a full and = complete engine-setup, but did not return after being shut down. There = must be some way I can get the engine running again? Please =20 On Thu, Jan 11, 2018 at 8:24 AM, Jayme <jaymef@gmail.com = <mailto:jaymef@gmail.com>> wrote: The hosts have all ready been fully updated with 4.2 packages though. =20 ex.=20 =20 ovirt-host.x86_64 = 4.2.0-1.el7.centos @ovirt-4.2 ovirt-host-dependencies.x86_64 = 4.2.0-1.el7.centos @ovirt-4.2 ovirt-host-deploy.noarch = 1.7.0-1.el7.centos @ovirt-4.2 ovirt-hosted-engine-ha.noarch = 2.2.2-1.el7.centos @ovirt-4.2 ovirt-hosted-engine-setup.noarch = 2.2.3-1.el7.centos @ovirt-4.2 =20 On Thu, Jan 11, 2018 at 8:16 AM, Martin Sivak <msivak@redhat.com = <mailto:msivak@redhat.com>> wrote: Hi, =20 yes, you need to upgrade the hosts. Just take the ovirt-hosted-engine-ha and ovirt-hosted-engine-setup packages from ovirt 4.2 repositories. =20 Martin =20 On Thu, Jan 11, 2018 at 11:40 AM, Jayme <jaymef@gmail.com = <mailto:jaymef@gmail.com>> wrote:
How do I upgrade the hosted engine packages when I can't reach it or = do you mean upgrade the hosts if so how exactly do I do that. As for the = missing VM it appears that the disk image is there but it's missing XML file I = have no idea why or how to recreate it.
On Jan 11, 2018 4:43 AM, "Martin Sivak" <msivak@redhat.com = <mailto:msivak@redhat.com>> wrote:
Hi,
you hit one known issue we already have fixes for (4.1 hosts with =
4.2
engine): = https://gerrit.ovirt.org/#/q/status:open+project:ovirt-hosted-engine-ha+br= anch:v2.1.z+topic:ovf_42_for_41 = <https://gerrit.ovirt.org/#/q/status:open+project:ovirt-hosted-engine-ha+b= ranch:v2.1.z+topic:ovf_42_for_41>
You can try hotfixing it by upgrading hosted engine packages to 4.2 = or applying the patches manually and installing python-lxml.
I am not sure what happened to your other VM.
Best regards
Martin Sivak
On Thu, Jan 11, 2018 at 6:15 AM, Jayme <jaymef@gmail.com = <mailto:jaymef@gmail.com>> wrote:
I performed Ovirt 4.2 upgrade on a 3 host cluster with NFS shared storage. The shared storage is mounted from one of the hosts.
I upgraded the hosted engine first, downloading the 4.2 rpm, = doing a yum update then engine setup which seemed to complete successfully, = at the end it powered down the hosted VM but it never came back up. I was = unable to start it.
I proceeded to upgrade the three hosts, ovirt 4.2 rpm and a full = yum update. I also rebooted each of the three hosts.
After some time the hosts did come back and almost all of the VMs = are running again and seem to be working ok with the exception of = two:
1. The hosted VM still will not start, I've tried everything I = can think of.
2. A VM that I know existed is not running and does not appear to = exist, I have no idea where it is or how to start it.
1. Hosted engine
=46rom one of the hosts I get a weird error trying to start it:
# hosted-engine --vm-start Command VM.getStats with args {'vmID': '4013c829-c9d7-4b72-90d5-6fe58137504c'} failed: (code=3D1, message=3DVirtual machine does not exist: {'vmId': u'4013c829-c9d7-4b72-90d5-6fe58137504c'})
=46rom the two other hosts I do not get the same error as above, = sometimes it appears to start but --vm-status shows errors such as: Engine = status : {"reason": "failed liveliness check", "health": "bad", "vm": = "up", "detail": "Up"}
Seeing these errors in syslog:
Jan 11 01:06:30 host0 libvirtd: 2018-01-11 05:06:30.473+0000: = 1910: error : qemuOpenFileAs:3183 : Failed to open file
= '/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/c2dde892-f978-= 4dfc-a421-c8e04cf387f9/23aa0a66-fa6c-4967-a1e5-fbe47c0cd705': No such file or directory
Jan 11 01:06:30 host0 libvirtd: 2018-01-11 05:06:30.473+0000: = 1910: error : qemuDomainStorageOpenStat:11492 : cannot stat file
= '/var/run/vdsm/storage/248f46f0-d793-4581-9810-c9d965e2f286/c2dde892-f978-= 4dfc-a421-c8e04cf387f9/23aa0a66-fa6c-4967-a1e5-fbe47c0cd705': Bad file descriptor
2. Missing VM. virsh -r list on each host does not show the VM = at all. I know it existed and is important. The log on one of the hosts = even shows that it started it recently then stopped in 10 or so minutes = later:
Jan 10 18:47:17 host3 systemd-machined: New machine qemu-9-Berna. Jan 10 18:47:17 host3 systemd: Started Virtual Machine = qemu-9-Berna. Jan 10 18:47:17 host3 systemd: Starting Virtual Machine = qemu-9-Berna. Jan 10 18:54:45 host3 systemd-machined: Machine qemu-9-Berna = terminated.
How can I find out the status of the "Berna" VM and get it = running again?
Thanks so much!
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users = <http://lists.ovirt.org/mailman/listinfo/users>
=20 =20
Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--Apple-Mail=_AD24D2E9-79CF-4789-A356-B91097F38271 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" class=3D"">Were = you running gluster under you shared storage? If so, you probably need = to setup ganesha nfs yourself.<div class=3D""><br class=3D""></div><div = class=3D"">If not, check your ha-agent logs and make sure it=E2=80=99s = mounting the storage properly and check for errors. Good luck!</div><div = class=3D""><br class=3D""><div><blockquote type=3D"cite" class=3D""><hr = style=3D"border:none;border-top:solid #B5C4DF 1.0pt;padding:0 0 0 = 0;margin:10px 0 5px 0;" class=3D""><span style=3D"margin: -1.3px 0.0px = 0.0px 0.0px" id=3D"RwhHeaderAttributes" class=3D""><font = face=3D"Helvetica" size=3D"4" color=3D"#000000" style=3D"font: 13.0px = Helvetica; color: #000000" class=3D""><b class=3D"">From:</b> Jayme = <<a href=3D"mailto:jaymef@gmail.com" = class=3D"">jaymef@gmail.com</a>></font></span><br class=3D""> <span style=3D"margin: -1.3px 0.0px 0.0px 0.0px" class=3D""><font = face=3D"Helvetica" size=3D"4" color=3D"#000000" style=3D"font: 13.0px = Helvetica; color: #000000" class=3D""><b class=3D"">Subject:</b> Re: = [ovirt-users] Some major problems after 4.2 upgrade, could really use = some assistance</font></span><br class=3D""> <span style=3D"margin: -1.3px 0.0px 0.0px 0.0px" class=3D""><font = face=3D"Helvetica" size=3D"4" color=3D"#000000" style=3D"font: 13.0px = Helvetica; color: #000000" class=3D""><b class=3D"">Date:</b> January = 11, 2018 at 12:28:32 PM CST</font></span><br class=3D""> <span style=3D"margin: -1.3px 0.0px 0.0px 0.0px" class=3D""><font = face=3D"Helvetica" size=3D"4" color=3D"#000000" style=3D"font: 13.0px = Helvetica; color: #000000" class=3D""><b class=3D"">To:</b> Martin = Sivak; <a href=3D"mailto:users@ovirt.org" = class=3D"">users@ovirt.org</a></font></span><br class=3D""> <br class=3D"Apple-interchange-newline"><div class=3D""><div dir=3D"ltr" = class=3D"">This is becoming critical for me, does anyone have any ideas = or=20 recommendations on what I can do to recover access to hosted VM? = As of=20 right now I have three hosts that are fully updated, they have the 4.2=20= repo and a full yum update was performed on them, there are no new=20 updates to apply. The hosted engine had updates as well as a full = and=20 complete engine-setup, but did not return after being shut down. = There=20 must be some way I can get the engine running again? = Please</div><div class=3D"gmail_extra"><br class=3D""><div = class=3D"gmail_quote">On Thu, Jan 11, 2018 at 8:24 AM, Jayme <span = dir=3D"ltr" class=3D""><<a href=3D"mailto:jaymef@gmail.com" = target=3D"_blank" class=3D"">jaymef@gmail.com</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr" = class=3D"">The hosts have all ready been fully updated with 4.2 packages = though.<div class=3D""><br class=3D""></div><div = class=3D"">ex. </div><div class=3D""><br class=3D""></div><div = class=3D""><div class=3D"">ovirt-host.x86_64 = = = 4.2.0-1.el7.centos = = @ovirt-4.2</div><div = class=3D"">ovirt-host-dependencies.x86_<wbr class=3D"">64 = = = 4.2.0-1.el7.centos = = @ovirt-4.2</div><div class=3D"">ovirt-host-deploy.noarch = = = 1.7.0-1.el7.centos = = @ovirt-4.2</div><div class=3D"">ovirt-hosted-engine-ha.noarch = = = 2.2.2-1.el7.centos = = @ovirt-4.2</div><div class=3D"">ovirt-hosted-engine-setup.<wbr = class=3D"">noarch = = 2.2.3-1.el7.centos = = @ovirt-4.2</div></div></div><div class=3D"HOEnZb"><div = class=3D"h5"><div class=3D"gmail_extra"><br class=3D""><div = class=3D"gmail_quote">On Thu, Jan 11, 2018 at 8:16 AM, Martin Sivak = <span dir=3D"ltr" class=3D""><<a href=3D"mailto:msivak@redhat.com" = target=3D"_blank" class=3D"">msivak@redhat.com</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br class=3D""> <br class=3D""> yes, you need to upgrade the hosts. Just take the<br class=3D""> ovirt-hosted-engine-ha and ovirt-hosted-engine-setup packages from<br = class=3D""> ovirt 4.2 repositories.<br class=3D""> <span class=3D"m_6061824483285160243HOEnZb"><font color=3D"#888888" = class=3D""><br class=3D""> Martin<br class=3D""> </font></span><div class=3D"m_6061824483285160243HOEnZb"><div = class=3D"m_6061824483285160243h5"><br class=3D""> On Thu, Jan 11, 2018 at 11:40 AM, Jayme <<a = href=3D"mailto:jaymef@gmail.com" target=3D"_blank" = class=3D"">jaymef@gmail.com</a>> wrote:<br class=3D""> > How do I upgrade the hosted engine packages when I can't reach it = or do you<br class=3D""> > mean upgrade the hosts if so how exactly do I do that. As for the = missing VM<br class=3D""> > it appears that the disk image is there but it's missing XML file I = have no<br class=3D""> > idea why or how to recreate it.<br class=3D""> ><br class=3D""> > On Jan 11, 2018 4:43 AM, "Martin Sivak" <<a = href=3D"mailto:msivak@redhat.com" target=3D"_blank" = class=3D"">msivak@redhat.com</a>> wrote:<br class=3D""> >><br class=3D""> >> Hi,<br class=3D""> >><br class=3D""> >> you hit one known issue we already have fixes for (4.1 hosts = with 4.2<br class=3D""> >> engine):<br class=3D""> >> <a = href=3D"https://gerrit.ovirt.org/#/q/status:open+project:ovirt-hosted-engi= ne-ha+branch:v2.1.z+topic:ovf_42_for_41" rel=3D"noreferrer" = target=3D"_blank" class=3D"">https://gerrit.ovirt.org/#/q/s<wbr = class=3D"">tatus:open+project:ovirt-hoste<wbr = class=3D"">d-engine-ha+branch:v2.1.z+<wbr = class=3D"">topic:ovf_42_for_41</a><br class=3D""> >><br class=3D""> >> You can try hotfixing it by upgrading hosted engine packages to = 4.2 or<br class=3D""> >> applying the patches manually and installing python-lxml.<br = class=3D""> >><br class=3D""> >> I am not sure what happened to your other VM.<br class=3D""> >><br class=3D""> >> Best regards<br class=3D""> >><br class=3D""> >> Martin Sivak<br class=3D""> >><br class=3D""> >> On Thu, Jan 11, 2018 at 6:15 AM, Jayme <<a = href=3D"mailto:jaymef@gmail.com" target=3D"_blank" = class=3D"">jaymef@gmail.com</a>> wrote:<br class=3D""> >> > I performed Ovirt 4.2 upgrade on a 3 host cluster with NFS = shared<br class=3D""> >> > storage.<br class=3D""> >> > The shared storage is mounted from one of the hosts.<br = class=3D""> >> ><br class=3D""> >> > I upgraded the hosted engine first, downloading the 4.2 = rpm, doing a yum<br class=3D""> >> > update then engine setup which seemed to complete = successfully, at the<br class=3D""> >> > end<br class=3D""> >> > it powered down the hosted VM but it never came back = up. I was unable<br class=3D""> >> > to<br class=3D""> >> > start it.<br class=3D""> >> ><br class=3D""> >> > I proceeded to upgrade the three hosts, ovirt 4.2 rpm and = a full yum<br class=3D""> >> > update.<br class=3D""> >> > I also rebooted each of the three hosts.<br class=3D""> >> ><br class=3D""> >> > After some time the hosts did come back and almost all of = the VMs are<br class=3D""> >> > running again and seem to be working ok with the exception = of two:<br class=3D""> >> ><br class=3D""> >> > 1. The hosted VM still will not start, I've tried = everything I can think<br class=3D""> >> > of.<br class=3D""> >> ><br class=3D""> >> > 2. A VM that I know existed is not running and does not = appear to exist,<br class=3D""> >> > I<br class=3D""> >> > have no idea where it is or how to start it.<br class=3D""> >> ><br class=3D""> >> > 1. Hosted engine<br class=3D""> >> ><br class=3D""> >> > =46rom one of the hosts I get a weird error trying to = start it:<br class=3D""> >> ><br class=3D""> >> > # hosted-engine --vm-start<br class=3D""> >> > Command VM.getStats with args {'vmID':<br class=3D""> >> > '4013c829-c9d7-4b72-90d5-6fe58<wbr class=3D"">137504c'} = failed:<br class=3D""> >> > (code=3D1, message=3DVirtual machine does not exist: = {'vmId':<br class=3D""> >> > u'4013c829-c9d7-4b72-90d5-6fe5<wbr class=3D"">8137504c'})<br= class=3D""> >> ><br class=3D""> >> > =46rom the two other hosts I do not get the same error as = above, sometimes<br class=3D""> >> > it<br class=3D""> >> > appears to start but --vm-status shows errors such = as: Engine status<br class=3D""> >> > : {"reason": "failed liveliness check", "health": "bad", = "vm": "up",<br class=3D""> >> > "detail": "Up"}<br class=3D""> >> ><br class=3D""> >> > Seeing these errors in syslog:<br class=3D""> >> ><br class=3D""> >> > Jan 11 01:06:30 host0 libvirtd: 2018-01-11 = 05:06:30.473+0000: 1910:<br class=3D""> >> > error :<br class=3D""> >> > qemuOpenFileAs:3183 : Failed to open file<br class=3D""> >> ><br class=3D""> >> > '/var/run/vdsm/storage/248f46f<wbr = class=3D"">0-d793-4581-9810-c9d965e2f286/<wbr = class=3D"">c2dde892-f978-4dfc-a421-<wbr = class=3D"">c8e04cf387f9/23aa0a66-fa6c-<wbr = class=3D"">4967-a1e5-fbe47c0cd705':<br class=3D""> >> > No such file or directory<br class=3D""> >> ><br class=3D""> >> > Jan 11 01:06:30 host0 libvirtd: 2018-01-11 = 05:06:30.473+0000: 1910:<br class=3D""> >> > error :<br class=3D""> >> > qemuDomainStorageOpenStat:1149<wbr class=3D"">2 : cannot = stat file<br class=3D""> >> ><br class=3D""> >> > '/var/run/vdsm/storage/248f46f<wbr = class=3D"">0-d793-4581-9810-c9d965e2f286/<wbr = class=3D"">c2dde892-f978-4dfc-a421-<wbr = class=3D"">c8e04cf387f9/23aa0a66-fa6c-<wbr = class=3D"">4967-a1e5-fbe47c0cd705':<br class=3D""> >> > Bad file descriptor<br class=3D""> >> ><br class=3D""> >> > 2. Missing VM. virsh -r list on each host does not = show the VM at all.<br class=3D""> >> > I<br class=3D""> >> > know it existed and is important. The log on one of = the hosts even<br class=3D""> >> > shows<br class=3D""> >> > that it started it recently then stopped in 10 or so = minutes later:<br class=3D""> >> ><br class=3D""> >> > Jan 10 18:47:17 host3 systemd-machined: New machine = qemu-9-Berna.<br class=3D""> >> > Jan 10 18:47:17 host3 systemd: Started Virtual Machine = qemu-9-Berna.<br class=3D""> >> > Jan 10 18:47:17 host3 systemd: Starting Virtual Machine = qemu-9-Berna.<br class=3D""> >> > Jan 10 18:54:45 host3 systemd-machined: Machine = qemu-9-Berna terminated.<br class=3D""> >> ><br class=3D""> >> > How can I find out the status of the "Berna" VM and get it = running<br class=3D""> >> > again?<br class=3D""> >> ><br class=3D""> >> > Thanks so much!<br class=3D""> >> ><br class=3D""> >> > ______________________________<wbr = class=3D"">_________________<br class=3D""> >> > Users mailing list<br class=3D""> >> > <a href=3D"mailto:Users@ovirt.org" target=3D"_blank" = class=3D"">Users@ovirt.org</a><br class=3D""> >> > <a href=3D"http://lists.ovirt.org/mailman/listinfo/users" = rel=3D"noreferrer" target=3D"_blank" = class=3D"">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/users</a><br class=3D""> >> ><br class=3D""> </div></div></blockquote></div><br class=3D""></div> </div></div></blockquote></div><br class=3D""></div> _______________________________________________<br class=3D"">Users = mailing list<br class=3D""><a href=3D"mailto:Users@ovirt.org" = class=3D"">Users@ovirt.org</a><br = class=3D"">http://lists.ovirt.org/mailman/listinfo/users<br = class=3D""></div></blockquote></div><br class=3D""></div></body></html>= --Apple-Mail=_AD24D2E9-79CF-4789-A356-B91097F38271--