Re: [ovirt-users] hosted-engine VM and services not working
--------=_MB3BBF1E10-B89F-4246-8ACD-AB3F4F14DC3C Content-Type: text/plain; format=flowed; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Yaniv I found a solution. Our Ovirt 3.6 AIO box was still running and had those VMs still=20 configured in their pre exported and switch off state. I removed any snap shots I found from those pre exported VMs, then=20 copied the disk image files and other bits from host01 (Ovirt v 4.1)=20 back into the Ovirt 3.6 AIO box, and were needed fixing the relevent IDs=20 to be what the Engine in the Ovirt 3.6 box expected. The VMs then started up properly again without hassle and with the=20 latest files on the Ovirt 3.6 AIO box. So now in the progress of rebuilding host01 with hosted-engine v4.1 Kind regards Andrew ------ Original Message ------ From: "Yaniv Kaul" <ykaul@redhat.com> To: "Andrew Dent" <adent@ctcroydon.com.au> Cc: "users" <users@ovirt.org> Sent: 18/06/2017 6:00:09 PM Subject: Re: [ovirt-users] hosted-engine VM and services not working
On Sat, Jun 17, 2017 at 12:50 AM, <adent@ctcroydon.com.au> wrote:
If I reinstall and the rerun the hosted-engine setup how do I get the=20 VMs in their current running state back into and being recognised by=20 the new hosted engine?
Current running state is again quite challenging. You'll need to fix=20 the hosted-engine.
Can import the storage domain? (not for running VMs) Y.
Kind regards
Andrew
On 17 Jun 2017, at 6:54 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Fri, Jun 16, 2017 at 9:11 AM, Andrew Dent <adent@ctcroydon.com.au>=20 wrote:
Hi
Well I've got myself into a fine mess.
host01 was setup with hosted-engine v4.1. This was successful. Imported 3 VMs from a v3.6 OVirt AIO instance. (This OVirt 3.6 is=20 still running with more VMs on it) Tried to add host02 to the new Ovirt 4.1 setup. This partially=20 succeeded but I couldn't add any storage domains to it. Cannot=20 remember why. In Ovirt engine UI I removed host02. I reinstalled host02 with Centos7, tried to add it and Ovirt UI told=20 me it was already there (but it wasn't listed in the UI). Renamed the reinstalled host02 to host03, changed the ipaddress,=20 reconfig the DNS server and added host03 into the Ovirt Engine UI. All good, and I was able to import more VMs to it. I was also able to shutdown a VM on host01 assign it to host03 and=20 start the VM. Cool, everything working. The above was all last couple of weeks.
This week I performed some yum updates on the Engine VM. No reboot. Today noticed that the Ovirt services in the Engine VM were in a=20 endless restart loop. They would be up for a 5 minutes and then die. Looking into /var/log/ovirt-engine/engine.log and I could only see=20 errors relating to host02. Ovirt was trying to find it and failing.=20 Then falling over. I ran "hosted-engine --clean-metadata" thinking it would cleanup and=20 remove bad references to hosts, but now realise that was a really=20 bad idea as it didn't do what I'd hoped. At this point the sequence below worked, I could login to Ovirt UI=20 but after 5 minutes the services would be off service ovirt-engine restart service ovirt-websocket-proxy restart service httpd restart
I saw some reference to having to remove hosts from the database by=20 hand in situations where under the hood of Ovirt a decommission host=20 was still listed, but wasn't showing in the GUI. So I removed reference to host02 (vds_id and host_id) in the=20 following tables in this order. vds_dynamic vds_statistics vds_static host_device
Now when I try to start ovirt-websocket it will not start service ovirt-websocket start Redirecting to /bin/systemctl start ovirt-websocket.service Failed to start ovirt-websocket.service: Unit not found.
I'm now thinking that I need to do the following in the engine VM # engine-cleanup # yum remove ovirt-engine # yum install=20 ovirt-engine # engine-setup But to run engine-cleanup I need to put the engine-vm into=20 maintenance mode and because of the --clean-metadata that I ran=20 earlier on host01 I cannot do that.
What is the best course of action from here?
To be honest, with all the steps taken above, I'd install everything=20 (including OS) from scratch... There's a bit too much mess to try to clean up properly here. Y.
Cheers
Andrew
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users=20 <http://lists.ovirt.org/mailman/listinfo/users>
</div></div></div> <div>------ Original Message ------</div> <div>From: "Yaniv Kaul" <<a href=3D"mailto:ykaul@redhat.com">ykaul@redha= t.com</a>></div> <div>To: "Andrew Dent" <<a href=3D"mailto:adent@ctcroydon.com.au">adent@= ctcroydon.com.au</a>></div> <div>Cc: "users" <<a href=3D"mailto:users@ovirt.org">users@ovirt.org</a>= ></div> <div>Sent: 18/06/2017 6:00:09 PM</div> <div>Subject: Re: [ovirt-users] hosted-engine VM and services not working</=
--------=_MB3BBF1E10-B89F-4246-8ACD-AB3F4F14DC3C Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <?xml version=3D"1.0" encoding=3D"utf-16"?><html><head><style id=3D"signatu= reStyle" type=3D"text/css"><!--#xa78294227b5f49c, #xa78294227b5f49c, #xa782= 94227b5f49c, #xa78294227b5f49c #xb71224f920234978acc74f4d23143069, #xa78294= 227b5f49c {font-family: Tahoma; font-size: 12pt;} #xa78294227b5f49c, #xa78294227b5f49c {font-family: 'Segoe UI'; font-size: 12pt;} #xa78294227b5f49c #xb71224f920234978acc74f4d23143069 p.MsoNormal, #xa782942= 27b5f49c p.MsoNormal {margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-seri= f;} #xa78294227b5f49c #xb71224f920234978acc74f4d23143069 div.WordSection1, #xa7= 8294227b5f49c div.WordSection1 {page: WordSection1;} #xa78294227b5f49c a:link {color: rgb(5, 99, 193); text-decoration: underline;} --></style><style type=3D"text/css"><![CDATA[#x3d04dd896a4a42b1a36732d91328= 7ec0{ font-family:'Segoe UI'; font-size:12pt; }#xad6e4c7b8f164c94bb061ede59833d9c{ font-family:'Segoe UI'; font-size:12pt; }]]><!--blockquote.cite {margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right: 0p= x; border-left-width: 1px; border-left-style: solid; border-left-color: rgb= (204, 204, 204);} blockquote.cite2 {margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right: 0p= x; border-left-width: 1px; border-left-style: solid; border-left-color: rgb= (204, 204, 204); margin-top: 3px; padding-top: 0px;} body {font-family: 'Segoe UI'; font-size: 12pt;} --></style></head><body><div>Hi Yaniv</div><div><br /></div><div>I found a= solution.=C2=A0</div><div>Our Ovirt 3.6 AIO box was still running and had t= hose VMs still configured in their pre exported and switch off state.=C2=A0= </div><div>I removed any snap shots I found from those pre exported VMs, th= en copied the disk image files and other bits from h<span style=3D"font-siz= e: 12pt;">ost01 (Ovirt v 4.1) back into the Ovirt 3.6 AIO box, and were nee= ded fixing the relevent IDs to be what the Engine in the Ovirt 3.6 box expe= cted.=C2=A0</span></div><div><span style=3D"font-size: 12pt;">The VMs then= started up properly again without hassle and with the latest files on the O= virt 3.6 AIO box.=C2=A0</span></div><div><br /></div><div><span style=3D"fo= nt-size: 12pt;">So now in the progress of rebuilding host01 with=C2=A0</spa= n><span style=3D"font-size: 12pt;">hosted-engine=C2=A0</span><span style=3D= "font-size: 12pt;">v4.1=C2=A0</span></div><div><br /></div><div id=3D"signa= ture_old"><div id=3D"xa78294227b5f49c"> <div class=3D"WordSection1"> <div id=3D"xb71224f920234978acc74f4d23143069"><div class=3D"WordSection1"><= p class=3D"MsoNormal"><font face=3D"Tahoma"><font size=3D"3" style=3D"font-= size: 12pt;">Kind r</font><span style=3D"font-size: 12pt;">egards</span></f= ont></p><p class=3D"MsoNormal"><font size=3D"3" style=3D"font-size: 12pt;"= face=3D"Tahoma"><br /></font></p><p class=3D"MsoNormal"><font size=3D"3" st= yle=3D"font-size: 12pt;" face=3D"Tahoma">Andrew=C2=A0</font></p></div></div= div><div><br /></div> <div id=3D"xbea74d01f04742a"><blockquote cite=3D"249bdf9e95f568dc6180880838= 41b2b46780224d@localhost" type=3D"cite" class=3D"cite2"> <div dir=3D"ltr"><br /><div class=3D"gmail_extra"><br /><div class=3D"gmail= _quote">On Sat, Jun 17, 2017 at 12:50 AM, <span dir=3D"ltr"><<a href=3D= "mailto:adent@ctcroydon.com.au">adent@ctcroydon.com.au</a>></span> wrote= :<br /><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-= left:1px #ccc solid;padding-left:1ex"><div dir=3D"auto"><div>If I reinstall = and the rerun the hosted-engine setup how do I get the VMs in their curren= t running state back into and being recognised by the new hosted engine?<br = /></div></div></blockquote><div><br /></div><div>Current running state is= again quite challenging. You'll need to fix the hosted-engine.</div><div><b= r /></div><div>Can import the storage domain? (not for running VMs)</div><d= iv>Y.</div><div><br /></div><blockquote class=3D"gmail_quote" style=3D"marg= in:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"auto= "><div><br />Kind regards<span class=3D"HOEnZb"><font color=3D"#888888"><di= v><br /></div><div>Andrew</div></font></span></div><span class=3D""><div><b= r />On 17 Jun 2017, at 6:54 AM, Yaniv Kaul <<a href=3D"mailto:ykaul@redh= at.com">ykaul@redhat.com</a>> wrote:<br /><br /></div></span><blockquote = type=3D"cite" class=3D"cite"><div><div dir=3D"ltr"><br /><div class=3D"gma= il_extra"><br /><div class=3D"gmail_quote"><span class=3D"">On Fri, Jun 16, = 2017 at 9:11 AM, Andrew Dent <span dir=3D"ltr"><<a href=3D"mailto:adent= @ctcroydon.com.au">adent@ctcroydon.com.au</a>></span> wrote:<br /></span=
<div><div class=3D"h5"><blockquote class=3D"gmail_quote" style=3D"margin:0 = 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u>
<div><div>Hi</div><div><br /></div><div>Well I've got myself into a fine me= ss.=C2=A0</div><div><br /></div><div>host01 was setup with hosted-engine v4= .1.=C2=A0<span style=3D"font-size:12pt">This was successful.=C2=A0</span></= div><div><span style=3D"font-size:12pt">Imported 3 VMs from a v3.6 OVirt AI= O instance. (This OVirt 3.6 is still running with more VMs on it)</span></d= iv><div>Tried to add host02 to the new Ovirt 4.1 setup. This partially succ= eeded but I couldn't add any storage domains to it. Cannot remember why.=C2= =A0</div><div>In Ovirt engine UI I removed host02.=C2=A0</div><div>I reinst= alled host02 with Centos7, tried to add it and Ovirt UI told me it was alre= ady there (but it wasn't listed in the UI).=C2=A0</div><div>Renamed the rei= nstalled host02 to host03, changed the ipaddress, reconfig the DNS server a= nd added host03 into the Ovirt Engine UI.=C2=A0</div><div>All good, and I w= as able to import more VMs to it.=C2=A0</div><div>I was also able to shutdo= wn a VM on host01 assign it to host03 and start the VM. Cool, everything wo= rking.=C2=A0</div><div>The above was all last couple of weeks.=C2=A0</div><= div><br /></div><div>This week I performed some yum updates on the Engine V= M. No reboot.=C2=A0</div><div>Today noticed that the Ovirt services in the= Engine VM were in a endless restart loop. They would be up for a 5 minutes= and then die.=C2=A0</div><div>Looking into=C2=A0/var/log/ovirt-engine/eng<w= br>ine.log and I could only see errors relating to host02. Ovirt was trying = to find it and failing. Then falling over.=C2=A0</wbr></div><div>I ran "ho= sted-engine --clean-metadata" thinking it would cleanup and remove bad refe= rences to hosts, but now realise that was a really bad idea as it didn't do = what I'd hoped.=C2=A0</div><div>At this point the sequence below worked, I = could login to Ovirt UI but after 5 minutes the services would be off</div=
<div><div>
<span style=3D"color:rgb(34,34,34);font-family:Verdana,Arial,Helvetica,sans= -serif;font-variant-ligatures:normal;background-color:rgb(255,255,255)">ser= vice ovirt-engine restart</span></div></div><div><span style=3D"color:rgb(3= 4,34,34);font-family:Verdana,Arial,Helvetica,sans-serif;font-variant-ligatu= res:normal;background-color:rgb(255,255,255)"><div> <span style=3D"font-variant-ligatures:normal">service ovirt-websocket-proxy = restart</span></div><div><span style=3D"font-variant-ligatures:normal"><di= v> <span style=3D"font-variant-ligatures:normal">service httpd restart</span><= /div></span></div></span></div><div><br /></div><div>I saw some reference t= o having to remove hosts from the database by hand in situations where unde= r the hood of Ovirt a decommission host was still listed, but wasn't showin= g in the GUI.=C2=A0</div><div>So I removed reference to host02 (vds_id and= host_id) in the following tables in this order.=C2=A0</div><div><span style= =3D"font-size:12pt">vds_dynamic</span></div><div><div id=3D"m_-869654554590= 4220606m_-2002812221470261923x0ec1289986994fb5ac309827edd33abe"> <div><div><span style=3D"font-size:12pt">vds_statistics</span></div><div><d= iv id=3D"m_-8696545545904220606m_-2002812221470261923x87d978c56880415e9594d= 7fe317f30f3"> <div><div><div id=3D"m_-8696545545904220606m_-2002812221470261923x0ec128998= 6994fb5ac309827edd33abe"><div><div>vds_static</div><div></div></div></div><= /div></div></div></div><div></div></div></div></div><div>host_device</div><= div><br /></div><div>Now when I try to start ovirt-websocket it will not st= art</div><div>service ovirt-websocket start<br />Redirecting to /bin/system= ctl start =C2=A0ovirt-websocket.service<br />Failed to start ovirt-websocke= t.service: Unit not found.</div><div><br /></div><div>I'm now thinking that = I need to do the following in the engine VM</div><div><div> <pre class=3D"m_-8696545545904220606m_-2002812221470261923highlight m_-8696= 545545904220606m_-2002812221470261923plaintext"><code>#=C2=A0engine-cleanup #=C2=A0yum=C2=A0remove=C2=A0ovirt-engine #=C2=A0yum=C2=A0install=C2=A0ovirt-engine #=C2=A0engine-setup=C2=A0</code></pre></div></div><div>But to run engine-cl= eanup I need to put the engine-vm into maintenance mode and because of the= --clean-metadata that I ran earlier on host01 I cannot do that.=C2=A0</div>= <div><br /></div><div>What is the best course of action from here?</div></d= iv></blockquote><div><br /></div></div></div><span class=3D""><div>To be ho= nest, with all the steps taken above, I'd install everything (including OS) = from scratch...</div><div>There's a bit too much mess to try to clean up p= roperly here.</div><div>Y.</div><div>=C2=A0</div><blockquote class=3D"gmail= _quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:= 1ex"><div><div><br /></div><div id=3D"m_-8696545545904220606m_-200281222147= 0261923signature_old"><div id=3D"m_-8696545545904220606m_-20028122214702619= 23xbaf36d997b7f497"> <div class=3D"m_-8696545545904220606m_-2002812221470261923WordSection1"> <div id=3D"m_-8696545545904220606m_-2002812221470261923xb71224f920234978acc= 74f4d23143069"><div class=3D"m_-8696545545904220606m_-2002812221470261923Wo= rdSection1"><p class=3D"MsoNormal"><font face=3D"Tahoma"><span style=3D"fon= t-size:16px">Cheers</span></font></p><span class=3D"m_-8696545545904220606H= OEnZb"><font color=3D"#888888"><p class=3D"MsoNormal"><br /></p><p class=3D= "MsoNormal"><font face=3D"Tahoma"><span style=3D"font-size:16px">Andrew</sp= an></font></p></font></span></div></div></div></div></div> </div><br />______________________________<wbr>_________________<br /> Users mailing list<br /> <a href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a><br /> <a href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer= ">http://lists.ovirt.org/mailman<wbr>/listinfo/users</wbr></a><br /> <br /></wbr></blockquote></span></div><br /></div></div> </div></blockquote></div></blockquote></div><br /></div></div> </blockquote></div> </body></html> --------=_MB3BBF1E10-B89F-4246-8ACD-AB3F4F14DC3C--
On Wed, Jun 21, 2017 at 5:20 AM, Andrew Dent <adent@ctcroydon.com.au> wrote:
Hi Yaniv
I found a solution. Our Ovirt 3.6 AIO box was still running and had those VMs still configured in their pre exported and switch off state. I removed any snap shots I found from those pre exported VMs, then copied the disk image files and other bits from host01 (Ovirt v 4.1) back into the Ovirt 3.6 AIO box, and were needed fixing the relevent IDs to be what the Engine in the Ovirt 3.6 box expected. The VMs then started up properly again without hassle and with the latest files on the Ovirt 3.6 AIO box.
Well done and kudos for the resourcefulness! Y.
So now in the progress of rebuilding host01 with hosted-engine v4.1
Kind regards
Andrew ------ Original Message ------ From: "Yaniv Kaul" <ykaul@redhat.com> To: "Andrew Dent" <adent@ctcroydon.com.au> Cc: "users" <users@ovirt.org> Sent: 18/06/2017 6:00:09 PM Subject: Re: [ovirt-users] hosted-engine VM and services not working
On Sat, Jun 17, 2017 at 12:50 AM, <adent@ctcroydon.com.au> wrote:
If I reinstall and the rerun the hosted-engine setup how do I get the VMs in their current running state back into and being recognised by the new hosted engine?
Current running state is again quite challenging. You'll need to fix the hosted-engine.
Can import the storage domain? (not for running VMs) Y.
Kind regards
Andrew
On 17 Jun 2017, at 6:54 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Fri, Jun 16, 2017 at 9:11 AM, Andrew Dent <adent@ctcroydon.com.au> wrote:
Hi
Well I've got myself into a fine mess.
host01 was setup with hosted-engine v4.1. This was successful. Imported 3 VMs from a v3.6 OVirt AIO instance. (This OVirt 3.6 is still running with more VMs on it) Tried to add host02 to the new Ovirt 4.1 setup. This partially succeeded but I couldn't add any storage domains to it. Cannot remember why. In Ovirt engine UI I removed host02. I reinstalled host02 with Centos7, tried to add it and Ovirt UI told me it was already there (but it wasn't listed in the UI). Renamed the reinstalled host02 to host03, changed the ipaddress, reconfig the DNS server and added host03 into the Ovirt Engine UI. All good, and I was able to import more VMs to it. I was also able to shutdown a VM on host01 assign it to host03 and start the VM. Cool, everything working. The above was all last couple of weeks.
This week I performed some yum updates on the Engine VM. No reboot. Today noticed that the Ovirt services in the Engine VM were in a endless restart loop. They would be up for a 5 minutes and then die. Looking into /var/log/ovirt-engine/engine.log and I could only see errors relating to host02. Ovirt was trying to find it and failing. Then falling over. I ran "hosted-engine --clean-metadata" thinking it would cleanup and remove bad references to hosts, but now realise that was a really bad idea as it didn't do what I'd hoped. At this point the sequence below worked, I could login to Ovirt UI but after 5 minutes the services would be off service ovirt-engine restart service ovirt-websocket-proxy restart service httpd restart
I saw some reference to having to remove hosts from the database by hand in situations where under the hood of Ovirt a decommission host was still listed, but wasn't showing in the GUI. So I removed reference to host02 (vds_id and host_id) in the following tables in this order. vds_dynamic vds_statistics vds_static host_device
Now when I try to start ovirt-websocket it will not start service ovirt-websocket start Redirecting to /bin/systemctl start ovirt-websocket.service Failed to start ovirt-websocket.service: Unit not found.
I'm now thinking that I need to do the following in the engine VM
# engine-cleanup # yum remove ovirt-engine # yum install ovirt-engine # engine-setup
But to run engine-cleanup I need to put the engine-vm into maintenance mode and because of the --clean-metadata that I ran earlier on host01 I cannot do that.
What is the best course of action from here?
To be honest, with all the steps taken above, I'd install everything (including OS) from scratch... There's a bit too much mess to try to clean up properly here. Y.
Cheers
Andrew
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (2)
-
Andrew Dent -
Yaniv Kaul