New subject: hosted-engine VM and services not working

Tuesday, 20 June 2017

--------=_MB3BBF1E10-B89F-4246-8ACD-AB3F4F14DC3C
Content-Type: text/plain; format=flowed; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Hi Yaniv

I found a solution.
Our Ovirt 3.6 AIO box was still running and had those VMs still=20
configured in their pre exported and switch off state.
I removed any snap shots I found from those pre exported VMs, then=20
copied the disk image files and other bits from host01 (Ovirt v 4.1)=20
back into the Ovirt 3.6 AIO box, and were needed fixing the relevent IDs=20
to be what the Engine in the Ovirt 3.6 box expected.
The VMs then started up properly again without hassle and with the=20
latest files on the Ovirt 3.6 AIO box.

So now in the progress of rebuilding host01 with hosted-engine v4.1

Kind regards

Andrew

------ Original Message ------
From: "Yaniv Kaul" <ykaul(a)redhat.com&gt;
To: "Andrew Dent" <adent(a)ctcroydon.com.au&gt;
Cc: "users" <users(a)ovirt.org&gt;
Sent: 18/06/2017 6:00:09 PM
Subject: Re: [ovirt-users] hosted-engine VM and services not working

...

On Sat, Jun 17, 2017 at 12:50 AM, <adent(a)ctcroydon.com.au&gt; wrote:
>If I reinstall and the rerun the hosted-engine setup how do I get the=20
>VMs in their current running state back into and being recognised by=20
>the new hosted engine?

Current running state is again quite challenging. You'll need to fix=20
the hosted-engine.

Can import the storage domain? (not for running VMs)
Y.

>
>Kind regards
>
>Andrew
>
>On 17 Jun 2017, at 6:54 AM, Yaniv Kaul <ykaul(a)redhat.com&gt; wrote:
>
>>
>>
>>On Fri, Jun 16, 2017 at 9:11 AM, Andrew Dent <adent(a)ctcroydon.com.au&gt;=20
>>wrote:
>>>Hi
>>>
>>>Well I've got myself into a fine mess.
>>>
>>>host01 was setup with hosted-engine v4.1. This was successful.
>>>Imported 3 VMs from a v3.6 OVirt AIO instance. (This OVirt 3.6 is=20
>>>still running with more VMs on it)
>>>Tried to add host02 to the new Ovirt 4.1 setup. This partially=20
>>>succeeded but I couldn't add any storage domains to it. Cannot=20
>>>remember why.
>>>In Ovirt engine UI I removed host02.
>>>I reinstalled host02 with Centos7, tried to add it and Ovirt UI told=20
>>>me it was already there (but it wasn't listed in the UI).
>>>Renamed the reinstalled host02 to host03, changed the ipaddress,=20
>>>reconfig the DNS server and added host03 into the Ovirt Engine UI.
>>>All good, and I was able to import more VMs to it.
>>>I was also able to shutdown a VM on host01 assign it to host03 and=20
>>>start the VM. Cool, everything working.
>>>The above was all last couple of weeks.
>>>
>>>This week I performed some yum updates on the Engine VM. No reboot.
>>>Today noticed that the Ovirt services in the Engine VM were in a=20
>>>endless restart loop. They would be up for a 5 minutes and then die.
>>>Looking into /var/log/ovirt-engine/engine.log and I could only see=20
>>>errors relating to host02. Ovirt was trying to find it and failing.=20
>>>Then falling over.
>>>I ran "hosted-engine --clean-metadata" thinking it would cleanup
and=20
>>>remove bad references to hosts, but now realise that was a really=20
>>>bad idea as it didn't do what I'd hoped.
>>>At this point the sequence below worked, I could login to Ovirt UI=20
>>>but after 5 minutes the services would be off
>>>service ovirt-engine restart
>>>service ovirt-websocket-proxy restart
>>>service httpd restart
>>>
>>>I saw some reference to having to remove hosts from the database by=20
>>>hand in situations where under the hood of Ovirt a decommission host=20
>>>was still listed, but wasn't showing in the GUI.
>>>So I removed reference to host02 (vds_id and host_id) in the=20
>>>following tables in this order.
>>>vds_dynamic
>>>vds_statistics
>>>vds_static
>>>host_device
>>>
>>>Now when I try to start ovirt-websocket it will not start
>>>service ovirt-websocket start
>>>Redirecting to /bin/systemctl start  ovirt-websocket.service
>>>Failed to start ovirt-websocket.service: Unit not found.
>>>
>>>I'm now thinking that I need to do the following in the engine VM
>>># engine-cleanup # yum remove ovirt-engine # yum install=20
>>>ovirt-engine # engine-setup
>>>But to run engine-cleanup I need to put the engine-vm into=20
>>>maintenance mode and because of the --clean-metadata that I ran=20
>>>earlier on host01 I cannot do that.
>>>
>>>What is the best course of action from here?
>>
>>To be honest, with all the steps taken above, I'd install everything=20
>>(including OS) from scratch...
>>There's a bit too much mess to try to clean up properly here.
>>Y.
>>
>>>
>>>Cheers
>>>
>>>
>>>
>>>Andrew
>>>
>>>
>>>_______________________________________________
>>>Users mailing list
&gt;&gt;&gt;Users(a)ovirt.org
>>>http://lists.ovirt.org/mailman/listinfo/users=20
>>><http://lists.ovirt.org/mailman/listinfo/users>
>>>
>>
 --------=_MB3BBF1E10-B89F-4246-8ACD-AB3F4F14DC3C
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<?xml version=3D"1.0"
encoding=3D"utf-16"?><html><head><style id=3D"signatu=
reStyle" type=3D"text/css"><!--#xa78294227b5f49c, #xa78294227b5f49c,
#xa782=
94227b5f49c, #xa78294227b5f49c #xb71224f920234978acc74f4d23143069, #xa78294=
227b5f49c
{font-family: Tahoma; font-size: 12pt;}
#xa78294227b5f49c, #xa78294227b5f49c
{font-family: 'Segoe UI'; font-size: 12pt;}
#xa78294227b5f49c #xb71224f920234978acc74f4d23143069 p.MsoNormal, #xa782942=
27b5f49c p.MsoNormal
{margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-seri=
f;}
#xa78294227b5f49c #xb71224f920234978acc74f4d23143069 div.WordSection1, #xa7=
8294227b5f49c div.WordSection1
{page: WordSection1;}
#xa78294227b5f49c a:link
{color: rgb(5, 99, 193); text-decoration: underline;}
--></style><style
type=3D"text/css"><![CDATA[#x3d04dd896a4a42b1a36732d91328=
7ec0{
	font-family:'Segoe UI';
	font-size:12pt;
}#xad6e4c7b8f164c94bb061ede59833d9c{
	font-family:'Segoe UI';
	font-size:12pt;
}]]><!--blockquote.cite
{margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right: 0p=
x; border-left-width: 1px; border-left-style: solid; border-left-color: rgb=
(204, 204, 204);}
blockquote.cite2
{margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right: 0p=
x; border-left-width: 1px; border-left-style: solid; border-left-color: rgb=
(204, 204, 204); margin-top: 3px; padding-top: 0px;}
body
{font-family: 'Segoe UI'; font-size: 12pt;}
--></style></head><body><div>Hi
Yaniv</div><div><br /></div><div>I found a=
 solution.=C2=A0</div><div>Our Ovirt 3.6 AIO box was still running and had t=
hose VMs still configured in their pre exported and switch off state.=C2=A0=
</div><div>I removed any snap shots I found from those pre exported VMs, th=
en copied the disk image files and other bits from h<span style=3D"font-siz=
e: 12pt;">ost01 (Ovirt v 4.1) back into the Ovirt 3.6 AIO box, and were nee=
ded fixing the relevent IDs to be what the Engine in the Ovirt 3.6 box expe=
cted.=C2=A0</span></div><div><span style=3D"font-size:
12pt;">The VMs then=
 started up properly again without hassle and with the latest files on the O=
virt 3.6 AIO box.=C2=A0</span></div><div><br
/></div><div><span style=3D"fo=
nt-size: 12pt;">So now in the progress of rebuilding host01 with=C2=A0</spa=
n><span style=3D"font-size:
12pt;">hosted-engine=C2=A0</span><span style=3D=
"font-size: 12pt;">v4.1=C2=A0</span></div><div><br
/></div><div id=3D"signa=
ture_old"><div id=3D"xa78294227b5f49c">

<div class=3D"WordSection1">

<div id=3D"xb71224f920234978acc74f4d23143069"><div
class=3D"WordSection1"><=
p class=3D"MsoNormal"><font face=3D"Tahoma"><font
size=3D"3" style=3D"font-=
size: 12pt;">Kind r</font><span style=3D"font-size:
12pt;">egards</span></f=
ont></p><p class=3D"MsoNormal"><font size=3D"3"
style=3D"font-size: 12pt;"=
 face=3D"Tahoma"><br /></font></p><p
class=3D"MsoNormal"><font size=3D"3" st=
yle=3D"font-size: 12pt;"
face=3D"Tahoma">Andrew=C2=A0</font></p></div></div=
...
</div></div></div> <div>------ Original
Message ------</div>
<div>From: "Yaniv Kaul" &lt;<a
href=3D"mailto:ykaul@redhat.com">ykaul@redha=
t.com</a>&gt;</div>
<div>To: "Andrew Dent" &lt;<a
href=3D"mailto:adent@ctcroydon.com.au">adent@=
ctcroydon.com.au</a>&gt;</div>
<div>Cc: "users" &lt;<a
href=3D"mailto:users@ovirt.org">users@ovirt.org</a>=
&gt;</div>
<div>Sent: 18/06/2017 6:00:09 PM</div>
<div>Subject: Re: [ovirt-users] hosted-engine VM and services not working</=
div><div><br /></div>
<div id=3D"xbea74d01f04742a"><blockquote
cite=3D"249bdf9e95f568dc6180880838=
41b2b46780224d@localhost" type=3D"cite" class=3D"cite2">
<div dir=3D"ltr"><br /><div
class=3D"gmail_extra"><br /><div class=3D"gmail=
_quote">On Sat, Jun 17, 2017 at 12:50 AM,  <span
dir=3D"ltr">&lt;<a href=3D=
"mailto:adent@ctcroydon.com.au">adent@ctcroydon.com.au</a>&gt;</span>
wrote=
:<br /><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0
.8ex;border-=
left:1px #ccc solid;padding-left:1ex"><div
dir=3D"auto"><div>If I reinstall =
and the rerun the hosted-engine setup how do I get the VMs in their curren=
t running state back into and being recognised by the new hosted engine?<br =
/></div></div></blockquote><div><br
/></div><div>Current running state is=
 again quite challenging. You'll need to fix the
hosted-engine.</div><div><b=
r /></div><div>Can import the storage domain? (not for running
VMs)</div><d=
iv>Y.</div><div><br /></div><blockquote
class=3D"gmail_quote" style=3D"marg=
in:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div
dir=3D"auto=
"><div><br />Kind regards<span
class=3D"HOEnZb"><font color=3D"#888888"><di=
v><br
/></div><div>Andrew</div></font></span></div><span
class=3D""><div><b=
r />On 17 Jun 2017, at 6:54 AM, Yaniv Kaul &lt;<a
href=3D"mailto:ykaul@redh=
at.com&quot;&gt;ykaul(a)redhat.com&lt;/a&gt;&amp;gt; wrote:<br /><br
/></div></span><blockquote =
type=3D"cite" class=3D"cite"><div><div
dir=3D"ltr"><br /><div class=3D"gma=
il_extra"><br /><div class=3D"gmail_quote"><span
class=3D"">On Fri, Jun 16, =
2017 at 9:11 AM, Andrew Dent <span dir=3D"ltr">&lt;<a
href=3D"mailto:adent=
@ctcroydon.com.au&quot;&gt;adent(a)ctcroydon.com.au&lt;/a&gt;&amp;gt;&lt;/span&gt;
wrote:<br /></span=
...
<div><div class=3D"h5"><blockquote
class=3D"gmail_quote" style=3D"margin:0 = 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex"><u></u>

<div><div>Hi</div><div><br /></div><div>Well
I've got myself into a fine me=
ss.=C2=A0</div><div><br /></div><div>host01 was setup with
hosted-engine v4=
.1.=C2=A0<span style=3D"font-size:12pt">This was
successful.=C2=A0</span></=
div><div><span style=3D"font-size:12pt">Imported 3 VMs from a
v3.6 OVirt AI=
O instance. (This OVirt 3.6 is still running with more VMs on it)</span></d=
iv><div>Tried to add host02 to the new Ovirt 4.1 setup. This partially succ=
eeded but I couldn't add any storage domains to it. Cannot remember why.=C2=
=A0</div><div>In Ovirt engine UI I removed
host02.=C2=A0</div><div>I reinst=
alled host02 with Centos7, tried to add it and Ovirt UI told me it was alre=
ady there (but it wasn't listed in the UI).=C2=A0</div><div>Renamed the
rei=
nstalled host02 to host03, changed the ipaddress, reconfig the DNS server a=
nd added host03 into the Ovirt Engine UI.=C2=A0</div><div>All good, and I w=
as able to import more VMs to it.=C2=A0</div><div>I was also able to shutdo=
wn a VM on host01 assign it to host03 and start the VM. Cool, everything wo=
rking.=C2=A0</div><div>The above was all last couple of
weeks.=C2=A0</div><=
div><br /></div><div>This week I performed some yum updates on the
Engine V=
M. No reboot.=C2=A0</div><div>Today noticed that the Ovirt services in the=
 Engine VM were in a endless restart loop. They would be up for a 5 minutes=
 and then die.=C2=A0</div><div>Looking
into=C2=A0/var/log/ovirt-engine/eng<w=
br>ine.log and I could only see errors relating to host02. Ovirt was trying =
to find it and failing. Then falling over.=C2=A0</wbr></div><div>I ran
"ho=
sted-engine --clean-metadata" thinking it would cleanup and remove bad refe=
rences to hosts, but now realise that was a really bad idea as it didn't do =
what I'd hoped.=C2=A0</div><div>At this point the sequence below worked, I
=
could login to Ovirt UI but after 5 minutes the services would be off</div=
...
<div><div> 
<span style=3D"color:rgb(34,34,34);font-family:Verdana,Arial,Helvetica,sans=
-serif;font-variant-ligatures:normal;background-color:rgb(255,255,255)">ser=
vice ovirt-engine restart</span></div></div><div><span
style=3D"color:rgb(3=
4,34,34);font-family:Verdana,Arial,Helvetica,sans-serif;font-variant-ligatu=
res:normal;background-color:rgb(255,255,255)"><div>

<span style=3D"font-variant-ligatures:normal">service
ovirt-websocket-proxy =
restart</span></div><div><span
style=3D"font-variant-ligatures:normal"><di=
v>

<span style=3D"font-variant-ligatures:normal">service httpd
restart</span><=
/div></span></div></span></div><div><br
/></div><div>I saw some reference t=
o having to remove hosts from the database by hand in situations where unde=
r the hood of Ovirt a decommission host was still listed, but wasn't showin=
g in the GUI.=C2=A0</div><div>So I removed reference to host02 (vds_id and=
 host_id) in the following tables in this order.=C2=A0</div><div><span
style=
=3D"font-size:12pt">vds_dynamic</span></div><div><div
id=3D"m_-869654554590=
4220606m_-2002812221470261923x0ec1289986994fb5ac309827edd33abe">

<div><div><span
style=3D"font-size:12pt">vds_statistics</span></div><div><d=
iv id=3D"m_-8696545545904220606m_-2002812221470261923x87d978c56880415e9594d=
7fe317f30f3">

<div><div><div
id=3D"m_-8696545545904220606m_-2002812221470261923x0ec128998=
6994fb5ac309827edd33abe"><div><div>vds_static</div><div></div></div></div><=
/div></div></div></div><div></div></div></div></div><div>host_device</div><=
div><br /></div><div>Now when I try to start ovirt-websocket it will
not st=
art</div><div>service ovirt-websocket start<br />Redirecting to
/bin/system=
ctl start =C2=A0ovirt-websocket.service<br />Failed to start ovirt-websocke=
t.service: Unit not found.</div><div><br
/></div><div>I'm now thinking that =
I need to do the following in the engine VM</div...
<div><div> <pre
class=3D"m_-8696545545904220606m_-2002812221470261923highlight m_-8696=
545545904220606m_-2002812221470261923plaintext"><code>#=C2=A0engine-cleanup
#=C2=A0yum=C2=A0remove=C2=A0ovirt-engine
#=C2=A0yum=C2=A0install=C2=A0ovirt-engine
#=C2=A0engine-setup=C2=A0</code></pre></div></div><div>But
to run engine-cl=
eanup I need to put the engine-vm into maintenance mode and because of the=
 --clean-metadata that I ran earlier on host01 I cannot do that.=C2=A0</div>=
<div><br /></div><div>What is the best course of action from
here?</div></d=
iv></blockquote><div><br
/></div></div></div><span class=3D""><div>To be
ho=
nest, with all the steps taken above, I'd install everything (including OS) =
from scratch...</div><div>There's a bit too much mess to try to clean up
p=
roperly
here.</div><div>Y.</div><div>=C2=A0</div><blockquote
class=3D"gmail=
_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:=
1ex"><div><div><br /></div><div
id=3D"m_-8696545545904220606m_-200281222147=
0261923signature_old"><div
id=3D"m_-8696545545904220606m_-20028122214702619=
23xbaf36d997b7f497">

<div class=3D"m_-8696545545904220606m_-2002812221470261923WordSection1">

<div id=3D"m_-8696545545904220606m_-2002812221470261923xb71224f920234978acc=
74f4d23143069"><div
class=3D"m_-8696545545904220606m_-2002812221470261923Wo=
rdSection1"><p class=3D"MsoNormal"><font
face=3D"Tahoma"><span style=3D"fon=
t-size:16px">Cheers</span></font></p><span
class=3D"m_-8696545545904220606H=
OEnZb"><font color=3D"#888888"><p
class=3D"MsoNormal"><br /></p><p class=3D=
"MsoNormal"><font face=3D"Tahoma"><span
style=3D"font-size:16px">Andrew</sp=
an></font></p></font></span></div></div...
</div></div></div> </div><br
/>______________________________<wbr>_________________<br />
Users mailing list<br />
<a href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a><br />
<a href=3D"http://lists.ovirt.org/mailman/listinfo/users"
rel=3D"noreferrer=
">http://lists.ovirt.org/mailman<wbr>/listinfo/users</wbr></a><br
/>
<br /></wbr></blockquote></span></div><br
/></div></div>

</div></blockquote></div></blockquote></div><br
/></div></div>
</blockquote></div>
</body></html>
--------=_MB3BBF1E10-B89F-4246-8ACD-AB3F4F14DC3C--

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Re: [ovirt-users] hosted-engine VM and services not working