Re: [ovirt-users] hosted-engine VM and services not working

21 Jun 2017

      --------=_MB3BBF1E10-B89F-4246-8ACD-AB3F4F14DC3C
Content-Type: text/plain; format=flowed; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Hi Yaniv

I found a solution.
Our Ovirt 3.6 AIO box was still running and had those VMs still=20
configured in their pre exported and switch off state.
I removed any snap shots I found from those pre exported VMs, then=20
copied the disk image files and other bits from host01 (Ovirt v 4.1)=20
back into the Ovirt 3.6 AIO box, and were needed fixing the relevent IDs=20
to be what the Engine in the Ovirt 3.6 box expected.
The VMs then started up properly again without hassle and with the=20
latest files on the Ovirt 3.6 AIO box.

So now in the progress of rebuilding host01 with hosted-engine v4.1

Kind regards

Andrew

------ Original Message ------
From: "Yaniv Kaul" <ykaul@redhat.com>
To: "Andrew Dent" <adent@ctcroydon.com.au>
Cc: "users" <users@ovirt.org>
Sent: 18/06/2017 6:00:09 PM
Subject: Re: [ovirt-users] hosted-engine VM and services not working
...
On Sat, Jun 17, 2017 at 12:50 AM, <adent@ctcroydon.com.au> wrote:
...
If I reinstall and the rerun the hosted-engine setup how do I get the=20
VMs in their current running state back into and being recognised by=20
the new hosted engine?
Current running state is again quite challenging. You'll need to fix=20
the hosted-engine.
Can import the storage domain? (not for running VMs)
Y.
...
Kind regards
Andrew
On 17 Jun 2017, at 6:54 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
...
On Fri, Jun 16, 2017 at 9:11 AM, Andrew Dent <adent@ctcroydon.com.au>=20
wrote:
...
Hi
Well I've got myself into a fine mess.
host01 was setup with hosted-engine v4.1. This was successful.
Imported 3 VMs from a v3.6 OVirt AIO instance. (This OVirt 3.6 is=20
still running with more VMs on it)
Tried to add host02 to the new Ovirt 4.1 setup. This partially=20
succeeded but I couldn't add any storage domains to it. Cannot=20
remember why.
In Ovirt engine UI I removed host02.
I reinstalled host02 with Centos7, tried to add it and Ovirt UI told=20
me it was already there (but it wasn't listed in the UI).
Renamed the reinstalled host02 to host03, changed the ipaddress,=20
reconfig the DNS server and added host03 into the Ovirt Engine UI.
All good, and I was able to import more VMs to it.
I was also able to shutdown a VM on host01 assign it to host03 and=20
start the VM. Cool, everything working.
The above was all last couple of weeks.
This week I performed some yum updates on the Engine VM. No reboot.
Today noticed that the Ovirt services in the Engine VM were in a=20
endless restart loop. They would be up for a 5 minutes and then die.
Looking into /var/log/ovirt-engine/engine.log and I could only see=20
errors relating to host02. Ovirt was trying to find it and failing.=20
Then falling over.
I ran "hosted-engine --clean-metadata" thinking it would cleanup and=20
remove bad references to hosts, but now realise that was a really=20
bad idea as it didn't do what I'd hoped.
At this point the sequence below worked, I could login to Ovirt UI=20
but after 5 minutes the services would be off
service ovirt-engine restart
service ovirt-websocket-proxy restart
service httpd restart
I saw some reference to having to remove hosts from the database by=20
hand in situations where under the hood of Ovirt a decommission host=20
was still listed, but wasn't showing in the GUI.
So I removed reference to host02 (vds_id and host_id) in the=20
following tables in this order.
vds_dynamic
vds_statistics
vds_static
host_device
Now when I try to start ovirt-websocket it will not start
service ovirt-websocket start
Redirecting to /bin/systemctl start  ovirt-websocket.service
Failed to start ovirt-websocket.service: Unit not found.
I'm now thinking that I need to do the following in the engine VM
# engine-cleanup # yum remove ovirt-engine # yum install=20
ovirt-engine # engine-setup
But to run engine-cleanup I need to put the engine-vm into=20
maintenance mode and because of the --clean-metadata that I ran=20
earlier on host01 I cannot do that.
What is the best course of action from here?
To be honest, with all the steps taken above, I'd install everything=20
(including OS) from scratch...
There's a bit too much mess to try to clean up properly here.
Y.
...
Cheers
Andrew
_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users=20
<http://lists.ovirt.org/mailman/listinfo/users>
...
</div></div></div>
<div>------ Original Message ------</div>
<div>From: "Yaniv Kaul" <<a href=3D"mailto:ykaul@redhat.com">ykaul@redha=
t.com</a>></div>
<div>To: "Andrew Dent" <<a href=3D"mailto:adent@ctcroydon.com.au">adent@=
ctcroydon.com.au</a>></div>
<div>Cc: "users" <<a href=3D"mailto:users@ovirt.org">users@ovirt.org</a>=
></div>
<div>Sent: 18/06/2017 6:00:09 PM</div>
<div>Subject: Re: [ovirt-users] hosted-engine VM and services not working</=
--------=_MB3BBF1E10-B89F-4246-8ACD-AB3F4F14DC3C
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<?xml version=3D"1.0" encoding=3D"utf-16"?><html><head><style id=3D"signatu=
reStyle" type=3D"text/css"><!--#xa78294227b5f49c, #xa78294227b5f49c, #xa782=
94227b5f49c, #xa78294227b5f49c #xb71224f920234978acc74f4d23143069, #xa78294=
227b5f49c
{font-family: Tahoma; font-size: 12pt;}
#xa78294227b5f49c, #xa78294227b5f49c
{font-family: 'Segoe UI'; font-size: 12pt;}
#xa78294227b5f49c #xb71224f920234978acc74f4d23143069 p.MsoNormal, #xa782942=
27b5f49c p.MsoNormal
{margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-seri=
f;}
#xa78294227b5f49c #xb71224f920234978acc74f4d23143069 div.WordSection1, #xa7=
8294227b5f49c div.WordSection1
{page: WordSection1;}
#xa78294227b5f49c a:link
{color: rgb(5, 99, 193); text-decoration: underline;}
--></style><style type=3D"text/css"><![CDATA[#x3d04dd896a4a42b1a36732d91328=
7ec0{
	font-family:'Segoe UI';
	font-size:12pt;
}#xad6e4c7b8f164c94bb061ede59833d9c{
	font-family:'Segoe UI';
	font-size:12pt;
}]]><!--blockquote.cite
{margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right: 0p=
x; border-left-width: 1px; border-left-style: solid; border-left-color: rgb=
(204, 204, 204);}
blockquote.cite2
{margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right: 0p=
x; border-left-width: 1px; border-left-style: solid; border-left-color: rgb=
(204, 204, 204); margin-top: 3px; padding-top: 0px;}
body
{font-family: 'Segoe UI'; font-size: 12pt;}
--></style></head><body><div>Hi Yaniv</div><div><br /></div><div>I found a=
 solution.=C2=A0</div><div>Our Ovirt 3.6 AIO box was still running and had t=
hose VMs still configured in their pre exported and switch off state.=C2=A0=
</div><div>I removed any snap shots I found from those pre exported VMs, th=
en copied the disk image files and other bits from h<span style=3D"font-siz=
e: 12pt;">ost01 (Ovirt v 4.1) back into the Ovirt 3.6 AIO box, and were nee=
ded fixing the relevent IDs to be what the Engine in the Ovirt 3.6 box expe=
cted.=C2=A0</span></div><div><span style=3D"font-size: 12pt;">The VMs then=
 started up properly again without hassle and with the latest files on the O=
virt 3.6 AIO box.=C2=A0</span></div><div><br /></div><div><span style=3D"fo=
nt-size: 12pt;">So now in the progress of rebuilding host01 with=C2=A0</spa=
n><span style=3D"font-size: 12pt;">hosted-engine=C2=A0</span><span style=3D=
"font-size: 12pt;">v4.1=C2=A0</span></div><div><br /></div><div id=3D"signa=
ture_old"><div id=3D"xa78294227b5f49c">

<div class=3D"WordSection1">

<div id=3D"xb71224f920234978acc74f4d23143069"><div class=3D"WordSection1"><=
p class=3D"MsoNormal"><font face=3D"Tahoma"><font size=3D"3" style=3D"font-=
size: 12pt;">Kind r</font><span style=3D"font-size: 12pt;">egards</span></f=
ont></p><p class=3D"MsoNormal"><font size=3D"3" style=3D"font-size: 12pt;"=
 face=3D"Tahoma"><br /></font></p><p class=3D"MsoNormal"><font size=3D"3" st=
yle=3D"font-size: 12pt;" face=3D"Tahoma">Andrew=C2=A0</font></p></div></div=
div><div><br /></div>
<div id=3D"xbea74d01f04742a"><blockquote cite=3D"249bdf9e95f568dc6180880838=
41b2b46780224d@localhost" type=3D"cite" class=3D"cite2">
<div dir=3D"ltr"><br /><div class=3D"gmail_extra"><br /><div class=3D"gmail=
_quote">On Sat, Jun 17, 2017 at 12:50 AM,  <span dir=3D"ltr"><<a href=3D=
"mailto:adent@ctcroydon.com.au">adent@ctcroydon.com.au</a>></span> wrote=
:<br /><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-=
left:1px #ccc solid;padding-left:1ex"><div dir=3D"auto"><div>If I reinstall =
and the rerun the hosted-engine setup how do I get the VMs in their curren=
t running state back into and being recognised by the new hosted engine?<br =
/></div></div></blockquote><div><br /></div><div>Current running state is=
 again quite challenging. You'll need to fix the hosted-engine.</div><div><b=
r /></div><div>Can import the storage domain? (not for running VMs)</div><d=
iv>Y.</div><div><br /></div><blockquote class=3D"gmail_quote" style=3D"marg=
in:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"auto=
"><div><br />Kind regards<span class=3D"HOEnZb"><font color=3D"#888888"><di=
v><br /></div><div>Andrew</div></font></span></div><span class=3D""><div><b=
r />On 17 Jun 2017, at 6:54 AM, Yaniv Kaul <<a href=3D"mailto:ykaul@redh=
at.com">ykaul@redhat.com</a>> wrote:<br /><br /></div></span><blockquote =
type=3D"cite" class=3D"cite"><div><div dir=3D"ltr"><br /><div class=3D"gma=
il_extra"><br /><div class=3D"gmail_quote"><span class=3D"">On Fri, Jun 16, =
2017 at 9:11 AM, Andrew Dent <span dir=3D"ltr"><<a href=3D"mailto:adent=
@ctcroydon.com.au">adent@ctcroydon.com.au</a>></span> wrote:<br /></span=
...
<div><div class=3D"h5"><blockquote class=3D"gmail_quote" style=3D"margin:0 =
0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u>
<div><div>Hi</div><div><br /></div><div>Well I've got myself into a fine me=
ss.=C2=A0</div><div><br /></div><div>host01 was setup with hosted-engine v4=
.1.=C2=A0<span style=3D"font-size:12pt">This was successful.=C2=A0</span></=
div><div><span style=3D"font-size:12pt">Imported 3 VMs from a v3.6 OVirt AI=
O instance. (This OVirt 3.6 is still running with more VMs on it)</span></d=
iv><div>Tried to add host02 to the new Ovirt 4.1 setup. This partially succ=
eeded but I couldn't add any storage domains to it. Cannot remember why.=C2=
=A0</div><div>In Ovirt engine UI I removed host02.=C2=A0</div><div>I reinst=
alled host02 with Centos7, tried to add it and Ovirt UI told me it was alre=
ady there (but it wasn't listed in the UI).=C2=A0</div><div>Renamed the rei=
nstalled host02 to host03, changed the ipaddress, reconfig the DNS server a=
nd added host03 into the Ovirt Engine UI.=C2=A0</div><div>All good, and I w=
as able to import more VMs to it.=C2=A0</div><div>I was also able to shutdo=
wn a VM on host01 assign it to host03 and start the VM. Cool, everything wo=
rking.=C2=A0</div><div>The above was all last couple of weeks.=C2=A0</div><=
div><br /></div><div>This week I performed some yum updates on the Engine V=
M. No reboot.=C2=A0</div><div>Today noticed that the Ovirt services in the=
 Engine VM were in a endless restart loop. They would be up for a 5 minutes=
 and then die.=C2=A0</div><div>Looking into=C2=A0/var/log/ovirt-engine/eng<w=
br>ine.log and I could only see errors relating to host02. Ovirt was trying =
to find it and failing. Then falling over.=C2=A0</wbr></div><div>I ran "ho=
sted-engine --clean-metadata" thinking it would cleanup and remove bad refe=
rences to hosts, but now realise that was a really bad idea as it didn't do =
what I'd hoped.=C2=A0</div><div>At this point the sequence below worked, I =
could login to Ovirt UI but after 5 minutes the services would be off</div=
...
<div><div>
<span style=3D"color:rgb(34,34,34);font-family:Verdana,Arial,Helvetica,sans=
-serif;font-variant-ligatures:normal;background-color:rgb(255,255,255)">ser=
vice ovirt-engine restart</span></div></div><div><span style=3D"color:rgb(3=
4,34,34);font-family:Verdana,Arial,Helvetica,sans-serif;font-variant-ligatu=
res:normal;background-color:rgb(255,255,255)"><div>

<span style=3D"font-variant-ligatures:normal">service ovirt-websocket-proxy =
restart</span></div><div><span style=3D"font-variant-ligatures:normal"><di=
v>

<span style=3D"font-variant-ligatures:normal">service httpd restart</span><=
/div></span></div></span></div><div><br /></div><div>I saw some reference t=
o having to remove hosts from the database by hand in situations where unde=
r the hood of Ovirt a decommission host was still listed, but wasn't showin=
g in the GUI.=C2=A0</div><div>So I removed reference to host02 (vds_id and=
 host_id) in the following tables in this order.=C2=A0</div><div><span style=
=3D"font-size:12pt">vds_dynamic</span></div><div><div id=3D"m_-869654554590=
4220606m_-2002812221470261923x0ec1289986994fb5ac309827edd33abe">

<div><div><span style=3D"font-size:12pt">vds_statistics</span></div><div><d=
iv id=3D"m_-8696545545904220606m_-2002812221470261923x87d978c56880415e9594d=
7fe317f30f3">

<div><div><div id=3D"m_-8696545545904220606m_-2002812221470261923x0ec128998=
6994fb5ac309827edd33abe"><div><div>vds_static</div><div></div></div></div><=
/div></div></div></div><div></div></div></div></div><div>host_device</div><=
div><br /></div><div>Now when I try to start ovirt-websocket it will not st=
art</div><div>service ovirt-websocket start<br />Redirecting to /bin/system=
ctl start =C2=A0ovirt-websocket.service<br />Failed to start ovirt-websocke=
t.service: Unit not found.</div><div><br /></div><div>I'm now thinking that =
I need to do the following in the engine VM</div><div><div>
<pre class=3D"m_-8696545545904220606m_-2002812221470261923highlight m_-8696=
545545904220606m_-2002812221470261923plaintext"><code>#=C2=A0engine-cleanup
#=C2=A0yum=C2=A0remove=C2=A0ovirt-engine
#=C2=A0yum=C2=A0install=C2=A0ovirt-engine
#=C2=A0engine-setup=C2=A0</code></pre></div></div><div>But to run engine-cl=
eanup I need to put the engine-vm into maintenance mode and because of the=
 --clean-metadata that I ran earlier on host01 I cannot do that.=C2=A0</div>=
<div><br /></div><div>What is the best course of action from here?</div></d=
iv></blockquote><div><br /></div></div></div><span class=3D""><div>To be ho=
nest, with all the steps taken above, I'd install everything (including OS) =
from scratch...</div><div>There's a bit too much mess to try to clean up p=
roperly here.</div><div>Y.</div><div>=C2=A0</div><blockquote class=3D"gmail=
_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:=
1ex"><div><div><br /></div><div id=3D"m_-8696545545904220606m_-200281222147=
0261923signature_old"><div id=3D"m_-8696545545904220606m_-20028122214702619=
23xbaf36d997b7f497">

<div class=3D"m_-8696545545904220606m_-2002812221470261923WordSection1">

<div id=3D"m_-8696545545904220606m_-2002812221470261923xb71224f920234978acc=
74f4d23143069"><div class=3D"m_-8696545545904220606m_-2002812221470261923Wo=
rdSection1"><p class=3D"MsoNormal"><font face=3D"Tahoma"><span style=3D"fon=
t-size:16px">Cheers</span></font></p><span class=3D"m_-8696545545904220606H=
OEnZb"><font color=3D"#888888"><p class=3D"MsoNormal"><br /></p><p class=3D=
"MsoNormal"><font face=3D"Tahoma"><span style=3D"font-size:16px">Andrew</sp=
an></font></p></font></span></div></div></div></div></div>
</div><br />______________________________<wbr>_________________<br />
Users mailing list<br />
<a href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a><br />
<a href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer=
">http://lists.ovirt.org/mailman<wbr>/listinfo/users</wbr></a><br />
<br /></wbr></blockquote></span></div><br /></div></div>

</div></blockquote></div></blockquote></div><br /></div></div>
</blockquote></div>
</body></html>
--------=_MB3BBF1E10-B89F-4246-8ACD-AB3F4F14DC3C--