hosted-engine VM and services not working

Friday, 16 June 2017

--------=_MBFFB0EF4D-0CCB-43B8-B6B6-961FF15C7403
Content-Type: text/plain; format=flowed; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Hi

Well I've got myself into a fine mess.

host01 was setup with hosted-engine v4.1. This was successful.
Imported 3 VMs from a v3.6 OVirt AIO instance. (This OVirt 3.6 is still=20
running with more VMs on it)
Tried to add host02 to the new Ovirt 4.1 setup. This partially succeeded=20
but I couldn't add any storage domains to it. Cannot remember why.
In Ovirt engine UI I removed host02.
I reinstalled host02 with Centos7, tried to add it and Ovirt UI told me=20
it was already there (but it wasn't listed in the UI).
Renamed the reinstalled host02 to host03, changed the ipaddress,=20
reconfig the DNS server and added host03 into the Ovirt Engine UI.
All good, and I was able to import more VMs to it.
I was also able to shutdown a VM on host01 assign it to host03 and start=20
the VM. Cool, everything working.
The above was all last couple of weeks.

This week I performed some yum updates on the Engine VM. No reboot.
Today noticed that the Ovirt services in the Engine VM were in a endless=20
restart loop. They would be up for a 5 minutes and then die.
Looking into /var/log/ovirt-engine/engine.log and I could only see=20
errors relating to host02. Ovirt was trying to find it and failing. Then=20
falling over.
I ran "hosted-engine --clean-metadata" thinking it would cleanup and=20
remove bad references to hosts, but now realise that was a really bad=20
idea as it didn't do what I'd hoped.
At this point the sequence below worked, I could login to Ovirt UI but=20
after 5 minutes the services would be off
service ovirt-engine restart
service ovirt-websocket-proxy restart
service httpd restart

I saw some reference to having to remove hosts from the database by hand=20
in situations where under the hood of Ovirt a decommission host was=20
still listed, but wasn't showing in the GUI.
So I removed reference to host02 (vds_id and host_id) in the following=20
tables in this order.
vds_dynamic
vds_statistics
vds_static
host_device

Now when I try to start ovirt-websocket it will not start
service ovirt-websocket start
Redirecting to /bin/systemctl start  ovirt-websocket.service
Failed to start ovirt-websocket.service: Unit not found.

I'm now thinking that I need to do the following in the engine VM
# engine-cleanup # yum remove ovirt-engine # yum install ovirt-engine #=20
engine-setup
But to run engine-cleanup I need to put the engine-vm into maintenance=20
mode and because of the --clean-metadata that I ran earlier on host01 I=20
cannot do that.

What is the best course of action from here?

Cheers

Andrew

--------=_MBFFB0EF4D-0CCB-43B8-B6B6-961FF15C7403
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<?xml version=3D"1.0"
encoding=3D"utf-16"?><html><head>

<style id=3D"signatureStyle"><!--#xbaf36d997b7f497, #xbaf36d997b7f497,
#xba=
f36d997b7f497, #xbaf36d997b7f497 #xb71224f920234978acc74f4d23143069, #xbaf3=
6d997b7f497
{font-family: Tahoma; font-size: 12pt;}
#xbaf36d997b7f497, #xbaf36d997b7f497
{font-family: 'Segoe UI'; font-size: 12pt;}
#xbaf36d997b7f497 #xb71224f920234978acc74f4d23143069 p.MsoNormal, #xbaf36d9=
97b7f497 p.MsoNormal
{margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-seri=
f;}
#xbaf36d997b7f497 #xb71224f920234978acc74f4d23143069 div.WordSection1, #xba=
f36d997b7f497 div.WordSection1
{page: WordSection1;}
#xbaf36d997b7f497 a:link
{color: rgb(5, 99, 193); text-decoration: underline;}
--></style>
<style><![CDATA[#x87d978c56880415e9594d7fe317f30f3 #x0ec1289986994fb5ac3098=
27edd33abe,#x87d978c56880415e9594d7fe317f30f3{
	font-family:Tahoma;
	font-size:12pt;
}#x0ec1289986994fb5ac309827edd33abe #xbaf36d997b7f497,#x0ec1289986994fb5ac3=
09827edd33abe #xbaf36d997b7f497,#x0ec1289986994fb5ac309827edd33abe #xbaf36d=
997b7f497,#x0ec1289986994fb5ac309827edd33abe #xbaf36d997b7f497 #xb71224f920=
234978acc74f4d23143069,#x0ec1289986994fb5ac309827edd33abe #xbaf36d997b7f497=
{
	font-family:Tahoma;
	font-size:12pt;
}
#x0ec1289986994fb5ac309827edd33abe #xbaf36d997b7f497,#x0ec1289986994fb5ac30=
9827edd33abe #xbaf36d997b7f497{
	font-family:'Segoe UI';
	font-size:12pt;
}
#x0ec1289986994fb5ac309827edd33abe #xbaf36d997b7f497 #xb71224f920234978acc7=
4f4d23143069 p.MsoNormal,#x0ec1289986994fb5ac309827edd33abe #xbaf36d997b7f4=
97 p.MsoNormal{
	margin:0cm 0cm 0.0001pt;
	font-size:11pt;
	font-family:Calibri,sans-serif;
}
#x0ec1289986994fb5ac309827edd33abe #xbaf36d997b7f497 #xb71224f920234978acc7=
4f4d23143069 div.WordSection1,#x0ec1289986994fb5ac309827edd33abe #xbaf36d99=
7b7f497 div.WordSection1{
	page:WordSection1;
}
#x0ec1289986994fb5ac309827edd33abe{
	font-family:Tahoma;
	font-size:12pt;
}]]><!--body
{font-family: Tahoma; font-size: 12pt;}
--></style>
</head>
<body><div>Hi</div><div><br /></div><div>Well
I've got myself into a fine m=
ess.=C2=A0</div><div><br /></div><div>host01 was setup with
hosted-engine v=
4.1.=C2=A0<span style=3D"font-size: 12pt;">This was
successful.=C2=A0</span=
...
</div><div><span style=3D"font-size:
12pt;">Imported 3 VMs from a v3.6 OVi= rt AIO instance. (This OVirt 3.6 is
still running with more VMs on it)</spa=
n></div><div>Tried to add host02 to the new Ovirt 4.1 setup. This partially
=
succeeded but I couldn't add any storage domains to it. Cannot remember wh=
y.=C2=A0</div><div>In Ovirt engine UI I removed
host02.=C2=A0</div><div>I r=
einstalled host02 with Centos7, tried to add it and Ovirt UI told me it was =
already there (but it wasn't listed in the UI).=C2=A0</div><div>Renamed
th=
e reinstalled host02 to host03, changed the ipaddress, reconfig the DNS ser=
ver and added host03 into the Ovirt Engine UI.=C2=A0</div><div>All good, an=
d I was able to import more VMs to it.=C2=A0</div><div>I was also able to s=
hutdown a VM on host01 assign it to host03 and start the VM. Cool, everythi=
ng working.=C2=A0</div><div>The above was all last couple of
weeks.=C2=A0</=
div><div><br /></div><div>This week I performed some yum
updates on the Eng=
ine VM. No reboot.=C2=A0</div><div>Today noticed that the Ovirt services in =
the Engine VM were in a endless restart loop. They would be up for a 5 min=
utes and then die.=C2=A0</div><div>Looking into=C2=A0/var/log/ovirt-engine/=
engine.log and I could only see errors relating to host02. Ovirt was trying =
to find it and failing. Then falling over.=C2=A0</div><div>I ran
"hosted-e=
ngine --clean-metadata" thinking it would cleanup and remove bad references =
to hosts, but now realise that was a really bad idea as it didn't do what=
 I'd hoped.=C2=A0</div><div>At this point the sequence below worked, I
could =
login to Ovirt UI but after 5 minutes the services would be off</div><div>=
<div>

<span style=3D"color: rgb(34, 34, 34); font-family: Verdana, Arial, Helveti=
ca, sans-serif; font-variant-ligatures: normal; orphans: 2; widows: 2; back=
ground-color: rgb(255, 255, 255);">service ovirt-engine
restart</span></div=
...
</div><div><span style=3D"color: rgb(34, 34, 34);
font-family: Verdana, Ar= ial, Helvetica, sans-serif; font-variant-ligatures:
normal; orphans: 2; wid=
ows: 2; background-color: rgb(255, 255, 255);"...
<div>

<span style=3D"font-variant-ligatures: normal;">service
ovirt-websocket-pro=
xy restart</span></div><div><span
style=3D"font-variant-ligatures: normal;"=
...
<div> 
<span style=3D"font-variant-ligatures: normal;">service httpd
restart</span=
...
</div></span></div></span></div><div><br
/></div><div>I saw some reference = to having to remove hosts from
the database by hand in situations where un=
der the hood of Ovirt a decommission host was still listed, but wasn't show=
ing in the GUI.=C2=A0</div><div>So I removed reference to host02 (vds_id an=
d host_id) in the following tables in this order.=C2=A0</div><div><span
sty=
le=3D"font-size:
12pt;">vds_dynamic</span></div><div><div
id=3D"x0ec1289986=
994fb5ac309827edd33abe">

<div><div><span style=3D"font-size:
12pt;">vds_statistics</span></div><div>=
<div id=3D"x87d978c56880415e9594d7fe317f30f3">

<div><div><div
id=3D"x0ec1289986994fb5ac309827edd33abe"><div><div>vds_stati=
c</div><div></div></div></div></div></div></div></div><div></div></div></di=
v></div><div>host_device</div><div><br
/></div><div>Now when I try to start =
ovirt-websocket it will not start</div><div>service ovirt-websocket
start<=
br />Redirecting to /bin/systemctl start =C2=A0ovirt-websocket.service<br /=
...
Failed to start ovirt-websocket.service: Unit not
found.</div><div><br /><= /div><div>I'm now
thinking that I need to do the following in the engine VM=
</div><div...
<div> <pre
class=3D"highlight plaintext"><code>#=C2=A0engine-cleanup
#=C2=A0yum=C2=A0remove=C2=A0ovirt-engine
#=C2=A0yum=C2=A0install=C2=A0ovirt-engine
#=C2=A0engine-setup=C2=A0</code></pre></div></div><div>But
to run engine-cl=
eanup I need to put the engine-vm into maintenance mode and because of the=
 --clean-metadata that I ran earlier on host01 I cannot do that.=C2=A0</div>=
<div><br /></div><div>What is the best course of action from
here?</div><di=
v><br /></div><div id=3D"signature_old"><div
id=3D"xbaf36d997b7f497">

<div class=3D"WordSection1">

<div id=3D"xb71224f920234978acc74f4d23143069"><div
class=3D"WordSection1"><=
p class=3D"MsoNormal"><font face=3D"Tahoma"><span
style=3D"font-size: 16px;=
">Cheers</span></font></p><p
class=3D"MsoNormal"><br /></p><p class=3D"MsoN=
ormal"><font face=3D"Tahoma"><span style=3D"font-size:
16px;">Andrew</span>=
</font></p></div></div></div></div></div>
</body></html>
--------=_MBFFB0EF4D-0CCB-43B8-B6B6-961FF15C7403--

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011