10:53 a.m.
--Apple-Mail=_C0B0F920-BBED-4F8E-B722-72D6ED3D1454
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=utf-8
On 09 Aug 2016, at 19:14, Nicol=C3=A1s <nicolas(a)devels.es>
wrote:
=20
Hi,
=20
It worked (thanks Ekin, I'd probably had not turned it off and it =
would
indeed been restarted), restarting libvirtd and vdsmd made all =
machines set their status to up. Seems the cuprit is libvirtd in this =
case, as I could see some errors in the log. I'm attaching the log FWIW.
Hi,
I suppose you use RHEL/CentOS, a libvirt responsiveness issue was fixed =
in https://rhn.redhat.com/errata/RHBA-2016-1290.html
Let us know if the same thing reproduces again
Thanks,
michal
=20
Thanks.
=20
El 07/08/16 a las 19:37, Ekin Mero=C4=9Flu escribi=C3=B3:
> Hi,
>=20
> Just a reminder, if you have power management configured, first turn =
that off
for the host - when you restart vdsmd with the power management =
configured, engine finds it not responding and tries to fence (e.g. =
reboot) the host.
>=20
> Other than that, restarting vdsmd has been safe in my experience...
>=20
> Regards, =20
>=20
> On Thu, Aug 4, 2016 at 6:10 PM, Nicol=C3=A1s <nicolas(a)devels.es =
<mailto:nicolas@devels.es>> wrote:
>=20
>=20
> El 04/08/16 a las 15:25, Arik Hadas escribi=C3=B3:
>=20
> ----- Original Message -----
> El 2016-08-04 08:24, Arik Hadas escribi=C3=B3:
> ----- Original Message -----
>=20
> El 04/08/16 a las 07:18, Arik Hadas escribi=C3=B3:
> ----- Original Message -----
> Hi,
>=20
> We're running oVirt 4.0.1 and today I found out that one of our hosts
> has all its VMs in an unknown state. I actually don't know how (and
> when) did this happen, but I'd like to restore service possibly =
without
> turning off these machines. The host is up, the VMs are up,
'qemu'
> process exists, no errors, it's just the VMs running on it that have =
a
> '?' where status is defined.
>=20
> Is it safe in this case to simply modify database and set those VM's
> status to 'up'? I remember having to do this a time ago when we faced
> storage issues, it didn't break anything back then. If not, is there =
a
> "safe" way to migrate those VMs to a different host and
restart the
> host
> that marked them as unknown?
> Hi Nicol=C3=A1s,
>=20
> I assume that the host these VMs are running on is empty in the
> webadmin,
> right? if that is the case then you've probably hit [1]. Changing =
their
> status to up is not the way to go since these VMs will not be =
monitored.
> Hi Arik,
>=20
> By "empty" you mean the webadmin reports the host being running 0 =
VMs?
> If so, that's not the case, actually the VM count seems to be
correct
> in
> relation to "qemu-*" processes (about 32 VMs), I can even see the
> machines in the "Virtual machines" tab of the host, it's just they =
are
> all marked with the '?' mark.
> No, I meant the 'Host' column in the Virtual Machines tab but if you
> see
> the VMs in the "Virtual machines" sub-tab of the host then run_on_vds
> points to the right host..
>=20
> The host is up in the webadmin as well?
> Can you share the engine log?
>=20
> Yes, the host is up in the webadmin, there are no issues with it, =
just
> the VMs running on it have the '?' mark. I've made 3
tests:
>=20
> 1) Restart engine: did not help
> 2) Check firewall, seems to be ok.
> 2) PostgreSQL: UPDATE vm_dynamic SET status =3D 1 WHERE status =3D 8; =
:
> After a while, I see lots of entries like this:
>=20
> 2016-08-04 09:23:10,910 WARN
> =
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (DefaultQuartzScheduler4) [6ad135b8] Correlation ID: null, Call =
Stack:
> null, Custom Event ID: -1, Message: VM xxx is not responding.
>=20
> I'm attaching the engine log, but I don't know when did this happen =
for
> the first time, though. If there's a manual way/command to
migrate =
VMs
> to a different host I'd appreciate a hint about it.
>=20
> Is it safe to restart vdsmd on this host?
> The engine log looks fine - the VMs are reported as not-responding =
for
> some reason. I would restart libvirtd and vdsmd then
>=20
> Is restarting those two daemons safe? I mean, will that stop all =
qemu-*
processes, so the VMs marked as unknown will stop?
>=20
>=20
> Thanks.
>=20
> Thanks.
>=20
> Yes, there is no other way to resolve it other than changing the DB =
but
> the change should be to update run_on_vds field of these VMs to
the =
host
> you know they are running on. Their status will then be updates
in 15
> sec.
>=20
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=3D1354494 =
<https://bugzilla.redhat.com/show_bug.cgi?id=3D1354494>
>=20
> Arik.
>=20
> Thanks.
>=20
> Nicol=C3=A1s
>=20
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/users =
<http://lists.ovirt.org/mailman/listinfo/users>
>=20
>=20
>=20
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/users =
<http://lists.ovirt.org/mailman/listinfo/users>
>=20
>=20
>=20
> --=20
> Ekin Mero=C4=9Flu Red Hat Certified Architect=20
>=20
> linuxera =C3=96zg=C3=BCr Yaz=C4=B1l=C4=B1m =C3=87=C3=B6z=C3=BCm ve =
Hizmetleri=20
> T +90 (850) 22 LINUX | GSM +90 (532) 137 77 04
> www.linuxera.com <http://www.linuxera.com/> | bilgi(a)linuxera.com =
<mailto:bilgi@linuxera.com>
<libvirtd.tar.gz>_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--Apple-Mail=_C0B0F920-BBED-4F8E-B722-72D6ED3D1454
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=utf-8
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><br class=3D""><div><blockquote
type=3D"cite" class=3D""><div =
class=3D"">On 09 Aug 2016, at 19:14, Nicol=C3=A1s <<a =
href=3D"mailto:nicolas@devels.es"
class=3D"">nicolas(a)devels.es</a>&gt; =
wrote:</div><br class=3D"Apple-interchange-newline"><div
class=3D"">
=20
<meta content=3D"text/html; charset=3Dutf-8" =
http-equiv=3D"Content-Type" class=3D"">
=20
<div bgcolor=3D"#FFFFFF" text=3D"#000000"
class=3D"">
Hi,<br class=3D"">
<br class=3D"">
It worked (thanks Ekin, I'd probably had not turned it off and it
would indeed been restarted), restarting libvirtd and vdsmd made all
machines set their status to up. Seems the cuprit is libvirtd in
this case, as I could see some errors in the log. I'm attaching the
log FWIW.<br
class=3D""></div></div></blockquote><div><br =
class=3D""></div><div>Hi,</div><div>I suppose you use
RHEL/CentOS, a =
libvirt responsiveness issue was fixed in <a =
href=3D"https://rhn.redhat.com/errata/RHBA-2016-1290.html" =
class=3D"">https://rhn.redhat.com/errata/RHBA-2016-1290.html...
v><div>Let us know if the same thing reproduces
again</div><div><br =
class=3D""></div><div>Thanks,</div><div>michal</div><div><br
=
class=3D""><blockquote type=3D"cite"
class=3D""><div class=3D""><div =
bgcolor=3D"#FFFFFF" text=3D"#000000" class=3D"">
<br class=3D"">
Thanks.<br class=3D"">
<br class=3D"">
<div class=3D"moz-cite-prefix">El 07/08/16 a las 19:37, Ekin =
Mero=C4=9Flu
escribi=C3=B3:<br class=3D"">
</div>
<blockquote =
cite=3D"mid:CAA0tEyOBT7KH+Xd2NMKpKY489DDbvS3wgfbXTXJi1vCCvoQNGA@mail.gmail=
.com" type=3D"cite" class=3D"">
<div dir=3D"ltr" class=3D"">
<div class=3D"gmail_default" =
style=3D"font-family:tahoma,sans-serif;font-size:small">Hi,</div>
<div class=3D"gmail_default" =
style=3D"font-family:tahoma,sans-serif;font-size:small"><br
class=3D"">
</div>
<div class=3D"gmail_default" =
style=3D"font-family:tahoma,sans-serif;font-size:small">Just a
reminder, if you have power management configured, first turn
that off for the host - when you restart vdsmd with the power
management configured, engine finds it not responding and
tries to fence (e.g. reboot) the host.</div>
<div class=3D"gmail_default" =
style=3D"font-family:tahoma,sans-serif;font-size:small"><br
class=3D"">
</div>
<div class=3D"gmail_default" =
style=3D"font-family:tahoma,sans-serif;font-size:small">Other
than that, restarting vdsmd has been safe in my =
experience...</div>
<div class=3D"gmail_default" =
style=3D"font-family:tahoma,sans-serif;font-size:small"><br
class=3D"">
</div>
<div class=3D"gmail_default" =
style=3D"font-family:tahoma,sans-serif;font-size:small">Regards,
</div>
</div>
<div class=3D"gmail_extra"><br class=3D"">
<div class=3D"gmail_quote">On Thu, Aug 4, 2016 at 6:10 PM, =
Nicol=C3=A1s
<span dir=3D"ltr" class=3D""><<a
moz-do-not-send=3D"true" =
href=3D"mailto:nicolas@devels.es" target=3D"_blank" =
class=3D"">nicolas(a)devels.es</a>&gt;</span>
wrote:<br class=3D"">
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex"><br =
class=3D"">
<br class=3D"">
El 04/08/16 a las 15:25, Arik Hadas escribi=C3=B3:<br =
class=3D"">
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class=3D"">
<div class=3D"h5">
<br class=3D"">
----- Original Message -----<br class=3D"">
<blockquote class=3D"gmail_quote" style=3D"margin:0 0
=
0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
El 2016-08-04 08:24, Arik Hadas escribi=C3=B3:<br =
class=3D"">
<blockquote class=3D"gmail_quote" style=3D"margin:0
=
0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
----- Original Message -----<br class=3D"">
<blockquote class=3D"gmail_quote"
style=3D"margin:0 =
0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<br class=3D"">
El 04/08/16 a las 07:18, Arik Hadas =
escribi=C3=B3:<br class=3D"">
<blockquote class=3D"gmail_quote" =
style=3D"margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
----- Original Message -----<br class=3D"">
<blockquote class=3D"gmail_quote" =
style=3D"margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
Hi,<br class=3D"">
<br class=3D"">
We're running oVirt 4.0.1 and today I found
out that one of our hosts<br class=3D"">
has all its VMs in an unknown state. I
actually don't know how (and<br
class=3D"">
when) did this happen, but I'd like to
restore service possibly without<br =
class=3D"">
turning off these machines. The host is up,
the VMs are up, 'qemu'<br class=3D"">
process exists, no errors, it's just the VMs
running on it that have a<br class=3D"">
'?' where status is defined.<br
class=3D"">
<br class=3D"">
Is it safe in this case to simply modify
database and set those VM's<br
class=3D"">
status to 'up'? I remember having to do this
a time ago when we faced<br class=3D"">
storage issues, it didn't break anything
back then. If not, is there a<br class=3D"">
"safe" way to migrate those VMs to a
different host and restart the<br class=3D"">
host<br class=3D"">
that marked them as unknown?<br class=3D"">
</blockquote>
Hi Nicol=C3=A1s,<br class=3D"">
<br class=3D"">
I assume that the host these VMs are running
on is empty in the<br class=3D"">
webadmin,<br class=3D"">
right? if that is the case then you've
probably hit [1]. Changing their<br class=3D"">
status to up is not the way to go since these
VMs will not be monitored.<br class=3D"">
</blockquote>
Hi Arik,<br class=3D"">
<br class=3D"">
By "empty" you mean the webadmin reports the
host being running 0 VMs?<br class=3D"">
If so, that's not the case, actually the VM
count seems to be correct<br class=3D"">
in<br class=3D"">
relation to "qemu-*" processes (about 32 VMs), I
can even see the<br class=3D"">
machines in the "Virtual machines" tab of the
host, it's just they are<br class=3D"">
all marked with the '?' mark.<br
class=3D"">
</blockquote>
No, I meant the 'Host' column in the Virtual
Machines tab but if you<br class=3D"">
see<br class=3D"">
the VMs in the "Virtual machines" sub-tab of the
host then run_on_vds<br class=3D"">
points to the right host..<br class=3D"">
<br class=3D"">
The host is up in the webadmin as well?<br =
class=3D"">
Can you share the engine log?<br class=3D"">
<br class=3D"">
</blockquote>
Yes, the host is up in the webadmin, there are no
issues with it, just<br class=3D"">
the VMs running on it have the '?' mark. I've made 3
tests:<br class=3D"">
<br class=3D"">
1) Restart engine: did not help<br class=3D"">
2) Check firewall, seems to be ok.<br class=3D"">
2) PostgreSQL: UPDATE vm_dynamic SET status =3D 1
WHERE status =3D 8; :<br class=3D"">
After a while, I see lots of entries like this:<br =
class=3D"">
<br class=3D"">
2016-08-04 09:23:10,910 WARN<br =
class=3D"">
[org.ovirt.engine.core.dal.dbb<wbr =
class=3D"">roker.auditloghandling.AuditLo<wbr
class=3D"">gDirector]<br =
class=3D"">
(DefaultQuartzScheduler4) [6ad135b8] Correlation ID:
null, Call Stack:<br class=3D"">
null, Custom Event ID: -1, Message: VM xxx is not
responding.<br class=3D"">
<br class=3D"">
I'm attaching the engine log, but I don't know when
did this happen for<br class=3D"">
the first time, though. If there's a manual
way/command to migrate VMs<br class=3D"">
to a different host I'd appreciate a hint about =
it.<br class=3D"">
<br class=3D"">
Is it safe to restart vdsmd on this host?<br =
class=3D"">
</blockquote>
</div>
</div>
The engine log looks fine - the VMs are reported as
not-responding for<br class=3D"">
some reason. I would restart libvirtd and vdsmd then<br =
class=3D"">
</blockquote>
<br class=3D"">
Is restarting those two daemons safe? I mean, will that stop
all qemu-* processes, so the VMs marked as unknown will
stop?
<div class=3D"HOEnZb">
<div class=3D"h5"><br class=3D"">
<br class=3D"">
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<blockquote class=3D"gmail_quote" style=3D"margin:0 0
=
0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
Thanks.<br class=3D"">
<br class=3D"">
<blockquote class=3D"gmail_quote" style=3D"margin:0
=
0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<blockquote class=3D"gmail_quote"
style=3D"margin:0 =
0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
Thanks.<br class=3D"">
<br class=3D"">
<blockquote class=3D"gmail_quote" =
style=3D"margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
Yes, there is no other way to resolve it other
than changing the DB but<br class=3D"">
the change should be to update run_on_vds
field of these VMs to the host<br class=3D"">
you know they are running on. Their status
will then be updates in 15<br class=3D"">
sec.<br class=3D"">
<br class=3D"">
[1] <a moz-do-not-send=3D"true" =
href=3D"https://bugzilla.redhat.com/show_bug.cgi?id=3D1354494" =
rel=3D"noreferrer" target=3D"_blank" =
class=3D"">https://bugzilla.redhat.com/sh<wbr =
class=3D"">ow_bug.cgi?id=3D1354494</a><br class=3D"">
<br class=3D"">
Arik.<br class=3D"">
<br class=3D"">
<blockquote class=3D"gmail_quote" =
style=3D"margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
Thanks.<br class=3D"">
<br class=3D"">
Nicol=C3=A1s<br class=3D"">
<br class=3D"">
______________________________<wbr =
class=3D"">_________________<br class=3D"">
Users mailing list<br class=3D"">
<a moz-do-not-send=3D"true" =
href=3D"mailto:Users@ovirt.org" target=3D"_blank" =
class=3D"">Users(a)ovirt.org</a><br class=3D"">
<a moz-do-not-send=3D"true" =
href=3D"http://lists.ovirt.org/mailman/listinfo/users"
rel=3D"noreferrer" =
target=3D"_blank" class=3D"">http://lists.ovirt.org/mailman<wbr
=
class=3D"">/listinfo/users</a><br class=3D"">
<br class=3D"">
</blockquote>
</blockquote>
<br class=3D"">
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<br class=3D"">
______________________________<wbr =
class=3D"">_________________<br class=3D"">
Users mailing list<br class=3D"">
<a moz-do-not-send=3D"true" =
href=3D"mailto:Users@ovirt.org" target=3D"_blank" =
class=3D"">Users(a)ovirt.org</a><br class=3D"">
<a moz-do-not-send=3D"true" =
href=3D"http://lists.ovirt.org/mailman/listinfo/users"
rel=3D"noreferrer" =
target=3D"_blank" class=3D"">http://lists.ovirt.org/mailman<wbr
=
class=3D"">/listinfo/users</a><br class=3D"">
</div>
</div>
</blockquote>
</div>
<br class=3D"">
<br clear=3D"all" class=3D"">
<div class=3D""><br class=3D"">
</div>
-- <br class=3D"">
<div class=3D"gmail_signature"
data-smartmail=3D"gmail_signature">=
<div dir=3D"ltr" class=3D"">
<div class=3D"">
<div dir=3D"ltr" class=3D"">
<div class=3D"">
<div dir=3D"ltr" class=3D"">
<div class=3D"">
<div dir=3D"ltr" class=3D"">
<div dir=3D"ltr" class=3D"">
<div dir=3D"ltr" class=3D"">
<div dir=3D"ltr" class=3D"">
<div dir=3D"ltr" class=3D"">
<div style=3D"text-align:left"
class=3D"">=
<table style=3D"font-family:Ubuntu" =
class=3D"">
<tbody class=3D"">
<tr class=3D"">
<td class=3D""><img =
moz-do-not-send=3D"true" =
src=3D"http://www.linuxera.com/files/RH_CertifiedArchitect_CMYK_smal...
class=3D""></td>
<td valign=3D"top" =
class=3D""><font face=3D"arial, helvetica,
sans-serif"
class=3D""><strong=
style=3D"font-size:small" class=3D"">Ekin
Mero=C4=9Flu</strong><i =
style=3D"font-size:small" class=3D""> Red
Hat Certified =
Architect</i><font size=3D"2"
class=3D""> </font><br class=3D"">
</font><font face=3D"arial,
helvetica, sans-serif" =
class=3D""><br class=3D"">
<font size=3D"2" =
class=3D"">linuxera
=C3=96zg=C3=BCr Yaz=C4=B1l=C4=
=B1m =C3=87=C3=B6z=C3=BCm ve
Hizmetleri </font><br =
class=3D"">
<strong =
style=3D"font-size:small" class=3D"">T</strong><font
size=3D"2" =
class=3D""> +90 (850) 22
=
LINUX | </font><strong style=3D"font-size:small" =
class=3D"">GSM</strong><font size=3D"2"
class=3D""> +90 (532) 137 =
77
04</font><br
class=3D"">
<font size=3D"2"
class=3D""><a=
moz-do-not-send=3D"true" href=3D"http://www.linuxera.com/" =
target=3D"_blank" class=3D"">www.linuxera.com</a>
| <a =
moz-do-not-send=3D"true" href=3D"mailto:bilgi@linuxera.com" =
target=3D"_blank"
class=3D"">bilgi(a)linuxera.com</a></font></font></td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br class=3D"">
</div>
<span =
id=3D"cid:76301FB7-7C1B-4C58-8D45-6F5F4DEE8F7A@brq.redhat.com"><libvirt=
d.tar.gz></span>_______________________________________________<br =
class=3D"">Users mailing list<br class=3D""><a =
href=3D"mailto:Users@ovirt.org"
class=3D"">Users(a)ovirt.org</a><br =
class=3D"">http://lists.ovirt.org/mailman/listinfo/users<br =
class=3D""></div></blockquote></div><br
class=3D""></body></html>=
--Apple-Mail=_C0B0F920-BBED-4F8E-B722-72D6ED3D1454--
![src=3D"http://www.linuxera.com/files/RH_CertifiedArchitect_CMYK_smal...](http://src=3D"http://www.linuxera.com/files/RH_CertifiedArchitect_CMYK_small.png%22=){kind=link}