Re: [ovirt-users] Migrate machines in unknown state?
El 04/08/16 a las 15:25, Arik Hadas escribió:
----- Original Message -----
El 2016-08-04 08:24, Arik Hadas escribió:
----- Original Message -----
El 04/08/16 a las 07:18, Arik Hadas escribió:
----- Original Message -----
Hi,
We're running oVirt 4.0.1 and today I found out that one of our hosts has all its VMs in an unknown state. I actually don't know how (and when) did this happen, but I'd like to restore service possibly without turning off these machines. The host is up, the VMs are up, 'qemu' process exists, no errors, it's just the VMs running on it that have a '?' where status is defined.
Is it safe in this case to simply modify database and set those VM's status to 'up'? I remember having to do this a time ago when we faced storage issues, it didn't break anything back then. If not, is there a "safe" way to migrate those VMs to a different host and restart the host that marked them as unknown? Hi Nicolás,
I assume that the host these VMs are running on is empty in the webadmin, right? if that is the case then you've probably hit [1]. Changing their status to up is not the way to go since these VMs will not be monitored. Hi Arik,
By "empty" you mean the webadmin reports the host being running 0 VMs? If so, that's not the case, actually the VM count seems to be correct in relation to "qemu-*" processes (about 32 VMs), I can even see the machines in the "Virtual machines" tab of the host, it's just they are all marked with the '?' mark.
No, I meant the 'Host' column in the Virtual Machines tab but if you see the VMs in the "Virtual machines" sub-tab of the host then run_on_vds points to the right host..
The host is up in the webadmin as well? Can you share the engine log?
Yes, the host is up in the webadmin, there are no issues with it, just the VMs running on it have the '?' mark. I've made 3 tests:
1) Restart engine: did not help 2) Check firewall, seems to be ok. 2) PostgreSQL: UPDATE vm_dynamic SET status = 1 WHERE status = 8; : After a while, I see lots of entries like this:
2016-08-04 09:23:10,910 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6ad135b8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM xxx is not responding.
I'm attaching the engine log, but I don't know when did this happen for the first time, though. If there's a manual way/command to migrate VMs to a different host I'd appreciate a hint about it.
Is it safe to restart vdsmd on this host? The engine log looks fine - the VMs are reported as not-responding for some reason. I would restart libvirtd and vdsmd then
Is restarting those two daemons safe? I mean, will that stop all qemu-* processes, so the VMs marked as unknown will stop?
Thanks.
Thanks.
Yes, there is no other way to resolve it other than changing the DB but the change should be to update run_on_vds field of these VMs to the host you know they are running on. Their status will then be updates in 15 sec.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1354494
Arik.
Thanks.
Nicolás
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Thu, Aug 4, 2016 at 6:10 PM, Nicolás <nicolas@devels.es> wrote:
El 04/08/16 a las 15:25, Arik Hadas escribió:
----- Original Message -----
El 2016-08-04 08:24, Arik Hadas escribió:
----- Original Message -----
El 04/08/16 a las 07:18, Arik Hadas escribió:
----- Original Message -----
> Hi, > > We're running oVirt 4.0.1 and today I found out that one of our hosts > has all its VMs in an unknown state. I actually don't know how (and > when) did this happen, but I'd like to restore service possibly > without > turning off these machines. The host is up, the VMs are up, 'qemu' > process exists, no errors, it's just the VMs running on it that have > a > '?' where status is defined. > > Is it safe in this case to simply modify database and set those VM's > status to 'up'? I remember having to do this a time ago when we faced > storage issues, it didn't break anything back then. If not, is there > a > "safe" way to migrate those VMs to a different host and restart the > host > that marked them as unknown? > Hi Nicolás,
I assume that the host these VMs are running on is empty in the webadmin, right? if that is the case then you've probably hit [1]. Changing their status to up is not the way to go since these VMs will not be monitored.
Hi Arik,
By "empty" you mean the webadmin reports the host being running 0 VMs? If so, that's not the case, actually the VM count seems to be correct in relation to "qemu-*" processes (about 32 VMs), I can even see the machines in the "Virtual machines" tab of the host, it's just they are all marked with the '?' mark.
No, I meant the 'Host' column in the Virtual Machines tab but if you see the VMs in the "Virtual machines" sub-tab of the host then run_on_vds points to the right host..
The host is up in the webadmin as well? Can you share the engine log?
Yes, the host is up in the webadmin, there are no issues with it, just the VMs running on it have the '?' mark. I've made 3 tests:
1) Restart engine: did not help 2) Check firewall, seems to be ok. 2) PostgreSQL: UPDATE vm_dynamic SET status = 1 WHERE status = 8; : After a while, I see lots of entries like this:
2016-08-04 09:23:10,910 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6ad135b8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM xxx is not responding.
I'm attaching the engine log, but I don't know when did this happen for the first time, though. If there's a manual way/command to migrate VMs to a different host I'd appreciate a hint about it.
Is it safe to restart vdsmd on this host?
The engine log looks fine - the VMs are reported as not-responding for some reason. I would restart libvirtd and vdsmd then
Is restarting those two daemons safe? I mean, will that stop all qemu-* processes, so the VMs marked as unknown will stop?
Neither should touch the qemu process, but re-connect to it as they restart. Y.
Thanks.
Thanks.
Yes, there is no other way to resolve it other than changing the DB but
the change should be to update run_on_vds field of these VMs to the host you know they are running on. Their status will then be updates in 15 sec.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1354494
Arik.
Thanks. > > Nicolás > > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > >
Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi, Just a reminder, if you have power management configured, first turn that off for the host - when you restart vdsmd with the power management configured, engine finds it not responding and tries to fence (e.g. reboot) the host. Other than that, restarting vdsmd has been safe in my experience... Regards, On Thu, Aug 4, 2016 at 6:10 PM, Nicolás <nicolas@devels.es> wrote:
El 04/08/16 a las 15:25, Arik Hadas escribió:
----- Original Message -----
El 2016-08-04 08:24, Arik Hadas escribió:
----- Original Message -----
El 04/08/16 a las 07:18, Arik Hadas escribió:
----- Original Message -----
> Hi, > > We're running oVirt 4.0.1 and today I found out that one of our hosts > has all its VMs in an unknown state. I actually don't know how (and > when) did this happen, but I'd like to restore service possibly > without > turning off these machines. The host is up, the VMs are up, 'qemu' > process exists, no errors, it's just the VMs running on it that have > a > '?' where status is defined. > > Is it safe in this case to simply modify database and set those VM's > status to 'up'? I remember having to do this a time ago when we faced > storage issues, it didn't break anything back then. If not, is there > a > "safe" way to migrate those VMs to a different host and restart the > host > that marked them as unknown? > Hi Nicolás,
I assume that the host these VMs are running on is empty in the webadmin, right? if that is the case then you've probably hit [1]. Changing their status to up is not the way to go since these VMs will not be monitored.
Hi Arik,
By "empty" you mean the webadmin reports the host being running 0 VMs? If so, that's not the case, actually the VM count seems to be correct in relation to "qemu-*" processes (about 32 VMs), I can even see the machines in the "Virtual machines" tab of the host, it's just they are all marked with the '?' mark.
No, I meant the 'Host' column in the Virtual Machines tab but if you see the VMs in the "Virtual machines" sub-tab of the host then run_on_vds points to the right host..
The host is up in the webadmin as well? Can you share the engine log?
Yes, the host is up in the webadmin, there are no issues with it, just the VMs running on it have the '?' mark. I've made 3 tests:
1) Restart engine: did not help 2) Check firewall, seems to be ok. 2) PostgreSQL: UPDATE vm_dynamic SET status = 1 WHERE status = 8; : After a while, I see lots of entries like this:
2016-08-04 09:23:10,910 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6ad135b8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM xxx is not responding.
I'm attaching the engine log, but I don't know when did this happen for the first time, though. If there's a manual way/command to migrate VMs to a different host I'd appreciate a hint about it.
Is it safe to restart vdsmd on this host?
The engine log looks fine - the VMs are reported as not-responding for some reason. I would restart libvirtd and vdsmd then
Is restarting those two daemons safe? I mean, will that stop all qemu-* processes, so the VMs marked as unknown will stop?
Thanks.
Thanks.
Yes, there is no other way to resolve it other than changing the DB but
the change should be to update run_on_vds field of these VMs to the host you know they are running on. Their status will then be updates in 15 sec.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1354494
Arik.
Thanks. > > Nicolás > > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > >
Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- *Ekin Meroğlu** Red Hat Certified Architect* linuxera Özgür Yazılım Çözüm ve Hizmetleri *T* +90 (850) 22 LINUX | *GSM* +90 (532) 137 77 04 www.linuxera.com | bilgi@linuxera.com
Hi, It worked (thanks Ekin, I'd probably had not turned it off and it would indeed been restarted), restarting libvirtd and vdsmd made all machines set their status to up. Seems the cuprit is libvirtd in this case, as I could see some errors in the log. I'm attaching the log FWIW. Thanks. El 07/08/16 a las 19:37, Ekin Meroğlu escribió:
Hi,
Just a reminder, if you have power management configured, first turn that off for the host - when you restart vdsmd with the power management configured, engine finds it not responding and tries to fence (e.g. reboot) the host.
Other than that, restarting vdsmd has been safe in my experience...
Regards,
On Thu, Aug 4, 2016 at 6:10 PM, Nicolás <nicolas@devels.es <mailto:nicolas@devels.es>> wrote:
El 04/08/16 a las 15:25, Arik Hadas escribió:
----- Original Message -----
El 2016-08-04 08:24, Arik Hadas escribió:
----- Original Message -----
El 04/08/16 a las 07:18, Arik Hadas escribió:
----- Original Message -----
Hi,
We're running oVirt 4.0.1 and today I found out that one of our hosts has all its VMs in an unknown state. I actually don't know how (and when) did this happen, but I'd like to restore service possibly without turning off these machines. The host is up, the VMs are up, 'qemu' process exists, no errors, it's just the VMs running on it that have a '?' where status is defined.
Is it safe in this case to simply modify database and set those VM's status to 'up'? I remember having to do this a time ago when we faced storage issues, it didn't break anything back then. If not, is there a "safe" way to migrate those VMs to a different host and restart the host that marked them as unknown?
Hi Nicolás,
I assume that the host these VMs are running on is empty in the webadmin, right? if that is the case then you've probably hit [1]. Changing their status to up is not the way to go since these VMs will not be monitored.
Hi Arik,
By "empty" you mean the webadmin reports the host being running 0 VMs? If so, that's not the case, actually the VM count seems to be correct in relation to "qemu-*" processes (about 32 VMs), I can even see the machines in the "Virtual machines" tab of the host, it's just they are all marked with the '?' mark.
No, I meant the 'Host' column in the Virtual Machines tab but if you see the VMs in the "Virtual machines" sub-tab of the host then run_on_vds points to the right host..
The host is up in the webadmin as well? Can you share the engine log?
Yes, the host is up in the webadmin, there are no issues with it, just the VMs running on it have the '?' mark. I've made 3 tests:
1) Restart engine: did not help 2) Check firewall, seems to be ok. 2) PostgreSQL: UPDATE vm_dynamic SET status = 1 WHERE status = 8; : After a while, I see lots of entries like this:
2016-08-04 09:23:10,910 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6ad135b8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM xxx is not responding.
I'm attaching the engine log, but I don't know when did this happen for the first time, though. If there's a manual way/command to migrate VMs to a different host I'd appreciate a hint about it.
Is it safe to restart vdsmd on this host?
The engine log looks fine - the VMs are reported as not-responding for some reason. I would restart libvirtd and vdsmd then
Is restarting those two daemons safe? I mean, will that stop all qemu-* processes, so the VMs marked as unknown will stop?
Thanks.
Thanks.
Yes, there is no other way to resolve it other than changing the DB but the change should be to update run_on_vds field of these VMs to the host you know they are running on. Their status will then be updates in 15 sec.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1354494 <https://bugzilla.redhat.com/show_bug.cgi?id=1354494>
Arik.
Thanks.
Nicolás
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users>
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users>
-- *Ekin Meroğlu*/ Red Hat Certified Architect/
linuxera Özgür Yazılım Çözüm ve Hizmetleri *T* +90 (850) 22 LINUX | *GSM* +90 (532) 137 77 04 www.linuxera.com <http://www.linuxera.com> | bilgi@linuxera.com <mailto:bilgi@linuxera.com>
--Apple-Mail=_C0B0F920-BBED-4F8E-B722-72D6ED3D1454 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8
On 09 Aug 2016, at 19:14, Nicol=C3=A1s <nicolas@devels.es> wrote: =20 Hi, =20 It worked (thanks Ekin, I'd probably had not turned it off and it = would indeed been restarted), restarting libvirtd and vdsmd made all = machines set their status to up. Seems the cuprit is libvirtd in this = case, as I could see some errors in the log. I'm attaching the log FWIW.
=20 Thanks. =20 El 07/08/16 a las 19:37, Ekin Mero=C4=9Flu escribi=C3=B3:
Hi, =20 Just a reminder, if you have power management configured, first turn =
=20 Other than that, restarting vdsmd has been safe in my experience... =20 Regards, =20 =20 On Thu, Aug 4, 2016 at 6:10 PM, Nicol=C3=A1s <nicolas@devels.es = <mailto:nicolas@devels.es>> wrote: =20 =20 El 04/08/16 a las 15:25, Arik Hadas escribi=C3=B3: =20 ----- Original Message ----- El 2016-08-04 08:24, Arik Hadas escribi=C3=B3: ----- Original Message ----- =20 El 04/08/16 a las 07:18, Arik Hadas escribi=C3=B3: ----- Original Message ----- Hi, =20 We're running oVirt 4.0.1 and today I found out that one of our hosts has all its VMs in an unknown state. I actually don't know how (and when) did this happen, but I'd like to restore service possibly = without turning off these machines. The host is up, the VMs are up, 'qemu' process exists, no errors, it's just the VMs running on it that have = a '?' where status is defined. =20 Is it safe in this case to simply modify database and set those VM's status to 'up'? I remember having to do this a time ago when we faced storage issues, it didn't break anything back then. If not, is there = a "safe" way to migrate those VMs to a different host and restart the host that marked them as unknown? Hi Nicol=C3=A1s, =20 I assume that the host these VMs are running on is empty in the webadmin, right? if that is the case then you've probably hit [1]. Changing =
Hi, I suppose you use RHEL/CentOS, a libvirt responsiveness issue was fixed = in https://rhn.redhat.com/errata/RHBA-2016-1290.html Let us know if the same thing reproduces again Thanks, michal that off for the host - when you restart vdsmd with the power management = configured, engine finds it not responding and tries to fence (e.g. = reboot) the host. their
status to up is not the way to go since these VMs will not be = monitored. Hi Arik, =20 By "empty" you mean the webadmin reports the host being running 0 = VMs? If so, that's not the case, actually the VM count seems to be correct in relation to "qemu-*" processes (about 32 VMs), I can even see the machines in the "Virtual machines" tab of the host, it's just they = are all marked with the '?' mark. No, I meant the 'Host' column in the Virtual Machines tab but if you see the VMs in the "Virtual machines" sub-tab of the host then run_on_vds points to the right host.. =20 The host is up in the webadmin as well? Can you share the engine log? =20 Yes, the host is up in the webadmin, there are no issues with it, = just the VMs running on it have the '?' mark. I've made 3 tests: =20 1) Restart engine: did not help 2) Check firewall, seems to be ok. 2) PostgreSQL: UPDATE vm_dynamic SET status =3D 1 WHERE status =3D 8; = : After a while, I see lots of entries like this: =20 2016-08-04 09:23:10,910 WARN = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6ad135b8] Correlation ID: null, Call = Stack: null, Custom Event ID: -1, Message: VM xxx is not responding. =20 I'm attaching the engine log, but I don't know when did this happen = for the first time, though. If there's a manual way/command to migrate = VMs to a different host I'd appreciate a hint about it. =20 Is it safe to restart vdsmd on this host? The engine log looks fine - the VMs are reported as not-responding = for some reason. I would restart libvirtd and vdsmd then =20 Is restarting those two daemons safe? I mean, will that stop all = qemu-* processes, so the VMs marked as unknown will stop? =20 =20 Thanks. =20 Thanks. =20 Yes, there is no other way to resolve it other than changing the DB = but the change should be to update run_on_vds field of these VMs to the = host you know they are running on. Their status will then be updates in 15 sec. =20 [1] https://bugzilla.redhat.com/show_bug.cgi?id=3D1354494 = <https://bugzilla.redhat.com/show_bug.cgi?id=3D1354494> =20 Arik. =20 Thanks. =20 Nicol=C3=A1s =20 _______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users = <http://lists.ovirt.org/mailman/listinfo/users> =20 =20 =20 _______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users = <http://lists.ovirt.org/mailman/listinfo/users> =20 =20 =20 --=20 Ekin Mero=C4=9Flu Red Hat Certified Architect=20 =20 linuxera =C3=96zg=C3=BCr Yaz=C4=B1l=C4=B1m =C3=87=C3=B6z=C3=BCm ve = Hizmetleri=20 T +90 (850) 22 LINUX | GSM +90 (532) 137 77 04 www.linuxera.com <http://www.linuxera.com/> | bilgi@linuxera.com = <mailto:bilgi@linuxera.com> <libvirtd.tar.gz>_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--Apple-Mail=_C0B0F920-BBED-4F8E-B722-72D6ED3D1454 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On 09 Aug 2016, at 19:14, Nicol=C3=A1s <<a = href=3D"mailto:nicolas@devels.es" class=3D"">nicolas@devels.es</a>> = wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""> =20 <meta content=3D"text/html; charset=3Dutf-8" = http-equiv=3D"Content-Type" class=3D""> =20 <div bgcolor=3D"#FFFFFF" text=3D"#000000" class=3D""> Hi,<br class=3D""> <br class=3D""> It worked (thanks Ekin, I'd probably had not turned it off and it would indeed been restarted), restarting libvirtd and vdsmd made all machines set their status to up. Seems the cuprit is libvirtd in this case, as I could see some errors in the log. I'm attaching the log FWIW.<br class=3D""></div></div></blockquote><div><br = class=3D""></div><div>Hi,</div><div>I suppose you use RHEL/CentOS, a = libvirt responsiveness issue was fixed in <a = href=3D"https://rhn.redhat.com/errata/RHBA-2016-1290.html" = class=3D"">https://rhn.redhat.com/errata/RHBA-2016-1290.html</a></div></di= v><div>Let us know if the same thing reproduces again</div><div><br = class=3D""></div><div>Thanks,</div><div>michal</div><div><br = class=3D""><blockquote type=3D"cite" class=3D""><div class=3D""><div = bgcolor=3D"#FFFFFF" text=3D"#000000" class=3D""> <br class=3D""> Thanks.<br class=3D""> <br class=3D""> <div class=3D"moz-cite-prefix">El 07/08/16 a las 19:37, Ekin = Mero=C4=9Flu escribi=C3=B3:<br class=3D""> </div> <blockquote = cite=3D"mid:CAA0tEyOBT7KH+Xd2NMKpKY489DDbvS3wgfbXTXJi1vCCvoQNGA@mail.gmail= .com" type=3D"cite" class=3D""> <div dir=3D"ltr" class=3D""> <div class=3D"gmail_default" = style=3D"font-family:tahoma,sans-serif;font-size:small">Hi,</div> <div class=3D"gmail_default" = style=3D"font-family:tahoma,sans-serif;font-size:small"><br class=3D""> </div> <div class=3D"gmail_default" = style=3D"font-family:tahoma,sans-serif;font-size:small">Just a reminder, if you have power management configured, first turn that off for the host - when you restart vdsmd with the power management configured, engine finds it not responding and tries to fence (e.g. reboot) the host.</div> <div class=3D"gmail_default" = style=3D"font-family:tahoma,sans-serif;font-size:small"><br class=3D""> </div> <div class=3D"gmail_default" = style=3D"font-family:tahoma,sans-serif;font-size:small">Other than that, restarting vdsmd has been safe in my = experience...</div> <div class=3D"gmail_default" = style=3D"font-family:tahoma,sans-serif;font-size:small"><br class=3D""> </div> <div class=3D"gmail_default" = style=3D"font-family:tahoma,sans-serif;font-size:small">Regards, </div> </div> <div class=3D"gmail_extra"><br class=3D""> <div class=3D"gmail_quote">On Thu, Aug 4, 2016 at 6:10 PM, = Nicol=C3=A1s <span dir=3D"ltr" class=3D""><<a moz-do-not-send=3D"true" = href=3D"mailto:nicolas@devels.es" target=3D"_blank" = class=3D"">nicolas@devels.es</a>></span> wrote:<br class=3D""> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br = class=3D""> <br class=3D""> El 04/08/16 a las 15:25, Arik Hadas escribi=C3=B3:<br = class=3D""> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div class=3D""> <div class=3D"h5"> <br class=3D""> ----- Original Message -----<br class=3D""> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 = 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> El 2016-08-04 08:24, Arik Hadas escribi=C3=B3:<br = class=3D""> <blockquote class=3D"gmail_quote" style=3D"margin:0 = 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> ----- Original Message -----<br class=3D""> <blockquote class=3D"gmail_quote" style=3D"margin:0 = 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <br class=3D""> El 04/08/16 a las 07:18, Arik Hadas = escribi=C3=B3:<br class=3D""> <blockquote class=3D"gmail_quote" = style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> ----- Original Message -----<br class=3D""> <blockquote class=3D"gmail_quote" = style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Hi,<br class=3D""> <br class=3D""> We're running oVirt 4.0.1 and today I found out that one of our hosts<br class=3D""> has all its VMs in an unknown state. I actually don't know how (and<br class=3D""> when) did this happen, but I'd like to restore service possibly without<br = class=3D""> turning off these machines. The host is up, the VMs are up, 'qemu'<br class=3D""> process exists, no errors, it's just the VMs running on it that have a<br class=3D""> '?' where status is defined.<br class=3D""> <br class=3D""> Is it safe in this case to simply modify database and set those VM's<br class=3D""> status to 'up'? I remember having to do this a time ago when we faced<br class=3D""> storage issues, it didn't break anything back then. If not, is there a<br class=3D""> "safe" way to migrate those VMs to a different host and restart the<br class=3D""> host<br class=3D""> that marked them as unknown?<br class=3D""> </blockquote> Hi Nicol=C3=A1s,<br class=3D""> <br class=3D""> I assume that the host these VMs are running on is empty in the<br class=3D""> webadmin,<br class=3D""> right? if that is the case then you've probably hit [1]. Changing their<br class=3D""> status to up is not the way to go since these VMs will not be monitored.<br class=3D""> </blockquote> Hi Arik,<br class=3D""> <br class=3D""> By "empty" you mean the webadmin reports the host being running 0 VMs?<br class=3D""> If so, that's not the case, actually the VM count seems to be correct<br class=3D""> in<br class=3D""> relation to "qemu-*" processes (about 32 VMs), I can even see the<br class=3D""> machines in the "Virtual machines" tab of the host, it's just they are<br class=3D""> all marked with the '?' mark.<br class=3D""> </blockquote> No, I meant the 'Host' column in the Virtual Machines tab but if you<br class=3D""> see<br class=3D""> the VMs in the "Virtual machines" sub-tab of the host then run_on_vds<br class=3D""> points to the right host..<br class=3D""> <br class=3D""> The host is up in the webadmin as well?<br = class=3D""> Can you share the engine log?<br class=3D""> <br class=3D""> </blockquote> Yes, the host is up in the webadmin, there are no issues with it, just<br class=3D""> the VMs running on it have the '?' mark. I've made 3 tests:<br class=3D""> <br class=3D""> 1) Restart engine: did not help<br class=3D""> 2) Check firewall, seems to be ok.<br class=3D""> 2) PostgreSQL: UPDATE vm_dynamic SET status =3D 1 WHERE status =3D 8; :<br class=3D""> After a while, I see lots of entries like this:<br = class=3D""> <br class=3D""> 2016-08-04 09:23:10,910 WARN<br = class=3D""> [org.ovirt.engine.core.dal.dbb<wbr = class=3D"">roker.auditloghandling.AuditLo<wbr class=3D"">gDirector]<br = class=3D""> (DefaultQuartzScheduler4) [6ad135b8] Correlation ID: null, Call Stack:<br class=3D""> null, Custom Event ID: -1, Message: VM xxx is not responding.<br class=3D""> <br class=3D""> I'm attaching the engine log, but I don't know when did this happen for<br class=3D""> the first time, though. If there's a manual way/command to migrate VMs<br class=3D""> to a different host I'd appreciate a hint about = it.<br class=3D""> <br class=3D""> Is it safe to restart vdsmd on this host?<br = class=3D""> </blockquote> </div> </div> The engine log looks fine - the VMs are reported as not-responding for<br class=3D""> some reason. I would restart libvirtd and vdsmd then<br = class=3D""> </blockquote> <br class=3D""> Is restarting those two daemons safe? I mean, will that stop all qemu-* processes, so the VMs marked as unknown will stop? <div class=3D"HOEnZb"> <div class=3D"h5"><br class=3D""> <br class=3D""> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 = 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Thanks.<br class=3D""> <br class=3D""> <blockquote class=3D"gmail_quote" style=3D"margin:0 = 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <blockquote class=3D"gmail_quote" style=3D"margin:0 = 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Thanks.<br class=3D""> <br class=3D""> <blockquote class=3D"gmail_quote" = style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Yes, there is no other way to resolve it other than changing the DB but<br class=3D""> the change should be to update run_on_vds field of these VMs to the host<br class=3D""> you know they are running on. Their status will then be updates in 15<br class=3D""> sec.<br class=3D""> <br class=3D""> [1] <a moz-do-not-send=3D"true" = href=3D"https://bugzilla.redhat.com/show_bug.cgi?id=3D1354494" = rel=3D"noreferrer" target=3D"_blank" = class=3D"">https://bugzilla.redhat.com/sh<wbr = class=3D"">ow_bug.cgi?id=3D1354494</a><br class=3D""> <br class=3D""> Arik.<br class=3D""> <br class=3D""> <blockquote class=3D"gmail_quote" = style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Thanks.<br class=3D""> <br class=3D""> Nicol=C3=A1s<br class=3D""> <br class=3D""> ______________________________<wbr = class=3D"">_________________<br class=3D""> Users mailing list<br class=3D""> <a moz-do-not-send=3D"true" = href=3D"mailto:Users@ovirt.org" target=3D"_blank" = class=3D"">Users@ovirt.org</a><br class=3D""> <a moz-do-not-send=3D"true" = href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer" = target=3D"_blank" class=3D"">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/users</a><br class=3D""> <br class=3D""> </blockquote> </blockquote> <br class=3D""> </blockquote> </blockquote> </blockquote> </blockquote> <br class=3D""> ______________________________<wbr = class=3D"">_________________<br class=3D""> Users mailing list<br class=3D""> <a moz-do-not-send=3D"true" = href=3D"mailto:Users@ovirt.org" target=3D"_blank" = class=3D"">Users@ovirt.org</a><br class=3D""> <a moz-do-not-send=3D"true" = href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer" = target=3D"_blank" class=3D"">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/users</a><br class=3D""> </div> </div> </blockquote> </div> <br class=3D""> <br clear=3D"all" class=3D""> <div class=3D""><br class=3D""> </div> -- <br class=3D""> <div class=3D"gmail_signature" data-smartmail=3D"gmail_signature">= <div dir=3D"ltr" class=3D""> <div class=3D""> <div dir=3D"ltr" class=3D""> <div class=3D""> <div dir=3D"ltr" class=3D""> <div class=3D""> <div dir=3D"ltr" class=3D""> <div dir=3D"ltr" class=3D""> <div dir=3D"ltr" class=3D""> <div dir=3D"ltr" class=3D""> <div dir=3D"ltr" class=3D""> <div style=3D"text-align:left" class=3D"">= <table style=3D"font-family:Ubuntu" = class=3D""> <tbody class=3D""> <tr class=3D""> <td class=3D""><img = moz-do-not-send=3D"true" = src=3D"http://www.linuxera.com/files/RH_CertifiedArchitect_CMYK_small.png"= class=3D""></td> <td valign=3D"top" = class=3D""><font face=3D"arial, helvetica, sans-serif" class=3D""><strong= style=3D"font-size:small" class=3D"">Ekin Mero=C4=9Flu</strong><i = style=3D"font-size:small" class=3D""> Red Hat Certified = Architect</i><font size=3D"2" class=3D""> </font><br class=3D""> </font><font face=3D"arial, helvetica, sans-serif" = class=3D""><br class=3D""> <font size=3D"2" = class=3D"">linuxera =C3=96zg=C3=BCr Yaz=C4=B1l=C4= =B1m =C3=87=C3=B6z=C3=BCm ve Hizmetleri </font><br = class=3D""> <strong = style=3D"font-size:small" class=3D"">T</strong><font size=3D"2" = class=3D""> +90 (850) 22 = LINUX | </font><strong style=3D"font-size:small" = class=3D"">GSM</strong><font size=3D"2" class=3D""> +90 (532) 137 = 77 04</font><br class=3D""> <font size=3D"2" class=3D""><a= moz-do-not-send=3D"true" href=3D"http://www.linuxera.com/" = target=3D"_blank" class=3D"">www.linuxera.com</a> | <a = moz-do-not-send=3D"true" href=3D"mailto:bilgi@linuxera.com" = target=3D"_blank" class=3D"">bilgi@linuxera.com</a></font></font></td> </tr> </tbody> </table> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </blockquote> <br class=3D""> </div> <span = id=3D"cid:76301FB7-7C1B-4C58-8D45-6F5F4DEE8F7A@brq.redhat.com"><libvirt= d.tar.gz></span>_______________________________________________<br = class=3D"">Users mailing list<br class=3D""><a = href=3D"mailto:Users@ovirt.org" class=3D"">Users@ovirt.org</a><br = class=3D"">http://lists.ovirt.org/mailman/listinfo/users<br = class=3D""></div></blockquote></div><br class=3D""></body></html>= --Apple-Mail=_C0B0F920-BBED-4F8E-B722-72D6ED3D1454--
participants (4)
-
Ekin Meroğlu -
Michal Skrivanek -
Nicolás -
Yaniv Kaul