oVirt Node 4.2.5.1 vm stuck on "Powering Down"
Hello, vm stuck (on two hosts) I have MySQL VM guest that was stuck in a way that "kill -9" to the MySQL server pid did nothing. from the shell I execute "init 0", this did not shut down the VM. from the ovirt-engine I'm trying to power off the machine, and it is not doing anything. the event shows:
VDSM kvm5 command DestroyVDS failed: Virtual machine destroy error
and:
Failed to power off VM mysql-production (Host: kvm5, User: admin@internal-authz).
Note, when I list the VM s on the hosts view, I see that that same VM is on two hosts (kvm7 and kvm5). It looks like the VM is stuck on two hosts, in the status "Powering Down" any ideas? Thank you
On Wed, Jan 2, 2019 at 10:51 AM maoz zadok <maozza@gmail.com> wrote:
Hello, vm stuck (on two hosts) I have MySQL VM guest that was stuck in a way that "kill -9" to the MySQL server pid did nothing. from the shell I execute "init 0", this did not shut down the VM.
from the ovirt-engine I'm trying to power off the machine, and it is not doing anything. the event shows:
VDSM kvm5 command DestroyVDS failed: Virtual machine destroy error
and:
Failed to power off VM mysql-production (Host: kvm5, User: admin@internal-authz).
Note, when I list the VM s on the hosts view, I see that that same VM is on two hosts (kvm7 and kvm5). It looks like the VM is stuck on two hosts, in the status "Powering Down"
any ideas?
No idea about this specific problem and how to solve it. Is this on the same system you restored from backup? Is this VM marked as HA (highly-available)? Did it move to a different host since you took the backup you now restored? If all are 'yes', then it's likely that the engine tried to start it (because it's HA) before it noticed that it's already running on another host (a fact that it could not know, if it was different from what's in the restored database). Sadly, this very likely leads to corruption. See also: https://bugzilla.redhat.com/1441322 Since I do not see 'Resetting HA VM status' in the restore logs you posted, it might be a different issue, but perhaps similar. If indeed you see two qemu processes on the hosts, I'd try to kill -9 both of them. Adding Arik. Good luck,
Thank you _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/E2HKUJZYUNC3VC...
-- Didi
Thank you Didi for the fast reply! The problem was before the restore from backup (that was the reason that I reboot the engine and the engine didn't come up, I dug a bigger hole :-( ), HA is not marked. but The VM was trying to migrate from kvm7 to kvm5 before the engine crashed. I don't see the qemu process on any of the hosts. "vdsm-client Host getVMList fullStatus=True" show the machine uuid only on one (kvm5) I'll try restarting the host (kvm5), let's see if it helps. On Wed, Jan 2, 2019 at 11:07 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Wed, Jan 2, 2019 at 10:51 AM maoz zadok <maozza@gmail.com> wrote:
Hello, vm stuck (on two hosts) I have MySQL VM guest that was stuck in a way that "kill -9" to the
MySQL server pid did nothing.
from the shell I execute "init 0", this did not shut down the VM.
from the ovirt-engine I'm trying to power off the machine, and it is not doing anything. the event shows:
VDSM kvm5 command DestroyVDS failed: Virtual machine destroy error
and:
Failed to power off VM mysql-production (Host: kvm5, User:
admin@internal-authz).
Note, when I list the VM s on the hosts view, I see that that same VM is on two hosts (kvm7 and kvm5). It looks like the VM is stuck on two hosts, in the status "Powering Down"
any ideas?
No idea about this specific problem and how to solve it.
Is this on the same system you restored from backup? Is this VM marked as HA (highly-available)? Did it move to a different host since you took the backup you now restored?
If all are 'yes', then it's likely that the engine tried to start it (because it's HA) before it noticed that it's already running on another host (a fact that it could not know, if it was different from what's in the restored database). Sadly, this very likely leads to corruption. See also:
https://bugzilla.redhat.com/1441322
Since I do not see 'Resetting HA VM status' in the restore logs you posted, it might be a different issue, but perhaps similar.
If indeed you see two qemu processes on the hosts, I'd try to kill -9 both of them.
Adding Arik.
Good luck,
Thank you _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/E2HKUJZYUNC3VC...
-- Didi
Host restart resolve the issue! On Wed, Jan 2, 2019 at 11:50 AM maoz zadok <maozza@gmail.com> wrote:
Thank you Didi for the fast reply! The problem was before the restore from backup (that was the reason that I reboot the engine and the engine didn't come up, I dug a bigger hole :-( ), HA is not marked. but The VM was trying to migrate from kvm7 to kvm5 before the engine crashed. I don't see the qemu process on any of the hosts. "vdsm-client Host getVMList fullStatus=True" show the machine uuid only on one (kvm5)
I'll try restarting the host (kvm5), let's see if it helps.
On Wed, Jan 2, 2019 at 11:07 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Wed, Jan 2, 2019 at 10:51 AM maoz zadok <maozza@gmail.com> wrote:
Hello, vm stuck (on two hosts) I have MySQL VM guest that was stuck in a way that "kill -9" to the
MySQL server pid did nothing.
from the shell I execute "init 0", this did not shut down the VM.
from the ovirt-engine I'm trying to power off the machine, and it is not doing anything. the event shows:
VDSM kvm5 command DestroyVDS failed: Virtual machine destroy error
and:
Failed to power off VM mysql-production (Host: kvm5, User:
admin@internal-authz).
Note, when I list the VM s on the hosts view, I see that that same VM is on two hosts (kvm7 and kvm5). It looks like the VM is stuck on two hosts, in the status "Powering Down"
any ideas?
No idea about this specific problem and how to solve it.
Is this on the same system you restored from backup? Is this VM marked as HA (highly-available)? Did it move to a different host since you took the backup you now restored?
If all are 'yes', then it's likely that the engine tried to start it (because it's HA) before it noticed that it's already running on another host (a fact that it could not know, if it was different from what's in the restored database). Sadly, this very likely leads to corruption. See also:
https://bugzilla.redhat.com/1441322
Since I do not see 'Resetting HA VM status' in the restore logs you posted, it might be a different issue, but perhaps similar.
If indeed you see two qemu processes on the hosts, I'd try to kill -9 both of them.
Adding Arik.
Good luck,
Thank you _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/E2HKUJZYUNC3VC...
-- Didi
Thanks for the report! :-) On Wed, Jan 2, 2019 at 4:47 PM maoz zadok <maozza@gmail.com> wrote:
Host restart resolve the issue!
On Wed, Jan 2, 2019 at 11:50 AM maoz zadok <maozza@gmail.com> wrote:
Thank you Didi for the fast reply! The problem was before the restore from backup (that was the reason that I reboot the engine and the engine didn't come up, I dug a bigger hole :-( ), HA is not marked. but The VM was trying to migrate from kvm7 to kvm5 before the engine crashed. I don't see the qemu process on any of the hosts. "vdsm-client Host getVMList fullStatus=True" show the machine uuid only on one (kvm5)
I'll try restarting the host (kvm5), let's see if it helps.
On Wed, Jan 2, 2019 at 11:07 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Wed, Jan 2, 2019 at 10:51 AM maoz zadok <maozza@gmail.com> wrote:
Hello, vm stuck (on two hosts) I have MySQL VM guest that was stuck in a way that "kill -9" to the MySQL server pid did nothing. from the shell I execute "init 0", this did not shut down the VM.
from the ovirt-engine I'm trying to power off the machine, and it is not doing anything. the event shows:
VDSM kvm5 command DestroyVDS failed: Virtual machine destroy error
and:
Failed to power off VM mysql-production (Host: kvm5, User: admin@internal-authz).
Note, when I list the VM s on the hosts view, I see that that same VM is on two hosts (kvm7 and kvm5). It looks like the VM is stuck on two hosts, in the status "Powering Down"
any ideas?
No idea about this specific problem and how to solve it.
Is this on the same system you restored from backup? Is this VM marked as HA (highly-available)? Did it move to a different host since you took the backup you now restored?
If all are 'yes', then it's likely that the engine tried to start it (because it's HA) before it noticed that it's already running on another host (a fact that it could not know, if it was different from what's in the restored database). Sadly, this very likely leads to corruption. See also:
https://bugzilla.redhat.com/1441322
Since I do not see 'Resetting HA VM status' in the restore logs you posted, it might be a different issue, but perhaps similar.
If indeed you see two qemu processes on the hosts, I'd try to kill -9 both of them.
Adding Arik.
Good luck,
Thank you _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/E2HKUJZYUNC3VC...
-- Didi
-- Didi
participants (2)
-
maoz zadok -
Yedidyah Bar David