Destroyed VM blocking hosts/filling logs
Hello, I've somehow gotten one of my VMs stuck in a state that ovirt seems to be rather confused about its existence of now. I'm running oVirt 4.3.10 and using oVirt Node on all the hosts. My engine and host event logs are now filling up very rapidly with this error: VDSM node217 command DestroyVDS failed: General Exception: ("'1048576'",) I was playing with hugetable support, and that error number or string looks suspiciously like the "hugetable size" custom property I set on the VM. This VM was migrated to another host at one point as well, and now that host is also generating the same error as well. When I try to move these hosts to maintenance mode, they get stuck in "Preparing for Maintenance" while it tries to migrate/deal with the VM that's not there any more. Forcibly rebooting the hosts does not change anything. The VM state/host seems to be captured somewhere persistent in this case. The VM in question is not running, and I can start it up on another host successfully, but ovirt still thinks it exists on the other 2 hosts no matter what I do. Is there perhaps some way to delete it from the engine database directly to straighten things out? Here's a dump of the vdsm log on one of the hosts. I haven't been able to pinpoint what the exact issue is or how to fix it, but hopefully someone here will have seen this before? 2021-04-03 04:40:35,515+0000 INFO (jsonrpc/1) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:10.100.0.210,58150, vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:48) 2021-04-03 04:40:35,516+0000 INFO (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Release VM resources (vm:5186) 2021-04-03 04:40:35,516+0000 WARN (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') trying to set state to Powering down when already Down (vm:626) 2021-04-03 04:40:35,516+0000 INFO (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection (guestagent:455) 2021-04-03 04:40:35,517+0000 INFO (jsonrpc/1) [vdsm.api] START teardownImage(sdUUID='a08af6be-3802-4bb1-9fa5-4b6a10227290', spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9', imgUUID='9c896907-59b0-4983-9478-b36b2c2eb01e', volUUID=None) from=::ffff:10.100.0.210,58150, task_id=fc946d20-126a-4fd0-9078-91 4b4a64b1d9 (api:48) 2021-04-03 04:40:35,518+0000 INFO (jsonrpc/1) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/a08af6be-3802-4bb1-9fa5-4b6a10227290/9c896907-59b0-4983-9478-b36b2c2eb01e' (fileSD:592) 2021-04-03 04:40:35,518+0000 INFO (jsonrpc/1) [vdsm.api] FINISH teardownImage return=None from=::ffff:10.100.0.210,58150, task_id=fc946d20-126a-4fd0-9078-914b4a64b1d9 (api:54) 2021-04-03 04:40:35,519+0000 INFO (jsonrpc/1) [vdsm.api] START teardownImage(sdUUID='b891448d-dd92-4a7b-a51a-22abc3d7da67', spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9', imgUUID='c0e95483-35f1-4a61-958e-4e308b70d3f8', volUUID=None) from=::ffff:10.100.0.210,58150, task_id=77c0fdca-e13a-44b5-9a00-29 0522b194b2 (api:48) 2021-04-03 04:40:35,520+0000 INFO (jsonrpc/1) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/b891448d-dd92-4a7b-a51a-22abc3d7da67/c0e95483-35f1-4a61-958e-4e308b70d3f8' (fileSD:592) 2021-04-03 04:40:35,520+0000 INFO (jsonrpc/1) [vdsm.api] FINISH teardownImage return=None from=::ffff:10.100.0.210,58150, task_id=77c0fdca-e13a-44b5-9a00-290522b194b2 (api:54) 2021-04-03 04:40:35,521+0000 INFO (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection (guestagent:455) 2021-04-03 04:40:35,521+0000 WARN (jsonrpc/1) [root] File: /var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.ovirt-guest-agent.0 already removed (fileutils:54) 2021-04-03 04:40:35,521+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing network: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:198) 2021-04-03 04:40:35,522+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing net user: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:205) 2021-04-03 04:40:35,526+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing network: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:198) 2021-04-03 04:40:35,526+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing net user: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:205) 2021-04-03 04:40:35,527+0000 WARN (jsonrpc/1) [root] File: /var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.org.qemu.guest_agent.0 already removed (fileutils:54) 2021-04-03 04:40:35,528+0000 WARN (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') timestamp already removed from stats cache (vm:2445) 2021-04-03 04:40:35,531+0000 ERROR (jsonrpc/1) [api] FINISH destroy error='1048576' (api:134) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 124, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 333, in destroy res = self.vm.destroy(gracefulAttempts) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5283, in destroy result = self.doDestroy(gracefulAttempts, reason) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5302, in doDestroy return self.releaseVm(gracefulAttempts) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5207, in releaseVm self._cleanup() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2382, in _cleanup self._cleanup_hugepages() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2393, in _cleanup_hugepages self.nr_hugepages, self.hugepagesz File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 243, in calculate_required_deallocation _preallocated_hugepages(vm_hugepagesz)) File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 262, in _preallocated_hugepages kernel_args['hugepagesz'] File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 291, in _cmdline_hugepagesz_to_kb }[cmdline] KeyError: '1048576' 2021-04-03 04:40:35,531+0000 INFO (jsonrpc/1) [api.virt] FINISH destroy return={'status': {'message': 'General Exception: ("\'1048576\'",)', 'code': 100}} from=::ffff:10.100.0.210,58150, vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:54) 2021-04-03 04:40:35,532+0000 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call VM.destroy failed (error 100) in 0.02 seconds (__init__:312)
Hi, You can try with the vdsm-client tool: https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html Stopping a VM: 1) Get the vmId: # vdsm-client Host getVMList fullStatus=True 2) Destroy the VM # vdsm-client VM destroy vmID=<vmID> *Regards,* *Shani Leviim* On Sat, Apr 3, 2021 at 7:50 AM <davidk@riavera.com> wrote:
Hello,
I've somehow gotten one of my VMs stuck in a state that ovirt seems to be rather confused about its existence of now. I'm running oVirt 4.3.10 and using oVirt Node on all the hosts.
My engine and host event logs are now filling up very rapidly with this error:
VDSM node217 command DestroyVDS failed: General Exception: ("'1048576'",)
I was playing with hugetable support, and that error number or string looks suspiciously like the "hugetable size" custom property I set on the VM.
This VM was migrated to another host at one point as well, and now that host is also generating the same error as well.
When I try to move these hosts to maintenance mode, they get stuck in "Preparing for Maintenance" while it tries to migrate/deal with the VM that's not there any more.
Forcibly rebooting the hosts does not change anything. The VM state/host seems to be captured somewhere persistent in this case.
The VM in question is not running, and I can start it up on another host successfully, but ovirt still thinks it exists on the other 2 hosts no matter what I do.
Is there perhaps some way to delete it from the engine database directly to straighten things out?
Here's a dump of the vdsm log on one of the hosts. I haven't been able to pinpoint what the exact issue is or how to fix it, but hopefully someone here will have seen this before?
2021-04-03 04:40:35,515+0000 INFO (jsonrpc/1) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:10.100.0.210,58150, vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:48) 2021-04-03 04:40:35,516+0000 INFO (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Release VM resources (vm:5186) 2021-04-03 04:40:35,516+0000 WARN (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') trying to set state to Powering down when already Down (vm:626) 2021-04-03 04:40:35,516+0000 INFO (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection (guestagent:455) 2021-04-03 04:40:35,517+0000 INFO (jsonrpc/1) [vdsm.api] START teardownImage(sdUUID='a08af6be-3802-4bb1-9fa5-4b6a10227290', spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9', imgUUID='9c896907-59b0-4983-9478-b36b2c2eb01e', volUUID=None) from=::ffff:10.100.0.210,58150, task_id=fc946d20-126a-4fd0-9078-91 4b4a64b1d9 (api:48) 2021-04-03 04:40:35,518+0000 INFO (jsonrpc/1) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/a08af6be-3802-4bb1-9fa5-4b6a10227290/9c896907-59b0-4983-9478-b36b2c2eb01e' (fileSD:592) 2021-04-03 04:40:35,518+0000 INFO (jsonrpc/1) [vdsm.api] FINISH teardownImage return=None from=::ffff:10.100.0.210,58150, task_id=fc946d20-126a-4fd0-9078-914b4a64b1d9 (api:54) 2021-04-03 04:40:35,519+0000 INFO (jsonrpc/1) [vdsm.api] START teardownImage(sdUUID='b891448d-dd92-4a7b-a51a-22abc3d7da67', spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9', imgUUID='c0e95483-35f1-4a61-958e-4e308b70d3f8', volUUID=None) from=::ffff:10.100.0.210,58150, task_id=77c0fdca-e13a-44b5-9a00-29 0522b194b2 (api:48) 2021-04-03 04:40:35,520+0000 INFO (jsonrpc/1) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/b891448d-dd92-4a7b-a51a-22abc3d7da67/c0e95483-35f1-4a61-958e-4e308b70d3f8' (fileSD:592) 2021-04-03 04:40:35,520+0000 INFO (jsonrpc/1) [vdsm.api] FINISH teardownImage return=None from=::ffff:10.100.0.210,58150, task_id=77c0fdca-e13a-44b5-9a00-290522b194b2 (api:54) 2021-04-03 04:40:35,521+0000 INFO (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection (guestagent:455) 2021-04-03 04:40:35,521+0000 WARN (jsonrpc/1) [root] File: /var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.ovirt-guest-agent.0 already removed (fileutils:54) 2021-04-03 04:40:35,521+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing network: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:198) 2021-04-03 04:40:35,522+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing net user: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:205) 2021-04-03 04:40:35,526+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing network: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:198) 2021-04-03 04:40:35,526+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing net user: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:205) 2021-04-03 04:40:35,527+0000 WARN (jsonrpc/1) [root] File: /var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.org.qemu.guest_agent.0 already removed (fileutils:54) 2021-04-03 04:40:35,528+0000 WARN (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') timestamp already removed from stats cache (vm:2445) 2021-04-03 04:40:35,531+0000 ERROR (jsonrpc/1) [api] FINISH destroy error='1048576' (api:134) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 124, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 333, in destroy res = self.vm.destroy(gracefulAttempts) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5283, in destroy result = self.doDestroy(gracefulAttempts, reason) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5302, in doDestroy return self.releaseVm(gracefulAttempts) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5207, in releaseVm self._cleanup() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2382, in _cleanup self._cleanup_hugepages() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2393, in _cleanup_hugepages self.nr_hugepages, self.hugepagesz File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 243, in calculate_required_deallocation _preallocated_hugepages(vm_hugepagesz)) File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 262, in _preallocated_hugepages kernel_args['hugepagesz'] File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 291, in _cmdline_hugepagesz_to_kb }[cmdline] KeyError: '1048576' 2021-04-03 04:40:35,531+0000 INFO (jsonrpc/1) [api.virt] FINISH destroy return={'status': {'message': 'General Exception: ("\'1048576\'",)', 'code': 100}} from=::ffff:10.100.0.210,58150, vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:54) 2021-04-03 04:40:35,532+0000 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call VM.destroy failed (error 100) in 0.02 seconds (__init__:312) _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZ...
Hello, This seems to be what the engine is trying to do, and failing at for some reason. eg: [root@ovirt-node217 ~]# vdsm-client Host getVMList fullStatus=True [ "8b3964bc-cd3f-4f13-84c6-1811193c93eb", "132668b6-9992-451f-95ac-dbcbeb03f5f1" ] For reference: [root@ovirt-node217 ~]# virsh -r list --all Id Name State ---------------------------------------------------- - vm-s2 shut off - vm-s1 shut off And in the console, it shows a count of "2" beside this host, but on the host detail page, under the virtual-machine tab, the list is empty (these VMs are actually running on a different host). [root@ovirt-node217 ~]# vdsm-client VM destroy vmID="8b3964bc-cd3f-4f13-84c6-1811193c93eb" vdsm-client: Command VM.destroy with args {'vmID': '8b3964bc-cd3f-4f13-84c6-1811193c93eb'} failed: (code=100, message=General Exception: ("'1048576'",)) I guess what I need is a way to remove/clean-up these VMs manually since ovirt does not seem to be able to do it by itself. This condition also blocks the host from being put into maintenance mode. When I reboot the host manually and "confirm host was rebooted", the VMs are still there and still stuck. Sincerely, David On 2021-04-07 6:01 a.m., Shani Leviim wrote:
Hi, You can try with the vdsm-client tool: https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html <https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html>
Stopping a VM: 1) Get the vmId: # vdsm-client Host getVMList fullStatus=True
2) Destroy the VM # vdsm-client VM destroy vmID=<vmID>
*Regards, * *Shani Leviim *
On Sat, Apr 3, 2021 at 7:50 AM <davidk@riavera.com <mailto:davidk@riavera.com>> wrote:
Hello,
I've somehow gotten one of my VMs stuck in a state that ovirt seems to be rather confused about its existence of now. I'm running oVirt 4.3.10 and using oVirt Node on all the hosts.
My engine and host event logs are now filling up very rapidly with this error:
VDSM node217 command DestroyVDS failed: General Exception: ("'1048576'",)
I was playing with hugetable support, and that error number or string looks suspiciously like the "hugetable size" custom property I set on the VM.
This VM was migrated to another host at one point as well, and now that host is also generating the same error as well.
When I try to move these hosts to maintenance mode, they get stuck in "Preparing for Maintenance" while it tries to migrate/deal with the VM that's not there any more.
Forcibly rebooting the hosts does not change anything. The VM state/host seems to be captured somewhere persistent in this case.
The VM in question is not running, and I can start it up on another host successfully, but ovirt still thinks it exists on the other 2 hosts no matter what I do.
Is there perhaps some way to delete it from the engine database directly to straighten things out?
Here's a dump of the vdsm log on one of the hosts. I haven't been able to pinpoint what the exact issue is or how to fix it, but hopefully someone here will have seen this before?
2021-04-03 04:40:35,515+0000 INFO (jsonrpc/1) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:10.100.0.210,58150, vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:48) 2021-04-03 04:40:35,516+0000 INFO (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Release VM resources (vm:5186) 2021-04-03 04:40:35,516+0000 WARN (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') trying to set state to Powering down when already Down (vm:626) 2021-04-03 04:40:35,516+0000 INFO (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection (guestagent:455) 2021-04-03 04:40:35,517+0000 INFO (jsonrpc/1) [vdsm.api] START teardownImage(sdUUID='a08af6be-3802-4bb1-9fa5-4b6a10227290', spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9', imgUUID='9c896907-59b0-4983-9478-b36b2c2eb01e', volUUID=None) from=::ffff:10.100.0.210,58150, task_id=fc946d20-126a-4fd0-9078-91 4b4a64b1d9 (api:48) 2021-04-03 04:40:35,518+0000 INFO (jsonrpc/1) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/a08af6be-3802-4bb1-9fa5-4b6a10227290/9c896907-59b0-4983-9478-b36b2c2eb01e' (fileSD:592) 2021-04-03 04:40:35,518+0000 INFO (jsonrpc/1) [vdsm.api] FINISH teardownImage return=None from=::ffff:10.100.0.210,58150, task_id=fc946d20-126a-4fd0-9078-914b4a64b1d9 (api:54) 2021-04-03 04:40:35,519+0000 INFO (jsonrpc/1) [vdsm.api] START teardownImage(sdUUID='b891448d-dd92-4a7b-a51a-22abc3d7da67', spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9', imgUUID='c0e95483-35f1-4a61-958e-4e308b70d3f8', volUUID=None) from=::ffff:10.100.0.210,58150, task_id=77c0fdca-e13a-44b5-9a00-29 0522b194b2 (api:48) 2021-04-03 04:40:35,520+0000 INFO (jsonrpc/1) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/b891448d-dd92-4a7b-a51a-22abc3d7da67/c0e95483-35f1-4a61-958e-4e308b70d3f8' (fileSD:592) 2021-04-03 04:40:35,520+0000 INFO (jsonrpc/1) [vdsm.api] FINISH teardownImage return=None from=::ffff:10.100.0.210,58150, task_id=77c0fdca-e13a-44b5-9a00-290522b194b2 (api:54) 2021-04-03 04:40:35,521+0000 INFO (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection (guestagent:455) 2021-04-03 04:40:35,521+0000 WARN (jsonrpc/1) [root] File: /var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.ovirt-guest-agent.0 already removed (fileutils:54) 2021-04-03 04:40:35,521+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing network: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:198) 2021-04-03 04:40:35,522+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing net user: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:205) 2021-04-03 04:40:35,526+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing network: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:198) 2021-04-03 04:40:35,526+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing net user: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:205) 2021-04-03 04:40:35,527+0000 WARN (jsonrpc/1) [root] File: /var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.org.qemu.guest_agent.0 already removed (fileutils:54) 2021-04-03 04:40:35,528+0000 WARN (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') timestamp already removed from stats cache (vm:2445) 2021-04-03 04:40:35,531+0000 ERROR (jsonrpc/1) [api] FINISH destroy error='1048576' (api:134) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 124, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 333, in destroy res = self.vm.destroy(gracefulAttempts) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5283, in destroy result = self.doDestroy(gracefulAttempts, reason) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5302, in doDestroy return self.releaseVm(gracefulAttempts) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5207, in releaseVm self._cleanup() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2382, in _cleanup self._cleanup_hugepages() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2393, in _cleanup_hugepages self.nr_hugepages, self.hugepagesz File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 243, in calculate_required_deallocation _preallocated_hugepages(vm_hugepagesz)) File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 262, in _preallocated_hugepages kernel_args['hugepagesz'] File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 291, in _cmdline_hugepagesz_to_kb }[cmdline] KeyError: '1048576' 2021-04-03 04:40:35,531+0000 INFO (jsonrpc/1) [api.virt] FINISH destroy return={'status': {'message': 'General Exception: ("\'1048576\'",)', 'code': 100}} from=::ffff:10.100.0.210,58150, vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:54) 2021-04-03 04:40:35,532+0000 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call VM.destroy failed (error 100) in 0.02 seconds (__init__:312) _______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html <https://www.ovirt.org/privacy-policy.html> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZ... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZXGYRSROHCO7LAS4VOM/>
Hi David, You can delete the VM from the DB using this command: SELECT DeleteVm('<vm-id>'); *Regards,* *Shani Leviim* On Wed, Apr 7, 2021 at 4:23 PM David Kerry <davidk@riavera.com> wrote:
Hello,
This seems to be what the engine is trying to do, and failing at for some reason.
eg:
[root@ovirt-node217 ~]# vdsm-client Host getVMList fullStatus=True [ "8b3964bc-cd3f-4f13-84c6-1811193c93eb", "132668b6-9992-451f-95ac-dbcbeb03f5f1" ]
For reference:
[root@ovirt-node217 ~]# virsh -r list --all Id Name State ---------------------------------------------------- - vm-s2 shut off - vm-s1 shut off
And in the console, it shows a count of "2" beside this host, but on the host detail page, under the virtual-machine tab, the list is empty (these VMs are actually running on a different host).
[root@ovirt-node217 ~]# vdsm-client VM destroy vmID="8b3964bc-cd3f-4f13-84c6-1811193c93eb" vdsm-client: Command VM.destroy with args {'vmID': '8b3964bc-cd3f-4f13-84c6-1811193c93eb'} failed: (code=100, message=General Exception: ("'1048576'",))
I guess what I need is a way to remove/clean-up these VMs manually since ovirt does not seem to be able to do it by itself.
This condition also blocks the host from being put into maintenance mode.
When I reboot the host manually and "confirm host was rebooted", the VMs are still there and still stuck.
Sincerely,
David
On 2021-04-07 6:01 a.m., Shani Leviim wrote:
Hi, You can try with the vdsm-client tool: https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html
Stopping a VM: 1) Get the vmId: # vdsm-client Host getVMList fullStatus=True
2) Destroy the VM # vdsm-client VM destroy vmID=<vmID>
*Regards, *
*Shani Leviim *
On Sat, Apr 3, 2021 at 7:50 AM <davidk@riavera.com> wrote:
Hello,
I've somehow gotten one of my VMs stuck in a state that ovirt seems to be rather confused about its existence of now. I'm running oVirt 4.3.10 and using oVirt Node on all the hosts.
My engine and host event logs are now filling up very rapidly with this error:
VDSM node217 command DestroyVDS failed: General Exception: ("'1048576'",)
I was playing with hugetable support, and that error number or string looks suspiciously like the "hugetable size" custom property I set on the VM.
This VM was migrated to another host at one point as well, and now that host is also generating the same error as well.
When I try to move these hosts to maintenance mode, they get stuck in "Preparing for Maintenance" while it tries to migrate/deal with the VM that's not there any more.
Forcibly rebooting the hosts does not change anything. The VM state/host seems to be captured somewhere persistent in this case.
The VM in question is not running, and I can start it up on another host successfully, but ovirt still thinks it exists on the other 2 hosts no matter what I do.
Is there perhaps some way to delete it from the engine database directly to straighten things out?
Here's a dump of the vdsm log on one of the hosts. I haven't been able to pinpoint what the exact issue is or how to fix it, but hopefully someone here will have seen this before?
2021-04-03 04:40:35,515+0000 INFO (jsonrpc/1) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:10.100.0.210,58150, vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:48) 2021-04-03 04:40:35,516+0000 INFO (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Release VM resources (vm:5186) 2021-04-03 04:40:35,516+0000 WARN (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') trying to set state to Powering down when already Down (vm:626) 2021-04-03 04:40:35,516+0000 INFO (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection (guestagent:455) 2021-04-03 04:40:35,517+0000 INFO (jsonrpc/1) [vdsm.api] START teardownImage(sdUUID='a08af6be-3802-4bb1-9fa5-4b6a10227290', spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9', imgUUID='9c896907-59b0-4983-9478-b36b2c2eb01e', volUUID=None) from=::ffff:10.100.0.210,58150, task_id=fc946d20-126a-4fd0-9078-91 4b4a64b1d9 (api:48) 2021-04-03 04:40:35,518+0000 INFO (jsonrpc/1) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/a08af6be-3802-4bb1-9fa5-4b6a10227290/9c896907-59b0-4983-9478-b36b2c2eb01e' (fileSD:592) 2021-04-03 04:40:35,518+0000 INFO (jsonrpc/1) [vdsm.api] FINISH teardownImage return=None from=::ffff:10.100.0.210,58150, task_id=fc946d20-126a-4fd0-9078-914b4a64b1d9 (api:54) 2021-04-03 04:40:35,519+0000 INFO (jsonrpc/1) [vdsm.api] START teardownImage(sdUUID='b891448d-dd92-4a7b-a51a-22abc3d7da67', spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9', imgUUID='c0e95483-35f1-4a61-958e-4e308b70d3f8', volUUID=None) from=::ffff:10.100.0.210,58150, task_id=77c0fdca-e13a-44b5-9a00-29 0522b194b2 (api:48) 2021-04-03 04:40:35,520+0000 INFO (jsonrpc/1) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/b891448d-dd92-4a7b-a51a-22abc3d7da67/c0e95483-35f1-4a61-958e-4e308b70d3f8' (fileSD:592) 2021-04-03 04:40:35,520+0000 INFO (jsonrpc/1) [vdsm.api] FINISH teardownImage return=None from=::ffff:10.100.0.210,58150, task_id=77c0fdca-e13a-44b5-9a00-290522b194b2 (api:54) 2021-04-03 04:40:35,521+0000 INFO (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection (guestagent:455) 2021-04-03 04:40:35,521+0000 WARN (jsonrpc/1) [root] File: /var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.ovirt-guest-agent.0 already removed (fileutils:54) 2021-04-03 04:40:35,521+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing network: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:198) 2021-04-03 04:40:35,522+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing net user: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:205) 2021-04-03 04:40:35,526+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing network: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:198) 2021-04-03 04:40:35,526+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing net user: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:205) 2021-04-03 04:40:35,527+0000 WARN (jsonrpc/1) [root] File: /var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.org.qemu.guest_agent.0 already removed (fileutils:54) 2021-04-03 04:40:35,528+0000 WARN (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') timestamp already removed from stats cache (vm:2445) 2021-04-03 04:40:35,531+0000 ERROR (jsonrpc/1) [api] FINISH destroy error='1048576' (api:134) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 124, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 333, in destroy res = self.vm.destroy(gracefulAttempts) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5283, in destroy result = self.doDestroy(gracefulAttempts, reason) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5302, in doDestroy return self.releaseVm(gracefulAttempts) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5207, in releaseVm self._cleanup() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2382, in _cleanup self._cleanup_hugepages() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2393, in _cleanup_hugepages self.nr_hugepages, self.hugepagesz File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 243, in calculate_required_deallocation _preallocated_hugepages(vm_hugepagesz)) File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 262, in _preallocated_hugepages kernel_args['hugepagesz'] File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 291, in _cmdline_hugepagesz_to_kb }[cmdline] KeyError: '1048576' 2021-04-03 04:40:35,531+0000 INFO (jsonrpc/1) [api.virt] FINISH destroy return={'status': {'message': 'General Exception: ("\'1048576\'",)', 'code': 100}} from=::ffff:10.100.0.210,58150, vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:54) 2021-04-03 04:40:35,532+0000 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call VM.destroy failed (error 100) in 0.02 seconds (__init__:312) _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZ...
Hi Shani, These VMs in particular are running just fine on other hosts (and I'd like to keep them that way, preferably). It looks like this command would delete the whole VM from the entire system instead of just removing the stuck/shutdown instances from the hosts it's not running on any more. Can you confirm this is what it would do? If so, is there another option to remove these stuck "ghost" VM instances from the hosts they are no longer running on? Thanks David On 2021-04-08 3:20 a.m., Shani Leviim wrote:
Hi David, You can delete the VM from the DB using this command: SELECT DeleteVm('<vm-id>');
*Regards, * *Shani Leviim *
On Wed, Apr 7, 2021 at 4:23 PM David Kerry <davidk@riavera.com <mailto:davidk@riavera.com>> wrote:
Hello,
This seems to be what the engine is trying to do, and failing at for some reason.
eg:
[root@ovirt-node217 ~]# vdsm-client Host getVMList fullStatus=True [ "8b3964bc-cd3f-4f13-84c6-1811193c93eb", "132668b6-9992-451f-95ac-dbcbeb03f5f1" ]
For reference:
[root@ovirt-node217 ~]# virsh -r list --all Id Name State ---------------------------------------------------- - vm-s2 shut off - vm-s1 shut off
And in the console, it shows a count of "2" beside this host, but on the host detail page, under the virtual-machine tab, the list is empty (these VMs are actually running on a different host).
[root@ovirt-node217 ~]# vdsm-client VM destroy vmID="8b3964bc-cd3f-4f13-84c6-1811193c93eb" vdsm-client: Command VM.destroy with args {'vmID': '8b3964bc-cd3f-4f13-84c6-1811193c93eb'} failed: (code=100, message=General Exception: ("'1048576'",))
I guess what I need is a way to remove/clean-up these VMs manually since ovirt does not seem to be able to do it by itself.
This condition also blocks the host from being put into maintenance mode.
When I reboot the host manually and "confirm host was rebooted", the VMs are still there and still stuck.
Sincerely,
David
On 2021-04-07 6:01 a.m., Shani Leviim wrote:
Hi, You can try with the vdsm-client tool: https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html <https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html>
Stopping a VM: 1) Get the vmId: # vdsm-client Host getVMList fullStatus=True
2) Destroy the VM # vdsm-client VM destroy vmID=<vmID>
*Regards, * *Shani Leviim *
On Sat, Apr 3, 2021 at 7:50 AM <davidk@riavera.com <mailto:davidk@riavera.com>> wrote:
Hello,
I've somehow gotten one of my VMs stuck in a state that ovirt seems to be rather confused about its existence of now. I'm running oVirt 4.3.10 and using oVirt Node on all the hosts.
My engine and host event logs are now filling up very rapidly with this error:
VDSM node217 command DestroyVDS failed: General Exception: ("'1048576'",)
I was playing with hugetable support, and that error number or string looks suspiciously like the "hugetable size" custom property I set on the VM.
This VM was migrated to another host at one point as well, and now that host is also generating the same error as well.
When I try to move these hosts to maintenance mode, they get stuck in "Preparing for Maintenance" while it tries to migrate/deal with the VM that's not there any more.
Forcibly rebooting the hosts does not change anything. The VM state/host seems to be captured somewhere persistent in this case.
The VM in question is not running, and I can start it up on another host successfully, but ovirt still thinks it exists on the other 2 hosts no matter what I do.
Is there perhaps some way to delete it from the engine database directly to straighten things out?
Here's a dump of the vdsm log on one of the hosts. I haven't been able to pinpoint what the exact issue is or how to fix it, but hopefully someone here will have seen this before?
2021-04-03 04:40:35,515+0000 INFO (jsonrpc/1) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:10.100.0.210,58150, vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:48) 2021-04-03 04:40:35,516+0000 INFO (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Release VM resources (vm:5186) 2021-04-03 04:40:35,516+0000 WARN (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') trying to set state to Powering down when already Down (vm:626) 2021-04-03 04:40:35,516+0000 INFO (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection (guestagent:455) 2021-04-03 04:40:35,517+0000 INFO (jsonrpc/1) [vdsm.api] START teardownImage(sdUUID='a08af6be-3802-4bb1-9fa5-4b6a10227290', spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9', imgUUID='9c896907-59b0-4983-9478-b36b2c2eb01e', volUUID=None) from=::ffff:10.100.0.210,58150, task_id=fc946d20-126a-4fd0-9078-91 4b4a64b1d9 (api:48) 2021-04-03 04:40:35,518+0000 INFO (jsonrpc/1) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/a08af6be-3802-4bb1-9fa5-4b6a10227290/9c896907-59b0-4983-9478-b36b2c2eb01e' (fileSD:592) 2021-04-03 04:40:35,518+0000 INFO (jsonrpc/1) [vdsm.api] FINISH teardownImage return=None from=::ffff:10.100.0.210,58150, task_id=fc946d20-126a-4fd0-9078-914b4a64b1d9 (api:54) 2021-04-03 04:40:35,519+0000 INFO (jsonrpc/1) [vdsm.api] START teardownImage(sdUUID='b891448d-dd92-4a7b-a51a-22abc3d7da67', spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9', imgUUID='c0e95483-35f1-4a61-958e-4e308b70d3f8', volUUID=None) from=::ffff:10.100.0.210,58150, task_id=77c0fdca-e13a-44b5-9a00-29 0522b194b2 (api:48) 2021-04-03 04:40:35,520+0000 INFO (jsonrpc/1) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/b891448d-dd92-4a7b-a51a-22abc3d7da67/c0e95483-35f1-4a61-958e-4e308b70d3f8' (fileSD:592) 2021-04-03 04:40:35,520+0000 INFO (jsonrpc/1) [vdsm.api] FINISH teardownImage return=None from=::ffff:10.100.0.210,58150, task_id=77c0fdca-e13a-44b5-9a00-290522b194b2 (api:54) 2021-04-03 04:40:35,521+0000 INFO (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection (guestagent:455) 2021-04-03 04:40:35,521+0000 WARN (jsonrpc/1) [root] File: /var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.ovirt-guest-agent.0 already removed (fileutils:54) 2021-04-03 04:40:35,521+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing network: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:198) 2021-04-03 04:40:35,522+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing net user: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:205) 2021-04-03 04:40:35,526+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing network: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:198) 2021-04-03 04:40:35,526+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing net user: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:205) 2021-04-03 04:40:35,527+0000 WARN (jsonrpc/1) [root] File: /var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.org.qemu.guest_agent.0 already removed (fileutils:54) 2021-04-03 04:40:35,528+0000 WARN (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') timestamp already removed from stats cache (vm:2445) 2021-04-03 04:40:35,531+0000 ERROR (jsonrpc/1) [api] FINISH destroy error='1048576' (api:134) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 124, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 333, in destroy res = self.vm.destroy(gracefulAttempts) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5283, in destroy result = self.doDestroy(gracefulAttempts, reason) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5302, in doDestroy return self.releaseVm(gracefulAttempts) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5207, in releaseVm self._cleanup() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2382, in _cleanup self._cleanup_hugepages() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2393, in _cleanup_hugepages self.nr_hugepages, self.hugepagesz File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 243, in calculate_required_deallocation _preallocated_hugepages(vm_hugepagesz)) File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 262, in _preallocated_hugepages kernel_args['hugepagesz'] File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 291, in _cmdline_hugepagesz_to_kb }[cmdline] KeyError: '1048576' 2021-04-03 04:40:35,531+0000 INFO (jsonrpc/1) [api.virt] FINISH destroy return={'status': {'message': 'General Exception: ("\'1048576\'",)', 'code': 100}} from=::ffff:10.100.0.210,58150, vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:54) 2021-04-03 04:40:35,532+0000 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call VM.destroy failed (error 100) in 0.02 seconds (__init__:312) _______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html <https://www.ovirt.org/privacy-policy.html> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZ... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZXGYRSROHCO7LAS4VOM/>
Hi David, Yes - this one will remove completely the VM from the DB. You can use the virsh command to delete the VM guests: https://www.cyberciti.biz/faq/howto-linux-delete-a-running-vm-guest-on-kvm/ *Regards,* *Shani Leviim* On Thu, Apr 8, 2021 at 4:32 PM David Kerry <davidk@riavera.com> wrote:
Hi Shani,
These VMs in particular are running just fine on other hosts (and I'd like to keep them that way, preferably).
It looks like this command would delete the whole VM from the entire system instead of just removing the stuck/shutdown instances from the hosts it's not running on any more.
Can you confirm this is what it would do? If so, is there another option to remove these stuck "ghost" VM instances from the hosts they are no longer running on?
Thanks
David
Hi David, You can delete the VM from the DB using this command: SELECT DeleteVm('<vm-id>');
*Regards, * *Shani Leviim *
On Wed, Apr 7, 2021 at 4:23 PM David Kerry <davidk@riavera.com <mailto: davidk@riavera.com>> wrote:
Hello,
This seems to be what the engine is trying to do, and failing at for some reason.
eg:
[root@ovirt-node217 ~]# vdsm-client Host getVMList fullStatus=True [ "8b3964bc-cd3f-4f13-84c6-1811193c93eb", "132668b6-9992-451f-95ac-dbcbeb03f5f1" ]
For reference:
[root@ovirt-node217 ~]# virsh -r list --all Id Name State ---------------------------------------------------- - vm-s2 shut off - vm-s1 shut off
And in the console, it shows a count of "2" beside this host, but on
On 2021-04-08 3:20 a.m., Shani Leviim wrote: the host detail
page, under the virtual-machine tab, the list is empty (these VMs
are actually
running on a different host).
[root@ovirt-node217 ~]# vdsm-client VM destroy
vmID="8b3964bc-cd3f-4f13-84c6-1811193c93eb"
vdsm-client: Command VM.destroy with args {'vmID':
'8b3964bc-cd3f-4f13-84c6-1811193c93eb'} failed:
(code=100, message=General Exception: ("'1048576'",))
I guess what I need is a way to remove/clean-up these VMs manually
since ovirt
does not seem to be able to do it by itself.
This condition also blocks the host from being put into maintenance
mode.
When I reboot the host manually and "confirm host was rebooted", the
VMs
are still there and still stuck.
Sincerely,
David
On 2021-04-07 6:01 a.m., Shani Leviim wrote:
Hi, You can try with the vdsm-client tool: https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html
Stopping a VM: 1) Get the vmId: # vdsm-client Host getVMList fullStatus=True
2) Destroy the VM # vdsm-client VM destroy vmID=<vmID>
*Regards, * *Shani Leviim *
On Sat, Apr 3, 2021 at 7:50 AM <davidk@riavera.com <mailto:
davidk@riavera.com>> wrote:
Hello,
I've somehow gotten one of my VMs stuck in a state that ovirt
seems to be rather confused about its
existence of now. I'm running oVirt 4.3.10 and using oVirt
Node on all the hosts.
My engine and host event logs are now filling up very rapidly
with this error:
VDSM node217 command DestroyVDS failed: General Exception:
("'1048576'",)
I was playing with hugetable support, and that error number or
string looks suspiciously
like the "hugetable size" custom property I set on the VM.
This VM was migrated to another host at one point as well, and
now that host is also
generating the same error as well.
When I try to move these hosts to maintenance mode, they get
stuck in "Preparing for
Maintenance" while it tries to migrate/deal with the VM that's
not there any more.
Forcibly rebooting the hosts does not change anything. The VM
state/host seems to be
captured somewhere persistent in this case.
The VM in question is not running, and I can start it up on
another host successfully,
but ovirt still thinks it exists on the other 2 hosts no matter
what I do.
Is there perhaps some way to delete it from the engine database
things out?
Here's a dump of the vdsm log on one of the hosts. I haven't
been able to pinpoint what
the exact issue is or how to fix it, but hopefully someone here
will have seen this before?
2021-04-03 04:40:35,515+0000 INFO (jsonrpc/1) [api.virt] START
destroy(gracefulAttempts=1) from=::ffff:10.100.0.210,58150, vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:48)
2021-04-03 04:40:35,516+0000 INFO (jsonrpc/1) [virt.vm]
(vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Release VM resources (vm:5186)
2021-04-03 04:40:35,516+0000 WARN (jsonrpc/1) [virt.vm]
(vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') trying to set state to Powering down when already Down (vm:626)
2021-04-03 04:40:35,516+0000 INFO (jsonrpc/1) [virt.vm]
(vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection (guestagent:455)
2021-04-03 04:40:35,517+0000 INFO (jsonrpc/1) [vdsm.api] START
teardownImage(sdUUID='a08af6be-3802-4bb1-9fa5-4b6a10227290', spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9', imgUUID='9c896907-59b0-4983-9478-b36b2c2eb01e', volUUID=None) from=::ffff:10.100.0.210,58150, task_id=fc946d20-126a-4fd0-9078-91
4b4a64b1d9 (api:48) 2021-04-03 04:40:35,518+0000 INFO (jsonrpc/1)
[storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/a08af6be-3802-4bb1-9fa5-4b6a10227290/9c896907-59b0-4983-9478-b36b2c2eb01e' (fileSD:592)
2021-04-03 04:40:35,518+0000 INFO (jsonrpc/1) [vdsm.api]
FINISH teardownImage return=None from=::ffff:10.100.0.210,58150, task_id=fc946d20-126a-4fd0-9078-914b4a64b1d9 (api:54)
2021-04-03 04:40:35,519+0000 INFO (jsonrpc/1) [vdsm.api] START
teardownImage(sdUUID='b891448d-dd92-4a7b-a51a-22abc3d7da67', spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9', imgUUID='c0e95483-35f1-4a61-958e-4e308b70d3f8', volUUID=None) from=::ffff:10.100.0.210,58150, task_id=77c0fdca-e13a-44b5-9a00-29
0522b194b2 (api:48) 2021-04-03 04:40:35,520+0000 INFO (jsonrpc/1)
[storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/b891448d-dd92-4a7b-a51a-22abc3d7da67/c0e95483-35f1-4a61-958e-4e308b70d3f8' (fileSD:592)
2021-04-03 04:40:35,520+0000 INFO (jsonrpc/1) [vdsm.api]
FINISH teardownImage return=None from=::ffff:10.100.0.210,58150, task_id=77c0fdca-e13a-44b5-9a00-290522b194b2 (api:54)
2021-04-03 04:40:35,521+0000 INFO (jsonrpc/1) [virt.vm]
(vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection (guestagent:455)
2021-04-03 04:40:35,521+0000 WARN (jsonrpc/1) [root] File:
/var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.ovirt-guest-agent.0 already removed (fileutils:54)
2021-04-03 04:40:35,521+0000 WARN (jsonrpc/1) [root]
Attempting to remove a non existing network: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:198)
2021-04-03 04:40:35,522+0000 WARN (jsonrpc/1) [root]
Attempting to remove a non existing net user: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:205)
2021-04-03 04:40:35,526+0000 WARN (jsonrpc/1) [root]
Attempting to remove a non existing network: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:198)
2021-04-03 04:40:35,526+0000 WARN (jsonrpc/1) [root]
Attempting to remove a non existing net user: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:205)
2021-04-03 04:40:35,527+0000 WARN (jsonrpc/1) [root] File:
/var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.org.qemu.guest_agent.0 already removed (fileutils:54)
2021-04-03 04:40:35,528+0000 WARN (jsonrpc/1) [virt.vm]
(vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') timestamp already removed from stats cache (vm:2445)
2021-04-03 04:40:35,531+0000 ERROR (jsonrpc/1) [api] FINISH
destroy error='1048576' (api:134)
Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/common/api.py",
ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line
333, in destroy
res = self.vm.destroy(gracefulAttempts) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
5283, in destroy
result = self.doDestroy(gracefulAttempts, reason) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
5302, in doDestroy
return self.releaseVm(gracefulAttempts) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
5207, in releaseVm
self._cleanup() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
2382, in _cleanup
self._cleanup_hugepages() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line
2393, in _cleanup_hugepages
self.nr_hugepages, self.hugepagesz File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py",
_preallocated_hugepages(vm_hugepagesz)) File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py",
kernel_args['hugepagesz'] File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py",
<https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html> directly to straighten line 124, in method line 243, in calculate_required_deallocation line 262, in _preallocated_hugepages line 291, in _cmdline_hugepagesz_to_kb
}[cmdline] KeyError: '1048576' 2021-04-03 04:40:35,531+0000 INFO (jsonrpc/1) [api.virt]
FINISH destroy return={'status': {'message': 'General Exception: ("\'1048576\'",)', 'code': 100}} from=::ffff:10.100.0.210,58150, vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:54)
2021-04-03 04:40:35,532+0000 INFO (jsonrpc/1)
[jsonrpc.JsonRpcServer] RPC call VM.destroy failed (error 100) in 0.02 seconds (__init__:312)
_______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:
users-leave@ovirt.org>
Privacy Statement: https://www.ovirt.org/privacy-policy.html <
https://www.ovirt.org/privacy-policy.html>
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/ < https://www.ovirt.org/community/about/community-guidelines/>
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZ... < https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZ...
Hi Shani, I actually came across that option and attempted it at one point, but vdsm has locked me out of using that command it seems. Eg: [root@ovirt-node217 ~]# virsh undefine vm-s2 Please enter your authentication name: admin Please enter your password: error: failed to connect to the hypervisor error: authentication failed: authentication failed No known username/password seems to work. Is there some magic user to use for this, or some way to bypass the authentication? Thanks David On 2021-04-08 10:10 a.m., Shani Leviim wrote:
Hi David, Yes - this one will remove completely the VM from the DB.
You can use the virsh command to delete the VM guests: https://www.cyberciti.biz/faq/howto-linux-delete-a-running-vm-guest-on-kvm/ <https://www.cyberciti.biz/faq/howto-linux-delete-a-running-vm-guest-on-kvm/>
*Regards, * *Shani Leviim *
On Thu, Apr 8, 2021 at 4:32 PM David Kerry <davidk@riavera.com <mailto:davidk@riavera.com>> wrote:
Hi Shani,
These VMs in particular are running just fine on other hosts (and I'd like to keep them that way, preferably).
It looks like this command would delete the whole VM from the entire system instead of just removing the stuck/shutdown instances from the hosts it's not running on any more.
Can you confirm this is what it would do? If so, is there another option to remove these stuck "ghost" VM instances from the hosts they are no longer running on?
Thanks
David
On 2021-04-08 3:20 a.m., Shani Leviim wrote: > Hi David, > You can delete the VM from the DB using this command: > SELECT DeleteVm('<vm-id>'); > > *Regards, > * > *Shani Leviim > * > > > On Wed, Apr 7, 2021 at 4:23 PM David Kerry <davidk@riavera.com <mailto:davidk@riavera.com> <mailto:davidk@riavera.com <mailto:davidk@riavera.com>>> wrote: > > Hello, > > This seems to be what the engine is trying to do, and failing at for some reason. > > eg: > > [root@ovirt-node217 ~]# vdsm-client Host getVMList fullStatus=True > [ > "8b3964bc-cd3f-4f13-84c6-1811193c93eb", > "132668b6-9992-451f-95ac-dbcbeb03f5f1" > ] > > For reference: > > [root@ovirt-node217 ~]# virsh -r list --all > Id Name State > ---------------------------------------------------- > - vm-s2 shut off > - vm-s1 shut off > > And in the console, it shows a count of "2" beside this host, but on the host detail > page, under the virtual-machine tab, the list is empty (these VMs are actually > running on a different host). > > [root@ovirt-node217 ~]# vdsm-client VM destroy vmID="8b3964bc-cd3f-4f13-84c6-1811193c93eb" > vdsm-client: Command VM.destroy with args {'vmID': '8b3964bc-cd3f-4f13-84c6-1811193c93eb'} failed: > (code=100, message=General Exception: ("'1048576'",)) > > I guess what I need is a way to remove/clean-up these VMs manually since ovirt > does not seem to be able to do it by itself. > > This condition also blocks the host from being put into maintenance mode. > > When I reboot the host manually and "confirm host was rebooted", the VMs > are still there and still stuck. > > Sincerely, > > David > > > On 2021-04-07 6:01 a.m., Shani Leviim wrote: >> Hi, >> You can try with the vdsm-client tool: >> https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html <https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html> <https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html <https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html>> >> >> Stopping a VM: >> 1) Get the vmId: >> # vdsm-client Host getVMList fullStatus=True >> >> 2) Destroy the VM >> # vdsm-client VM destroy vmID=<vmID> >> >> *Regards, >> * >> *Shani Leviim >> * >> >> >> On Sat, Apr 3, 2021 at 7:50 AM <davidk@riavera.com <mailto:davidk@riavera.com> <mailto:davidk@riavera.com <mailto:davidk@riavera.com>>> wrote: >> >> Hello, >> >> I've somehow gotten one of my VMs stuck in a state that ovirt seems to be rather confused about its >> existence of now. I'm running oVirt 4.3.10 and using oVirt Node on all the hosts. >> >> My engine and host event logs are now filling up very rapidly with this error: >> >> VDSM node217 command DestroyVDS failed: General Exception: ("'1048576'",) >> >> I was playing with hugetable support, and that error number or string looks suspiciously >> like the "hugetable size" custom property I set on the VM. >> >> This VM was migrated to another host at one point as well, and now that host is also >> generating the same error as well. >> >> When I try to move these hosts to maintenance mode, they get stuck in "Preparing for >> Maintenance" while it tries to migrate/deal with the VM that's not there any more. >> >> Forcibly rebooting the hosts does not change anything. The VM state/host seems to be >> captured somewhere persistent in this case. >> >> The VM in question is not running, and I can start it up on another host successfully, >> but ovirt still thinks it exists on the other 2 hosts no matter what I do. >> >> Is there perhaps some way to delete it from the engine database directly to straighten >> things out? >> >> Here's a dump of the vdsm log on one of the hosts. I haven't been able to pinpoint what >> the exact issue is or how to fix it, but hopefully someone here will have seen this before? >> >> 2021-04-03 04:40:35,515+0000 INFO (jsonrpc/1) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:10.100.0.210,58150, vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:48) >> 2021-04-03 04:40:35,516+0000 INFO (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Release VM resources (vm:5186) >> 2021-04-03 04:40:35,516+0000 WARN (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') trying to set state to Powering down when already Down (vm:626) >> 2021-04-03 04:40:35,516+0000 INFO (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection (guestagent:455) >> 2021-04-03 04:40:35,517+0000 INFO (jsonrpc/1) [vdsm.api] START teardownImage(sdUUID='a08af6be-3802-4bb1-9fa5-4b6a10227290', spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9', imgUUID='9c896907-59b0-4983-9478-b36b2c2eb01e', volUUID=None) from=::ffff:10.100.0.210,58150, task_id=fc946d20-126a-4fd0-9078-91 >> 4b4a64b1d9 (api:48) >> 2021-04-03 04:40:35,518+0000 INFO (jsonrpc/1) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/a08af6be-3802-4bb1-9fa5-4b6a10227290/9c896907-59b0-4983-9478-b36b2c2eb01e' (fileSD:592) >> 2021-04-03 04:40:35,518+0000 INFO (jsonrpc/1) [vdsm.api] FINISH teardownImage return=None from=::ffff:10.100.0.210,58150, task_id=fc946d20-126a-4fd0-9078-914b4a64b1d9 (api:54) >> 2021-04-03 04:40:35,519+0000 INFO (jsonrpc/1) [vdsm.api] START teardownImage(sdUUID='b891448d-dd92-4a7b-a51a-22abc3d7da67', spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9', imgUUID='c0e95483-35f1-4a61-958e-4e308b70d3f8', volUUID=None) from=::ffff:10.100.0.210,58150, task_id=77c0fdca-e13a-44b5-9a00-29 >> 0522b194b2 (api:48) >> 2021-04-03 04:40:35,520+0000 INFO (jsonrpc/1) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/b891448d-dd92-4a7b-a51a-22abc3d7da67/c0e95483-35f1-4a61-958e-4e308b70d3f8' (fileSD:592) >> 2021-04-03 04:40:35,520+0000 INFO (jsonrpc/1) [vdsm.api] FINISH teardownImage return=None from=::ffff:10.100.0.210,58150, task_id=77c0fdca-e13a-44b5-9a00-290522b194b2 (api:54) >> 2021-04-03 04:40:35,521+0000 INFO (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection (guestagent:455) >> 2021-04-03 04:40:35,521+0000 WARN (jsonrpc/1) [root] File: /var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.ovirt-guest-agent.0 already removed (fileutils:54) >> 2021-04-03 04:40:35,521+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing network: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:198) >> 2021-04-03 04:40:35,522+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing net user: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:205) >> 2021-04-03 04:40:35,526+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing network: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:198) >> 2021-04-03 04:40:35,526+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing net user: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:205) >> 2021-04-03 04:40:35,527+0000 WARN (jsonrpc/1) [root] File: /var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.org.qemu.guest_agent.0 already removed (fileutils:54) >> 2021-04-03 04:40:35,528+0000 WARN (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') timestamp already removed from stats cache (vm:2445) >> 2021-04-03 04:40:35,531+0000 ERROR (jsonrpc/1) [api] FINISH destroy error='1048576' (api:134) >> Traceback (most recent call last): >> File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 124, in method >> ret = func(*args, **kwargs) >> File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 333, in destroy >> res = self.vm.destroy(gracefulAttempts) >> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5283, in destroy >> result = self.doDestroy(gracefulAttempts, reason) >> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5302, in doDestroy >> return self.releaseVm(gracefulAttempts) >> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5207, in releaseVm >> self._cleanup() >> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2382, in _cleanup >> self._cleanup_hugepages() >> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2393, in _cleanup_hugepages >> self.nr_hugepages, self.hugepagesz >> File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 243, in calculate_required_deallocation >> _preallocated_hugepages(vm_hugepagesz)) >> File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 262, in _preallocated_hugepages >> kernel_args['hugepagesz'] >> File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 291, in _cmdline_hugepagesz_to_kb >> }[cmdline] >> KeyError: '1048576' >> 2021-04-03 04:40:35,531+0000 INFO (jsonrpc/1) [api.virt] FINISH destroy return={'status': {'message': 'General Exception: ("\'1048576\'",)', 'code': 100}} from=::ffff:10.100.0.210,58150, vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:54) >> 2021-04-03 04:40:35,532+0000 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call VM.destroy failed (error 100) in 0.02 seconds (__init__:312) >> _______________________________________________ >> Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> <mailto:users@ovirt.org <mailto:users@ovirt.org>> >> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>> >> Privacy Statement: https://www.ovirt.org/privacy-policy.html <https://www.ovirt.org/privacy-policy.html> <https://www.ovirt.org/privacy-policy.html <https://www.ovirt.org/privacy-policy.html>> >> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> <https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/>> >> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZ... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZXGYRSROHCO7LAS4VOM/> <https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZ... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZXGYRSROHCO7LAS4VOM/>> >>
You can find your virsh user and password in /etc/ovirt-hosted-engine/virsh_auth.conf The content should be something like this: sudo cat /etc/ovirt-hosted-engine/virsh_auth.conf [credentials-vdsm] authname=vdsm@ovirt password=mypassword *Regards,* *Shani Leviim* On Thu, Apr 8, 2021 at 5:20 PM David Kerry <davidk@riavera.com> wrote:
Hi Shani,
I actually came across that option and attempted it at one point, but vdsm has locked me out of using that command it seems.
Eg:
[root@ovirt-node217 ~]# virsh undefine vm-s2 Please enter your authentication name: admin Please enter your password: error: failed to connect to the hypervisor error: authentication failed: authentication failed
No known username/password seems to work.
Is there some magic user to use for this, or some way to bypass the authentication?
Thanks
David
On 2021-04-08 10:10 a.m., Shani Leviim wrote:
Hi David, Yes - this one will remove completely the VM from the DB.
You can use the virsh command to delete the VM guests:
https://www.cyberciti.biz/faq/howto-linux-delete-a-running-vm-guest-on-kvm/ < https://www.cyberciti.biz/faq/howto-linux-delete-a-running-vm-guest-on-kvm/
*Regards, * *Shani Leviim *
On Thu, Apr 8, 2021 at 4:32 PM David Kerry <davidk@riavera.com <mailto:
davidk@riavera.com>> wrote:
Hi Shani,
These VMs in particular are running just fine on other hosts (and I'd like to keep them that way, preferably).
It looks like this command would delete the whole VM from the entire system instead of just removing the stuck/shutdown instances from the hosts it's not running on any more.
Can you confirm this is what it would do? If so, is there another option to remove these stuck "ghost" VM instances from the hosts
they are
no longer running on?
Thanks
David
On 2021-04-08 3:20 a.m., Shani Leviim wrote: > Hi David, > You can delete the VM from the DB using this command: > SELECT DeleteVm('<vm-id>'); > > *Regards, > * > *Shani Leviim > * > > > On Wed, Apr 7, 2021 at 4:23 PM David Kerry <davidk@riavera.com
<mailto:davidk@riavera.com> <mailto:davidk@riavera.com <mailto: davidk@riavera.com>>> wrote:
> > Hello, > > This seems to be what the engine is trying to do, and failing
at for some reason.
> > eg: > > [root@ovirt-node217 ~]# vdsm-client Host getVMList
fullStatus=True
> [ > "8b3964bc-cd3f-4f13-84c6-1811193c93eb", > "132668b6-9992-451f-95ac-dbcbeb03f5f1" > ] > > For reference: > > [root@ovirt-node217 ~]# virsh -r list --all > Id Name State > ---------------------------------------------------- > - vm-s2 shut off > - vm-s1 shut off > > And in the console, it shows a count of "2" beside this host,
but on the host detail
> page, under the virtual-machine tab, the list is empty (these
VMs are actually
> running on a different host). > > [root@ovirt-node217 ~]# vdsm-client VM destroy
vmID="8b3964bc-cd3f-4f13-84c6-1811193c93eb"
> vdsm-client: Command VM.destroy with args {'vmID':
'8b3964bc-cd3f-4f13-84c6-1811193c93eb'} failed:
> (code=100, message=General Exception: ("'1048576'",)) > > I guess what I need is a way to remove/clean-up these VMs
manually since ovirt
> does not seem to be able to do it by itself. > > This condition also blocks the host from being put into
maintenance mode.
> > When I reboot the host manually and "confirm host was
rebooted", the VMs
> are still there and still stuck. > > Sincerely, > > David > > > On 2021-04-07 6:01 a.m., Shani Leviim wrote: >> Hi, >> You can try with the vdsm-client tool: >>
https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html < https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html> < https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html < https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html>>
>> >> Stopping a VM: >> 1) Get the vmId: >> # vdsm-client Host getVMList fullStatus=True >> >> 2) Destroy the VM >> # vdsm-client VM destroy vmID=<vmID> >> >> *Regards, >> * >> *Shani Leviim >> * >> >> >> On Sat, Apr 3, 2021 at 7:50 AM <davidk@riavera.com <mailto:
davidk@riavera.com> <mailto:davidk@riavera.com <mailto:davidk@riavera.com>>> wrote:
>> >> Hello, >> >> I've somehow gotten one of my VMs stuck in a state that
ovirt seems to be rather confused about its
>> existence of now. I'm running oVirt 4.3.10 and using
oVirt Node on all the hosts.
>> >> My engine and host event logs are now filling up very
rapidly with this error:
>> >> VDSM node217 command DestroyVDS failed: General
Exception: ("'1048576'",)
>> >> I was playing with hugetable support, and that error
number or string looks suspiciously
>> like the "hugetable size" custom property I set on the
VM.
>> >> This VM was migrated to another host at one point as
well, and now that host is also
>> generating the same error as well. >> >> When I try to move these hosts to maintenance mode, they
get stuck in "Preparing for
>> Maintenance" while it tries to migrate/deal with the VM
that's not there any more.
>> >> Forcibly rebooting the hosts does not change anything.
The VM state/host seems to be
>> captured somewhere persistent in this case. >> >> The VM in question is not running, and I can start it up
on another host successfully,
>> but ovirt still thinks it exists on the other 2 hosts no
matter what I do.
>> >> Is there perhaps some way to delete it from the engine
database directly to straighten
>> things out? >> >> Here's a dump of the vdsm log on one of the hosts. I
haven't been able to pinpoint what
>> the exact issue is or how to fix it, but hopefully
someone here will have seen this before?
>> >> 2021-04-03 04:40:35,515+0000 INFO (jsonrpc/1)
[api.virt] START destroy(gracefulAttempts=1) from=::ffff:10.100.0.210,58150, vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:48)
>> 2021-04-03 04:40:35,516+0000 INFO (jsonrpc/1) [virt.vm]
(vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Release VM resources (vm:5186)
>> 2021-04-03 04:40:35,516+0000 WARN (jsonrpc/1) [virt.vm]
(vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') trying to set state to Powering down when already Down (vm:626)
>> 2021-04-03 04:40:35,516+0000 INFO (jsonrpc/1) [virt.vm]
(vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection (guestagent:455)
>> 2021-04-03 04:40:35,517+0000 INFO (jsonrpc/1)
[vdsm.api] START teardownImage(sdUUID='a08af6be-3802-4bb1-9fa5-4b6a10227290', spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9', imgUUID='9c896907-59b0-4983-9478-b36b2c2eb01e', volUUID=None) from=::ffff:10.100.0.210,58150, task_id=fc946d20-126a-4fd0-9078-91
>> 4b4a64b1d9 (api:48) >> 2021-04-03 04:40:35,518+0000 INFO (jsonrpc/1)
[storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/a08af6be-3802-4bb1-9fa5-4b6a10227290/9c896907-59b0-4983-9478-b36b2c2eb01e' (fileSD:592)
>> 2021-04-03 04:40:35,518+0000 INFO (jsonrpc/1)
[vdsm.api] FINISH teardownImage return=None from=::ffff:10.100.0.210,58150, task_id=fc946d20-126a-4fd0-9078-914b4a64b1d9 (api:54)
>> 2021-04-03 04:40:35,519+0000 INFO (jsonrpc/1)
[vdsm.api] START teardownImage(sdUUID='b891448d-dd92-4a7b-a51a-22abc3d7da67', spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9', imgUUID='c0e95483-35f1-4a61-958e-4e308b70d3f8', volUUID=None) from=::ffff:10.100.0.210,58150, task_id=77c0fdca-e13a-44b5-9a00-29
>> 0522b194b2 (api:48) >> 2021-04-03 04:40:35,520+0000 INFO (jsonrpc/1)
[storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/b891448d-dd92-4a7b-a51a-22abc3d7da67/c0e95483-35f1-4a61-958e-4e308b70d3f8' (fileSD:592)
>> 2021-04-03 04:40:35,520+0000 INFO (jsonrpc/1)
[vdsm.api] FINISH teardownImage return=None from=::ffff:10.100.0.210,58150, task_id=77c0fdca-e13a-44b5-9a00-290522b194b2 (api:54)
>> 2021-04-03 04:40:35,521+0000 INFO (jsonrpc/1) [virt.vm]
(vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection (guestagent:455)
>> 2021-04-03 04:40:35,521+0000 WARN (jsonrpc/1) [root]
File: /var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.ovirt-guest-agent.0 already removed (fileutils:54)
>> 2021-04-03 04:40:35,521+0000 WARN (jsonrpc/1) [root]
Attempting to remove a non existing network: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:198)
>> 2021-04-03 04:40:35,522+0000 WARN (jsonrpc/1) [root]
Attempting to remove a non existing net user: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:205)
>> 2021-04-03 04:40:35,526+0000 WARN (jsonrpc/1) [root]
Attempting to remove a non existing network: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:198)
>> 2021-04-03 04:40:35,526+0000 WARN (jsonrpc/1) [root]
Attempting to remove a non existing net user: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:205)
>> 2021-04-03 04:40:35,527+0000 WARN (jsonrpc/1) [root]
File: /var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.org.qemu.guest_agent.0 already removed (fileutils:54)
>> 2021-04-03 04:40:35,528+0000 WARN (jsonrpc/1) [virt.vm]
(vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') timestamp already removed from stats cache (vm:2445)
>> 2021-04-03 04:40:35,531+0000 ERROR (jsonrpc/1) [api]
FINISH destroy error='1048576' (api:134)
>> Traceback (most recent call last): >> File
"/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 124, in method
>> ret = func(*args, **kwargs) >> File "/usr/lib/python2.7/site-packages/vdsm/API.py",
line 333, in destroy
>> res = self.vm.destroy(gracefulAttempts) >> File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5283, in destroy
>> result = self.doDestroy(gracefulAttempts, reason) >> File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5302, in doDestroy
>> return self.releaseVm(gracefulAttempts) >> File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5207, in releaseVm
>> self._cleanup() >> File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2382, in _cleanup
>> self._cleanup_hugepages() >> File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2393, in _cleanup_hugepages
>> self.nr_hugepages, self.hugepagesz >> File
"/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 243, in calculate_required_deallocation
>> _preallocated_hugepages(vm_hugepagesz)) >> File
"/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 262, in _preallocated_hugepages
>> kernel_args['hugepagesz'] >> File
"/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 291, in _cmdline_hugepagesz_to_kb
>> }[cmdline] >> KeyError: '1048576' >> 2021-04-03 04:40:35,531+0000 INFO (jsonrpc/1)
[api.virt] FINISH destroy return={'status': {'message': 'General Exception: ("\'1048576\'",)', 'code': 100}} from=::ffff:10.100.0.210,58150, vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:54)
>> 2021-04-03 04:40:35,532+0000 INFO (jsonrpc/1)
[jsonrpc.JsonRpcServer] RPC call VM.destroy failed (error 100) in 0.02 seconds (__init__:312)
>> _______________________________________________ >> Users mailing list -- users@ovirt.org <mailto:
users@ovirt.org> <mailto:users@ovirt.org <mailto:users@ovirt.org>>
>> To unsubscribe send an email to users-leave@ovirt.org
<mailto:users-leave@ovirt.org> <mailto:users-leave@ovirt.org <mailto: users-leave@ovirt.org>>
>> Privacy Statement:
https://www.ovirt.org/privacy-policy.html < https://www.ovirt.org/privacy-policy.html> < https://www.ovirt.org/privacy-policy.html < https://www.ovirt.org/privacy-policy.html>>
>> oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/ < https://www.ovirt.org/community/about/community-guidelines/> < https://www.ovirt.org/community/about/community-guidelines/ < https://www.ovirt.org/community/about/community-guidelines/>>
>> List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZ... < https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZXGYRSROHCO7LAS4VOM/> < https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZ... < https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZ...
>>
Hi Shani, Thank you! I've successfully nuked the zombie VMs from this host, and after a reboot, everything is back to normal again. Not sure how these VMs got stuck like this in the first place, but at least I have an option for cleaning it up now. David Kerry On 2021-04-08 10:28 a.m., Shani Leviim wrote:
You can find your virsh user and password in /etc/ovirt-hosted-engine/virsh_auth.conf The content should be something like this:
sudo cat /etc/ovirt-hosted-engine/virsh_auth.conf [credentials-vdsm] authname=vdsm@ovirt password=mypassword
*Regards, * *Shani Leviim *
On Thu, Apr 8, 2021 at 5:20 PM David Kerry <davidk@riavera.com <mailto:davidk@riavera.com>> wrote:
Hi Shani,
I actually came across that option and attempted it at one point, but vdsm has locked me out of using that command it seems.
Eg:
[root@ovirt-node217 ~]# virsh undefine vm-s2 Please enter your authentication name: admin Please enter your password: error: failed to connect to the hypervisor error: authentication failed: authentication failed
No known username/password seems to work.
Is there some magic user to use for this, or some way to bypass the authentication?
Thanks
David
On 2021-04-08 10:10 a.m., Shani Leviim wrote: > Hi David, > Yes - this one will remove completely the VM from the DB. > > You can use the virsh command to delete the VM guests: > https://www.cyberciti.biz/faq/howto-linux-delete-a-running-vm-guest-on-kvm/ <https://www.cyberciti.biz/faq/howto-linux-delete-a-running-vm-guest-on-kvm/> <https://www.cyberciti.biz/faq/howto-linux-delete-a-running-vm-guest-on-kvm/ <https://www.cyberciti.biz/faq/howto-linux-delete-a-running-vm-guest-on-kvm/>> > > *Regards, > * > *Shani Leviim > * > > > On Thu, Apr 8, 2021 at 4:32 PM David Kerry <davidk@riavera.com <mailto:davidk@riavera.com> <mailto:davidk@riavera.com <mailto:davidk@riavera.com>>> wrote: > > Hi Shani, > > These VMs in particular are running just fine on other hosts (and > I'd like to keep them that way, preferably). > > It looks like this command would delete the whole VM from the > entire system instead of just removing the stuck/shutdown instances > from the hosts it's not running on any more. > > Can you confirm this is what it would do? If so, is there another > option to remove these stuck "ghost" VM instances from the hosts they are > no longer running on? > > > Thanks > > David > > > On 2021-04-08 3:20 a.m., Shani Leviim wrote: > > Hi David, > > You can delete the VM from the DB using this command: > > SELECT DeleteVm('<vm-id>'); > > > > *Regards, > > * > > *Shani Leviim > > * > > > > > > On Wed, Apr 7, 2021 at 4:23 PM David Kerry <davidk@riavera.com <mailto:davidk@riavera.com> <mailto:davidk@riavera.com <mailto:davidk@riavera.com>> <mailto:davidk@riavera.com <mailto:davidk@riavera.com> <mailto:davidk@riavera.com <mailto:davidk@riavera.com>>>> wrote: > > > > Hello, > > > > This seems to be what the engine is trying to do, and failing at for some reason. > > > > eg: > > > > [root@ovirt-node217 ~]# vdsm-client Host getVMList fullStatus=True > > [ > > "8b3964bc-cd3f-4f13-84c6-1811193c93eb", > > "132668b6-9992-451f-95ac-dbcbeb03f5f1" > > ] > > > > For reference: > > > > [root@ovirt-node217 ~]# virsh -r list --all > > Id Name State > > ---------------------------------------------------- > > - vm-s2 shut off > > - vm-s1 shut off > > > > And in the console, it shows a count of "2" beside this host, but on the host detail > > page, under the virtual-machine tab, the list is empty (these VMs are actually > > running on a different host). > > > > [root@ovirt-node217 ~]# vdsm-client VM destroy vmID="8b3964bc-cd3f-4f13-84c6-1811193c93eb" > > vdsm-client: Command VM.destroy with args {'vmID': '8b3964bc-cd3f-4f13-84c6-1811193c93eb'} failed: > > (code=100, message=General Exception: ("'1048576'",)) > > > > I guess what I need is a way to remove/clean-up these VMs manually since ovirt > > does not seem to be able to do it by itself. > > > > This condition also blocks the host from being put into maintenance mode. > > > > When I reboot the host manually and "confirm host was rebooted", the VMs > > are still there and still stuck. > > > > Sincerely, > > > > David > > > > > > On 2021-04-07 6:01 a.m., Shani Leviim wrote: > >> Hi, > >> You can try with the vdsm-client tool: > >> https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html <https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html> <https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html <https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html>> <https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html <https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html> <https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html <https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html>>> > >> > >> Stopping a VM: > >> 1) Get the vmId: > >> # vdsm-client Host getVMList fullStatus=True > >> > >> 2) Destroy the VM > >> # vdsm-client VM destroy vmID=<vmID> > >> > >> *Regards, > >> * > >> *Shani Leviim > >> * > >> > >> > >> On Sat, Apr 3, 2021 at 7:50 AM <davidk@riavera.com <mailto:davidk@riavera.com> <mailto:davidk@riavera.com <mailto:davidk@riavera.com>> <mailto:davidk@riavera.com <mailto:davidk@riavera.com> <mailto:davidk@riavera.com <mailto:davidk@riavera.com>>>> wrote: > >> > >> Hello, > >> > >> I've somehow gotten one of my VMs stuck in a state that ovirt seems to be rather confused about its > >> existence of now. I'm running oVirt 4.3.10 and using oVirt Node on all the hosts. > >> > >> My engine and host event logs are now filling up very rapidly with this error: > >> > >> VDSM node217 command DestroyVDS failed: General Exception: ("'1048576'",) > >> > >> I was playing with hugetable support, and that error number or string looks suspiciously > >> like the "hugetable size" custom property I set on the VM. > >> > >> This VM was migrated to another host at one point as well, and now that host is also > >> generating the same error as well. > >> > >> When I try to move these hosts to maintenance mode, they get stuck in "Preparing for > >> Maintenance" while it tries to migrate/deal with the VM that's not there any more. > >> > >> Forcibly rebooting the hosts does not change anything. The VM state/host seems to be > >> captured somewhere persistent in this case. > >> > >> The VM in question is not running, and I can start it up on another host successfully, > >> but ovirt still thinks it exists on the other 2 hosts no matter what I do. > >> > >> Is there perhaps some way to delete it from the engine database directly to straighten > >> things out? > >> > >> Here's a dump of the vdsm log on one of the hosts. I haven't been able to pinpoint what > >> the exact issue is or how to fix it, but hopefully someone here will have seen this before? > >> > >> 2021-04-03 04:40:35,515+0000 INFO (jsonrpc/1) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:10.100.0.210,58150, vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:48) > >> 2021-04-03 04:40:35,516+0000 INFO (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Release VM resources (vm:5186) > >> 2021-04-03 04:40:35,516+0000 WARN (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') trying to set state to Powering down when already Down (vm:626) > >> 2021-04-03 04:40:35,516+0000 INFO (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection (guestagent:455) > >> 2021-04-03 04:40:35,517+0000 INFO (jsonrpc/1) [vdsm.api] START teardownImage(sdUUID='a08af6be-3802-4bb1-9fa5-4b6a10227290', spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9', imgUUID='9c896907-59b0-4983-9478-b36b2c2eb01e', volUUID=None) from=::ffff:10.100.0.210,58150, task_id=fc946d20-126a-4fd0-9078-91 > >> 4b4a64b1d9 (api:48) > >> 2021-04-03 04:40:35,518+0000 INFO (jsonrpc/1) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/a08af6be-3802-4bb1-9fa5-4b6a10227290/9c896907-59b0-4983-9478-b36b2c2eb01e' (fileSD:592) > >> 2021-04-03 04:40:35,518+0000 INFO (jsonrpc/1) [vdsm.api] FINISH teardownImage return=None from=::ffff:10.100.0.210,58150, task_id=fc946d20-126a-4fd0-9078-914b4a64b1d9 (api:54) > >> 2021-04-03 04:40:35,519+0000 INFO (jsonrpc/1) [vdsm.api] START teardownImage(sdUUID='b891448d-dd92-4a7b-a51a-22abc3d7da67', spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9', imgUUID='c0e95483-35f1-4a61-958e-4e308b70d3f8', volUUID=None) from=::ffff:10.100.0.210,58150, task_id=77c0fdca-e13a-44b5-9a00-29 > >> 0522b194b2 (api:48) > >> 2021-04-03 04:40:35,520+0000 INFO (jsonrpc/1) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/b891448d-dd92-4a7b-a51a-22abc3d7da67/c0e95483-35f1-4a61-958e-4e308b70d3f8' (fileSD:592) > >> 2021-04-03 04:40:35,520+0000 INFO (jsonrpc/1) [vdsm.api] FINISH teardownImage return=None from=::ffff:10.100.0.210,58150, task_id=77c0fdca-e13a-44b5-9a00-290522b194b2 (api:54) > >> 2021-04-03 04:40:35,521+0000 INFO (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection (guestagent:455) > >> 2021-04-03 04:40:35,521+0000 WARN (jsonrpc/1) [root] File: /var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.ovirt-guest-agent.0 already removed (fileutils:54) > >> 2021-04-03 04:40:35,521+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing network: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:198) > >> 2021-04-03 04:40:35,522+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing net user: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:205) > >> 2021-04-03 04:40:35,526+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing network: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:198) > >> 2021-04-03 04:40:35,526+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing net user: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:205) > >> 2021-04-03 04:40:35,527+0000 WARN (jsonrpc/1) [root] File: /var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.org.qemu.guest_agent.0 already removed (fileutils:54) > >> 2021-04-03 04:40:35,528+0000 WARN (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') timestamp already removed from stats cache (vm:2445) > >> 2021-04-03 04:40:35,531+0000 ERROR (jsonrpc/1) [api] FINISH destroy error='1048576' (api:134) > >> Traceback (most recent call last): > >> File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 124, in method > >> ret = func(*args, **kwargs) > >> File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 333, in destroy > >> res = self.vm.destroy(gracefulAttempts) > >> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5283, in destroy > >> result = self.doDestroy(gracefulAttempts, reason) > >> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5302, in doDestroy > >> return self.releaseVm(gracefulAttempts) > >> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5207, in releaseVm > >> self._cleanup() > >> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2382, in _cleanup > >> self._cleanup_hugepages() > >> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2393, in _cleanup_hugepages > >> self.nr_hugepages, self.hugepagesz > >> File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 243, in calculate_required_deallocation > >> _preallocated_hugepages(vm_hugepagesz)) > >> File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 262, in _preallocated_hugepages > >> kernel_args['hugepagesz'] > >> File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 291, in _cmdline_hugepagesz_to_kb > >> }[cmdline] > >> KeyError: '1048576' > >> 2021-04-03 04:40:35,531+0000 INFO (jsonrpc/1) [api.virt] FINISH destroy return={'status': {'message': 'General Exception: ("\'1048576\'",)', 'code': 100}} from=::ffff:10.100.0.210,58150, vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:54) > >> 2021-04-03 04:40:35,532+0000 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call VM.destroy failed (error 100) in 0.02 seconds (__init__:312) > >> _______________________________________________ > >> Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> <mailto:users@ovirt.org <mailto:users@ovirt.org>> <mailto:users@ovirt.org <mailto:users@ovirt.org> <mailto:users@ovirt.org <mailto:users@ovirt.org>>> > >> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>> <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org> <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>>> > >> Privacy Statement: https://www.ovirt.org/privacy-policy.html <https://www.ovirt.org/privacy-policy.html> <https://www.ovirt.org/privacy-policy.html <https://www.ovirt.org/privacy-policy.html>> <https://www.ovirt.org/privacy-policy.html <https://www.ovirt.org/privacy-policy.html> <https://www.ovirt.org/privacy-policy.html <https://www.ovirt.org/privacy-policy.html>>> > >> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> <https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/>> <https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> <https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/>>> > >> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZ... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZXGYRSROHCO7LAS4VOM/> <https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZ... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZXGYRSROHCO7LAS4VOM/>> <https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZ... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZXGYRSROHCO7LAS4VOM/> <https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZ... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZXGYRSROHCO7LAS4VOM/>>> > >> >
username: vdsm@ovirrt, password: shibboleth On Thu, Apr 8, 2021 at 5:25 PM David Kerry <davidk@riavera.com> wrote:
Hi Shani,
I actually came across that option and attempted it at one point, but vdsm has locked me out of using that command it seems.
Eg:
[root@ovirt-node217 ~]# virsh undefine vm-s2 Please enter your authentication name: admin Please enter your password: error: failed to connect to the hypervisor error: authentication failed: authentication failed
No known username/password seems to work.
Is there some magic user to use for this, or some way to bypass the authentication?
Thanks
David
On 2021-04-08 10:10 a.m., Shani Leviim wrote:
Hi David, Yes - this one will remove completely the VM from the DB.
You can use the virsh command to delete the VM guests: https://www.cyberciti.biz/faq/howto-linux-delete-a-running-vm-guest-on-kvm/ <https://www.cyberciti.biz/faq/howto-linux-delete-a-running-vm-guest-on-kvm/>
*Regards, * *Shani Leviim *
On Thu, Apr 8, 2021 at 4:32 PM David Kerry <davidk@riavera.com <mailto:davidk@riavera.com>> wrote:
Hi Shani,
These VMs in particular are running just fine on other hosts (and I'd like to keep them that way, preferably).
It looks like this command would delete the whole VM from the entire system instead of just removing the stuck/shutdown instances from the hosts it's not running on any more.
Can you confirm this is what it would do? If so, is there another option to remove these stuck "ghost" VM instances from the hosts they are no longer running on?
Thanks
David
On 2021-04-08 3:20 a.m., Shani Leviim wrote: > Hi David, > You can delete the VM from the DB using this command: > SELECT DeleteVm('<vm-id>'); > > *Regards, > * > *Shani Leviim > * > > > On Wed, Apr 7, 2021 at 4:23 PM David Kerry <davidk@riavera.com <mailto:davidk@riavera.com> <mailto:davidk@riavera.com <mailto:davidk@riavera.com>>> wrote: > > Hello, > > This seems to be what the engine is trying to do, and failing at for some reason. > > eg: > > [root@ovirt-node217 ~]# vdsm-client Host getVMList fullStatus=True > [ > "8b3964bc-cd3f-4f13-84c6-1811193c93eb", > "132668b6-9992-451f-95ac-dbcbeb03f5f1" > ] > > For reference: > > [root@ovirt-node217 ~]# virsh -r list --all > Id Name State > ---------------------------------------------------- > - vm-s2 shut off > - vm-s1 shut off > > And in the console, it shows a count of "2" beside this host, but on the host detail > page, under the virtual-machine tab, the list is empty (these VMs are actually > running on a different host). > > [root@ovirt-node217 ~]# vdsm-client VM destroy vmID="8b3964bc-cd3f-4f13-84c6-1811193c93eb" > vdsm-client: Command VM.destroy with args {'vmID': '8b3964bc-cd3f-4f13-84c6-1811193c93eb'} failed: > (code=100, message=General Exception: ("'1048576'",)) > > I guess what I need is a way to remove/clean-up these VMs manually since ovirt > does not seem to be able to do it by itself. > > This condition also blocks the host from being put into maintenance mode. > > When I reboot the host manually and "confirm host was rebooted", the VMs > are still there and still stuck. > > Sincerely, > > David > > > On 2021-04-07 6:01 a.m., Shani Leviim wrote: >> Hi, >> You can try with the vdsm-client tool: >> https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html <https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html> <https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html <https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html>> >> >> Stopping a VM: >> 1) Get the vmId: >> # vdsm-client Host getVMList fullStatus=True >> >> 2) Destroy the VM >> # vdsm-client VM destroy vmID=<vmID> >> >> *Regards, >> * >> *Shani Leviim >> * >> >> >> On Sat, Apr 3, 2021 at 7:50 AM <davidk@riavera.com <mailto:davidk@riavera.com> <mailto:davidk@riavera.com <mailto:davidk@riavera.com>>> wrote: >> >> Hello, >> >> I've somehow gotten one of my VMs stuck in a state that ovirt seems to be rather confused about its >> existence of now. I'm running oVirt 4.3.10 and using oVirt Node on all the hosts. >> >> My engine and host event logs are now filling up very rapidly with this error: >> >> VDSM node217 command DestroyVDS failed: General Exception: ("'1048576'",) >> >> I was playing with hugetable support, and that error number or string looks suspiciously >> like the "hugetable size" custom property I set on the VM. >> >> This VM was migrated to another host at one point as well, and now that host is also >> generating the same error as well. >> >> When I try to move these hosts to maintenance mode, they get stuck in "Preparing for >> Maintenance" while it tries to migrate/deal with the VM that's not there any more. >> >> Forcibly rebooting the hosts does not change anything. The VM state/host seems to be >> captured somewhere persistent in this case. >> >> The VM in question is not running, and I can start it up on another host successfully, >> but ovirt still thinks it exists on the other 2 hosts no matter what I do. >> >> Is there perhaps some way to delete it from the engine database directly to straighten >> things out? >> >> Here's a dump of the vdsm log on one of the hosts. I haven't been able to pinpoint what >> the exact issue is or how to fix it, but hopefully someone here will have seen this before? >> >> 2021-04-03 04:40:35,515+0000 INFO (jsonrpc/1) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:10.100.0.210,58150, vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:48) >> 2021-04-03 04:40:35,516+0000 INFO (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Release VM resources (vm:5186) >> 2021-04-03 04:40:35,516+0000 WARN (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') trying to set state to Powering down when already Down (vm:626) >> 2021-04-03 04:40:35,516+0000 INFO (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection (guestagent:455) >> 2021-04-03 04:40:35,517+0000 INFO (jsonrpc/1) [vdsm.api] START teardownImage(sdUUID='a08af6be-3802-4bb1-9fa5-4b6a10227290', spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9', imgUUID='9c896907-59b0-4983-9478-b36b2c2eb01e', volUUID=None) from=::ffff:10.100.0.210,58150, task_id=fc946d20-126a-4fd0-9078-91 >> 4b4a64b1d9 (api:48) >> 2021-04-03 04:40:35,518+0000 INFO (jsonrpc/1) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/a08af6be-3802-4bb1-9fa5-4b6a10227290/9c896907-59b0-4983-9478-b36b2c2eb01e' (fileSD:592) >> 2021-04-03 04:40:35,518+0000 INFO (jsonrpc/1) [vdsm.api] FINISH teardownImage return=None from=::ffff:10.100.0.210,58150, task_id=fc946d20-126a-4fd0-9078-914b4a64b1d9 (api:54) >> 2021-04-03 04:40:35,519+0000 INFO (jsonrpc/1) [vdsm.api] START teardownImage(sdUUID='b891448d-dd92-4a7b-a51a-22abc3d7da67', spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9', imgUUID='c0e95483-35f1-4a61-958e-4e308b70d3f8', volUUID=None) from=::ffff:10.100.0.210,58150, task_id=77c0fdca-e13a-44b5-9a00-29 >> 0522b194b2 (api:48) >> 2021-04-03 04:40:35,520+0000 INFO (jsonrpc/1) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/b891448d-dd92-4a7b-a51a-22abc3d7da67/c0e95483-35f1-4a61-958e-4e308b70d3f8' (fileSD:592) >> 2021-04-03 04:40:35,520+0000 INFO (jsonrpc/1) [vdsm.api] FINISH teardownImage return=None from=::ffff:10.100.0.210,58150, task_id=77c0fdca-e13a-44b5-9a00-290522b194b2 (api:54) >> 2021-04-03 04:40:35,521+0000 INFO (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection (guestagent:455) >> 2021-04-03 04:40:35,521+0000 WARN (jsonrpc/1) [root] File: /var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.ovirt-guest-agent.0 already removed (fileutils:54) >> 2021-04-03 04:40:35,521+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing network: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:198) >> 2021-04-03 04:40:35,522+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing net user: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:205) >> 2021-04-03 04:40:35,526+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing network: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:198) >> 2021-04-03 04:40:35,526+0000 WARN (jsonrpc/1) [root] Attempting to remove a non existing net user: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:205) >> 2021-04-03 04:40:35,527+0000 WARN (jsonrpc/1) [root] File: /var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.org.qemu.guest_agent.0 already removed (fileutils:54) >> 2021-04-03 04:40:35,528+0000 WARN (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') timestamp already removed from stats cache (vm:2445) >> 2021-04-03 04:40:35,531+0000 ERROR (jsonrpc/1) [api] FINISH destroy error='1048576' (api:134) >> Traceback (most recent call last): >> File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 124, in method >> ret = func(*args, **kwargs) >> File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 333, in destroy >> res = self.vm.destroy(gracefulAttempts) >> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5283, in destroy >> result = self.doDestroy(gracefulAttempts, reason) >> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5302, in doDestroy >> return self.releaseVm(gracefulAttempts) >> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5207, in releaseVm >> self._cleanup() >> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2382, in _cleanup >> self._cleanup_hugepages() >> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2393, in _cleanup_hugepages >> self.nr_hugepages, self.hugepagesz >> File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 243, in calculate_required_deallocation >> _preallocated_hugepages(vm_hugepagesz)) >> File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 262, in _preallocated_hugepages >> kernel_args['hugepagesz'] >> File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 291, in _cmdline_hugepagesz_to_kb >> }[cmdline] >> KeyError: '1048576' >> 2021-04-03 04:40:35,531+0000 INFO (jsonrpc/1) [api.virt] FINISH destroy return={'status': {'message': 'General Exception: ("\'1048576\'",)', 'code': 100}} from=::ffff:10.100.0.210,58150, vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:54) >> 2021-04-03 04:40:35,532+0000 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call VM.destroy failed (error 100) in 0.02 seconds (__init__:312) >> _______________________________________________ >> Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> <mailto:users@ovirt.org <mailto:users@ovirt.org>> >> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>> >> Privacy Statement: https://www.ovirt.org/privacy-policy.html <https://www.ovirt.org/privacy-policy.html> <https://www.ovirt.org/privacy-policy.html <https://www.ovirt.org/privacy-policy.html>> >> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> <https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/>> >> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZ... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZXGYRSROHCO7LAS4VOM/> <https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZ... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T6QZXGYRSROHCO7LAS4VOM/>> >>
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/KZ4YZHKUA3THMK...
participants (4)
-
Benny Zlotnik -
David Kerry -
davidk＠riavera.com
-
Shani Leviim