[ovirt-users] Re: Destroyed VM blocking hosts/filling logs

Wednesday, 7 April 2021

Hello,

This seems to be what the engine is trying to do, and failing at for some reason.

eg:

[root@ovirt-node217 ~]# vdsm-client Host getVMList fullStatus=True
[
     "8b3964bc-cd3f-4f13-84c6-1811193c93eb",
     "132668b6-9992-451f-95ac-dbcbeb03f5f1"
]

For reference:

[root@ovirt-node217 ~]# virsh -r list --all
  Id    Name                           State
----------------------------------------------------
  -     vm-s2                          shut off
  -     vm-s1                          shut off

And in the console, it shows a count of "2" beside this host, but on the host
detail
page, under the virtual-machine tab, the list is empty (these VMs are actually
running on a different host).

[root@ovirt-node217 ~]# vdsm-client VM destroy
vmID="8b3964bc-cd3f-4f13-84c6-1811193c93eb"
vdsm-client: Command VM.destroy with args {'vmID':
'8b3964bc-cd3f-4f13-84c6-1811193c93eb'} failed:
(code=100, message=General Exception: ("'1048576'",))

I guess what I need is a way to remove/clean-up these VMs manually since ovirt
does not seem to be able to do it by itself.

This condition also blocks the host from being put into maintenance mode.

When I reboot the host manually and "confirm host was rebooted", the VMs
are still there and still stuck.

Sincerely,

David

On 2021-04-07 6:01 a.m., Shani Leviim wrote:
...
 Hi,
 You can try with the vdsm-client tool:
 https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html
<https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html>

 Stopping a VM:
 1) Get the vmId:
 # vdsm-client Host getVMList fullStatus=True

 2) Destroy the VM
 # vdsm-client VM destroy vmID=<vmID>

 *Regards,
 *
 *Shani Leviim
 *

 On Sat, Apr 3, 2021 at 7:50 AM <davidk(a)riavera.com
<mailto:davidk@riavera.com>> wrote:

     Hello,

     I've somehow gotten one of my VMs stuck in a state that ovirt seems to be rather
confused about its
     existence of now.  I'm running oVirt 4.3.10 and using oVirt Node on all the
hosts.

     My engine and host event logs are now filling up very rapidly with this error:

     VDSM node217 command DestroyVDS failed: General Exception:
("'1048576'",)

     I was playing with hugetable support, and that error number or string looks
suspiciously
     like the "hugetable size" custom property I set on the VM.

     This VM was migrated to another host at one point as well, and now that host is also
     generating the same error as well.

     When I try to move these hosts to maintenance mode, they get stuck in "Preparing
for
     Maintenance" while it tries to migrate/deal with the VM that's not there any
more.

     Forcibly rebooting the hosts does not change anything.  The VM state/host seems to
be
     captured somewhere persistent in this case.

     The VM in question is not running, and I can start it up on another host
successfully,
     but ovirt still thinks it exists on the other 2 hosts no matter what I do.

     Is there perhaps some way to delete it from the engine database directly to
straighten
     things out?

     Here's a dump of the vdsm log on one of the hosts.  I haven't been able to
pinpoint what
     the exact issue is or how to fix it, but hopefully someone here will have seen this
before?

     2021-04-03 04:40:35,515+0000 INFO  (jsonrpc/1) [api.virt] START
destroy(gracefulAttempts=1) from=::ffff:10.100.0.210,58150,
vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:48)
     2021-04-03 04:40:35,516+0000 INFO  (jsonrpc/1) [virt.vm]
(vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Release VM resources (vm:5186)
     2021-04-03 04:40:35,516+0000 WARN  (jsonrpc/1) [virt.vm]
(vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') trying to set state to Powering down
when already Down (vm:626)
     2021-04-03 04:40:35,516+0000 INFO  (jsonrpc/1) [virt.vm]
(vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection
(guestagent:455)
     2021-04-03 04:40:35,517+0000 INFO  (jsonrpc/1) [vdsm.api] START
teardownImage(sdUUID='a08af6be-3802-4bb1-9fa5-4b6a10227290',
spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9',
imgUUID='9c896907-59b0-4983-9478-b36b2c2eb01e', volUUID=None)
from=::ffff:10.100.0.210,58150, task_id=fc946d20-126a-4fd0-9078-91
     4b4a64b1d9 (api:48)
     2021-04-03 04:40:35,518+0000 INFO  (jsonrpc/1) [storage.StorageDomain] Removing image
rundir link
u'/var/run/vdsm/storage/a08af6be-3802-4bb1-9fa5-4b6a10227290/9c896907-59b0-4983-9478-b36b2c2eb01e'
(fileSD:592)
     2021-04-03 04:40:35,518+0000 INFO  (jsonrpc/1) [vdsm.api] FINISH teardownImage
return=None from=::ffff:10.100.0.210,58150, task_id=fc946d20-126a-4fd0-9078-914b4a64b1d9
(api:54)
     2021-04-03 04:40:35,519+0000 INFO  (jsonrpc/1) [vdsm.api] START
teardownImage(sdUUID='b891448d-dd92-4a7b-a51a-22abc3d7da67',
spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9',
imgUUID='c0e95483-35f1-4a61-958e-4e308b70d3f8', volUUID=None)
from=::ffff:10.100.0.210,58150, task_id=77c0fdca-e13a-44b5-9a00-29
     0522b194b2 (api:48)
     2021-04-03 04:40:35,520+0000 INFO  (jsonrpc/1) [storage.StorageDomain] Removing image
rundir link
u'/var/run/vdsm/storage/b891448d-dd92-4a7b-a51a-22abc3d7da67/c0e95483-35f1-4a61-958e-4e308b70d3f8'
(fileSD:592)
     2021-04-03 04:40:35,520+0000 INFO  (jsonrpc/1) [vdsm.api] FINISH teardownImage
return=None from=::ffff:10.100.0.210,58150, task_id=77c0fdca-e13a-44b5-9a00-290522b194b2
(api:54)
     2021-04-03 04:40:35,521+0000 INFO  (jsonrpc/1) [virt.vm]
(vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection
(guestagent:455)
     2021-04-03 04:40:35,521+0000 WARN  (jsonrpc/1) [root] File:
/var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.ovirt-guest-agent.0
already removed (fileutils:54)
     2021-04-03 04:40:35,521+0000 WARN  (jsonrpc/1) [root] Attempting to remove a non
existing network: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:198)
     2021-04-03 04:40:35,522+0000 WARN  (jsonrpc/1) [root] Attempting to remove a non
existing net user: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:205)
     2021-04-03 04:40:35,526+0000 WARN  (jsonrpc/1) [root] Attempting to remove a non
existing network: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:198)
     2021-04-03 04:40:35,526+0000 WARN  (jsonrpc/1) [root] Attempting to remove a non
existing net user: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:205)
     2021-04-03 04:40:35,527+0000 WARN  (jsonrpc/1) [root] File:
/var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.org.qemu.guest_agent.0
already removed (fileutils:54)
     2021-04-03 04:40:35,528+0000 WARN  (jsonrpc/1) [virt.vm]
(vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') timestamp already removed from stats
cache (vm:2445)
     2021-04-03 04:40:35,531+0000 ERROR (jsonrpc/1) [api] FINISH destroy
error='1048576' (api:134)
     Traceback (most recent call last):
       File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 124, in
method
         ret = func(*args, **kwargs)
       File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 333, in
destroy
         res = self.vm.destroy(gracefulAttempts)
       File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5283, in
destroy
         result = self.doDestroy(gracefulAttempts, reason)
       File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5302, in
doDestroy
         return self.releaseVm(gracefulAttempts)
       File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5207, in
releaseVm
         self._cleanup()
       File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2382, in
_cleanup
         self._cleanup_hugepages()
       File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2393, in
_cleanup_hugepages
         self.nr_hugepages, self.hugepagesz
       File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 243, in
calculate_required_deallocation
         _preallocated_hugepages(vm_hugepagesz))
       File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 262, in
_preallocated_hugepages
         kernel_args['hugepagesz']
       File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 291, in
_cmdline_hugepagesz_to_kb
         }[cmdline]
     KeyError: '1048576'
     2021-04-03 04:40:35,531+0000 INFO  (jsonrpc/1) [api.virt] FINISH destroy
return={'status': {'message': 'General Exception:
("\'1048576\'",)', 'code': 100}}
from=::ffff:10.100.0.210,58150, vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:54)
     2021-04-03 04:40:35,532+0000 INFO  (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call
VM.destroy failed (error 100) in 0.02 seconds (__init__:312)
     _______________________________________________
     Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org>
     To unsubscribe send an email to users-leave(a)ovirt.org
<mailto:users-leave@ovirt.org>
     Privacy Statement: https://www.ovirt.org/privacy-policy.html
<https://www.ovirt.org/privacy-policy.html>
     oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
<https://www.ovirt.org/community/about/community-guidelines/>
     List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T...
<https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T...

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

[ovirt-users] Re: Destroyed VM blocking hosts/filling logs