Hi, You can try with the vdsm-client tool: https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html Stopping a VM: 1) Get the vmId: # vdsm-client Host getVMList fullStatus=True 2) Destroy the VM # vdsm-client VM destroy vmID=<vmID> *Regards,* *Shani Leviim* On Sat, Apr 3, 2021 at 7:50 AM <davidk(a)riavera.com&gt; wrote:

Hi David, You can delete the VM from the DB using this command: SELECT DeleteVm('<vm-id>'); *Regards,* *Shani Leviim* On Wed, Apr 7, 2021 at 4:23 PM David Kerry <davidk(a)riavera.com&gt; wrote:

Hi David, Yes - this one will remove completely the VM from the DB. You can use the virsh command to delete the VM guests: https://www.cyberciti.biz/faq/howto-linux-delete-a-running-vm-guest-on-kvm/ *Regards,* *Shani Leviim* On Thu, Apr 8, 2021 at 4:32 PM David Kerry <davidk(a)riavera.com&gt; wrote:

You can find your virsh user and password in /etc/ovirt-hosted-engine/virsh_auth.conf The content should be something like this: sudo cat /etc/ovirt-hosted-engine/virsh_auth.conf [credentials-vdsm] authname=vdsm@ovirt password=mypassword *Regards,* *Shani Leviim* On Thu, Apr 8, 2021 at 5:20 PM David Kerry <davidk(a)riavera.com&gt; wrote:

Hi Shani, Thank you! I've successfully nuked the zombie VMs from this host, and after a reboot, everything is back to normal again. Not sure how these VMs got stuck like this in the first place, but at least I have an option for cleaning it up now. David Kerry On 2021-04-08 10:28 a.m., Shani Leviim wrote: > You can find your virsh user and password in /etc/ovirt-hosted-engine/virsh_auth.conf > The content should be something like this: > > sudo cat /etc/ovirt-hosted-engine/virsh_auth.conf > [credentials-vdsm] > authname=vdsm@ovirt > password=mypassword > > > *Regards, > * > *Shani Leviim > * > > > On Thu, Apr 8, 2021 at 5:20 PM David Kerry <davidk(a)riavera.com <mailto:davidk@riavera.com>> wrote: > > Hi Shani, > > I actually came across that option and attempted it at one point, > but vdsm has locked me out of using that command it seems. > > Eg: > > [root@ovirt-node217 ~]# virsh undefine vm-s2 > Please enter your authentication name: admin > Please enter your password: > error: failed to connect to the hypervisor > error: authentication failed: authentication failed > > No known username/password seems to work. > > Is there some magic user to use for this, or some way > to bypass the authentication? > > Thanks > > David > > On 2021-04-08 10:10 a.m., Shani Leviim wrote: > > Hi David, > > Yes - this one will remove completely the VM from the DB. > > > > You can use the virsh command to delete the VM guests: > > https://www.cyberciti.biz/faq/howto-linux-delete-a-running-vm-guest-on-kvm/ <https://www.cyberciti.biz/faq/howto-linux-delete-a-running-vm-guest-on-kv... <https://www.cyberciti.biz/faq/howto-linux-delete-a-running-vm-guest-on-kvm/ <https://www.cyberciti.biz/faq/howto-linux-delete-a-running-vm-guest-on-kv... > > > > *Regards, > > * > > *Shani Leviim > > * > > > > > > On Thu, Apr 8, 2021 at 4:32 PM David Kerry <davidk(a)riavera.com <mailto:davidk@riavera.com> <mailto:davidk@riavera.com <mailto:davidk@riavera.com>>> wrote: > > > >     Hi Shani, > > > >     These VMs in particular are running just fine on other hosts (and > >     I'd like to keep them that way, preferably). > > > >     It looks like this command would delete the whole VM from the > >     entire system instead of just removing the stuck/shutdown instances > >     from the hosts it's not running on any more. > > > >     Can you confirm this is what it would do?  If so, is there another > >     option to remove these stuck "ghost" VM instances from the hosts they are > >     no longer running on? > > > > > >     Thanks > > > >     David > > > > > >     On 2021-04-08 3:20 a.m., Shani Leviim wrote: > >      > Hi David, > >      > You can delete the VM from the DB using this command: > >      > SELECT DeleteVm('<vm-id>'); > >      > > >      > *Regards, > >      > * > >      > *Shani Leviim > >      > * > >      > > >      > > >      > On Wed, Apr 7, 2021 at 4:23 PM David Kerry <davidk(a)riavera.com <mailto:davidk@riavera.com> <mailto:davidk@riavera.com <mailto:davidk@riavera.com>> <mailto:davidk@riavera.com <mailto:davidk@riavera.com> <mailto:davidk@riavera.com <mailto:davidk@riavera.com>>>> wrote: > >      > > >      >     Hello, > >      > > >      >     This seems to be what the engine is trying to do, and failing at for some reason. > >      > > >      >     eg: > >      > > >      >     [root@ovirt-node217 ~]# vdsm-client Host getVMList fullStatus=True > >      >     [ > >      >          "8b3964bc-cd3f-4f13-84c6-1811193c93eb", > >      >          "132668b6-9992-451f-95ac-dbcbeb03f5f1" > >      >     ] > >      > > >      >     For reference: > >      > > >      >     [root@ovirt-node217 ~]# virsh -r list --all > >      >       Id    Name                           State > >      >     ---------------------------------------------------- > >      >       -     vm-s2                          shut off > >      >       -     vm-s1                          shut off > >      > > >      >     And in the console, it shows a count of "2" beside this host, but on the host detail > >      >     page, under the virtual-machine tab, the list is empty (these VMs are actually > >      >     running on a different host). > >      > > >      >     [root@ovirt-node217 ~]# vdsm-client VM destroy vmID="8b3964bc-cd3f-4f13-84c6-1811193c93eb" > >      >     vdsm-client: Command VM.destroy with args {'vmID': '8b3964bc-cd3f-4f13-84c6-1811193c93eb'} failed: > >      >     (code=100, message=General Exception: ("'1048576'",)) > >      > > >      >     I guess what I need is a way to remove/clean-up these VMs manually since ovirt > >      >     does not seem to be able to do it by itself. > >      > > >      >     This condition also blocks the host from being put into maintenance mode. > >      > > >      >     When I reboot the host manually and "confirm host was rebooted", the VMs > >      >     are still there and still stuck. > >      > > >      >     Sincerely, > >      > > >      >     David > >      > > >      > > >      >     On 2021-04-07 6:01 a.m., Shani Leviim wrote: > >      >>     Hi, > >      >>     You can try with the vdsm-client tool: > >      >> https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html <https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html> <https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html <https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html>> <https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html <https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html> <https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html <https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html>&g... > >      >> > >      >>     Stopping a VM: > >      >>     1) Get the vmId: > >      >>     # vdsm-client Host getVMList fullStatus=True > >      >> > >      >>     2) Destroy the VM > >      >>     # vdsm-client VM destroy vmID=<vmID> > >      >> > >      >>     *Regards, > >      >>     * > >      >>     *Shani Leviim > >      >>     * > >      >> > >      >> > >      >>     On Sat, Apr 3, 2021 at 7:50 AM <davidk(a)riavera.com <mailto:davidk@riavera.com> <mailto:davidk@riavera.com <mailto:davidk@riavera.com>> <mailto:davidk@riavera.com <mailto:davidk@riavera.com> <mailto:davidk@riavera.com <mailto:davidk@riavera.com>>>> wrote: > >      >> > >      >>         Hello, > >      >> > >      >>         I've somehow gotten one of my VMs stuck in a state that ovirt seems to be rather confused about its > >      >>         existence of now.  I'm running oVirt 4.3.10 and using oVirt Node on all the hosts. > >      >> > >      >>         My engine and host event logs are now filling up very rapidly with this error: > >      >> > >      >>         VDSM node217 command DestroyVDS failed: General Exception: ("'1048576'",) > >      >> > >      >>         I was playing with hugetable support, and that error number or string looks suspiciously > >      >>         like the "hugetable size" custom property I set on the VM. > >      >> > >      >>         This VM was migrated to another host at one point as well, and now that host is also > >      >>         generating the same error as well. > >      >> > >      >>         When I try to move these hosts to maintenance mode, they get stuck in "Preparing for > >      >>         Maintenance" while it tries to migrate/deal with the VM that's not there any more. > >      >> > >      >>         Forcibly rebooting the hosts does not change anything.  The VM state/host seems to be > >      >>         captured somewhere persistent in this case. > >      >> > >      >>         The VM in question is not running, and I can start it up on another host successfully, > >      >>         but ovirt still thinks it exists on the other 2 hosts no matter what I do. > >      >> > >      >>         Is there perhaps some way to delete it from the engine database directly to straighten > >      >>         things out? > >      >> > >      >>         Here's a dump of the vdsm log on one of the hosts.  I haven't been able to pinpoint what > >      >>         the exact issue is or how to fix it, but hopefully someone here will have seen this before? > >      >> > >      >>         2021-04-03 04:40:35,515+0000 INFO  (jsonrpc/1) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:10.100.0.210,58150, vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:48) > >      >>         2021-04-03 04:40:35,516+0000 INFO  (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Release VM resources (vm:5186) > >      >>         2021-04-03 04:40:35,516+0000 WARN  (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') trying to set state to Powering down when already Down (vm:626) > >      >>         2021-04-03 04:40:35,516+0000 INFO  (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection (guestagent:455) > >      >>         2021-04-03 04:40:35,517+0000 INFO  (jsonrpc/1) [vdsm.api] START teardownImage(sdUUID='a08af6be-3802-4bb1-9fa5-4b6a10227290', spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9', imgUUID='9c896907-59b0-4983-9478-b36b2c2eb01e', volUUID=None) from=::ffff:10.100.0.210,58150, task_id=fc946d20-126a-4fd0-9078-91 > >      >>         4b4a64b1d9 (api:48) > >      >>         2021-04-03 04:40:35,518+0000 INFO  (jsonrpc/1) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/a08af6be-3802-4bb1-9fa5-4b6a10227290/9c896907-59b0-4983-9478-b36b2c2eb01e' (fileSD:592) > >      >>         2021-04-03 04:40:35,518+0000 INFO  (jsonrpc/1) [vdsm.api] FINISH teardownImage return=None from=::ffff:10.100.0.210,58150, task_id=fc946d20-126a-4fd0-9078-914b4a64b1d9 (api:54) > >      >>         2021-04-03 04:40:35,519+0000 INFO  (jsonrpc/1) [vdsm.api] START teardownImage(sdUUID='b891448d-dd92-4a7b-a51a-22abc3d7da67', spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9', imgUUID='c0e95483-35f1-4a61-958e-4e308b70d3f8', volUUID=None) from=::ffff:10.100.0.210,58150, task_id=77c0fdca-e13a-44b5-9a00-29 > >      >>         0522b194b2 (api:48) > >      >>         2021-04-03 04:40:35,520+0000 INFO  (jsonrpc/1) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/b891448d-dd92-4a7b-a51a-22abc3d7da67/c0e95483-35f1-4a61-958e-4e308b70d3f8' (fileSD:592) > >      >>         2021-04-03 04:40:35,520+0000 INFO  (jsonrpc/1) [vdsm.api] FINISH teardownImage return=None from=::ffff:10.100.0.210,58150, task_id=77c0fdca-e13a-44b5-9a00-290522b194b2 (api:54) > >      >>         2021-04-03 04:40:35,521+0000 INFO  (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection (guestagent:455) > >      >>         2021-04-03 04:40:35,521+0000 WARN  (jsonrpc/1) [root] File: /var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.ovirt-guest-agent.0 already removed (fileutils:54) > >      >>         2021-04-03 04:40:35,521+0000 WARN  (jsonrpc/1) [root] Attempting to remove a non existing network: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:198) > >      >>         2021-04-03 04:40:35,522+0000 WARN  (jsonrpc/1) [root] Attempting to remove a non existing net user: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:205) > >      >>         2021-04-03 04:40:35,526+0000 WARN  (jsonrpc/1) [root] Attempting to remove a non existing network: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:198) > >      >>         2021-04-03 04:40:35,526+0000 WARN  (jsonrpc/1) [root] Attempting to remove a non existing net user: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:205) > >      >>         2021-04-03 04:40:35,527+0000 WARN  (jsonrpc/1) [root] File: /var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.org.qemu.guest_agent.0 already removed (fileutils:54) > >      >>         2021-04-03 04:40:35,528+0000 WARN  (jsonrpc/1) [virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') timestamp already removed from stats cache (vm:2445) > >      >>         2021-04-03 04:40:35,531+0000 ERROR (jsonrpc/1) [api] FINISH destroy error='1048576' (api:134) > >      >>         Traceback (most recent call last): > >      >>           File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 124, in method > >      >>             ret = func(*args, **kwargs) > >      >>           File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 333, in destroy > >      >>             res = self.vm.destroy(gracefulAttempts) > >      >>           File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5283, in destroy > >      >>             result = self.doDestroy(gracefulAttempts, reason) > >      >>           File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5302, in doDestroy > >      >>             return self.releaseVm(gracefulAttempts) > >      >>           File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5207, in releaseVm > >      >>             self._cleanup() > >      >>           File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2382, in _cleanup > >      >>             self._cleanup_hugepages() > >      >>           File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2393, in _cleanup_hugepages > >      >>             self.nr_hugepages, self.hugepagesz > >      >>           File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 243, in calculate_required_deallocation > >      >>             _preallocated_hugepages(vm_hugepagesz)) > >      >>           File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 262, in _preallocated_hugepages > >      >>             kernel_args['hugepagesz'] > >      >>           File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 291, in _cmdline_hugepagesz_to_kb > >      >>             }[cmdline] > >      >>         KeyError: '1048576' > >      >>         2021-04-03 04:40:35,531+0000 INFO  (jsonrpc/1) [api.virt] FINISH destroy return={'status': {'message': 'General Exception: ("\'1048576\'",)', 'code': 100}} from=::ffff:10.100.0.210,58150, vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:54) > >      >>         2021-04-03 04:40:35,532+0000 INFO  (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call VM.destroy failed (error 100) in 0.02 seconds (__init__:312) > >      >>         _______________________________________________ > >      >>         Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org> <mailto:users@ovirt.org <mailto:users@ovirt.org>> <mailto:users@ovirt.org <mailto:users@ovirt.org> <mailto:users@ovirt.org <mailto:users@ovirt.org>>> > >      >>         To unsubscribe send an email to users-leave(a)ovirt.org <mailto:users-leave@ovirt.org> <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>> <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org> <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>>> > >      >>         Privacy Statement: https://www.ovirt.org/privacy-policy.html <https://www.ovirt.org/privacy-policy.html> <https://www.ovirt.org/privacy-policy.html <https://www.ovirt.org/privacy-policy.html>> <https://www.ovirt.org/privacy-policy.html <https://www.ovirt.org/privacy-policy.html> <https://www.ovirt.org/privacy-policy.html <https://www.ovirt.org/privacy-policy.html>>> > >      >>         oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> <https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/>> <https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> <https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/>>> > >      >>         List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T... <https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T... > >      >> > > >