10:15 a.m.
Hi Shani,
Thank you! I've successfully nuked the zombie VMs from this host, and after a
reboot,
everything is back to normal again.
Not sure how these VMs got stuck like this in the first place, but at least I have
an option for cleaning it up now.
David Kerry
On 2021-04-08 10:28 a.m., Shani Leviim wrote:
> You can find your virsh user and password in
/etc/ovirt-hosted-engine/virsh_auth.conf
> The content should be something like this:
>
> sudo cat /etc/ovirt-hosted-engine/virsh_auth.conf
> [credentials-vdsm]
> authname=vdsm@ovirt
> password=mypassword
>
>
> *Regards,
> *
> *Shani Leviim
> *
>
>
> On Thu, Apr 8, 2021 at 5:20 PM David Kerry <davidk(a)riavera.com
<mailto:davidk@riavera.com>> wrote:
>
> Hi Shani,
>
> I actually came across that option and attempted it at one point,
> but vdsm has locked me out of using that command it seems.
>
> Eg:
>
> [root@ovirt-node217 ~]# virsh undefine vm-s2
> Please enter your authentication name: admin
> Please enter your password:
> error: failed to connect to the hypervisor
> error: authentication failed: authentication failed
>
> No known username/password seems to work.
>
> Is there some magic user to use for this, or some way
> to bypass the authentication?
>
> Thanks
>
> David
>
> On 2021-04-08 10:10 a.m., Shani Leviim wrote:
> > Hi David,
> > Yes - this one will remove completely the VM from the DB.
> >
> > You can use the virsh command to delete the VM guests:
> > https://www.cyberciti.biz/faq/howto-linux-delete-a-running-vm-guest-on-kvm/
<https://www.cyberciti.biz/faq/howto-linux-delete-a-running-vm-guest-on-kv...
<https://www.cyberciti.biz/faq/howto-linux-delete-a-running-vm-guest-on-kvm/
<https://www.cyberciti.biz/faq/howto-linux-delete-a-running-vm-guest-on-kv...
> >
> > *Regards,
> > *
> > *Shani Leviim
> > *
> >
> >
> > On Thu, Apr 8, 2021 at 4:32 PM David Kerry <davidk(a)riavera.com
<mailto:davidk@riavera.com> <mailto:davidk@riavera.com
<mailto:davidk@riavera.com>>> wrote:
> >
> > Hi Shani,
> >
> > These VMs in particular are running just fine on other hosts (and
> > I'd like to keep them that way, preferably).
> >
> > It looks like this command would delete the whole VM from the
> > entire system instead of just removing the stuck/shutdown instances
> > from the hosts it's not running on any more.
> >
> > Can you confirm this is what it would do? If so, is there another
> > option to remove these stuck "ghost" VM instances from the
hosts they are
> > no longer running on?
> >
> >
> > Thanks
> >
> > David
> >
> >
> > On 2021-04-08 3:20 a.m., Shani Leviim wrote:
> > > Hi David,
> > > You can delete the VM from the DB using this command:
> > > SELECT DeleteVm('<vm-id>');
> > >
> > > *Regards,
> > > *
> > > *Shani Leviim
> > > *
> > >
> > >
> > > On Wed, Apr 7, 2021 at 4:23 PM David Kerry <davidk(a)riavera.com
<mailto:davidk@riavera.com> <mailto:davidk@riavera.com
<mailto:davidk@riavera.com>> <mailto:davidk@riavera.com
<mailto:davidk@riavera.com> <mailto:davidk@riavera.com
<mailto:davidk@riavera.com>>>> wrote:
> > >
> > > Hello,
> > >
> > > This seems to be what the engine is trying to do, and failing
at for some reason.
> > >
> > > eg:
> > >
> > > [root@ovirt-node217 ~]# vdsm-client Host getVMList
fullStatus=True
> > > [
> > > "8b3964bc-cd3f-4f13-84c6-1811193c93eb",
> > > "132668b6-9992-451f-95ac-dbcbeb03f5f1"
> > > ]
> > >
> > > For reference:
> > >
> > > [root@ovirt-node217 ~]# virsh -r list --all
> > > Id Name State
> > > ----------------------------------------------------
> > > - vm-s2 shut off
> > > - vm-s1 shut off
> > >
> > > And in the console, it shows a count of "2" beside
this host, but on the host detail
> > > page, under the virtual-machine tab, the list is empty (these
VMs are actually
> > > running on a different host).
> > >
> > > [root@ovirt-node217 ~]# vdsm-client VM destroy
vmID="8b3964bc-cd3f-4f13-84c6-1811193c93eb"
> > > vdsm-client: Command VM.destroy with args {'vmID':
'8b3964bc-cd3f-4f13-84c6-1811193c93eb'} failed:
> > > (code=100, message=General Exception:
("'1048576'",))
> > >
> > > I guess what I need is a way to remove/clean-up these VMs
manually since ovirt
> > > does not seem to be able to do it by itself.
> > >
> > > This condition also blocks the host from being put into
maintenance mode.
> > >
> > > When I reboot the host manually and "confirm host was
rebooted", the VMs
> > > are still there and still stuck.
> > >
> > > Sincerely,
> > >
> > > David
> > >
> > >
> > > On 2021-04-07 6:01 a.m., Shani Leviim wrote:
> > >> Hi,
> > >> You can try with the vdsm-client tool:
> > >>
https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html
<https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html>
<https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html
<https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html>>
<https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html
<https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html>
<https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html
<https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html>&g...
> > >>
> > >> Stopping a VM:
> > >> 1) Get the vmId:
> > >> # vdsm-client Host getVMList fullStatus=True
> > >>
> > >> 2) Destroy the VM
> > >> # vdsm-client VM destroy vmID=<vmID>
> > >>
> > >> *Regards,
> > >> *
> > >> *Shani Leviim
> > >> *
> > >>
> > >>
> > >> On Sat, Apr 3, 2021 at 7:50 AM <davidk(a)riavera.com
<mailto:davidk@riavera.com> <mailto:davidk@riavera.com
<mailto:davidk@riavera.com>> <mailto:davidk@riavera.com
<mailto:davidk@riavera.com> <mailto:davidk@riavera.com
<mailto:davidk@riavera.com>>>> wrote:
> > >>
> > >> Hello,
> > >>
> > >> I've somehow gotten one of my VMs stuck in a
state that ovirt seems to be rather confused about its
> > >> existence of now. I'm running oVirt 4.3.10 and
using oVirt Node on all the hosts.
> > >>
> > >> My engine and host event logs are now filling up very
rapidly with this error:
> > >>
> > >> VDSM node217 command DestroyVDS failed: General
Exception: ("'1048576'",)
> > >>
> > >> I was playing with hugetable support, and that error
number or string looks suspiciously
> > >> like the "hugetable size" custom property I
set on the VM.
> > >>
> > >> This VM was migrated to another host at one point as
well, and now that host is also
> > >> generating the same error as well.
> > >>
> > >> When I try to move these hosts to maintenance mode,
they get stuck in "Preparing for
> > >> Maintenance" while it tries to migrate/deal with
the VM that's not there any more.
> > >>
> > >> Forcibly rebooting the hosts does not change
anything. The VM state/host seems to be
> > >> captured somewhere persistent in this case.
> > >>
> > >> The VM in question is not running, and I can start it
up on another host successfully,
> > >> but ovirt still thinks it exists on the other 2 hosts
no matter what I do.
> > >>
> > >> Is there perhaps some way to delete it from the
engine database directly to straighten
> > >> things out?
> > >>
> > >> Here's a dump of the vdsm log on one of the
hosts. I haven't been able to pinpoint what
> > >> the exact issue is or how to fix it, but hopefully
someone here will have seen this before?
> > >>
> > >> 2021-04-03 04:40:35,515+0000 INFO (jsonrpc/1)
[api.virt] START destroy(gracefulAttempts=1) from=::ffff:10.100.0.210,58150,
vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:48)
> > >> 2021-04-03 04:40:35,516+0000 INFO (jsonrpc/1)
[virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Release VM resources
(vm:5186)
> > >> 2021-04-03 04:40:35,516+0000 WARN (jsonrpc/1)
[virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') trying to set state to
Powering down when already Down (vm:626)
> > >> 2021-04-03 04:40:35,516+0000 INFO (jsonrpc/1)
[virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection
(guestagent:455)
> > >> 2021-04-03 04:40:35,517+0000 INFO (jsonrpc/1)
[vdsm.api] START teardownImage(sdUUID='a08af6be-3802-4bb1-9fa5-4b6a10227290',
spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9',
imgUUID='9c896907-59b0-4983-9478-b36b2c2eb01e', volUUID=None)
from=::ffff:10.100.0.210,58150, task_id=fc946d20-126a-4fd0-9078-91
> > >> 4b4a64b1d9 (api:48)
> > >> 2021-04-03 04:40:35,518+0000 INFO (jsonrpc/1)
[storage.StorageDomain] Removing image rundir link
u'/var/run/vdsm/storage/a08af6be-3802-4bb1-9fa5-4b6a10227290/9c896907-59b0-4983-9478-b36b2c2eb01e'
(fileSD:592)
> > >> 2021-04-03 04:40:35,518+0000 INFO (jsonrpc/1)
[vdsm.api] FINISH teardownImage return=None from=::ffff:10.100.0.210,58150,
task_id=fc946d20-126a-4fd0-9078-914b4a64b1d9 (api:54)
> > >> 2021-04-03 04:40:35,519+0000 INFO (jsonrpc/1)
[vdsm.api] START teardownImage(sdUUID='b891448d-dd92-4a7b-a51a-22abc3d7da67',
spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9',
imgUUID='c0e95483-35f1-4a61-958e-4e308b70d3f8', volUUID=None)
from=::ffff:10.100.0.210,58150, task_id=77c0fdca-e13a-44b5-9a00-29
> > >> 0522b194b2 (api:48)
> > >> 2021-04-03 04:40:35,520+0000 INFO (jsonrpc/1)
[storage.StorageDomain] Removing image rundir link
u'/var/run/vdsm/storage/b891448d-dd92-4a7b-a51a-22abc3d7da67/c0e95483-35f1-4a61-958e-4e308b70d3f8'
(fileSD:592)
> > >> 2021-04-03 04:40:35,520+0000 INFO (jsonrpc/1)
[vdsm.api] FINISH teardownImage return=None from=::ffff:10.100.0.210,58150,
task_id=77c0fdca-e13a-44b5-9a00-290522b194b2 (api:54)
> > >> 2021-04-03 04:40:35,521+0000 INFO (jsonrpc/1)
[virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection
(guestagent:455)
> > >> 2021-04-03 04:40:35,521+0000 WARN (jsonrpc/1) [root]
File:
/var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.ovirt-guest-agent.0
already removed (fileutils:54)
> > >> 2021-04-03 04:40:35,521+0000 WARN (jsonrpc/1) [root]
Attempting to remove a non existing network:
ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:198)
> > >> 2021-04-03 04:40:35,522+0000 WARN (jsonrpc/1) [root]
Attempting to remove a non existing net user:
ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:205)
> > >> 2021-04-03 04:40:35,526+0000 WARN (jsonrpc/1) [root]
Attempting to remove a non existing network:
ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:198)
> > >> 2021-04-03 04:40:35,526+0000 WARN (jsonrpc/1) [root]
Attempting to remove a non existing net user:
ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 (libvirtnetwork:205)
> > >> 2021-04-03 04:40:35,527+0000 WARN (jsonrpc/1) [root]
File:
/var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.org.qemu.guest_agent.0
already removed (fileutils:54)
> > >> 2021-04-03 04:40:35,528+0000 WARN (jsonrpc/1)
[virt.vm] (vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') timestamp already removed
from stats cache (vm:2445)
> > >> 2021-04-03 04:40:35,531+0000 ERROR (jsonrpc/1) [api]
FINISH destroy error='1048576' (api:134)
> > >> Traceback (most recent call last):
> > >> File
"/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 124, in method
> > >> ret = func(*args, **kwargs)
> > >> File
"/usr/lib/python2.7/site-packages/vdsm/API.py", line 333, in destroy
> > >> res = self.vm.destroy(gracefulAttempts)
> > >> File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5283, in destroy
> > >> result = self.doDestroy(gracefulAttempts,
reason)
> > >> File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5302, in doDestroy
> > >> return self.releaseVm(gracefulAttempts)
> > >> File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5207, in releaseVm
> > >> self._cleanup()
> > >> File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2382, in _cleanup
> > >> self._cleanup_hugepages()
> > >> File
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2393, in
_cleanup_hugepages
> > >> self.nr_hugepages, self.hugepagesz
> > >> File
"/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 243, in
calculate_required_deallocation
> > >> _preallocated_hugepages(vm_hugepagesz))
> > >> File
"/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 262, in
_preallocated_hugepages
> > >> kernel_args['hugepagesz']
> > >> File
"/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 291, in
_cmdline_hugepagesz_to_kb
> > >> }[cmdline]
> > >> KeyError: '1048576'
> > >> 2021-04-03 04:40:35,531+0000 INFO (jsonrpc/1)
[api.virt] FINISH destroy return={'status': {'message': 'General
Exception: ("\'1048576\'",)', 'code': 100}}
from=::ffff:10.100.0.210,58150, vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:54)
> > >> 2021-04-03 04:40:35,532+0000 INFO (jsonrpc/1)
[jsonrpc.JsonRpcServer] RPC call VM.destroy failed (error 100) in 0.02 seconds
(__init__:312)
> > >> _______________________________________________
> > >> Users mailing list -- users(a)ovirt.org
<mailto:users@ovirt.org> <mailto:users@ovirt.org
<mailto:users@ovirt.org>> <mailto:users@ovirt.org
<mailto:users@ovirt.org> <mailto:users@ovirt.org
<mailto:users@ovirt.org>>>
> > >> To unsubscribe send an email to users-leave(a)ovirt.org
<mailto:users-leave@ovirt.org> <mailto:users-leave@ovirt.org
<mailto:users-leave@ovirt.org>> <mailto:users-leave@ovirt.org
<mailto:users-leave@ovirt.org> <mailto:users-leave@ovirt.org
<mailto:users-leave@ovirt.org>>>
> > >> Privacy Statement:
https://www.ovirt.org/privacy-policy.html
<https://www.ovirt.org/privacy-policy.html>
<https://www.ovirt.org/privacy-policy.html
<https://www.ovirt.org/privacy-policy.html>>
<https://www.ovirt.org/privacy-policy.html
<https://www.ovirt.org/privacy-policy.html>
<https://www.ovirt.org/privacy-policy.html
<https://www.ovirt.org/privacy-policy.html>>>
> > >> oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
<https://www.ovirt.org/community/about/community-guidelines/>
<https://www.ovirt.org/community/about/community-guidelines/
<https://www.ovirt.org/community/about/community-guidelines/>>
<https://www.ovirt.org/community/about/community-guidelines/
<https://www.ovirt.org/community/about/community-guidelines/>
<https://www.ovirt.org/community/about/community-guidelines/
<https://www.ovirt.org/community/about/community-guidelines/>>>
> > >> List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T...
<https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T...
<https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T...
<https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T...
<https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T...
<https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T...
<https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T...
<https://lists.ovirt.org/archives/list/users@ovirt.org/message/KBABFNZIJ6T...
> > >>
> >
>