- Users - oVirt List Archives

oVirt 4.2.5 : VM snapshot creation does not work : command HSMGetAllTasksStatusesVDS failed: Could not acquire resource
by Aleksey Maksimov 17 Aug '18

17 Aug '18

Hello We use oVirt 4.2.5.2-1.el7 (Hosted engine / 4 hosts in cluster / about twenty virtual machines) Virtual machine disks are located on the Data Domain from FC SAN. Snapshots of all virtual machines are created normally. But for one virtual machine, we can not create a snapshot. When we try to create a snapshot in the oVirt web console, we see such errors: Aug 13, 2018, 1:05:06 PM Failed to complete snapshot 'KOM-APP14_BACKUP01' creation for VM 'KOM-APP14'. Aug 13, 2018, 1:05:01 PM VDSM KOM-VM14 command HSMGetAllTasksStatusesVDS failed: Could not acquire resource. Probably resource factory threw an exception.: () Aug 13, 2018, 1:05:00 PM Snapshot 'KOM-APP14_BACKUP01' creation for VM 'KOM-APP14' was initiated by petya@sub.holding.com@sub.holding.com-authz. At this time on the server with the role of "SPM" in the vdsm.log we see this: ... 2018-08-13 05:05:06,471-0500 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.getStats succeeded in 0.00 seconds (__init__:573) 2018-08-13 05:05:06,478-0500 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Image.deleteVolumes succeeded in 0.05 seconds (__init__:573) 2018-08-13 05:05:06,478-0500 INFO (tasks/3) [storage.ThreadPool.WorkerThread] START task bb45ae7e-77e9-4fec-9ee2-8e1f0ad3d589 (cmd=<bound method Task.commit of <vdsm.storage.task.Task instance at 0x7f06b85a2128>>, args=None) (threadPool:208) 2018-08-13 05:05:07,009-0500 WARN (tasks/3) [storage.ResourceManager] Resource factory failed to create resource '01_img_6db73566-0f7f-4438-a9ef-6815075f45ea.cdf1751b-64d3-42bc-b9ef-b0174c7ea068'. Canceling request. (resourceManager:543) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/resourceManager.py", line 539, in registerResource obj = namespaceObj.factory.createResource(name, lockType) File "/usr/lib/python2.7/site-packages/vdsm/storage/resourceFactories.py", line 193, in createResource lockType) File "/usr/lib/python2.7/site-packages/vdsm/storage/resourceFactories.py", line 122, in __getResourceCandidatesList imgUUID=resourceName) File "/usr/lib/python2.7/site-packages/vdsm/storage/image.py", line 213, in getChain if srcVol.isLeaf(): File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 1430, in isLeaf return self._manifest.isLeaf() File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 139, in isLeaf return self.getVolType() == sc.type2name(sc.LEAF_VOL) File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 135, in getVolType self.voltype = self.getMetaParam(sc.VOLTYPE) File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 119, in getMetaParam meta = self.getMetadata() File "/usr/lib/python2.7/site-packages/vdsm/storage/blockVolume.py", line 112, in getMetadata md = VolumeMetadata.from_lines(lines) File "/usr/lib/python2.7/site-packages/vdsm/storage/volumemetadata.py", line 103, in from_lines "Missing metadata key: %s: found: %s" % (e, md)) MetaDataKeyNotFoundError: Meta Data key not found error: ("Missing metadata key: 'DOMAIN': found: {'NONE': '######################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################'}",) 2018-08-13 05:05:07,010-0500 WARN (tasks/3) [storage.ResourceManager.Request] (ResName='01_img_6db73566-0f7f-4438-a9ef-6815075f45ea.cdf1751b-64d3-42bc-b9ef-b0174c7ea068', ReqID='3d924e5e-60d1-47b0-86a7-c63585b56f09') Tried to cancel a processed request (resourceManager:187) 2018-08-13 05:05:07,010-0500 ERROR (tasks/3) [storage.TaskManager.Task] (Task='bb45ae7e-77e9-4fec-9ee2-8e1f0ad3d589') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 336, in run return self.cmd(*self.argslist, **self.argsdict) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1966, in deleteVolume with rm.acquireResource(img_ns, imgUUID, rm.EXCLUSIVE): File "/usr/lib/python2.7/site-packages/vdsm/storage/resourceManager.py", line 1025, in acquireResource return _manager.acquireResource(namespace, name, lockType, timeout=timeout) File "/usr/lib/python2.7/site-packages/vdsm/storage/resourceManager.py", line 475, in acquireResource raise se.ResourceAcqusitionFailed() ResourceAcqusitionFailed: Could not acquire resource. Probably resource factory threw an exception.: () 2018-08-13 05:05:07,059-0500 INFO (tasks/3) [storage.ThreadPool.WorkerThread] FINISH task bb45ae7e-77e9-4fec-9ee2-8e1f0ad3d589 (threadPool:210) 2018-08-13 05:05:07,246-0500 INFO (jsonrpc/1) [root] /usr/libexec/vdsm/hooks/after_get_caps/50_openstacknet: rc=0 err= (hooks:110) 2018-08-13 05:05:07,660-0500 INFO (jsonrpc/1) [root] /usr/libexec/vdsm/hooks/after_get_caps/openstacknet_utils.py: rc=0 err= (hooks:110) 2018-08-13 05:05:08,152-0500 INFO (jsonrpc/1) [root] /usr/libexec/vdsm/hooks/after_get_caps/ovirt_provider_ovn_hook: rc=0 err= (hooks:110) Please help us to solve this problem.

3 8

Clarifications regarding ovirt-provider-ovn network
by Anurag Porripireddi 17 Aug '18

17 Aug '18

Hi, I had some questions a overt-provider-ovn based external network which is built on top of an physical network. I notice that VLAN tagging gets greyed out when I try to create an external provider network. Is it not supported by overt right now?I am on version 4.2.5.3. Also, if I create NICs with the external provider network built atop a physical network on VMs that are on different hosts, but the same cluster, I see they are unable to ping each other, though it seems to work fine if they are on the same host. Is that elected behaviour? Could you you please help me understand why? Thanks, Anurag

3 3

[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Unable to resolve address\n"}
by Ariez Ahito 17 Aug '18

17 Aug '18

failed hosted-engine deployment i tried using command line but encountered this error i also tried downloading ovirt-node iso and using WEBUI but encountered same problem i deployed this hosted-engine 3.5 without any problems. any thoughts?

3 5

Hosted Engine Deployment Fails - Ansible Playbook
by jeremy_tourville＠hotmail.com 17 Aug '18

17 Aug '18

Can someone help me troubleshoot what my issue is? I have tried to setup hosted engine using static and DHCP addresses and both times it fails. Many thanks in advance! [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The system may not be provisioned according to the playbook results: please check the logs for the issue, fix accordingly or re-deploy from scratch.\n"} [ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook [root@vmh /]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp96s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovirtmgmt state UP group default qlen 1000 link/ether 0c:c4:7a:f9:b9:88 brd ff:ff:ff:ff:ff:ff 3: enp96s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether 0c:c4:7a:f9:b9:89 brd ff:ff:ff:ff:ff:ff 20: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 0c:c4:7a:f9:b9:88 brd ff:ff:ff:ff:ff:ff inet 172.30.50.2/24 brd 172.30.50.255 scope global dynamic ovirtmgmt valid_lft 84266sec preferred_lft 84266sec inet6 fe80::ec4:7aff:fef9:b988/64 scope link valid_lft forever preferred_lft forever 21: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:54:00:e2:1c:89 brd ff:ff:ff:ff:ff:ff inet 192.168.124.1/24 brd 192.168.124.255 scope global virbr0 valid_lft forever preferred_lft forever 22: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000 link/ether 52:54:00:e2:1c:89 brd ff:ff:ff:ff:ff:ff 23: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UNKNOWN group default qlen 1000 link/ether fe:16:3e:52:19:f4 brd ff:ff:ff:ff:ff:ff inet6 fe80::fc16:3eff:fe52:19f4/64 scope link valid_lft forever preferred_lft forever 24: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 82:43:87:7f:eb:62 brd ff:ff:ff:ff:ff:ff [root@vmh /]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/onn_vmh-ovirt--node--ng--4.2.5.1--0.20180731.0+1 145G 3.1G 134G 3% / devtmpfs 63G 0 63G 0% /dev tmpfs 63G 4.0K 63G 1% /dev/shm tmpfs 63G 19M 63G 1% /run tmpfs 63G 0 63G 0% /sys/fs/cgroup /dev/mapper/onn_vmh-var 15G 4.5G 9.4G 33% /var /dev/mapper/onn_vmh-tmp 976M 4.1M 905M 1% /tmp /dev/mapper/onn_vmh-home 976M 2.6M 907M 1% /home /dev/mapper/PNY_CS900_240GB_SSD_PNY14182241350103ED2p1 976M 365M 545M 41% /boot /dev/mapper/onn_vmh-var_log 7.8G 70M 7.3G 1% /var/log /dev/mapper/onn_vmh-var_log_audit 2.0G 9.2M 1.8G 1% /var/log/audit /dev/mapper/3600605b00a2faca222fb4da81ac9bdb1p1 7.4T 93M 7.1T 1% /srv tmpfs 13G 0 13G 0% /run/user/0 I have pasted my log file- https://pastebin.com/dyYksxaC

1 0

ovirt-ansible download/upload of snapshots for backup
by Николаев Алексей 16 Aug '18

16 Aug '18

2 2

Issue with NFS and Storage domain setup
by Inquirer Guy 16 Aug '18

16 Aug '18

Hi Ovirt, I successfully installed both ovirt-engine(ENGINE01) and ovirt node(NODE01) on a separate machines. I also created a FreeNAS(NAS01) with NFS share and already connected to my NODE01, all of these server though I haven't setup a DNS server, was manually added hostname on every machines and I can lookup and ping on them without a problem, I was able to add the NODE01 to my ENGINE01 as well. My issue was when I tried creating a storage domain on my ENGINE01, I did the below steps before running the engine-setup while also following the guide on the ovirt url: https://www.ovirt.org/documentation/how-to/troubleshooting/troubleshooting-… #touch /etc/exports #systemctl start rpcbind nfs-server #systemctl enable rpcbind nfs-server #engine-setup #mkdir /var/lib/exports/data #chown vdsm:kvm /var/lib/exports/data I added the 2 just in case but I have tried each alone but all fails #vi /etc/exports /var/lib/exports/data *(rw,sync,no_subtree_check,all_squash,anonuid=36,anongid=36) /var/lib/exports/data 0.0.0.0/0.0.0.0(rw) #systemctl restart rpc-statd nfs-server Once I started to add my storage domain I get the below error Attached is the engine log for your reference. Hope you guys can help me with these, Im really interested with this great product. Thanks!

3 3

Snapshots - Preview button greyed out
by Leo David 16 Aug '18

16 Aug '18

Hi Everyone, I am encountering an issue on my oVirt 4.2 self-hosted engine installation. Suddenly the "Preview" button in snapshots menu became inactive, I just cannot revert any previews snapshots. This feature worked fine over the last month, but now it seems to be just inactive. Any thoughts on this ? Thank you very much ! Leo -- Best regards, Leo David

3 3

Patches for L1FT?
by Eduardo Mayoral 16 Aug '18

16 Aug '18

Hi, For mitigation of the recently announced L1TF vulnerability, is it sufficient to update the compute nodes to the updated kernel? Are any other updates to KVM / vdsm / ovirt-engine required? Also, for the concurrent variant. Should we disable hyperthreading altogether? Is there any remediation (even if expensive from a performance view), that can be enabled? Thanks for your help! -- Eduardo Mayoral.

2 2

Do I need to set VM vnic profile for GlusterFS network?
by Jayme 16 Aug '18

16 Aug '18

I followed this guide to get my three node HCI cluster up and running: https://www.ovirt.org/blog/2018/02/up-and-running-with-ovirt-4-2-and-gluste… The docs were a bit out of date but most things matched up. I have configured an internal GlusterFS network on a separate subnet on 10Gbe switch and have set it as the glusterFS and migration network. Is this all that is needed to be done for all glusterFS traffic to pass through that network or must the glusterFS vnic profile also be selected for each VM? I'm not sure if my VMs should be using ovirtmgmnt vnic or GlusterFS vnic.

2 1

storage domain sync problem and fail to run vm after change LUN mapping
by dvotrak 16 Aug '18

16 Aug '18

Environment: Ovirt version: 4.2.5 Centos 7.5 During a planned maintenance window, multipath configuration was changed according to the storage vendor guidelines. The new configuration modify user_friendly_names from no to yes, so the multipath -ll output changed from this "360060160a6213400fa2d31acbbfce511 dm-8 DGC" to this: "mpathh (360060160a6213400fa2d31acbbfce511) dm-8 DGC". After this change, the following errors are displayed: 1) when attempting to start a VM with direct LUN: Failed to run VM <VMNAME> on Host <HYPERVISOR> VM <VMNAME> is down. Exit message: Bad volume specification <LUN_ID> Inspecting the VM disk, the lun id remains the previous one (360060160a6213400fa2d31acbbfce511) insted of the new one (mpathh) Remove/detach and re-add/re-attach the disk to VM doesen't help. 2) randomly: Storage domains with IDs [<ID>] could not be synchronized. To synchronize them, please move them to maintenance and then activate. Moving it to maintenance and then activate it doesen't help. Inspecting storage domain luns, almost all change the lun id to the new ones (mpathXX), but this one remain in "the old style" (maybe because lun_id is a primary key and this one was already taken?) Any idea on how to resolve these problems? Thanks Sent with [ProtonMail](https://protonmail.com) Secure Email.

2 3