- Users - oVirt List Archives

Gluster setup 3 Node - Now only showing single node setup in setup Wizard
by rob.downer＠orbitalsystems.co.uk 25 Nov '19

25 Nov '19

I have set up 3 Nodes with a separate volume for Gluster, I have set up the two networks and DNS works fine SSH has been set up for Gluster and you can login via ssh to the other two hosts from the host used to set up. When going to Virtualisation > Setup Gluster and Hosted Engine only single node shows up. I have restarted all 3 machines. All nodes machines show up in Dashboard etc... I have set this up before but it all worked, I erased everything and set up again with the separate volume to be used for Gluster storage

3 4

Beginning oVirt / Hyperconverged
by email＠christian-reiss.de 25 Nov '19

25 Nov '19

Hey folks! Quick question, really: I have 4 servers of identical hardware. The documentation says "you need 3", not "you need 3 or more"; is it possible to run hyperconverged with 4 servers (in same rack, let's neglect the possibility of 2 failing servers (split brain))? Also: I have a nice seperate server (low power footprint, big on cpu (8) and has 32gb ram)-- would love to use this as engine server only. Can I initiate a hyperconverged system with 4 working host starting from a stand-alone engine? Or does "hyperconverged" a must for HA engine on the cluster? One last one: Installing ovirt nodes, obviously. This will by default create a lvm across the entire RAID volume; can I simply create folders (with correct permissions) and use those as gluster bricks? Or do I need to partition the RAID in a special way? Thanks for your pointers, Chris.

4 3

Windows 2019 Server - Memory Usage
by Joseph Goldman 25 Nov '19

25 Nov '19

Hi List, My Windows Server 2019 installs dont seem to report memory usage to the ovirt engine. They have the ovirt-guest-agent installed, is there something else I need to do to get them reporting back to the engine/hypervisor? Thanks, Joe

1 0

VM Lock state location
by Joseph Goldman 25 Nov '19

25 Nov '19

Hi *, Trying to figure out where a VM's lockstate is stored. I have a few VM's that seem to be stuck in lock state. I have done an ./unlock_entity.sh -t all, and a -q gives me nothing in a lock state. I have jumped on psql DB direct and looked through as many tables as I can looking for a locked status, everything seems fine. In the engine, on these VM's if I try to do any actions I get a message similar to: 'Failed to Acquire Lock to object 'EngineLock:{exclusiveLocks='[6c2fc524-2f13-4cad-9108-347be2e88d1a=VM]', sharedLocks=''}'' When this issue first started, I was trying to do backups with snapshots, I got a lot of messages similar to this: /var/log/ovirt-engine/engine.log-20191121.gz:2019-11-20 11:44:30,441+10 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-41) [6cafc1ef] Trying to release a shared lock for key: '6 c2fc524-2f13-4cad-9108-347be2e88d1aVM' , but lock does not exist /var/log/ovirt-engine/engine.log-20191121.gz:2019-11-20 11:44:30,441+10 INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferImageStatusCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-41) [6cafc1ef] Lock freed to object ' EngineLock:{exclusiveLocks='', sharedLocks='[6c2fc524-2f13-4cad-9108-347be2e88d1a=VM, ab5d4e18-f0c6-4472-8d2e-8c3bd1ff1a6a=DISK]'}' I have made sure there is nothing in async_tasks, commands and the _entities tables for both. There was some command_entities, but I took a backup of the engine and decided to run a delete on that table (There was about 5 stuck records there) but that hasn't helped The affected VMs, cant delete snapshots, cant delete their disks, cant delete the VM, cant boot the VMs - they are just stuck in the engine. My ultimate question is, where is the engine looking up that lockstate as I believe it is stuck locked in error, so I can go there and manually force a removal of the lock state and at least delete these VM's - as ultimately I have cloned them to get the VM's back up and running. Can anyone point me in the right direction? Willing to pay consulting fee if you believe you have the answer for this one. Thanks, Joe

3 4

oVirt Admin Portal unaccessible via chrome (firefox works)
by Strahil Nikolov 24 Nov '19

24 Nov '19

Hello Community, I have a constantly loading chrome on my openSuSE 15.1 (and my android phone), while firefox has no issues . Can someone test accessing the oVirt Admin portal via chrome on x86_64 Linux ? Best Regards,Strahil Nikolov

5 5

Ovirt 4.3.6 USB passthrough not working with iommu_intel=on iommu=pt
by Don Dupuis 24 Nov '19

24 Nov '19

I am trying to pass through a USB Dongle to a Windows virtual machine in ovirt 4.3.6. This will work if these options aren't used. Below is the output of lsusb with no iommu enable: Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 001 Device 005: ID 1604:10c0 Tascam Bus 001 Device 004: ID 1604:10c0 Tascam Bus 001 Device 003: ID 1604:10c0 Tascam Bus 001 Device 002: ID 07f2:0001 Microcomputer Applications, Inc. KEYLOK II Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub The device I am passing through is Bus 001 Device 002. This is the error I get in vdsmd.log: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 867, in _startUnderlyingVm self._run() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2890, in _run self._domDependentInit() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2466, in _domDependentInit self._vmDependentInit() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2480, in _vmDependentInit self._getUnderlyingVmDevicesInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2448, in _getUnderlyingVmDevicesInfo vmdevices.common.update_device_info(self, self._devices) File "/usr/lib/python2.7/site-packages/vdsm/virt/vmdevices/common.py", line 94, in update_device_info hostdevice.HostDevice.update_device_info(vm, devices[hwclass.HOSTDEV]) File "/usr/lib/python2.7/site-packages/vdsm/virt/vmdevices/hostdevice.py", line 426, in update_device_info vm, device_conf, device_xml) File "/usr/lib/python2.7/site-packages/vdsm/virt/vmdevices/hostdevice.py", line 223, in update_from_xml if host_address == dev.hostAddress: AttributeError: 'MdevDevice' object has no attribute 'hostAddress' 2019-11-18 18:20:06,173-0600 INFO (vm/9944089c) [virt.vm] (vmId='9944089c-2109-4352-a0c6-7d0d9e04bb3f') Changed state to Down: 'MdevDevice' object has no attribute 'hostAddress' (code=1) (vm:1690) I am using kernel 3.10.0-957.12.1.el7.x86_64 vdsm is vdsm-4.30.33-1.el7.x86_64 libvirt is libvirt-5.0.0-1.el7.x86_64 qemu is qemu-img-ev-2.12.0-33.1.el7.x86_64 Any help with this issue would be appreciated. Thanks Don

4 9

metrics
by suporte＠logicworks.pt 24 Nov '19

24 Nov '19

Hi, I'm trying to install metrics for Version 4.3.6.7-1.el7 Do I need a RH subscription to downloading the RHEL guest image? (qcow_url:) Thanks -- Jose Ferradeira http://www.logicworks.pt

4 17

Export of disk possible?
by Gianluca Cecchi 24 Nov '19

24 Nov '19

Hello, in 4.3 I can: - Storage -> Domains -> Select Domain -> Disks -> Select Disk -> Download and this will download the disk image to my client, that then I can upload to a different environment - Storage -> Disks -> Select Disk -> Export But then, even if I have an export domain active I get this kind of window that I don't understand.... https://drive.google.com/file/d/1UU7NaoQ9F01oEkciUgz29HW-4dOPQVeF/view?usp=… Perhaps it means another DC managed by the same Engine is needed? How can I export a virtual disk from an engine environment to another engine environment? Yes, I can artificially create a VM and attach the disk to it and then export the VM, but is there a functionality to directly export a disk to an export domain? Thanks, Gianluca

2 1

Dm-VDO in Gluster
by rob.downer＠orbitalsystems.co.uk 23 Nov '19

23 Nov '19

I have 3 nodes set up with a 2TB Partition on each for Gluster set up. The drives are all SSD should I enable DM-VDO ? has anyone one got a simple / Pro / Con list ? I understand response is slower if enabled also some CPU hit both are acceptable. Thanks

1 0

Howto recover/reinstall a failed oVirt node in a 3 node HCI cluster
by Timo Eissler 22 Nov '19

22 Nov '19

Hello, i can't find the documentation on howto recover from a single host OS disk failure in a 3 node HCI cluster. I already searched the ovirt, RHV and RHHI documentation but i can't find the corresponding documentation. So can anyone please give me some hints? In our dev lab i have 3 servers with 2 disks each. sda for ovirt node and sdb for gluster. Let's assume the disk sda of a single node is defunct and needs to be replaced. So i grab a new disk and reinstall ovirt node from PXE. After the installation is finished i would like to reinstall the host from ovirt admin portal, but that doesn't work because of an invalid fingerprint. So how can i re-add that host to ovirt and gluster? Kind regards, Timo

2 1

python ovirtsdk
by serphentos＠sharedlink.tk 22 Nov '19

22 Nov '19

Hello. Can someone explain how to attach logical network to host via ovirtsdk? I saw a example of attaching networks to cluster or dc, but not a host https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/assign_n…

2 2

Re: Fail to attach/create disk from template when running via Ansible
by Vrgotic, Marko 22 Nov '19

22 Nov '19

Hi Ondra, I have rebuilt the staging with 4.3.6 and still same issue. Images are there, I can again create VM via UI, but not via Ansible. Here is the output of the complete run form engine.log: 2019-11-12 12:09:00,457Z INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-49) [] User admin@internal successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2019-11-12 12:09:00,978Z INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-49) [] User admin@internal successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2019-11-12 12:09:01,538Z INFO [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-49) [233de27] Running command: CreateUserSessionCommand internal: false. 2019-11-12 12:09:01,549Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-49) [233de27] EVENT_ID: USER_VDC_LOGIN(30), User admin@internal-authz connecting from '172.16.105.206' using session 'bsgNpfGJTNtzuuQWSyXVJqHhpLK4TY9TQRERWxxzGtpHonoqdk7WIHGZ7mID2+QkBKH9bPp8HVhyvDu66tzPnQ==' logged in. 2019-11-12 12:09:01,691Z INFO [org.ovirt.engine.core.bll.AddVmCommand] (default task-49) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Lock Acquired to object 'EngineLock:{exclusiveLocks='[centos-testvm-a910=VM_NAME]', sharedLocks='[400688e8-3c5b-4cca-9ef8-bbb6c79d54ed=DISK, 819aecf6-c83d-4470-9915-e309390d8098=TEMPLATE]'}' 2019-11-12 12:09:01,773Z INFO [org.ovirt.engine.core.bll.AddVmCommand] (default task-49) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Running command: AddVmCommand internal: false. Entities affected : ID: d3c90169-ecc8-46a0-a4ec-7c3d99b81b61 Type: ClusterAction group CREATE_VM with role type USER, ID: 819aecf6-c83d-4470-9915-e309390d8098 Type: VmTemplateAction group CREATE_VM with role type USER, ID: 78bfcb57-3a0d-4a71-a3f8-480ebae37c74 Type: StorageAction group CREATE_DISK with role type USER 2019-11-12 12:09:01,866Z INFO [org.ovirt.engine.core.bll.AddRngDeviceCommand] (default task-49) [4d4703a1] Running command: AddRngDeviceCommand internal: true. Entities affected : ID: 6343c55a-c67e-4d75-a767-dfc9412006e6 Type: VMAction group EDIT_VM_PROPERTIES with role type USER 2019-11-12 12:09:01,876Z INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (default task-49) [4d4703a1] START, SetVmStatusVDSCommand( SetVmStatusVDSCommandParameters:{vmId='6343c55a-c67e-4d75-a767-dfc9412006e6', status='ImageLocked', exitStatus='Normal'}), log id: 4eedcefb 2019-11-12 12:09:01,887Z INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (default task-49) [4d4703a1] FINISH, SetVmStatusVDSCommand, return: , log id: 4eedcefb 2019-11-12 12:09:01,906Z INFO [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] (default task-49) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Running command: CreateSnapshotFromTemplateCommand internal: true. Entities affected : ID: 78bfcb57-3a0d-4a71-a3f8-480ebae37c74 Type: Storage 2019-11-12 12:09:01,928Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateVolumeVDSCommand] (default task-49) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] START, CreateVolumeVDSCommand( CreateVolumeVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', storageDomainId='78bfcb57-3a0d-4a71-a3f8-480ebae37c74', imageGroupId='56ddac45-1913-4d24-af97-2f969fc3fa2e', imageSizeInBytes='8589934592', volumeFormat='COW', newImageId='7270dd54-a075-4b14-aca3-7c2a118057f0', imageType='Sparse', newImageDescription='', imageInitialSizeInBytes='0', imageId='767c6837-5191-4650-a49d-abca7d69a708', sourceImageGroupId='400688e8-3c5b-4cca-9ef8-bbb6c79d54ed'}), log id: 2dfac68e 2019-11-12 12:09:02,009Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateVolumeVDSCommand] (default task-49) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] FINISH, CreateVolumeVDSCommand, return: 7270dd54-a075-4b14-aca3-7c2a118057f0, log id: 2dfac68e 2019-11-12 12:09:02,014Z INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (default task-49) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 'fb915b7e-4814-4e93-a3ef-4f40c4e3c41b' 2019-11-12 12:09:02,014Z INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (default task-49) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] CommandMultiAsyncTasks::attachTask: Attaching task '0eb09758-113f-43b9-aa67-97d2e5a57e05' to command 'fb915b7e-4814-4e93-a3ef-4f40c4e3c41b'. 2019-11-12 12:09:02,027Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (default task-49) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Adding task '0eb09758-113f-43b9-aa67-97d2e5a57e05' (Parent Command 'CreateSnapshotFromTemplate', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters'), polling hasn't started yet.. 2019-11-12 12:09:02,098Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-49) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] EVENT_ID: USER_ADD_VM_STARTED(37), VM centos-testvm-a910 creation was initiated by admin@internal-authz. 2019-11-12 12:09:02,098Z INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (default task-49) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] BaseAsyncTask::startPollingTask: Starting to poll task '0eb09758-113f-43b9-aa67-97d2e5a57e05'. 2019-11-12 12:09:02,758Z INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Command 'AddVm' (id: '97d4103b-acef-4b55-a379-71a1baa04a4f') waiting on child command id: 'fb915b7e-4814-4e93-a3ef-4f40c4e3c41b' type:'CreateSnapshotFromTemplate' to complete 2019-11-12 12:09:04,773Z INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-52) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Command 'AddVm' (id: '97d4103b-acef-4b55-a379-71a1baa04a4f') waiting on child command id: 'fb915b7e-4814-4e93-a3ef-4f40c4e3c41b' type:'CreateSnapshotFromTemplate' to complete 2019-11-12 12:09:06,117Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-26) [] Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now 2019-11-12 12:09:06,121Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-26) [] Failed in 'HSMGetAllTasksStatusesVDS' method 2019-11-12 12:09:06,128Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-26) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM ovirt-staging-hv-02.avinity.tv command HSMGetAllTasksStatusesVDS failed: Invalid parameter: 'capacity=8589934592' 2019-11-12 12:09:06,128Z INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-26) [] SPMAsyncTask::PollTask: Polling task '0eb09758-113f-43b9-aa67-97d2e5a57e05' (Parent Command 'CreateSnapshotFromTemplate', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned status 'finished', result 'cleanSuccess'. 2019-11-12 12:09:06,131Z ERROR [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-26) [] BaseAsyncTask::logEndTaskFailure: Task '0eb09758-113f-43b9-aa67-97d2e5a57e05' (Parent Command 'CreateSnapshotFromTemplate', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with failure: -- Result: 'cleanSuccess' -- Message: 'VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = Invalid parameter: 'capacity=8589934592', code = 100', -- Exception: 'VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = Invalid parameter: 'capacity=8589934592', code = 100' 2019-11-12 12:09:06,133Z INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-26) [] CommandAsyncTask::endActionIfNecessary: All tasks of command 'fb915b7e-4814-4e93-a3ef-4f40c4e3c41b' has ended -> executing 'endAction' 2019-11-12 12:09:06,133Z INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-26) [] CommandAsyncTask::endAction: Ending action for '1' tasks (command ID: 'fb915b7e-4814-4e93-a3ef-4f40c4e3c41b'): calling endAction '. 2019-11-12 12:09:06,133Z INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-274926) [] CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction 'CreateSnapshotFromTemplate', 2019-11-12 12:09:06,137Z INFO [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] (EE-ManagedThreadFactory-engine-Thread-274926) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Command [id=fb915b7e-4814-4e93-a3ef-4f40c4e3c41b]: Updating status to 'FAILED', The command end method logic will be executed by one of its parent commands. 2019-11-12 12:09:06,137Z INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-274926) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'CreateSnapshotFromTemplate' completed, handling the result. 2019-11-12 12:09:06,137Z INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-274926) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'CreateSnapshotFromTemplate' succeeded, clearing tasks. 2019-11-12 12:09:06,137Z INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-274926) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] SPMAsyncTask::ClearAsyncTask: Attempting to clear task '0eb09758-113f-43b9-aa67-97d2e5a57e05' 2019-11-12 12:09:06,137Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-274926) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', taskId='0eb09758-113f-43b9-aa67-97d2e5a57e05'}), log id: baaf58a 2019-11-12 12:09:06,138Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-274926) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] START, HSMClearTaskVDSCommand(HostName = ovirt-staging-hv-02.avinity.tv, HSMTaskGuidBaseVDSCommandParameters:{hostId='3c70c563-e32e-4352-b1e1-f6bcc41f77e2', taskId='0eb09758-113f-43b9-aa67-97d2e5a57e05'}), log id: 7321c57d 2019-11-12 12:09:06,147Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-274926) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] FINISH, HSMClearTaskVDSCommand, return: , log id: 7321c57d 2019-11-12 12:09:06,147Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engine-Thread-274926) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] FINISH, SPMClearTaskVDSCommand, return: , log id: baaf58a 2019-11-12 12:09:06,152Z INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engine-Thread-274926) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] BaseAsyncTask::removeTaskFromDB: Removed task '0eb09758-113f-43b9-aa67-97d2e5a57e05' from DataBase 2019-11-12 12:09:06,152Z INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-274926) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 'fb915b7e-4814-4e93-a3ef-4f40c4e3c41b' 2019-11-12 12:09:08,790Z INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-34) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Command 'AddVm' id: '97d4103b-acef-4b55-a379-71a1baa04a4f' child commands '[fb915b7e-4814-4e93-a3ef-4f40c4e3c41b]' executions were completed, status 'FAILED' 2019-11-12 12:09:09,821Z ERROR [org.ovirt.engine.core.bll.AddVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Ending command 'org.ovirt.engine.core.bll.AddVmCommand' with failure. 2019-11-12 12:09:09,825Z ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand' with failure. 2019-11-12 12:09:09,835Z INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [44f6ef0c] Running command: RemoveImageCommand internal: true. Entities affected : ID: 78bfcb57-3a0d-4a71-a3f8-480ebae37c74 Type: Storage 2019-11-12 12:09:09,873Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [44f6ef0c] START, DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', storageDomainId='78bfcb57-3a0d-4a71-a3f8-480ebae37c74', imageGroupId='56ddac45-1913-4d24-af97-2f969fc3fa2e', postZeros='false', discard='false', forceDelete='false'}), log id: 28915db3 2019-11-12 12:09:09,907Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [44f6ef0c] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS failed: Image does not exist in domain: u'image=56ddac45-1913-4d24-af97-2f969fc3fa2e, domain=78bfcb57-3a0d-4a71-a3f8-480ebae37c74' 2019-11-12 12:09:09,907Z ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [44f6ef0c] Command 'DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', storageDomainId='78bfcb57-3a0d-4a71-a3f8-480ebae37c74', imageGroupId='56ddac45-1913-4d24-af97-2f969fc3fa2e', postZeros='false', discard='false', forceDelete='false'})' execution failed: IRSGenericException: IRSErrorException: Image does not exist in domain: u'image=56ddac45-1913-4d24-af97-2f969fc3fa2e, domain=78bfcb57-3a0d-4a71-a3f8-480ebae37c74' 2019-11-12 12:09:09,907Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [44f6ef0c] FINISH, DeleteImageGroupVDSCommand, return: , log id: 28915db3 2019-11-12 12:09:09,907Z INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [44f6ef0c] Disk '56ddac45-1913-4d24-af97-2f969fc3fa2e' doesn't exist on storage domain '78bfcb57-3a0d-4a71-a3f8-480ebae37c74', rolling forward 2019-11-12 12:09:09,918Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [44f6ef0c] Removed task '25bf448d-9271-4a85-ad9b-9c63012753ac' from DataBase 2019-11-12 12:09:09,983Z INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [] transaction rolled back 2019-11-12 12:09:09,983Z INFO [org.ovirt.engine.core.bll.AddVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [] Lock freed to object 'EngineLock:{exclusiveLocks='[centos-testvm-a910=VM_NAME]', sharedLocks='[400688e8-3c5b-4cca-9ef8-bbb6c79d54ed=DISK, 819aecf6-c83d-4470-9915-e309390d8098=TEMPLATE]'}' 2019-11-12 12:09:10,001Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [] EVENT_ID: USER_ADD_VM_FINISHED_FAILURE(60), Failed to complete VM centos-testvm-a910 creation. 2019-11-12 12:09:10,002Z ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [] Failed invoking callback end method 'onFailed' for command '97d4103b-acef-4b55-a379-71a1baa04a4f' with exception 'null', the callback is marked for end method retries 2019-11-12 12:09:12,019Z ERROR [org.ovirt.engine.core.bll.AddVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Ending command 'org.ovirt.engine.core.bll.AddVmCommand' with failure. 2019-11-12 12:09:12,023Z ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand' with failure. 2019-11-12 12:09:12,041Z INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [1b861e85] Running command: RemoveImageCommand internal: true. Entities affected : ID: 78bfcb57-3a0d-4a71-a3f8-480ebae37c74 Type: Storage 2019-11-12 12:09:12,063Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [1b861e85] START, DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', storageDomainId='78bfcb57-3a0d-4a71-a3f8-480ebae37c74', imageGroupId='56ddac45-1913-4d24-af97-2f969fc3fa2e', postZeros='false', discard='false', forceDelete='false'}), log id: 376f65dc 2019-11-12 12:09:12,090Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [1b861e85] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS failed: Image does not exist in domain: u'image=56ddac45-1913-4d24-af97-2f969fc3fa2e, domain=78bfcb57-3a0d-4a71-a3f8-480ebae37c74' 2019-11-12 12:09:12,090Z ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [1b861e85] Command 'DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', storageDomainId='78bfcb57-3a0d-4a71-a3f8-480ebae37c74', imageGroupId='56ddac45-1913-4d24-af97-2f969fc3fa2e', postZeros='false', discard='false', forceDelete='false'})' execution failed: IRSGenericException: IRSErrorException: Image does not exist in domain: u'image=56ddac45-1913-4d24-af97-2f969fc3fa2e, domain=78bfcb57-3a0d-4a71-a3f8-480ebae37c74' 2019-11-12 12:09:12,090Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [1b861e85] FINISH, DeleteImageGroupVDSCommand, return: , log id: 376f65dc 2019-11-12 12:09:12,090Z INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [1b861e85] Disk '56ddac45-1913-4d24-af97-2f969fc3fa2e' doesn't exist on storage domain '78bfcb57-3a0d-4a71-a3f8-480ebae37c74', rolling forward 2019-11-12 12:09:12,101Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [1b861e85] Removed task 'ebe0e879-8595-44db-9d64-e237afd6c8e8' from DataBase 2019-11-12 12:09:12,157Z INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] transaction rolled back 2019-11-12 12:09:12,157Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Trying to release a shared lock for key: '400688e8-3c5b-4cca-9ef8-bbb6c79d54edDISK' , but lock does not exist 2019-11-12 12:09:12,157Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Trying to release a shared lock for key: '819aecf6-c83d-4470-9915-e309390d8098TEMPLATE' , but lock does not exist 2019-11-12 12:09:12,157Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Trying to release exclusive lock which does not exist, lock key: 'centos-testvm-a910VM_NAME' 2019-11-12 12:09:12,157Z INFO [org.ovirt.engine.core.bll.AddVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Lock freed to object 'EngineLock:{exclusiveLocks='[centos-testvm-a910=VM_NAME]', sharedLocks='[400688e8-3c5b-4cca-9ef8-bbb6c79d54ed=DISK, 819aecf6-c83d-4470-9915-e309390d8098=TEMPLATE]'}' 2019-11-12 12:09:12,172Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] EVENT_ID: USER_ADD_VM_FINISHED_FAILURE(60), Failed to complete VM centos-testvm-a910 creation. 2019-11-12 12:09:12,172Z ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Failed invoking callback end method 'onFailed' for command '97d4103b-acef-4b55-a379-71a1baa04a4f' with exception 'null', the callback is marked for end method retries 2019-11-12 12:09:16,192Z ERROR [org.ovirt.engine.core.bll.AddVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Ending command 'org.ovirt.engine.core.bll.AddVmCommand' with failure. 2019-11-12 12:09:16,195Z ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand' with failure. 2019-11-12 12:09:16,215Z INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [8f3849f] Running command: RemoveImageCommand internal: true. Entities affected : ID: 78bfcb57-3a0d-4a71-a3f8-480ebae37c74 Type: Storage 2019-11-12 12:09:16,240Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [8f3849f] START, DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', storageDomainId='78bfcb57-3a0d-4a71-a3f8-480ebae37c74', imageGroupId='56ddac45-1913-4d24-af97-2f969fc3fa2e', postZeros='false', discard='false', forceDelete='false'}), log id: 75027448 2019-11-12 12:09:16,263Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [8f3849f] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS failed: Image does not exist in domain: u'image=56ddac45-1913-4d24-af97-2f969fc3fa2e, domain=78bfcb57-3a0d-4a71-a3f8-480ebae37c74' 2019-11-12 12:09:16,263Z ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [8f3849f] Command 'DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', storageDomainId='78bfcb57-3a0d-4a71-a3f8-480ebae37c74', imageGroupId='56ddac45-1913-4d24-af97-2f969fc3fa2e', postZeros='false', discard='false', forceDelete='false'})' execution failed: IRSGenericException: IRSErrorException: Image does not exist in domain: u'image=56ddac45-1913-4d24-af97-2f969fc3fa2e, domain=78bfcb57-3a0d-4a71-a3f8-480ebae37c74' 2019-11-12 12:09:16,263Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [8f3849f] FINISH, DeleteImageGroupVDSCommand, return: , log id: 75027448 2019-11-12 12:09:16,263Z INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [8f3849f] Disk '56ddac45-1913-4d24-af97-2f969fc3fa2e' doesn't exist on storage domain '78bfcb57-3a0d-4a71-a3f8-480ebae37c74', rolling forward 2019-11-12 12:09:16,274Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [8f3849f] Removed task '688274fc-0596-44b5-b167-36c86b8b8e5d' from DataBase 2019-11-12 12:09:16,341Z INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [] transaction rolled back 2019-11-12 12:09:16,341Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [] Trying to release a shared lock for key: '400688e8-3c5b-4cca-9ef8-bbb6c79d54edDISK' , but lock does not exist 2019-11-12 12:09:16,341Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [] Trying to release a shared lock for key: '819aecf6-c83d-4470-9915-e309390d8098TEMPLATE' , but lock does not exist 2019-11-12 12:09:16,342Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [] Trying to release exclusive lock which does not exist, lock key: 'centos-testvm-a910VM_NAME' 2019-11-12 12:09:16,342Z INFO [org.ovirt.engine.core.bll.AddVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [] Lock freed to object 'EngineLock:{exclusiveLocks='[centos-testvm-a910=VM_NAME]', sharedLocks='[400688e8-3c5b-4cca-9ef8-bbb6c79d54ed=DISK, 819aecf6-c83d-4470-9915-e309390d8098=TEMPLATE]'}' 2019-11-12 12:09:16,357Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [] EVENT_ID: USER_ADD_VM_FINISHED_FAILURE(60), Failed to complete VM centos-testvm-a910 creation. 2019-11-12 12:09:16,357Z ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedThreadFactory-engineScheduled-Thread-78) [] Failed invoking callback end method 'onFailed' for command '97d4103b-acef-4b55-a379-71a1baa04a4f' with exception 'null', the callback is marked for end method retries 2019-11-12 12:09:24,376Z ERROR [org.ovirt.engine.core.bll.AddVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Ending command 'org.ovirt.engine.core.bll.AddVmCommand' with failure. 2019-11-12 12:09:24,380Z ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand' with failure. 2019-11-12 12:09:24,387Z INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [153e3c43] Running command: RemoveImageCommand internal: true. Entities affected : ID: 78bfcb57-3a0d-4a71-a3f8-480ebae37c74 Type: Storage 2019-11-12 12:09:24,419Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [153e3c43] START, DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', storageDomainId='78bfcb57-3a0d-4a71-a3f8-480ebae37c74', imageGroupId='56ddac45-1913-4d24-af97-2f969fc3fa2e', postZeros='false', discard='false', forceDelete='false'}), log id: 29a560b1 2019-11-12 12:09:24,444Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [153e3c43] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS failed: Image does not exist in domain: u'image=56ddac45-1913-4d24-af97-2f969fc3fa2e, domain=78bfcb57-3a0d-4a71-a3f8-480ebae37c74' 2019-11-12 12:09:24,444Z ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [153e3c43] Command 'DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', storageDomainId='78bfcb57-3a0d-4a71-a3f8-480ebae37c74', imageGroupId='56ddac45-1913-4d24-af97-2f969fc3fa2e', postZeros='false', discard='false', forceDelete='false'})' execution failed: IRSGenericException: IRSErrorException: Image does not exist in domain: u'image=56ddac45-1913-4d24-af97-2f969fc3fa2e, domain=78bfcb57-3a0d-4a71-a3f8-480ebae37c74' 2019-11-12 12:09:24,444Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [153e3c43] FINISH, DeleteImageGroupVDSCommand, return: , log id: 29a560b1 2019-11-12 12:09:24,445Z INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [153e3c43] Disk '56ddac45-1913-4d24-af97-2f969fc3fa2e' doesn't exist on storage domain '78bfcb57-3a0d-4a71-a3f8-480ebae37c74', rolling forward 2019-11-12 12:09:24,455Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [153e3c43] Removed task '13b2490d-09a6-4230-a5d3-7b86b536e8c3' from DataBase 2019-11-12 12:09:24,509Z INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [] transaction rolled back 2019-11-12 12:09:24,509Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [] Trying to release a shared lock for key: '400688e8-3c5b-4cca-9ef8-bbb6c79d54edDISK' , but lock does not exist 2019-11-12 12:09:24,509Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [] Trying to release a shared lock for key: '819aecf6-c83d-4470-9915-e309390d8098TEMPLATE' , but lock does not exist 2019-11-12 12:09:24,509Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [] Trying to release exclusive lock which does not exist, lock key: 'centos-testvm-a910VM_NAME' 2019-11-12 12:09:24,509Z INFO [org.ovirt.engine.core.bll.AddVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [] Lock freed to object 'EngineLock:{exclusiveLocks='[centos-testvm-a910=VM_NAME]', sharedLocks='[400688e8-3c5b-4cca-9ef8-bbb6c79d54ed=DISK, 819aecf6-c83d-4470-9915-e309390d8098=TEMPLATE]'}' 2019-11-12 12:09:24,527Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [] EVENT_ID: USER_ADD_VM_FINISHED_FAILURE(60), Failed to complete VM centos-testvm-a910 creation. 2019-11-12 12:09:24,527Z ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [] Failed invoking callback end method 'onFailed' for command '97d4103b-acef-4b55-a379-71a1baa04a4f' with exception 'null', the callback is marked for end method retries 2019-11-12 12:09:34,547Z ERROR [org.ovirt.engine.core.bll.AddVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Ending command 'org.ovirt.engine.core.bll.AddVmCommand' with failure. 2019-11-12 12:09:34,551Z ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand' with failure. 2019-11-12 12:09:34,568Z INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [28afe05e] Running command: RemoveImageCommand internal: true. Entities affected : ID: 78bfcb57-3a0d-4a71-a3f8-480ebae37c74 Type: Storage 2019-11-12 12:09:34,591Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [28afe05e] START, DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', storageDomainId='78bfcb57-3a0d-4a71-a3f8-480ebae37c74', imageGroupId='56ddac45-1913-4d24-af97-2f969fc3fa2e', postZeros='false', discard='false', forceDelete='false'}), log id: 207e5265 2019-11-12 12:09:34,617Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [28afe05e] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS failed: Image does not exist in domain: u'image=56ddac45-1913-4d24-af97-2f969fc3fa2e, domain=78bfcb57-3a0d-4a71-a3f8-480ebae37c74' 2019-11-12 12:09:34,617Z ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [28afe05e] Command 'DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', storageDomainId='78bfcb57-3a0d-4a71-a3f8-480ebae37c74', imageGroupId='56ddac45-1913-4d24-af97-2f969fc3fa2e', postZeros='false', discard='false', forceDelete='false'})' execution failed: IRSGenericException: IRSErrorException: Image does not exist in domain: u'image=56ddac45-1913-4d24-af97-2f969fc3fa2e, domain=78bfcb57-3a0d-4a71-a3f8-480ebae37c74' 2019-11-12 12:09:34,618Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [28afe05e] FINISH, DeleteImageGroupVDSCommand, return: , log id: 207e5265 2019-11-12 12:09:34,618Z INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [28afe05e] Disk '56ddac45-1913-4d24-af97-2f969fc3fa2e' doesn't exist on storage domain '78bfcb57-3a0d-4a71-a3f8-480ebae37c74', rolling forward 2019-11-12 12:09:34,629Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [28afe05e] Removed task 'ad58e8d8-13cc-4ae4-8785-9df252b38c1f' from DataBase 2019-11-12 12:09:34,688Z INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [] transaction rolled back 2019-11-12 12:09:34,688Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [] Trying to release a shared lock for key: '400688e8-3c5b-4cca-9ef8-bbb6c79d54edDISK' , but lock does not exist 2019-11-12 12:09:34,688Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [] Trying to release a shared lock for key: '819aecf6-c83d-4470-9915-e309390d8098TEMPLATE' , but lock does not exist 2019-11-12 12:09:34,688Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [] Trying to release exclusive lock which does not exist, lock key: 'centos-testvm-a910VM_NAME' 2019-11-12 12:09:34,688Z INFO [org.ovirt.engine.core.bll.AddVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [] Lock freed to object 'EngineLock:{exclusiveLocks='[centos-testvm-a910=VM_NAME]', sharedLocks='[400688e8-3c5b-4cca-9ef8-bbb6c79d54ed=DISK, 819aecf6-c83d-4470-9915-e309390d8098=TEMPLATE]'}' 2019-11-12 12:09:34,707Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [] EVENT_ID: USER_ADD_VM_FINISHED_FAILURE(60), Failed to complete VM centos-testvm-a910 creation. 2019-11-12 12:09:34,707Z ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedThreadFactory-engineScheduled-Thread-33) [] Failed invoking callback end method 'onFailed' for command '97d4103b-acef-4b55-a379-71a1baa04a4f' with exception 'null', the callback is marked for end method retries 2019-11-12 12:09:42,871Z INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-75) [58fc7cd2] Lock Acquired to object 'EngineLock:{exclusiveLocks='[02a18cd9-ead2-44c7-9923-8fe84d1865ee=PROVIDER]', sharedLocks=''}' 2019-11-12 12:09:42,880Z INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-75) [58fc7cd2] Running command: SyncNetworkProviderCommand internal: true. 2019-11-12 12:09:42,975Z INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-49) [] User admin@internal successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2019-11-12 12:09:43,096Z INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-75) [58fc7cd2] Lock freed to object 'EngineLock:{exclusiveLocks='[02a18cd9-ead2-44c7-9923-8fe84d1865ee=PROVIDER]', sharedLocks=''}' 2019-11-12 12:09:44,729Z ERROR [org.ovirt.engine.core.bll.AddVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Ending command 'org.ovirt.engine.core.bll.AddVmCommand' with failure. 2019-11-12 12:09:44,732Z ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand' with failure. 2019-11-12 12:09:44,757Z INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [14057067] Running command: RemoveImageCommand internal: true. Entities affected : ID: 78bfcb57-3a0d-4a71-a3f8-480ebae37c74 Type: Storage 2019-11-12 12:09:44,780Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [14057067] START, DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', storageDomainId='78bfcb57-3a0d-4a71-a3f8-480ebae37c74', imageGroupId='56ddac45-1913-4d24-af97-2f969fc3fa2e', postZeros='false', discard='false', forceDelete='false'}), log id: aaea0bc 2019-11-12 12:09:44,804Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [14057067] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS failed: Image does not exist in domain: u'image=56ddac45-1913-4d24-af97-2f969fc3fa2e, domain=78bfcb57-3a0d-4a71-a3f8-480ebae37c74' 2019-11-12 12:09:44,804Z ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [14057067] Command 'DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='47523e52-311f-4829-a731-3d91f7885974', ignoreFailoverLimit='false', storageDomainId='78bfcb57-3a0d-4a71-a3f8-480ebae37c74', imageGroupId='56ddac45-1913-4d24-af97-2f969fc3fa2e', postZeros='false', discard='false', forceDelete='false'})' execution failed: IRSGenericException: IRSErrorException: Image does not exist in domain: u'image=56ddac45-1913-4d24-af97-2f969fc3fa2e, domain=78bfcb57-3a0d-4a71-a3f8-480ebae37c74' 2019-11-12 12:09:44,804Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [14057067] FINISH, DeleteImageGroupVDSCommand, return: , log id: aaea0bc 2019-11-12 12:09:44,804Z INFO [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [14057067] Disk '56ddac45-1913-4d24-af97-2f969fc3fa2e' doesn't exist on storage domain '78bfcb57-3a0d-4a71-a3f8-480ebae37c74', rolling forward 2019-11-12 12:09:44,815Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [14057067] Removed task 'c964f397-f054-4d81-98d6-052ce5a30168' from DataBase 2019-11-12 12:09:44,887Z INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [] transaction rolled back 2019-11-12 12:09:44,887Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [] Trying to release a shared lock for key: '400688e8-3c5b-4cca-9ef8-bbb6c79d54edDISK' , but lock does not exist 2019-11-12 12:09:44,887Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [] Trying to release a shared lock for key: '819aecf6-c83d-4470-9915-e309390d8098TEMPLATE' , but lock does not exist 2019-11-12 12:09:44,888Z WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [] Trying to release exclusive lock which does not exist, lock key: 'centos-testvm-a910VM_NAME' 2019-11-12 12:09:44,888Z INFO [org.ovirt.engine.core.bll.AddVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [] Lock freed to object 'EngineLock:{exclusiveLocks='[centos-testvm-a910=VM_NAME]', sharedLocks='[400688e8-3c5b-4cca-9ef8-bbb6c79d54ed=DISK, 819aecf6-c83d-4470-9915-e309390d8098=TEMPLATE]'}' 2019-11-12 12:09:44,903Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [] EVENT_ID: USER_ADD_VM_FINISHED_FAILURE(60), Failed to complete VM centos-testvm-a910 creation. 2019-11-12 12:09:44,924Z ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Failed invoking callback end method 'onFailed' for command '97d4103b-acef-4b55-a379-71a1baa04a4f' with exception 'null', the callback is marked for end method retries but max number of retries have been attempted. The command will be marked as Failed. 2019-11-12 12:09:44,925Z INFO [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Exception in invoking callback of command AddVm (97d4103b-acef-4b55-a379-71a1baa04a4f): NullPointerException: 2019-11-12 12:09:44,925Z ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Error invoking callback method 'onFailed' for 'FAILED' command '97d4103b-acef-4b55-a379-71a1baa04a4f' 2019-11-12 12:09:44,925Z ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [7748e6ab-7fb3-4ec5-9b2a-c7012a050efa] Exception: javax.ejb.EJBTransactionRolledbackException at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:203) [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:420) [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:148) [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81) [weld-ejb-3.1.1.Final.jar:3.1.1.Final] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:618) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81) at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view2.endAction(Unknown Source) [bll.jar:] at sun.reflect.GeneratedMethodAccessor1071.invoke(Unknown Source) [:1.8.0_222] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_222] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_222] at org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:410) [weld-core-impl-3.1.1.Final.jar:3.1.1.Final] at org.jboss.weld.module.ejb.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:134) [weld-ejb-3.1.1.Final.jar:3.1.1.Final] at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) [weld-core-impl-3.1.1.Final.jar:3.1.1.Final] at org.jboss.weld.module.ejb.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPointPropagatingEnterpriseTargetBeanInstance.java:68) [weld-ejb-3.1.1.Final.jar:3.1.1.Final] at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:106) [weld-core-impl-3.1.1.Final.jar:3.1.1.Final] at org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$BackendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.endAction(Unknown Source) [bll.jar:] at org.ovirt.engine.core.bll.VmCommand.endActionOnDisks(VmCommand.java:266) [bll.jar:] at org.ovirt.engine.core.bll.AddVmCommand.endWithFailure(AddVmCommand.java:1452) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.internalEndWithFailure(CommandBase.java:736) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.endActionInTransactionScope(CommandBase.java:658) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2003) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:202) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:137) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.endAction(CommandBase.java:520) [bll.jar:] at org.ovirt.engine.core.bll.ChildCommandsCallbackBase.endAction(ChildCommandsCallbackBase.java:134) [bll.jar:] at org.ovirt.engine.core.bll.ChildCommandsCallbackBase.onFailed(ChildCommandsCallbackBase.java:174) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.endCallback(CommandCallbacksPoller.java:67) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethodsImpl(CommandCallbacksPoller.java:166) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:109) [bll.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_222] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [rt.jar:1.8.0_222] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:383) [javax.enterprise.concurrent-1.0.jar:] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:534) [javax.enterprise.concurrent-1.0.jar:] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_222] at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) [javax.enterprise.concurrent-1.0.jar:] Caused by: java.lang.NullPointerException Please take a look. ----- kind regards/met vrindelijke groet Marko Vrgotic ActiveVideo On 10/10/2019, 14:23, "Vrgotic, Marko" <M.Vrgotic(a)activevideo.com> wrote: Hi, I could not check as in the meantime I destroyed my staging setup. Re-provisioning it now and will retest with ansible. On 10/10/2019, 10:51, "Vrgotic, Marko" <M.Vrgotic(a)activevideo.com> wrote: Hi Ondra, That is true, But why does it work from Portal? Image ID is not differ when run when ansible against same template? I will check anyway. On 10/10/2019, 09:47, "Ondra Machacek" <omachace(a)redhat.com> wrote: The error message is: Image does not exist in domain: u'image=d680a46a-7188-45f6-b6a7-3333f830c0b4, domain=935de3ac-a735-4fc8-9161-26bfd751ffc7' Can you please check if the image with id 'd680a46a-7188-45f6-b6a7-3333f830c0b4' exists on storage domain with id '935de3ac-a735-4fc8-9161-26bfd751ffc7'? On 08/10/2019 13:05, Vrgotic, Marko wrote: > Dear oVirt, > > Just recently, I have upgraded oVirt staging environment from 4.3.4 to > 4.3.6 release. > > Since then I noticed the issue when creating VMs from template, using > Ansible. When creating VM from template using UI, all works well. > > Tests were executed with same user, with SuperUser priviledges. > > pip freeze > > *ansible==2.7.13* > > asn1crypto==1.0.0 > > bcrypt==3.1.7 > > cffi==1.12.3 > > cryptography==2.7 > > dnspython==1.16.0 > > ipaddress==1.0.22 > > Jinja2==2.10.1 > > lxml==4.4.1 > > MarkupSafe==1.1.1 > > netaddr==0.7.19 > > *ovirt-engine-sdk-python==4.3.3* > > paramiko==2.6.0 > > pycparser==2.19 > > pycurl==7.43.0.3 > > PyNaCl==1.3.0 > > PyYAML==5.1.2 > > six==1.12.0 > > *Logs from engine: * > > 2019-10-08 09:57:42,294Z INFO [org.ovirt.engine.core.bll.AddVmCommand] > (default task-23) [067ec489-ae6c-4871-8ab6-8296016ca1ce] Lock Acquired > to object 'EngineLock:{exclusiveLocks='[centos-testvm-024=VM_NAME]', > sharedLocks='[9ac6f4ad-58d0-4a7e-b424-91f2d76abcac=TEMPLATE, > a7e42574-be60-4c94-94a5-cc4b30fdb16f=DISK]'}' > > 2019-10-08 09:57:42,299Z INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43687) [390b4c75] FINISH, > HSMClearTaskVDSCommand, return: , log id: be932a0 > > 2019-10-08 09:57:42,299Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43687) [390b4c75] FINISH, > SPMClearTaskVDSCommand, return: , log id: 3e25883 > > 2019-10-08 09:57:42,303Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43687) [390b4c75] > BaseAsyncTask::removeTaskFromDB: Removed task > '7e1d68a2-bb49-4456-b228-db851fe6603c' from DataBase > > 2019-10-08 09:57:42,303Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43687) [390b4c75] > CommandAsyncTask::HandleEndActionResult [within thread]: Removing > CommandMultiAsyncTasks object for entity > '4f8a6897-66e8-453b-b706-246cb8505b3e' > > 2019-10-08 09:57:42,375Z INFO [org.ovirt.engine.core.bll.AddVmCommand] > (default task-23) [067ec489-ae6c-4871-8ab6-8296016ca1ce] Running > command: AddVmCommand internal: false. Entities affected : ID: > 2e428504-c339-11e9-87ef-00163e3ec101 Type: ClusterAction group CREATE_VM > with role type USER, ID: 9ac6f4ad-58d0-4a7e-b424-91f2d76abcac Type: > VmTemplateAction group CREATE_VM with role type USER, ID: > 935de3ac-a735-4fc8-9161-26bfd751ffc7 Type: StorageAction group > CREATE_DISK with role type USER > > 2019-10-08 09:57:42,488Z INFO > [org.ovirt.engine.core.bll.AddRngDeviceCommand] (default task-23) > [332c57d4] Running command: AddRngDeviceCommand internal: true. Entities > affected : ID: d5719cf0-2a64-4a28-bbf2-edc1bb05b777 Type: VMAction > group EDIT_VM_PROPERTIES with role type USER > > 2019-10-08 09:57:42,498Z INFO > [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (default > task-23) [332c57d4] START, SetVmStatusVDSCommand( > SetVmStatusVDSCommandParameters:{vmId='d5719cf0-2a64-4a28-bbf2-edc1bb05b777', > status='ImageLocked', exitStatus='Normal'}), log id: 2fe004e2 > > 2019-10-08 09:57:42,507Z INFO > [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (default > task-23) [332c57d4] FINISH, SetVmStatusVDSCommand, return: , log id: > 2fe004e2 > > 2019-10-08 09:57:42,528Z INFO > [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] > (default task-23) [067ec489-ae6c-4871-8ab6-8296016ca1ce] Running > command: CreateSnapshotFromTemplateCommand internal: true. Entities > affected : ID: 935de3ac-a735-4fc8-9161-26bfd751ffc7 Type: Storage > > 2019-10-08 09:57:42,554Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.CreateVolumeVDSCommand] > (default task-23) [067ec489-ae6c-4871-8ab6-8296016ca1ce] START, > CreateVolumeVDSCommand( > CreateVolumeVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='d680a46a-7188-45f6-b6a7-3333f830c0b4', > imageSizeInBytes='8589934592', volumeFormat='COW', > newImageId='9a96018c-ce7d-4c77-a91a-30fc16c7b2ba', imageType='Sparse', > newImageDescription='', imageInitialSizeInBytes='0', > imageId='f9e0c4d9-81dd-4e44-8a71-33e55d4399c6', > sourceImageGroupId='a7e42574-be60-4c94-94a5-cc4b30fdb16f'}), log id: > 37b29c7d > > 2019-10-08 09:57:42,639Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.CreateVolumeVDSCommand] > (default task-23) [067ec489-ae6c-4871-8ab6-8296016ca1ce] FINISH, > CreateVolumeVDSCommand, return: 9a96018c-ce7d-4c77-a91a-30fc16c7b2ba, > log id: 37b29c7d > > 2019-10-08 09:57:42,645Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (default task-23) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] CommandAsyncTask::Adding > CommandMultiAsyncTasks object for command > 'f1665b66-2032-4a60-944e-c04beace088e' > > 2019-10-08 09:57:42,645Z INFO > [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (default task-23) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] > CommandMultiAsyncTasks::attachTask: Attaching task > '99abc310-b44b-435c-8d43-922b782a69dc' to command > 'f1665b66-2032-4a60-944e-c04beace088e'. > > 2019-10-08 09:57:42,660Z INFO > [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (default task-23) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Adding task > '99abc310-b44b-435c-8d43-922b782a69dc' (Parent Command > 'CreateSnapshotFromTemplate', Parameters Type > 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters'), polling > hasn't started yet.. > > 2019-10-08 09:57:42,934Z INFO > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (default task-23) [067ec489-ae6c-4871-8ab6-8296016ca1ce] EVENT_ID: > USER_ADD_VM_STARTED(37), VM centos-testvm-024 creation was initiated by > mteam@internal-authz. > > 2019-10-08 09:57:42,935Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (default task-23) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] BaseAsyncTask::startPollingTask: > Starting to poll task '99abc310-b44b-435c-8d43-922b782a69dc'. > > 2019-10-08 09:57:44,384Z INFO > [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] > (EE-ManagedThreadFactory-engineScheduled-Thread-74) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Command 'AddVm' (id: > '1b65f576-365d-4e8b-a920-621516cbdaed') waiting on child command id: > 'f1665b66-2032-4a60-944e-c04beace088e' type:'CreateSnapshotFromTemplate' > to complete > > 2019-10-08 09:57:48,393Z INFO > [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] > (EE-ManagedThreadFactory-engineScheduled-Thread-37) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Command 'AddVm' (id: > '1b65f576-365d-4e8b-a920-621516cbdaed') waiting on child command id: > 'f1665b66-2032-4a60-944e-c04beace088e' type:'CreateSnapshotFromTemplate' > to complete > > 2019-10-08 09:57:48,394Z INFO > [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] > (EE-ManagedThreadFactory-engineScheduled-Thread-37) [5830ba58] Command > 'AddDisk' id: '937776b8-b94d-4178-b9c2-e39fe42cb06c' child commands > '[63409b0b-749a-45d5-85a5-baf55d5fa2a4]' executions were completed, > status 'SUCCEEDED' > > 2019-10-08 09:57:48,394Z INFO > [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] > (EE-ManagedThreadFactory-engineScheduled-Thread-37) [5830ba58] Command > 'AddDisk' id: '937776b8-b94d-4178-b9c2-e39fe42cb06c' Updating status to > 'SUCCEEDED', The command end method logic will be executed by one of its > parent commands. > > 2019-10-08 09:57:48,398Z INFO > [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] > (EE-ManagedThreadFactory-engineScheduled-Thread-37) [5830ba58] Command > 'AddDisk' id: 'fdc5571e-a925-4400-a54a-a1ef1dcd23ae' child commands > '[54d68cae-6585-4351-aaf7-fe687905acf4]' executions were completed, > status 'SUCCEEDED' > > 2019-10-08 09:57:48,398Z INFO > [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] > (EE-ManagedThreadFactory-engineScheduled-Thread-37) [5830ba58] Command > 'AddDisk' id: 'fdc5571e-a925-4400-a54a-a1ef1dcd23ae' Updating status to > 'SUCCEEDED', The command end method logic will be executed by one of its > parent commands. > > 2019-10-08 09:57:48,401Z INFO > [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] > (EE-ManagedThreadFactory-engineScheduled-Thread-37) [5830ba58] Command > 'ProcessOvfUpdateForStorageDomain' (id: > 'bdcdae3a-00f6-4176-9d27-0bb0f04912a3') waiting on child command id: > 'b4050edd-62a7-4b83-a4ab-a21c83bf3ade' > type:'CreateOvfVolumeForStorageDomain' to complete > > 2019-10-08 09:57:48,402Z INFO > [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] > (EE-ManagedThreadFactory-engineScheduled-Thread-37) [5830ba58] Command > 'CreateOvfVolumeForStorageDomain' id: > 'b4050edd-62a7-4b83-a4ab-a21c83bf3ade' child commands > '[937776b8-b94d-4178-b9c2-e39fe42cb06c]' executions were completed, > status 'SUCCEEDED' > > 2019-10-08 09:57:48,404Z INFO > [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] > (EE-ManagedThreadFactory-engineScheduled-Thread-37) [5830ba58] Command > 'CreateOvfVolumeForStorageDomain' id: > '5fd47910-95b2-4387-8e26-ea2c80b078a0' child commands > '[fdc5571e-a925-4400-a54a-a1ef1dcd23ae]' executions were completed, > status 'SUCCEEDED' > > 2019-10-08 09:57:49,416Z INFO > [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [390b4c75] Command > 'ProcessOvfUpdateForStorageDomain' id: > '849bd16c-00d2-48a2-8fde-631af0dfe1b5' child commands > '[a80a3657-c813-42c1-a43d-27442b5dc238, > 4f8a6897-66e8-453b-b706-246cb8505b3e]' executions were completed, status > 'SUCCEEDED' > > 2019-10-08 09:57:49,418Z INFO > [org.ovirt.engine.core.bll.storage.ovfstore.CreateOvfVolumeForStorageDomainCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [5830ba58] Ending > command > 'org.ovirt.engine.core.bll.storage.ovfstore.CreateOvfVolumeForStorageDomainCommand' > successfully. > > 2019-10-08 09:57:49,420Z INFO > [org.ovirt.engine.core.bll.storage.disk.AddDiskCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [5830ba58] Ending > command 'org.ovirt.engine.core.bll.storage.disk.AddDiskCommand' > successfully. > > 2019-10-08 09:57:49,426Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.AddImageFromScratchCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-3) > [5830ba58] Ending command > 'org.ovirt.engine.core.bll.storage.disk.image.AddImageFromScratchCommand' successfully. > > 2019-10-08 09:57:49,427Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [5830ba58] START, > GetImageInfoVDSCommand( > GetImageInfoVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='66984468-bbfa-4941-8c6b-df72a6d920f2', > imageId='7db55211-041a-40c8-85a6-a2ff1bd182ce'}), log id: 6c02a73c > > 2019-10-08 09:57:49,429Z INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [5830ba58] START, > GetVolumeInfoVDSCommand(HostName = ovirt-staging-hv-01.avinity.tv, > GetVolumeInfoVDSCommandParameters:{hostId='bfc4c379-bd5f-45be-98a4-f0469dc3817c', > storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='66984468-bbfa-4941-8c6b-df72a6d920f2', > imageId='7db55211-041a-40c8-85a6-a2ff1bd182ce'}), log id: 2e131d9d > > 2019-10-08 09:57:49,456Z INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [5830ba58] FINISH, > GetVolumeInfoVDSCommand, return: > org.ovirt.engine.core.common.businessentities.storage.DiskImage@26992dd0, log > id: 2e131d9d > > 2019-10-08 09:57:49,456Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [5830ba58] FINISH, > GetImageInfoVDSCommand, return: > org.ovirt.engine.core.common.businessentities.storage.DiskImage@26992dd0, log > id: 6c02a73c > > 2019-10-08 09:57:49,464Z WARN > [org.ovirt.engine.core.bll.storage.disk.AddDiskCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [] VM is null - no > unlocking > > 2019-10-08 09:57:49,481Z INFO > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [] EVENT_ID: > USER_ADD_DISK_FINISHED_SUCCESS(2,021), The disk 'OVF_STORE' was > successfully added. > > 2019-10-08 09:57:49,490Z INFO > [org.ovirt.engine.core.bll.storage.ovfstore.CreateOvfVolumeForStorageDomainCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [5830ba58] Ending > command > 'org.ovirt.engine.core.bll.storage.ovfstore.CreateOvfVolumeForStorageDomainCommand' > successfully. > > 2019-10-08 09:57:49,491Z INFO > [org.ovirt.engine.core.bll.storage.disk.AddDiskCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [5830ba58] Ending > command 'org.ovirt.engine.core.bll.storage.disk.AddDiskCommand' > successfully. > > 2019-10-08 09:57:49,495Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.AddImageFromScratchCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-3) > [5830ba58] Ending command > 'org.ovirt.engine.core.bll.storage.disk.image.AddImageFromScratchCommand' successfully. > > 2019-10-08 09:57:49,497Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [5830ba58] START, > GetImageInfoVDSCommand( > GetImageInfoVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='ee208fd3-e05c-4ba8-aba9-fa521d69a870', > imageId='484c1eed-b03a-4fa0-a12d-0087294dec86'}), log id: 7a3dad31 > > 2019-10-08 09:57:49,498Z INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [5830ba58] START, > GetVolumeInfoVDSCommand(HostName = ovirt-staging-hv-01.avinity.tv, > GetVolumeInfoVDSCommandParameters:{hostId='bfc4c379-bd5f-45be-98a4-f0469dc3817c', > storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='ee208fd3-e05c-4ba8-aba9-fa521d69a870', > imageId='484c1eed-b03a-4fa0-a12d-0087294dec86'}), log id: 5b85c4a5 > > 2019-10-08 09:57:49,522Z INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [5830ba58] FINISH, > GetVolumeInfoVDSCommand, return: > org.ovirt.engine.core.common.businessentities.storage.DiskImage@13555e4, > log id: 5b85c4a5 > > 2019-10-08 09:57:49,522Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [5830ba58] FINISH, > GetImageInfoVDSCommand, return: > org.ovirt.engine.core.common.businessentities.storage.DiskImage@13555e4, > log id: 7a3dad31 > > 2019-10-08 09:57:49,529Z WARN > [org.ovirt.engine.core.bll.storage.disk.AddDiskCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [] VM is null - no > unlocking > > 2019-10-08 09:57:49,540Z INFO > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-3) [] EVENT_ID: > USER_ADD_DISK_FINISHED_SUCCESS(2,021), The disk 'OVF_STORE' was > successfully added. > > 2019-10-08 09:57:50,552Z INFO > [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [390b4c75] Ending > command > 'org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand' > successfully. > > 2019-10-08 09:57:50,575Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] START, > SetVolumeDescriptionVDSCommand( > SetVolumeDescriptionVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='ee208fd3-e05c-4ba8-aba9-fa521d69a870', > imageId='484c1eed-b03a-4fa0-a12d-0087294dec86'}), log id: 7e77a9a8 > > 2019-10-08 09:57:50,576Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] -- > executeIrsBrokerCommand: calling 'setVolumeDescription', parameters: > > 2019-10-08 09:57:50,576Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > spUUID=2e3b3484-c339-11e9-8d02-00163e3ec101 > > 2019-10-08 09:57:50,576Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > sdUUID=935de3ac-a735-4fc8-9161-26bfd751ffc7 > > 2019-10-08 09:57:50,576Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > imageGroupGUID=ee208fd3-e05c-4ba8-aba9-fa521d69a870 > > 2019-10-08 09:57:50,576Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > volUUID=484c1eed-b03a-4fa0-a12d-0087294dec86 > > 2019-10-08 09:57:50,576Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > description={"Updated":false,"Last Updated":null,"Storage > Domains":[{"uuid":"935de3ac-a735-4fc8-9161-26bfd751ffc7"}],"Disk > Description":"OVF_STORE"} > > 2019-10-08 09:57:50,606Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] FINISH, > SetVolumeDescriptionVDSCommand, return: , log id: 7e77a9a8 > > 2019-10-08 09:57:50,614Z INFO > [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] Lock > Acquired to object 'EngineLock:{exclusiveLocks='', > sharedLocks='[bfc4c379-bd5f-45be-98a4-f0469dc3817c=VDS_EXECUTION]'}' > > 2019-10-08 09:57:50,623Z INFO > [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] Running > command: UploadStreamCommand internal: true. Entities affected : ID: > 935de3ac-a735-4fc8-9161-26bfd751ffc7 Type: Storage > > 2019-10-08 09:57:50,624Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] START, > UploadStreamVDSCommand(HostName = ovirt-staging-hv-01.avinity.tv, > UploadStreamVDSCommandParameters:{hostId='bfc4c379-bd5f-45be-98a4-f0469dc3817c'}), > log id: 531a4a4a > > 2019-10-08 09:57:50,625Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] -- > executeVdsBrokerCommand, parameters: > > 2019-10-08 09:57:50,625Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > spUUID=2e3b3484-c339-11e9-8d02-00163e3ec101 > > 2019-10-08 09:57:50,625Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > sdUUID=935de3ac-a735-4fc8-9161-26bfd751ffc7 > > 2019-10-08 09:57:50,625Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > imageGUID=ee208fd3-e05c-4ba8-aba9-fa521d69a870 > > 2019-10-08 09:57:50,625Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > volUUID=484c1eed-b03a-4fa0-a12d-0087294dec86 > > 2019-10-08 09:57:50,625Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ size=30720 > > 2019-10-08 09:57:51,331Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] FINISH, > UploadStreamVDSCommand, return: , log id: 531a4a4a > > 2019-10-08 09:57:51,342Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] > CommandAsyncTask::Adding CommandMultiAsyncTasks object for command > 'c11f4d3c-f85b-43c5-986f-a517aed66d7e' > > 2019-10-08 09:57:51,342Z INFO > [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] > CommandMultiAsyncTasks::attachTask: Attaching task > 'e5d00ff5-eaa2-48e9-ab10-d6d06a01e89d' to command > 'c11f4d3c-f85b-43c5-986f-a517aed66d7e'. > > 2019-10-08 09:57:51,351Z INFO > [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] Adding > task 'e5d00ff5-eaa2-48e9-ab10-d6d06a01e89d' (Parent Command > 'UploadStream', Parameters Type > 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters'), polling > hasn't started yet.. > > 2019-10-08 09:57:51,355Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] > BaseAsyncTask::startPollingTask: Starting to poll task > 'e5d00ff5-eaa2-48e9-ab10-d6d06a01e89d'. > > 2019-10-08 09:57:51,358Z INFO > [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] Lock freed > to object 'EngineLock:{exclusiveLocks='', > sharedLocks='[bfc4c379-bd5f-45be-98a4-f0469dc3817c=VDS_EXECUTION]'}' > > 2019-10-08 09:57:51,360Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] START, > SetVolumeDescriptionVDSCommand( > SetVolumeDescriptionVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='ee208fd3-e05c-4ba8-aba9-fa521d69a870', > imageId='484c1eed-b03a-4fa0-a12d-0087294dec86'}), log id: 238169f6 > > 2019-10-08 09:57:51,360Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] -- > executeIrsBrokerCommand: calling 'setVolumeDescription', parameters: > > 2019-10-08 09:57:51,360Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > spUUID=2e3b3484-c339-11e9-8d02-00163e3ec101 > > 2019-10-08 09:57:51,360Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > sdUUID=935de3ac-a735-4fc8-9161-26bfd751ffc7 > > 2019-10-08 09:57:51,360Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > imageGroupGUID=ee208fd3-e05c-4ba8-aba9-fa521d69a870 > > 2019-10-08 09:57:51,360Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > volUUID=484c1eed-b03a-4fa0-a12d-0087294dec86 > > 2019-10-08 09:57:51,360Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > description={"Updated":true,"Size":30720,"Last Updated":"Tue Oct 08 > 09:57:50 UTC 2019","Storage > Domains":[{"uuid":"935de3ac-a735-4fc8-9161-26bfd751ffc7"}],"Disk > Description":"OVF_STORE"} > > 2019-10-08 09:57:51,389Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] FINISH, > SetVolumeDescriptionVDSCommand, return: , log id: 238169f6 > > 2019-10-08 09:57:51,596Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] START, > SetVolumeDescriptionVDSCommand( > SetVolumeDescriptionVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='66984468-bbfa-4941-8c6b-df72a6d920f2', > imageId='7db55211-041a-40c8-85a6-a2ff1bd182ce'}), log id: 59bde70f > > 2019-10-08 09:57:51,627Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] -- > executeIrsBrokerCommand: calling 'setVolumeDescription', parameters: > > 2019-10-08 09:57:51,627Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > spUUID=2e3b3484-c339-11e9-8d02-00163e3ec101 > > 2019-10-08 09:57:51,627Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > sdUUID=935de3ac-a735-4fc8-9161-26bfd751ffc7 > > 2019-10-08 09:57:51,627Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > imageGroupGUID=66984468-bbfa-4941-8c6b-df72a6d920f2 > > 2019-10-08 09:57:51,627Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > volUUID=7db55211-041a-40c8-85a6-a2ff1bd182ce > > 2019-10-08 09:57:51,627Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > description={"Updated":false,"Last Updated":null,"Storage > Domains":[{"uuid":"935de3ac-a735-4fc8-9161-26bfd751ffc7"}],"Disk > Description":"OVF_STORE"} > > 2019-10-08 09:57:51,664Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] FINISH, > SetVolumeDescriptionVDSCommand, return: , log id: 59bde70f > > 2019-10-08 09:57:51,672Z INFO > [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] Lock > Acquired to object 'EngineLock:{exclusiveLocks='', > sharedLocks='[bfc4c379-bd5f-45be-98a4-f0469dc3817c=VDS_EXECUTION]'}' > > 2019-10-08 09:57:51,683Z INFO > [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] Running > command: UploadStreamCommand internal: true. Entities affected : ID: > 935de3ac-a735-4fc8-9161-26bfd751ffc7 Type: Storage > > 2019-10-08 09:57:51,684Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] START, > UploadStreamVDSCommand(HostName = ovirt-staging-hv-01.avinity.tv, > UploadStreamVDSCommandParameters:{hostId='bfc4c379-bd5f-45be-98a4-f0469dc3817c'}), > log id: 1cb812b6 > > 2019-10-08 09:57:51,684Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] -- > executeVdsBrokerCommand, parameters: > > 2019-10-08 09:57:51,684Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > spUUID=2e3b3484-c339-11e9-8d02-00163e3ec101 > > 2019-10-08 09:57:51,684Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > sdUUID=935de3ac-a735-4fc8-9161-26bfd751ffc7 > > 2019-10-08 09:57:51,684Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > imageGUID=66984468-bbfa-4941-8c6b-df72a6d920f2 > > 2019-10-08 09:57:51,684Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > volUUID=7db55211-041a-40c8-85a6-a2ff1bd182ce > > 2019-10-08 09:57:51,684Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ size=30720 > > 2019-10-08 09:57:51,933Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] FINISH, > UploadStreamVDSCommand, return: , log id: 1cb812b6 > > 2019-10-08 09:57:51,943Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] > CommandAsyncTask::Adding CommandMultiAsyncTasks object for command > 'd0c07988-0e43-4e6d-b86c-e736b28d4537' > > 2019-10-08 09:57:51,943Z INFO > [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] > CommandMultiAsyncTasks::attachTask: Attaching task > 'ba88edd0-8188-472e-8f59-aa3f90e63181' to command > 'd0c07988-0e43-4e6d-b86c-e736b28d4537'. > > 2019-10-08 09:57:51,952Z INFO > [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] Adding > task 'ba88edd0-8188-472e-8f59-aa3f90e63181' (Parent Command > 'UploadStream', Parameters Type > 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters'), polling > hasn't started yet.. > > 2019-10-08 09:57:51,958Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] > BaseAsyncTask::startPollingTask: Starting to poll task > 'ba88edd0-8188-472e-8f59-aa3f90e63181'. > > 2019-10-08 09:57:51,961Z INFO > [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] Lock freed > to object 'EngineLock:{exclusiveLocks='', > sharedLocks='[bfc4c379-bd5f-45be-98a4-f0469dc3817c=VDS_EXECUTION]'}' > > 2019-10-08 09:57:51,962Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] START, > SetVolumeDescriptionVDSCommand( > SetVolumeDescriptionVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='66984468-bbfa-4941-8c6b-df72a6d920f2', > imageId='7db55211-041a-40c8-85a6-a2ff1bd182ce'}), log id: 213b3075 > > 2019-10-08 09:57:51,962Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] -- > executeIrsBrokerCommand: calling 'setVolumeDescription', parameters: > > 2019-10-08 09:57:51,962Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > spUUID=2e3b3484-c339-11e9-8d02-00163e3ec101 > > 2019-10-08 09:57:51,962Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > sdUUID=935de3ac-a735-4fc8-9161-26bfd751ffc7 > > 2019-10-08 09:57:51,962Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > imageGroupGUID=66984468-bbfa-4941-8c6b-df72a6d920f2 > > 2019-10-08 09:57:51,962Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > volUUID=7db55211-041a-40c8-85a6-a2ff1bd182ce > > 2019-10-08 09:57:51,962Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] ++ > description={"Updated":true,"Size":30720,"Last Updated":"Tue Oct 08 > 09:57:50 UTC 2019","Storage > Domains":[{"uuid":"935de3ac-a735-4fc8-9161-26bfd751ffc7"}],"Disk > Description":"OVF_STORE"} > > 2019-10-08 09:57:52,028Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-1) [5830ba58] FINISH, > SetVolumeDescriptionVDSCommand, return: , log id: 213b3075 > > 2019-10-08 09:57:52,224Z INFO > [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] Polling and > updating Async Tasks: 7 tasks, 3 tasks to poll now > > 2019-10-08 09:57:52,228Z ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] Failed in > 'HSMGetAllTasksStatusesVDS' method > > 2019-10-08 09:57:52,235Z ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] EVENT_ID: > VDS_BROKER_COMMAND_FAILURE(10,802), VDSM ovirt-staging-hv-01.avinity.tv > command HSMGetAllTasksStatusesVDS failed: Invalid parameter: 'size=16777216' > > 2019-10-08 09:57:52,235Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] > SPMAsyncTask::PollTask: Polling task > '99abc310-b44b-435c-8d43-922b782a69dc' (Parent Command > 'CreateSnapshotFromTemplate', Parameters Type > 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned > status 'finished', result 'cleanSuccess'. > > 2019-10-08 09:57:52,240Z ERROR > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] > BaseAsyncTask::logEndTaskFailure: Task > '99abc310-b44b-435c-8d43-922b782a69dc' (Parent Command > 'CreateSnapshotFromTemplate', Parameters Type > 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended > with failure: > > -- Result: 'cleanSuccess' > > -- Message: 'VDSGenericException: VDSErrorException: Failed to > HSMGetAllTasksStatusesVDS, error = Invalid parameter: 'size=16777216', > code = 100', > > -- Exception: 'VDSGenericException: VDSErrorException: Failed to > HSMGetAllTasksStatusesVDS, error = Invalid parameter: 'size=16777216', > code = 100' > > 2019-10-08 09:57:52,241Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] > CommandAsyncTask::endActionIfNecessary: All tasks of command > 'f1665b66-2032-4a60-944e-c04beace088e' has ended -> executing 'endAction' > > 2019-10-08 09:57:52,241Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] > CommandAsyncTask::endAction: Ending action for '1' tasks (command ID: > 'f1665b66-2032-4a60-944e-c04beace088e'): calling endAction '. > > 2019-10-08 09:57:52,241Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] > SPMAsyncTask::PollTask: Polling task > 'e5d00ff5-eaa2-48e9-ab10-d6d06a01e89d' (Parent Command 'UploadStream', > Parameters Type > 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned > status 'finished', result 'success'. > > 2019-10-08 09:57:52,241Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] > BaseAsyncTask::onTaskEndSuccess: Task > 'e5d00ff5-eaa2-48e9-ab10-d6d06a01e89d' (Parent Command 'UploadStream', > Parameters Type > 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended > successfully. > > 2019-10-08 09:57:52,241Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43693) [] > CommandAsyncTask::endCommandAction [within thread] context: Attempting > to endAction 'CreateSnapshotFromTemplate', > > 2019-10-08 09:57:52,242Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] > CommandAsyncTask::endActionIfNecessary: All tasks of command > 'c11f4d3c-f85b-43c5-986f-a517aed66d7e' has ended -> executing 'endAction' > > 2019-10-08 09:57:52,243Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] > CommandAsyncTask::endAction: Ending action for '1' tasks (command ID: > 'c11f4d3c-f85b-43c5-986f-a517aed66d7e'): calling endAction '. > > 2019-10-08 09:57:52,243Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] > SPMAsyncTask::PollTask: Polling task > 'ba88edd0-8188-472e-8f59-aa3f90e63181' (Parent Command 'UploadStream', > Parameters Type > 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned > status 'finished', result 'success'. > > 2019-10-08 09:57:52,243Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] > BaseAsyncTask::onTaskEndSuccess: Task > 'ba88edd0-8188-472e-8f59-aa3f90e63181' (Parent Command 'UploadStream', > Parameters Type > 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended > successfully. > > 2019-10-08 09:57:52,243Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43694) [] > CommandAsyncTask::endCommandAction [within thread] context: Attempting > to endAction 'UploadStream', > > 2019-10-08 09:57:52,244Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] > CommandAsyncTask::endActionIfNecessary: All tasks of command > 'd0c07988-0e43-4e6d-b86c-e736b28d4537' has ended -> executing 'endAction' > > 2019-10-08 09:57:52,244Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] > CommandAsyncTask::endAction: Ending action for '1' tasks (command ID: > 'd0c07988-0e43-4e6d-b86c-e736b28d4537'): calling endAction '. > > 2019-10-08 09:57:52,245Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43695) [] > CommandAsyncTask::endCommandAction [within thread] context: Attempting > to endAction 'UploadStream', > > 2019-10-08 09:57:52,245Z INFO > [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] > (EE-ManagedThreadFactory-engine-Thread-43693) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Command > [id=f1665b66-2032-4a60-944e-c04beace088e]: Updating status to 'FAILED', > The command end method logic will be executed by one of its parent commands. > > 2019-10-08 09:57:52,247Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43693) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] > CommandAsyncTask::HandleEndActionResult [within thread]: endAction for > action type 'CreateSnapshotFromTemplate' completed, handling the result. > > 2019-10-08 09:57:52,247Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43693) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] > CommandAsyncTask::HandleEndActionResult [within thread]: endAction for > action type 'CreateSnapshotFromTemplate' succeeded, clearing tasks. > > 2019-10-08 09:57:52,247Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43693) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] SPMAsyncTask::ClearAsyncTask: > Attempting to clear task '99abc310-b44b-435c-8d43-922b782a69dc' > > 2019-10-08 09:57:52,247Z INFO > [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] > (EE-ManagedThreadFactory-engine-Thread-43695) [5830ba58] Ending command > 'org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand' > successfully. > > 2019-10-08 09:57:52,247Z INFO > [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] > (EE-ManagedThreadFactory-engine-Thread-43694) [5830ba58] Ending command > 'org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand' > successfully. > > 2019-10-08 09:57:52,247Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43693) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] START, SPMClearTaskVDSCommand( > SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > taskId='99abc310-b44b-435c-8d43-922b782a69dc'}), log id: 66a4e67 > > 2019-10-08 09:57:52,247Z INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43693) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] START, > HSMClearTaskVDSCommand(HostName = ovirt-staging-hv-01.avinity.tv, > HSMTaskGuidBaseVDSCommandParameters:{hostId='bfc4c379-bd5f-45be-98a4-f0469dc3817c', > taskId='99abc310-b44b-435c-8d43-922b782a69dc'}), log id: 7ec6d8b8 > > 2019-10-08 09:57:52,249Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43694) [5830ba58] > CommandAsyncTask::HandleEndActionResult [within thread]: endAction for > action type 'UploadStream' completed, handling the result. > > 2019-10-08 09:57:52,249Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43694) [5830ba58] > CommandAsyncTask::HandleEndActionResult [within thread]: endAction for > action type 'UploadStream' succeeded, clearing tasks. > > 2019-10-08 09:57:52,249Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43694) [5830ba58] > SPMAsyncTask::ClearAsyncTask: Attempting to clear task > 'e5d00ff5-eaa2-48e9-ab10-d6d06a01e89d' > > 2019-10-08 09:57:52,249Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43694) [5830ba58] START, > SPMClearTaskVDSCommand( > SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > taskId='e5d00ff5-eaa2-48e9-ab10-d6d06a01e89d'}), log id: 356927d > > 2019-10-08 09:57:52,250Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43695) [5830ba58] > CommandAsyncTask::HandleEndActionResult [within thread]: endAction for > action type 'UploadStream' completed, handling the result. > > 2019-10-08 09:57:52,250Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43695) [5830ba58] > CommandAsyncTask::HandleEndActionResult [within thread]: endAction for > action type 'UploadStream' succeeded, clearing tasks. > > 2019-10-08 09:57:52,250Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43695) [5830ba58] > SPMAsyncTask::ClearAsyncTask: Attempting to clear task > 'ba88edd0-8188-472e-8f59-aa3f90e63181' > > 2019-10-08 09:57:52,251Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43695) [5830ba58] START, > SPMClearTaskVDSCommand( > SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > taskId='ba88edd0-8188-472e-8f59-aa3f90e63181'}), log id: 75844e34 > > 2019-10-08 09:57:52,259Z INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43693) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] FINISH, HSMClearTaskVDSCommand, > return: , log id: 7ec6d8b8 > > 2019-10-08 09:57:52,259Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43693) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] FINISH, SPMClearTaskVDSCommand, > return: , log id: 66a4e67 > > 2019-10-08 09:57:52,260Z INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43694) [5830ba58] START, > HSMClearTaskVDSCommand(HostName = ovirt-staging-hv-01.avinity.tv, > HSMTaskGuidBaseVDSCommandParameters:{hostId='bfc4c379-bd5f-45be-98a4-f0469dc3817c', > taskId='e5d00ff5-eaa2-48e9-ab10-d6d06a01e89d'}), log id: 3b239769 > > 2019-10-08 09:57:52,263Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43693) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] BaseAsyncTask::removeTaskFromDB: > Removed task '99abc310-b44b-435c-8d43-922b782a69dc' from DataBase > > 2019-10-08 09:57:52,263Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43693) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] > CommandAsyncTask::HandleEndActionResult [within thread]: Removing > CommandMultiAsyncTasks object for entity > 'f1665b66-2032-4a60-944e-c04beace088e' > > 2019-10-08 09:57:52,297Z INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43694) [5830ba58] FINISH, > HSMClearTaskVDSCommand, return: , log id: 3b239769 > > 2019-10-08 09:57:52,297Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43694) [5830ba58] FINISH, > SPMClearTaskVDSCommand, return: , log id: 356927d > > 2019-10-08 09:57:52,298Z INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43695) [5830ba58] START, > HSMClearTaskVDSCommand(HostName = ovirt-staging-hv-01.avinity.tv, > HSMTaskGuidBaseVDSCommandParameters:{hostId='bfc4c379-bd5f-45be-98a4-f0469dc3817c', > taskId='ba88edd0-8188-472e-8f59-aa3f90e63181'}), log id: 32d44e1 > > 2019-10-08 09:57:52,301Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43694) [5830ba58] > BaseAsyncTask::removeTaskFromDB: Removed task > 'e5d00ff5-eaa2-48e9-ab10-d6d06a01e89d' from DataBase > > 2019-10-08 09:57:52,301Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43694) [5830ba58] > CommandAsyncTask::HandleEndActionResult [within thread]: Removing > CommandMultiAsyncTasks object for entity > 'c11f4d3c-f85b-43c5-986f-a517aed66d7e' > > 2019-10-08 09:57:52,314Z INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43695) [5830ba58] FINISH, > HSMClearTaskVDSCommand, return: , log id: 32d44e1 > > 2019-10-08 09:57:52,314Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-43695) [5830ba58] FINISH, > SPMClearTaskVDSCommand, return: , log id: 75844e34 > > 2019-10-08 09:57:52,317Z INFO > [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43695) [5830ba58] > BaseAsyncTask::removeTaskFromDB: Removed task > 'ba88edd0-8188-472e-8f59-aa3f90e63181' from DataBase > > 2019-10-08 09:57:52,317Z INFO > [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] > (EE-ManagedThreadFactory-engine-Thread-43695) [5830ba58] > CommandAsyncTask::HandleEndActionResult [within thread]: Removing > CommandMultiAsyncTasks object for entity > 'd0c07988-0e43-4e6d-b86c-e736b28d4537' > > 2019-10-08 09:57:54,039Z INFO > [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] > (EE-ManagedThreadFactory-engineScheduled-Thread-44) [5830ba58] Command > 'ProcessOvfUpdateForStorageDomain' id: > 'bdcdae3a-00f6-4176-9d27-0bb0f04912a3' child commands > '[b4050edd-62a7-4b83-a4ab-a21c83bf3ade, > 5fd47910-95b2-4387-8e26-ea2c80b078a0, > c11f4d3c-f85b-43c5-986f-a517aed66d7e, > d0c07988-0e43-4e6d-b86c-e736b28d4537]' executions were completed, status > 'SUCCEEDED' > > 2019-10-08 09:57:55,046Z INFO > [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-40) [5830ba58] Ending > command > 'org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand' > successfully. > > 2019-10-08 09:57:58,077Z INFO > [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] > (EE-ManagedThreadFactory-engineScheduled-Thread-76) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Command 'AddVm' id: > '1b65f576-365d-4e8b-a920-621516cbdaed' child commands > '[f1665b66-2032-4a60-944e-c04beace088e]' executions were completed, > status 'FAILED' > > 2019-10-08 09:57:59,094Z ERROR [org.ovirt.engine.core.bll.AddVmCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-27) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Ending command > 'org.ovirt.engine.core.bll.AddVmCommand' with failure. > > 2019-10-08 09:57:59,099Z ERROR > [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-27) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Ending command > 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand' > with failure. > > 2019-10-08 09:57:59,104Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-27) [5794ed92] Running > command: RemoveImageCommand internal: true. Entities affected : ID: > 935de3ac-a735-4fc8-9161-26bfd751ffc7 Type: Storage > > 2019-10-08 09:57:59,126Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-27) [5794ed92] START, > DeleteImageGroupVDSCommand( > DeleteImageGroupVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='d680a46a-7188-45f6-b6a7-3333f830c0b4', postZeros='false', > discard='false', forceDelete='false'}), log id: 69861a78 > > 2019-10-08 09:57:59,139Z ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-27) [5794ed92] EVENT_ID: > IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS > failed: Image does not exist in domain: > u'image=d680a46a-7188-45f6-b6a7-3333f830c0b4, > domain=935de3ac-a735-4fc8-9161-26bfd751ffc7' > > 2019-10-08 09:57:59,139Z ERROR > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-27) [5794ed92] Command > 'DeleteImageGroupVDSCommand( > DeleteImageGroupVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='d680a46a-7188-45f6-b6a7-3333f830c0b4', postZeros='false', > discard='false', forceDelete='false'})' execution failed: > IRSGenericException: IRSErrorException: Image does not exist in domain: > u'image=d680a46a-7188-45f6-b6a7-3333f830c0b4, > domain=935de3ac-a735-4fc8-9161-26bfd751ffc7' > > 2019-10-08 09:57:59,139Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-27) [5794ed92] FINISH, > DeleteImageGroupVDSCommand, return: , log id: 69861a78 > > 2019-10-08 09:57:59,139Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-27) [5794ed92] Disk > 'd680a46a-7188-45f6-b6a7-3333f830c0b4' doesn't exist on storage domain > '935de3ac-a735-4fc8-9161-26bfd751ffc7', rolling forward > > 2019-10-08 09:57:59,149Z INFO > [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-27) [5794ed92] Removed > task 'b4c4c723-4ba1-4d5a-86e6-24658a293a04' from DataBase > > 2019-10-08 09:57:59,220Z INFO > [org.ovirt.engine.core.utils.transaction.TransactionSupport] > (EE-ManagedThreadFactory-engineScheduled-Thread-27) [] transaction > rolled back > > 2019-10-08 09:57:59,220Z INFO [org.ovirt.engine.core.bll.AddVmCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-27) [] Lock freed to > object 'EngineLock:{exclusiveLocks='[centos-testvm-024=VM_NAME]', > sharedLocks='[9ac6f4ad-58d0-4a7e-b424-91f2d76abcac=TEMPLATE, > a7e42574-be60-4c94-94a5-cc4b30fdb16f=DISK]'}' > > 2019-10-08 09:57:59,233Z ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-27) [] EVENT_ID: > USER_ADD_VM_FINISHED_FAILURE(60), Failed to complete VM > centos-testvm-024 creation. > > 2019-10-08 09:57:59,233Z ERROR > [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] > (EE-ManagedThreadFactory-engineScheduled-Thread-27) [] Failed invoking > callback end method 'onFailed' for command > '1b65f576-365d-4e8b-a920-621516cbdaed' with exception 'null', the > callback is marked for end method retries > > 2019-10-08 09:58:01,252Z ERROR [org.ovirt.engine.core.bll.AddVmCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Ending command > 'org.ovirt.engine.core.bll.AddVmCommand' with failure. > > 2019-10-08 09:58:01,257Z ERROR > [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Ending command > 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand' > with failure. > > 2019-10-08 09:58:01,270Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [19251787] Running > command: RemoveImageCommand internal: true. Entities affected : ID: > 935de3ac-a735-4fc8-9161-26bfd751ffc7 Type: Storage > > 2019-10-08 09:58:01,308Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [19251787] START, > DeleteImageGroupVDSCommand( > DeleteImageGroupVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='d680a46a-7188-45f6-b6a7-3333f830c0b4', postZeros='false', > discard='false', forceDelete='false'}), log id: 35fec55 > > 2019-10-08 09:58:01,324Z ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [19251787] EVENT_ID: > IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS > failed: Image does not exist in domain: > u'image=d680a46a-7188-45f6-b6a7-3333f830c0b4, > domain=935de3ac-a735-4fc8-9161-26bfd751ffc7' > > 2019-10-08 09:58:01,324Z ERROR > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [19251787] Command > 'DeleteImageGroupVDSCommand( > DeleteImageGroupVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='d680a46a-7188-45f6-b6a7-3333f830c0b4', postZeros='false', > discard='false', forceDelete='false'})' execution failed: > IRSGenericException: IRSErrorException: Image does not exist in domain: > u'image=d680a46a-7188-45f6-b6a7-3333f830c0b4, > domain=935de3ac-a735-4fc8-9161-26bfd751ffc7' > > 2019-10-08 09:58:01,324Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [19251787] FINISH, > DeleteImageGroupVDSCommand, return: , log id: 35fec55 > > 2019-10-08 09:58:01,324Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [19251787] Disk > 'd680a46a-7188-45f6-b6a7-3333f830c0b4' doesn't exist on storage domain > '935de3ac-a735-4fc8-9161-26bfd751ffc7', rolling forward > > 2019-10-08 09:58:01,335Z INFO > [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [19251787] Removed > task '6ed0a844-5363-4afd-bffc-a80b6ace69ea' from DataBase > > 2019-10-08 09:58:01,395Z INFO > [org.ovirt.engine.core.utils.transaction.TransactionSupport] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] transaction > rolled back > > 2019-10-08 09:58:01,395Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] Trying to release > a shared lock for key: '9ac6f4ad-58d0-4a7e-b424-91f2d76abcacTEMPLATE' , > but lock does not exist > > 2019-10-08 09:58:01,395Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] Trying to release > a shared lock for key: 'a7e42574-be60-4c94-94a5-cc4b30fdb16fDISK' , but > lock does not exist > > 2019-10-08 09:58:01,395Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] Trying to release > exclusive lock which does not exist, lock key: 'centos-testvm-024VM_NAME' > > 2019-10-08 09:58:01,395Z INFO [org.ovirt.engine.core.bll.AddVmCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] Lock freed to > object 'EngineLock:{exclusiveLocks='[centos-testvm-024=VM_NAME]', > sharedLocks='[9ac6f4ad-58d0-4a7e-b424-91f2d76abcac=TEMPLATE, > a7e42574-be60-4c94-94a5-cc4b30fdb16f=DISK]'}' > > 2019-10-08 09:58:01,407Z ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] EVENT_ID: > USER_ADD_VM_FINISHED_FAILURE(60), Failed to complete VM > centos-testvm-024 creation. > > 2019-10-08 09:58:01,407Z ERROR > [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] > (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] Failed invoking > callback end method 'onFailed' for command > '1b65f576-365d-4e8b-a920-621516cbdaed' with exception 'null', the > callback is marked for end method retries > > 2019-10-08 09:58:05,427Z ERROR [org.ovirt.engine.core.bll.AddVmCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Ending command > 'org.ovirt.engine.core.bll.AddVmCommand' with failure. > > 2019-10-08 09:58:05,430Z ERROR > [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Ending command > 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand' > with failure. > > 2019-10-08 09:58:05,437Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [cf91364] Running > command: RemoveImageCommand internal: true. Entities affected : ID: > 935de3ac-a735-4fc8-9161-26bfd751ffc7 Type: Storage > > 2019-10-08 09:58:05,452Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [cf91364] START, > DeleteImageGroupVDSCommand( > DeleteImageGroupVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='d680a46a-7188-45f6-b6a7-3333f830c0b4', postZeros='false', > discard='false', forceDelete='false'}), log id: 6ac726b6 > > 2019-10-08 09:58:05,466Z ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [cf91364] EVENT_ID: > IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS > failed: Image does not exist in domain: > u'image=d680a46a-7188-45f6-b6a7-3333f830c0b4, > domain=935de3ac-a735-4fc8-9161-26bfd751ffc7' > > 2019-10-08 09:58:05,466Z ERROR > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [cf91364] Command > 'DeleteImageGroupVDSCommand( > DeleteImageGroupVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='d680a46a-7188-45f6-b6a7-3333f830c0b4', postZeros='false', > discard='false', forceDelete='false'})' execution failed: > IRSGenericException: IRSErrorException: Image does not exist in domain: > u'image=d680a46a-7188-45f6-b6a7-3333f830c0b4, > domain=935de3ac-a735-4fc8-9161-26bfd751ffc7' > > 2019-10-08 09:58:05,466Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [cf91364] FINISH, > DeleteImageGroupVDSCommand, return: , log id: 6ac726b6 > > 2019-10-08 09:58:05,467Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [cf91364] Disk > 'd680a46a-7188-45f6-b6a7-3333f830c0b4' doesn't exist on storage domain > '935de3ac-a735-4fc8-9161-26bfd751ffc7', rolling forward > > 2019-10-08 09:58:05,477Z INFO > [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [cf91364] Removed > task '31bd7a11-203d-4459-ac54-3c398c85771d' from DataBase > > 2019-10-08 09:58:05,490Z INFO > [org.ovirt.engine.core.utils.transaction.TransactionSupport] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [] transaction > rolled back > > 2019-10-08 09:58:05,490Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [] Trying to release > a shared lock for key: '9ac6f4ad-58d0-4a7e-b424-91f2d76abcacTEMPLATE' , > but lock does not exist > > 2019-10-08 09:58:05,490Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [] Trying to release > a shared lock for key: 'a7e42574-be60-4c94-94a5-cc4b30fdb16fDISK' , but > lock does not exist > > 2019-10-08 09:58:05,490Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [] Trying to release > exclusive lock which does not exist, lock key: 'centos-testvm-024VM_NAME' > > 2019-10-08 09:58:05,490Z INFO [org.ovirt.engine.core.bll.AddVmCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [] Lock freed to > object 'EngineLock:{exclusiveLocks='[centos-testvm-024=VM_NAME]', > sharedLocks='[9ac6f4ad-58d0-4a7e-b424-91f2d76abcac=TEMPLATE, > a7e42574-be60-4c94-94a5-cc4b30fdb16f=DISK]'}' > > 2019-10-08 09:58:05,503Z ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [] EVENT_ID: > USER_ADD_VM_FINISHED_FAILURE(60), Failed to complete VM > centos-testvm-024 creation. > > 2019-10-08 09:58:05,503Z ERROR > [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] > (EE-ManagedThreadFactory-engineScheduled-Thread-17) [] Failed invoking > callback end method 'onFailed' for command > '1b65f576-365d-4e8b-a920-621516cbdaed' with exception 'null', the > callback is marked for end method retries > > 2019-10-08 09:58:13,523Z ERROR [org.ovirt.engine.core.bll.AddVmCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Ending command > 'org.ovirt.engine.core.bll.AddVmCommand' with failure. > > 2019-10-08 09:58:13,527Z ERROR > [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Ending command > 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand' > with failure. > > 2019-10-08 09:58:13,533Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [56103b5a] Running > command: RemoveImageCommand internal: true. Entities affected : ID: > 935de3ac-a735-4fc8-9161-26bfd751ffc7 Type: Storage > > 2019-10-08 09:58:13,548Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [56103b5a] START, > DeleteImageGroupVDSCommand( > DeleteImageGroupVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='d680a46a-7188-45f6-b6a7-3333f830c0b4', postZeros='false', > discard='false', forceDelete='false'}), log id: 33164673 > > 2019-10-08 09:58:13,560Z ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [56103b5a] EVENT_ID: > IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS > failed: Image does not exist in domain: > u'image=d680a46a-7188-45f6-b6a7-3333f830c0b4, > domain=935de3ac-a735-4fc8-9161-26bfd751ffc7' > > 2019-10-08 09:58:13,560Z ERROR > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [56103b5a] Command > 'DeleteImageGroupVDSCommand( > DeleteImageGroupVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='d680a46a-7188-45f6-b6a7-3333f830c0b4', postZeros='false', > discard='false', forceDelete='false'})' execution failed: > IRSGenericException: IRSErrorException: Image does not exist in domain: > u'image=d680a46a-7188-45f6-b6a7-3333f830c0b4, > domain=935de3ac-a735-4fc8-9161-26bfd751ffc7' > > 2019-10-08 09:58:13,560Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [56103b5a] FINISH, > DeleteImageGroupVDSCommand, return: , log id: 33164673 > > 2019-10-08 09:58:13,560Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [56103b5a] Disk > 'd680a46a-7188-45f6-b6a7-3333f830c0b4' doesn't exist on storage domain > '935de3ac-a735-4fc8-9161-26bfd751ffc7', rolling forward > > 2019-10-08 09:58:13,570Z INFO > [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [56103b5a] Removed > task 'ce6882de-2d89-4298-bdcf-86d621a54084' from DataBase > > 2019-10-08 09:58:13,583Z INFO > [org.ovirt.engine.core.utils.transaction.TransactionSupport] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [] transaction > rolled back > > 2019-10-08 09:58:13,583Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [] Trying to release > a shared lock for key: '9ac6f4ad-58d0-4a7e-b424-91f2d76abcacTEMPLATE' , > but lock does not exist > > 2019-10-08 09:58:13,583Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [] Trying to release > a shared lock for key: 'a7e42574-be60-4c94-94a5-cc4b30fdb16fDISK' , but > lock does not exist > > 2019-10-08 09:58:13,583Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [] Trying to release > exclusive lock which does not exist, lock key: 'centos-testvm-024VM_NAME' > > 2019-10-08 09:58:13,583Z INFO [org.ovirt.engine.core.bll.AddVmCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [] Lock freed to > object 'EngineLock:{exclusiveLocks='[centos-testvm-024=VM_NAME]', > sharedLocks='[9ac6f4ad-58d0-4a7e-b424-91f2d76abcac=TEMPLATE, > a7e42574-be60-4c94-94a5-cc4b30fdb16f=DISK]'}' > > 2019-10-08 09:58:13,595Z ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [] EVENT_ID: > USER_ADD_VM_FINISHED_FAILURE(60), Failed to complete VM > centos-testvm-024 creation. > > 2019-10-08 09:58:13,595Z ERROR > [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] > (EE-ManagedThreadFactory-engineScheduled-Thread-35) [] Failed invoking > callback end method 'onFailed' for command > '1b65f576-365d-4e8b-a920-621516cbdaed' with exception 'null', the > callback is marked for end method retries > > 2019-10-08 09:58:23,616Z ERROR [org.ovirt.engine.core.bll.AddVmCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Ending command > 'org.ovirt.engine.core.bll.AddVmCommand' with failure. > > 2019-10-08 09:58:23,620Z ERROR > [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Ending command > 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand' > with failure. > > 2019-10-08 09:58:23,627Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [48f2f9bd] Running > command: RemoveImageCommand internal: true. Entities affected : ID: > 935de3ac-a735-4fc8-9161-26bfd751ffc7 Type: Storage > > 2019-10-08 09:58:23,642Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [48f2f9bd] START, > DeleteImageGroupVDSCommand( > DeleteImageGroupVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='d680a46a-7188-45f6-b6a7-3333f830c0b4', postZeros='false', > discard='false', forceDelete='false'}), log id: 7346a5bd > > 2019-10-08 09:58:23,663Z ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [48f2f9bd] EVENT_ID: > IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS > failed: Image does not exist in domain: > u'image=d680a46a-7188-45f6-b6a7-3333f830c0b4, > domain=935de3ac-a735-4fc8-9161-26bfd751ffc7' > > 2019-10-08 09:58:23,663Z ERROR > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [48f2f9bd] Command > 'DeleteImageGroupVDSCommand( > DeleteImageGroupVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='d680a46a-7188-45f6-b6a7-3333f830c0b4', postZeros='false', > discard='false', forceDelete='false'})' execution failed: > IRSGenericException: IRSErrorException: Image does not exist in domain: > u'image=d680a46a-7188-45f6-b6a7-3333f830c0b4, > domain=935de3ac-a735-4fc8-9161-26bfd751ffc7' > > 2019-10-08 09:58:23,663Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [48f2f9bd] FINISH, > DeleteImageGroupVDSCommand, return: , log id: 7346a5bd > > 2019-10-08 09:58:23,663Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [48f2f9bd] Disk > 'd680a46a-7188-45f6-b6a7-3333f830c0b4' doesn't exist on storage domain > '935de3ac-a735-4fc8-9161-26bfd751ffc7', rolling forward > > 2019-10-08 09:58:23,682Z INFO > [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [48f2f9bd] Removed > task '8483357d-6d4f-418e-9751-70fcb69685cb' from DataBase > > 2019-10-08 09:58:23,695Z INFO > [org.ovirt.engine.core.utils.transaction.TransactionSupport] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [] transaction > rolled back > > 2019-10-08 09:58:23,695Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [] Trying to release > a shared lock for key: '9ac6f4ad-58d0-4a7e-b424-91f2d76abcacTEMPLATE' , > but lock does not exist > > 2019-10-08 09:58:23,695Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [] Trying to release > a shared lock for key: 'a7e42574-be60-4c94-94a5-cc4b30fdb16fDISK' , but > lock does not exist > > 2019-10-08 09:58:23,695Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [] Trying to release > exclusive lock which does not exist, lock key: 'centos-testvm-024VM_NAME' > > 2019-10-08 09:58:23,695Z INFO [org.ovirt.engine.core.bll.AddVmCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [] Lock freed to > object 'EngineLock:{exclusiveLocks='[centos-testvm-024=VM_NAME]', > sharedLocks='[9ac6f4ad-58d0-4a7e-b424-91f2d76abcac=TEMPLATE, > a7e42574-be60-4c94-94a5-cc4b30fdb16f=DISK]'}' > > 2019-10-08 09:58:23,706Z ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [] EVENT_ID: > USER_ADD_VM_FINISHED_FAILURE(60), Failed to complete VM > centos-testvm-024 creation. > > 2019-10-08 09:58:23,706Z ERROR > [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] > (EE-ManagedThreadFactory-engineScheduled-Thread-71) [] Failed invoking > callback end method 'onFailed' for command > '1b65f576-365d-4e8b-a920-621516cbdaed' with exception 'null', the > callback is marked for end method retries > > 2019-10-08 09:58:33,725Z ERROR [org.ovirt.engine.core.bll.AddVmCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Ending command > 'org.ovirt.engine.core.bll.AddVmCommand' with failure. > > 2019-10-08 09:58:33,729Z ERROR > [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Ending command > 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand' > with failure. > > 2019-10-08 09:58:33,736Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) [3eea0406] Running > command: RemoveImageCommand internal: true. Entities affected : ID: > 935de3ac-a735-4fc8-9161-26bfd751ffc7 Type: Storage > > 2019-10-08 09:58:33,752Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) [3eea0406] START, > DeleteImageGroupVDSCommand( > DeleteImageGroupVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='d680a46a-7188-45f6-b6a7-3333f830c0b4', postZeros='false', > discard='false', forceDelete='false'}), log id: 6d120001 > > 2019-10-08 09:58:33,767Z ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) [3eea0406] EVENT_ID: > IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS > failed: Image does not exist in domain: > u'image=d680a46a-7188-45f6-b6a7-3333f830c0b4, > domain=935de3ac-a735-4fc8-9161-26bfd751ffc7' > > 2019-10-08 09:58:33,767Z ERROR > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) [3eea0406] Command > 'DeleteImageGroupVDSCommand( > DeleteImageGroupVDSCommandParameters:{storagePoolId='2e3b3484-c339-11e9-8d02-00163e3ec101', > ignoreFailoverLimit='false', > storageDomainId='935de3ac-a735-4fc8-9161-26bfd751ffc7', > imageGroupId='d680a46a-7188-45f6-b6a7-3333f830c0b4', postZeros='false', > discard='false', forceDelete='false'})' execution failed: > IRSGenericException: IRSErrorException: Image does not exist in domain: > u'image=d680a46a-7188-45f6-b6a7-3333f830c0b4, > domain=935de3ac-a735-4fc8-9161-26bfd751ffc7' > > 2019-10-08 09:58:33,767Z INFO > [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) [3eea0406] FINISH, > DeleteImageGroupVDSCommand, return: , log id: 6d120001 > > 2019-10-08 09:58:33,767Z INFO > [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) [3eea0406] Disk > 'd680a46a-7188-45f6-b6a7-3333f830c0b4' doesn't exist on storage domain > '935de3ac-a735-4fc8-9161-26bfd751ffc7', rolling forward > > 2019-10-08 09:58:33,778Z INFO > [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) [3eea0406] Removed > task 'bfb5e88a-0e62-4e03-b02b-a5a0b36f84db' from DataBase > > 2019-10-08 09:58:33,791Z INFO > [org.ovirt.engine.core.utils.transaction.TransactionSupport] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) [] transaction > rolled back > > 2019-10-08 09:58:33,791Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) [] Trying to release > a shared lock for key: '9ac6f4ad-58d0-4a7e-b424-91f2d76abcacTEMPLATE' , > but lock does not exist > > 2019-10-08 09:58:33,791Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) [] Trying to release > a shared lock for key: 'a7e42574-be60-4c94-94a5-cc4b30fdb16fDISK' , but > lock does not exist > > 2019-10-08 09:58:33,791Z WARN > [org.ovirt.engine.core.bll.lock.InMemoryLockManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) [] Trying to release > exclusive lock which does not exist, lock key: 'centos-testvm-024VM_NAME' > > 2019-10-08 09:58:33,791Z INFO [org.ovirt.engine.core.bll.AddVmCommand] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) [] Lock freed to > object 'EngineLock:{exclusiveLocks='[centos-testvm-024=VM_NAME]', > sharedLocks='[9ac6f4ad-58d0-4a7e-b424-91f2d76abcac=TEMPLATE, > a7e42574-be60-4c94-94a5-cc4b30fdb16f=DISK]'}' > > 2019-10-08 09:58:33,804Z ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) [] EVENT_ID: > USER_ADD_VM_FINISHED_FAILURE(60), Failed to complete VM > centos-testvm-024 creation. > > 2019-10-08 09:58:33,824Z ERROR > [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Failed invoking callback end > method 'onFailed' for command '1b65f576-365d-4e8b-a920-621516cbdaed' > with exception 'null', the callback is marked for end method retries but > max number of retries have been attempted. The command will be marked as > Failed. > > 2019-10-08 09:58:33,824Z INFO > [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Exception in invoking callback of > command AddVm (1b65f576-365d-4e8b-a920-621516cbdaed): NullPointerException: > > 2019-10-08 09:58:33,824Z ERROR > [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Error invoking callback method > 'onFailed' for 'FAILED' command '1b65f576-365d-4e8b-a920-621516cbdaed' > > 2019-10-08 09:58:33,824Z ERROR > [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] > (EE-ManagedThreadFactory-engineScheduled-Thread-63) > [067ec489-ae6c-4871-8ab6-8296016ca1ce] Exception: > javax.ejb.EJBTransactionRolledbackException > > at > org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:203) > [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] > > at > org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:420) [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] > > at > org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:148) > [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] > > at > org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) > > at > org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) > > at > org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81) > [weld-ejb-3.1.1.Final.jar:3.1.1.Final] > > at > org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) > > at > org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) > > at > org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] > > at > org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) > > at > org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) > [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] > > at > org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) > > at > org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) > [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] > > at > org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) > > at > org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) > [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] > > at > org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) > > at > org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] > > at > org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) > > at > org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) > [wildfly-ejb3-17.0.1.Final.jar:17.0.1.Final] > > at > org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) > > at > org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > > at > org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) > > at > org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) > > at > org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) > > at > org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) > > at > org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:618) > > at > org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) > > at > org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) > > at > org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) > > at > org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) > > at > org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) > > at > org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81) > > at > org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view3.endAction(Unknown > Source) [bll.jar:] > > at > sun.reflect.GeneratedMethodAccessor1032.invoke(Unknown Source) [:1.8.0_222] > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > [rt.jar:1.8.0_222] > > at java.lang.reflect.Method.invoke(Method.java:498) > [rt.jar:1.8.0_222] > > at > org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:410) > [weld-core-impl-3.1.1.Final.jar:3.1.1.Final] > > at > org.jboss.weld.module.ejb.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:134) > [weld-ejb-3.1.1.Final.jar:3.1.1.Final] > > at > org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) > [weld-core-impl-3.1.1.Final.jar:3.1.1.Final] > > at > org.jboss.weld.module.ejb.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPointPropagatingEnterpriseTargetBeanInstance.java:68) > [weld-ejb-3.1.1.Final.jar:3.1.1.Final] > > at > org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:106) > [weld-core-impl-3.1.1.Final.jar:3.1.1.Final] > > at > org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$BackendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.endAction(Unknown > Source) [bll.jar:] > > at > org.ovirt.engine.core.bll.VmCommand.endActionOnDisks(VmCommand.java:266) > [bll.jar:] > > at > org.ovirt.engine.core.bll.AddVmCommand.endWithFailure(AddVmCommand.java:1452) > [bll.jar:] > > at > org.ovirt.engine.core.bll.CommandBase.internalEndWithFailure(CommandBase.java:736) > [bll.jar:] > > at > org.ovirt.engine.core.bll.CommandBase.endActionInTransactionScope(CommandBase.java:658) > [bll.jar:] > > at > org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2003) > [bll.jar:] > > at > org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:202) > [utils.jar:] > > at > org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:137) > [utils.jar:] > > at > org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:105) > [utils.jar:] > > at > org.ovirt.engine.core.bll.CommandBase.endAction(CommandBase.java:520) > [bll.jar:] > > at > org.ovirt.engine.core.bll.ChildCommandsCallbackBase.endAction(ChildCommandsCallbackBase.java:134) > [bll.jar:] > > at > org.ovirt.engine.core.bll.ChildCommandsCallbackBase.onFailed(ChildCommandsCallbackBase.java:174) > [bll.jar:] > > at > org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.endCallback(CommandCallbacksPoller.java:67) > [bll.jar:] > > at > org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethodsImpl(CommandCallbacksPoller.java:166) > [bll.jar:] > > at > org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:109) > [bll.jar:] > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [rt.jar:1.8.0_222] > > at > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > [rt.jar:1.8.0_222] > > at > org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:383) > [javax.enterprise.concurrent-1.0.jar:] > > at > org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:534) > [javax.enterprise.concurrent-1.0.jar:] > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [rt.jar:1.8.0_222] > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [rt.jar:1.8.0_222] > > at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_222] > > at > org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) > [javax.enterprise.concurrent-1.0.jar:] > > Caused by: java.lang.NullPointerException > > *Ansible output:* > > The full traceback is: > > Traceback (most recent call last): > > File > "/var/folders/40/w2c8fp151854mddz_4n3czwm0000gn/T/ansible_ovirt_vm_payload_uy4lqway/__main__.py", > line 2024, in main > > clone_permissions=module.params['clone_permissions'], > > File > "/var/folders/40/w2c8fp151854mddz_4n3czwm0000gn/T/ansible_ovirt_vm_payload_uy4lqway/ansible_ovirt_vm_payload.zip/ansible/module_utils/ovirt.py", > line 610, in create > > poll_interval=self._module.params['poll_interval'], > > File > "/var/folders/40/w2c8fp151854mddz_4n3czwm0000gn/T/ansible_ovirt_vm_payload_uy4lqway/ansible_ovirt_vm_payload.zip/ansible/module_utils/ovirt.py", > line 341, in wait > > raise Exception("Timeout exceed while waiting on result state of > the entity.") > > Exception: Timeout exceed while waiting on result state of the entity. > > fatal: [centos-testvm-024.avinity.tv -> localhost]: FAILED! => { > > "changed": false, > > "invocation": { > > "module_args": { > > "affinity_group_mappings": [], > > "affinity_label_mappings": [], > > "allow_partial_import": null, > > "ballooning_enabled": null, > > "boot_devices": null, > > "boot_menu": null, > > "cd_iso": null, > > "clone": false, > > "clone_permissions": false, > > "cloud_init": { > > "authorized_ssh_keys": "ssh-rsa removed pubkey", > > "host_name": "centos-testvm-024.avinity.tv", > > "user_name": "centos" > > }, > > "cloud_init_nics": [], > > "cloud_init_persist": false, > > "cluster": "avshared1", > > "cluster_mappings": [], > > "comment": "oVirt Template Bakery Validation", > > "cpu_cores": null, > > "cpu_mode": null, > > "cpu_pinning": null, > > "cpu_shares": null, > > "cpu_sockets": null, > > "cpu_threads": null, > > "custom_compatibility_version": null, > > "custom_properties": null, > > "delete_protected": null, > > "description": null, > > "disk_format": "cow", > > "disks": [], > > "domain_mappings": [], > > "fetch_nested": false, > > "force": false, > > "graphical_console": null, > > "high_availability": null, > > "high_availability_priority": null, > > "host": null, > > "host_devices": null, > > "id": null, > > "initrd_path": null, > > "instance_type": "Medium", > > "io_threads": null, > > "kernel_params": null, > > "kernel_path": null, > > "kvm": null, > > "lease": null, > > "lun_mappings": [], > > "memory": null, > > "memory_guaranteed": null, > > "memory_max": null, > > "name": "centos-testvm-024", > > "nested_attributes": [], > > "nics": [ > > { > > "interface": "virtio", > > "name": "nic1", > > "profile_name": "tenant1" > > } > > ], > > "numa_nodes": [], > > "numa_tune_mode": null, > > "operating_system": null, > > "placement_policy": null, > > "poll_interval": 3, > > "quota_id": null, > > "reassign_bad_macs": null, > > "rng_device": null, > > "role_mappings": [], > > "serial_console": null, > > "serial_policy": null, > > "serial_policy_value": null, > > "smartcard_enabled": null, > > "soundcard_enabled": null, > > "sso": null, > > "state": "running", > > "stateless": null, > > "storage_domain": null, > > "sysprep": null, > > "template": "av-centos-75-baseimage_nfs9000", > > "template_version": null, > > "ticket": null, > > "timeout": 180, > > "timezone": null, > > "type": null, > > "usb_support": null, > > "use_latest_template_version": null, > > "vmware": null, > > "vnic_profile_mappings": [], > > "wait": true, > > "watchdog": null, > > "xen": null > > } > > }, > > "msg": "Timeout exceed while waiting on result state of the entity." > > } > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/EB5AXLUNU6LRV… >

2 11

How to tell a migrated engine it is not running as a hosted engine anymore?
by rens.verhage＠topicus.nl 21 Nov '19

21 Nov '19

I migrated a hosted ovirt-engine to a standalone machine. To do so, I created a backup with scope=all and imported this back-up into a newly installed engine. Everything was working fine. Not much later, I made a mistake and wanted to start over, so I decided to do an engine-cleanup and re-import my back-up. However, when I perform the engine-cleanup, I get this error message: [ ERROR ] It seems that you are running your engine inside of the hosted-engine VM and are not in "Global Maintenance" mode. In that case you should put the system into the "Global Maintenance" mode before running engine-setup, or the hosted-engine HA agent might kill the machine, which might corrupt your data. This is not true. The engine is now running as a standalone server. There must be a config parameter somewhere in a config file or in the database that tells the engine what mode it is running in, but I haven't found it yet. I can't find any documentation for this issue, all migration guides tell you how to migrate from standalone to hosted engine, not the other way around. How can I fix this?

1 0

[ANN] oVirt 4.3.7 is now generally available
by Sandro Bonazzola 21 Nov '19

21 Nov '19

The oVirt Project is pleased to announce the general availability of oVirt 4.3.7 as of November 21st, 2019. This update is the seventh in a series of stabilization updates to the 4.3 series. This release is available now on x86_64 architecture for: * Red Hat Enterprise Linux 7.7 or later (but < 8) * CentOS Linux (or similar) 7.7 or later (but < 8) This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: * Red Hat Enterprise Linux 7.7 or later (but < 8) * CentOS Linux (or similar) 7.7 or later (but < 8) * oVirt Node 4.3 (available for x86_64 only) See the release notes [1] for installation / upgrade instructions and a list of new features and bugs fixed. Notes: - oVirt Appliance is already available - oVirt Node is already available[2] oVirt Node and Appliance have been updated including: - oVirt 4.3.7: http://www.ovirt.org/release/4.3.7/ - Latest CentOS 7.7 updates including: - CEBA-2019:3077 CentOS 7 binutils BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-October/023463.html> - CEBA-2019:3057 CentOS 7 lvm2 BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-October/023475.html> - CEBA-2019:3081 CentOS 7 firewalld BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-October/023478.html> - CEBA-2019:3054 CentOS 7 hostname BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-October/023483.html> - CESA-2019:3128 Important CentOS 7 java-1.8.0-openjdk Security Update <https://lists.centos.org/pipermail/centos-announce/2019-October/023492.html> - (CESA not published yet, Important: kernel security update) https://access.redhat.com/errata/RHSA-2019:3889 - CEEA-2019:3846 CentOS 7 microcode_ctl Enhancement Update <https://lists.centos.org/pipermail/centos-announce/2019-November/023515.html> - CESA-2019:2964 Important CentOS 7 patch Security Update <https://lists.centos.org/pipermail/centos-announce/2019-October/023495.html> - CEBA-2019:3068 CentOS 7 procps-ng BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-October/023471.html> - CEBA-2019:3061 CentOS 7 rsyslog BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-October/023466.html> - CESA-2019:3197 Important CentOS 7 sudo Security Update <https://lists.centos.org/pipermail/centos-announce/2019-October/023499.html> - CEBA-2019:3072 CentOS 7 systemd BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-October/023460.html> - CEBA-2019:3056 CentOS 7 dnsmasq BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-October/023485.html> - CEBA-2019:3070 CentOS 7 ipa BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-October/023486.html> - CEBA-2019:3071 CentOS 7 libguestfs BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-October/023459.html> - CEBA-2019:3066 CentOS 7 sos BugFix Update <https://lists.centos.org/pipermail/centos-announce/2019-October/023461.html> - latest CentOS Virt and Storage SIG updates: - Ansible 2.9.1: https://github.com/ansible/ansible/blob/stable-2.9/changelogs/CHANGELOG-v2.… - Glusterfs 6.6: https://docs.gluster.org/en/latest/release-notes/6.6/ - QEMU KVM EV 2.12.0-33.1.el7_7.4 : https://cbs.centos.org/koji/buildinfo?buildID=27696 Given the amount of security fixes provided by this release, upgrade is recommended as soon as practical. Additional Resources: * Read more about the oVirt 4.3.7 release highlights: http://www.ovirt.org/release/4.3.7/ * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.3.7/ [2] http://resources.ovirt.org/pub/ovirt-4.3/iso/ -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/>*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 0

Can't start VM. An old snapshot is hold a vm state
by serphentos＠sharedlink.tk 21 Nov '19

21 Nov '19

Hello. I have a two vm's that can't start because a old snapshot (auto-generated then we migrate a disk to another storage) I tried to unlock entity and start vm or remove snapshot but still receive an errors.

1 0

port-mirror in Linux bridge
by p.staniforth＠leedsbeckett.ac.uk 20 Nov '19

20 Nov '19

Hello, Does anyone know how the port-mirror is implemented in oVirt? I am trying to investigate intermittent problems we are having with snort in oVirt VMs. Regards, Paul S.

1 0

Newbie Questions
by Dirk Streubel 20 Nov '19

20 Nov '19

Hello, i'am a new user and i have a few questions, maybe somebody can Help. It is possible to install the Server Management on a Raspberry PI 4 with 4GB. I find in the official Re Hat Document "Red Hat Virtualization Manager Hardware Requirements" that 4GB RAM is enough. And the PI has a 128 GB Flash Card. Would it work? Have somebody tested it? Another Question: Since yesterday i am trying to ad a host in my environment on ovirt Version 4.3.6. My Server Management runs under Fedora 31 under KVM/Virt-Manager without any problems. Adding a new Host doesn't work. I have tried to ad another VM with 8 Cores and 32 GB RAM as a host. This is not working, maybe VMs in VMs under ovirt is not possible, i don't know. So i try a physical Machine here as a Host. This and the other Installations fails with the Information: no route to Host. I found a bug and it says that the bug is closed. In the bug stand that maybe something wrong with the openvswitch. So i tried the following commands on the Server Management: ovn-nbctl show and ovn-sbctl show and the output was nothing. journalctl -xfe on the host that i want to migrate gave my the output: Nov 17 10:30:38 hypervisor1.linux.fritz.box sshd[12632]: Accepted keyboard-interactive/pam for root from 10.2.0.99 port 43260 ssh2 Nov 17 10:30:38 hypervisor1.linux.fritz.box systemd-logind[960]: New session 5 of user root. -- Subject: A new session 5 has been created for user root -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- Documentation: http://www.freedesktop.org/wiki/Software/systemd/multiseat -- -- A new session with the ID 5 has been created for the user root. -- -- The leading process of the session is 12632. Nov 17 10:30:38 hypervisor1.linux.fritz.box sshd[12632]: pam_unix(sshd:session): session opened for user root by (uid=0) Nov 17 10:30:38 hypervisor1.linux.fritz.box systemd[1]: Started Session 5 of user root. -- Subject: Unit session-5.scope has finished start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit session-5.scope has finished starting up. -- -- The start-up result is done. Nov 17 10:30:39 hypervisor1.linux.fritz.box sshd[12632]: pam_unix(sshd:session): session closed for user root Nov 17 10:30:39 hypervisor1.linux.fritz.box systemd-logind[960]: Removed session 5. -- Subject: Session 5 has been terminated -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- Documentation: http://www.freedesktop.org/wiki/Software/systemd/multiseat -- -- A session with the ID 5 has been terminated. Nov 17 10:30:39 hypervisor1.linux.fritz.box sshd[12654]: Accepted keyboard-interactive/pam for root from 10.2.0.99 port 43264 ssh2 Nov 17 10:30:39 hypervisor1.linux.fritz.box systemd-logind[960]: New session 6 of user root. -- Subject: A new session 6 has been created for user root -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- Documentation: http://www.freedesktop.org/wiki/Software/systemd/multiseat -- -- A new session with the ID 6 has been created for the user root. -- -- The leading process of the session is 12654. Nov 17 10:30:39 hypervisor1.linux.fritz.box systemd[1]: Started Session 6 of user root. -- Subject: Unit session-6.scope has finished start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit session-6.scope has finished starting up. -- -- The start-up result is done. Nov 17 10:30:39 hypervisor1.linux.fritz.box sshd[12654]: pam_unix(sshd:session): session opened for user root by (uid=0) Nov 17 10:30:54 hypervisor1.linux.fritz.box sshd[12654]: pam_unix(sshd:session): session closed for user root Nov 17 10:30:54 hypervisor1.linux.fritz.box systemd-logind[960]: Removed session 6. -- Subject: Session 6 has been terminated -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- Documentation: http://www.freedesktop.org/wiki/Software/systemd/multiseat -- -- A session with the ID 6 has been terminated. Maybe i mix some things, i don't know :( Regards Dirk

5 8

Re: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing
by Strahil 20 Nov '19

20 Nov '19

Hi Sahina, I have a strange situation: 1. When I try to access the file via 'sudo -u vdsm dd if=disk of=test bs=4M' the command fails on aprox 60MB. 2. If I run same command as root , remove the file and then run again via vdsm user -> this time no i/o error reported. My guess is that I need to check what's going on the bricks themselve ... Best Regards, Strahil Nikolov

1 0

Re: Warning: out of sync hosts ovirt-node3 ?
by Strahil 20 Nov '19

20 Nov '19

Go to Compute -> Hosts -> select the host and go to network tab. Click on it and check if it is 'out of sync'. There is a button (somewhere on the right ) to sync all networks. Best Regards, Strahil NikolovOn Nov 14, 2019 09:34, wangyu13476969128(a)126.com wrote: > > In the ovirt-engine web interface, compute-->clusters, there is a cluster with the following information: > Warning: out of sync hosts ovirt-node3 ? > > What is the cause ? How can I solve this problem ? > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QIKGY3HLQJXR5…

4 4

Newbie Thin Client question
by crege7＠gmail.com 20 Nov '19

20 Nov '19

I am new to this whole technology and I can't find anything about thin clients. I was wondering if I can even use oVirt with a thin client, if anyone knows how help would be much appreciated. I am also wondering if there is a way to use a Raspberry Pi as a thin client.

3 2

Gluster & Hyper Converged setup
by rob.downer＠orbitalsystems.co.uk 19 Nov '19

19 Nov '19

Hi, Gluster will not set up and fails... can anyone see why ? /etc/hosts set up for both backend Gluster network and front end, also LAN DNS set up on the subnet for the front end. TASK [gluster.infra/roles/backend_setup : Set PV data alignment for JBOD] ****** task path: /etc/ansible/roles/gluster.infra/roles/backend_setup/tasks/vg_create.yml:17 skipping: [gfs2.gluster.private] => {"changed": false, "skip_reason": "Conditional result was False"} skipping: [gfs1.gluster.private] => {"changed": false, "skip_reason": "Conditional result was False"} skipping: [gfs3.gluster.private] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [gluster.infra/roles/backend_setup : Set PV data alignment for RAID] ****** task path: /etc/ansible/roles/gluster.infra/roles/backend_setup/tasks/vg_create.yml:24 ok: [gfs2.gluster.private] => {"ansible_facts": {"pv_dataalign": "3072K\n"}, "changed": false} ok: [gfs1.gluster.private] => {"ansible_facts": {"pv_dataalign": "3072K\n"}, "changed": false} ok: [gfs3.gluster.private] => {"ansible_facts": {"pv_dataalign": "3072K\n"}, "changed": false} TASK [gluster.infra/roles/backend_setup : Set VG physical extent size for RAID] *** task path: /etc/ansible/roles/gluster.infra/roles/backend_setup/tasks/vg_create.yml:34 ok: [gfs2.gluster.private] => {"ansible_facts": {"vg_pesize": "3072K\n"}, "changed": false} ok: [gfs1.gluster.private] => {"ansible_facts": {"vg_pesize": "3072K\n"}, "changed": false} ok: [gfs3.gluster.private] => {"ansible_facts": {"vg_pesize": "3072K\n"}, "changed": false} TASK [gluster.infra/roles/backend_setup : Create volume groups] **************** task path: /etc/ansible/roles/gluster.infra/roles/backend_setup/tasks/vg_create.yml:46 failed: [gfs1.gluster.private] (item={u'vgname': u'gluster_vg_sdb', u'pvname': u'/dev/sdb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"pvname": "/dev/sdb", "vgname": "gluster_vg_sdb"}, "msg": "Device /dev/sdb not found."} failed: [gfs3.gluster.private] (item={u'vgname': u'gluster_vg_sdb', u'pvname': u'/dev/sdb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"pvname": "/dev/sdb", "vgname": "gluster_vg_sdb"}, "msg": "Device /dev/sdb not found."} failed: [gfs2.gluster.private] (item={u'vgname': u'gluster_vg_sdb', u'pvname': u'/dev/sdb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"pvname": "/dev/sdb", "vgname": "gluster_vg_sdb"}, "msg": "Device /dev/sdb not found."} NO MORE HOSTS LEFT ************************************************************* NO MORE HOSTS LEFT ************************************************************* PLAY RECAP ********************************************************************* gfs1.gluster.private : ok=10 changed=0 unreachable=0 failed=1 skipped=16 rescued=0 ignored=0 gfs2.gluster.private : ok=11 changed=1 unreachable=0 failed=1 skipped=16 rescued=0 ignored=0 gfs3.gluster.private : ok=10 changed=0 unreachable=0 failed=1 skipped=16 rescued=0 ignored=0 0 / 0 Reply

4 9

Wrong CPU?
by Christian Reiss 19 Nov '19

19 Nov '19

Hey folks, running an AMD Ryzen CPU here: processor : 0 vendor_id : AuthenticAMD cpu family : 23 model : 49 model name : AMD EPYC 7282 16-Core Processor However, libvirt is detecting this as EPYC-IBPB without the ssbd flags? <cpu> <arch>x86_64</arch> <model>EPYC-IBPB</model> <vendor>AMD</vendor> <microcode version='137367580'/> <counter name='tsc' frequency='2799999000'/> <topology sockets='1' cores='16' threads='2'/> <feature name='ht'/> <feature name='osxsave'/> <feature name='cmt'/> <feature name='clwb'/> <feature name='umip'/> <feature name='xsaves'/> <feature name='mbm_total'/> <feature name='mbm_local'/> <feature name='cmp_legacy'/> <feature name='extapic'/> <feature name='ibs'/> <feature name='skinit'/> <feature name='wdt'/> <feature name='tce'/> <feature name='topoext'/> <feature name='perfctr_core'/> <feature name='perfctr_nb'/> <feature name='invtsc'/> <feature name='wbnoinvd'/> <pages unit='KiB' size='4'/> <pages unit='KiB' size='2048'/> <pages unit='KiB' size='1048576'/> </cpu> [root@node01 ~]# grep ssbd /var/cache/libvirt/qemu/capabilities/*.xml <property name='ssbd' type='boolean' value='false'/> <property name='virt-ssbd' type='boolean' value='false'/> <property name='ssbd' type='boolean' value='false'/> <property name='virt-ssbd' type='boolean' value='false'/> But the flag is there: [root@node01 ~]# grep ssbd /proc/cpuinfo | tail -n1 flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl xtopology nonstop_tsc extd_apicid aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb cat_l3 cdp_l3 hw_pstate sme retpoline_amd ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip overflow_recov succor smca I tried adding "options kvm_amd avic=1" as well as "options kvm_amd avic=0" to /etc/modprobe.d/kvm.conf (always with reboots), adding mitigations=off to grub.. I can't think of any other solution. I just can't get the oVirt engine running with the ssbd flag. Seems cpu can do this, oVirt can do this, libvirt does not detect the cpu correctly or at least ignores it. But the hosted engine demands it. I am at a loss. Any help is oh-so-greatly appreciated. -Chris. -- Christian Reiss - email(a)christian-reiss.de /"\ ASCII Ribbon support(a)alpha-labs.net \ / Campaign X against HTML WEB alpha-labs.net / \ in eMails GPG Retrieval https://gpg.christian-reiss.de GPG ID ABCD43C5, 0x44E29126ABCD43C5 GPG fingerprint = 9549 F537 2596 86BA 733C A4ED 44E2 9126 ABCD 43C5 "It's better to reign in hell than to serve in heaven.", John Milton, Paradise lost.

3 6

[ANN] oVirt 4.3.7 Fourth Release Candidate is now available for testing
by Sandro Bonazzola 19 Nov '19

19 Nov '19

The oVirt Project is pleased to announce the availability of the oVirt 4.3.7 Fourth Release Candidate for testing, as of November 19th, 2019. This update is a release candidate of the seventh in a series of stabilization updates to the 4.3 series. This is pre-release software. This pre-release should not to be used in production. This release is available now on x86_64 architecture for: * Red Hat Enterprise Linux 7.7 or later (but <8) * CentOS Linux (or similar) 7.7 or later (but <8) This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: * Red Hat Enterprise Linux 7.7 or later (but <8) * CentOS Linux (or similar) 7.7 or later (but <8) * oVirt Node 4.3 (available for x86_64 only) has been built consuming CentOS 7.7 Release See the release notes [1] for known issues, new features and bugs fixed. While testing this release candidate please note that oVirt node now includes: - ansible 2.9.1 - GlusterFS 6.6 Notes: - oVirt Appliance is already available - oVirt Node is already available Additional Resources: * Read more about the oVirt 4.3.7 release highlights: http://www.ovirt.org/release/4.3.7/ * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.3.7/ [2] http://resources.ovirt.org/pub/ovirt-4.3-pre/iso/ -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/>*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 0

FOSDEM 2020 Virtualization & IaaS Devroom CfP
by Piotr Kliczewski 18 Nov '19

18 Nov '19

Friendly reminder that there are 2 weeks before the submission deadline. Room day update: This year Virt and IaaS room will be on the 2nd of February. See you all at FOSDEM!

1 0

Gluster setup
by rob.downer＠orbitalsystems.co.uk 18 Nov '19

18 Nov '19

I have set up a 3 node system. Gluster has its own backend network and I have tried entering the FQDN hosts via ssh as follows... gfs1.gluster.private 10.10.45.11 gfs2.gluster.private 10.10.45.12 gfs3.gluster.private 10.10.45.13 I entered at /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 gfs1.gluster.private 10.10.45.11 gfs2.gluster.private 10.10.45.12 gfs3.gluster.private 10.10.45.13 but on the CLI host gfs1.gluster.private returns [root@ovirt1 etc]# host gfs1.gluster.private Host gfs1.gluster.private not found: 3(NXDOMAIN) [root@ovirt1 etc]# I guess this is the wrong hosts file, resolver.conf lists files first for lookup...

4 9