[ovirt-users] oVIRT 4.0.6 / Pools creation issue

I'm running a cluster of oVIRT on 4.0.6 and we are using purely NFS storage for ISO/EXPORT/DATA domain. We are using a Tintri NFS appliance. What I can't seem to figure out is when I go to create a Pool from a template it fails. I have tried to export the template to export domain, then delete the template and re-import the template, however it fails with the logs below from the engine.log. It appears to give error message trying to create snapshot from Template? We use to be on a gluster volumes be we migrated everything off gluster. I don't see any mention of gluster errors below so hoping someone might have an idea? 2017-01-25 16:55:30,200 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler3) [6e36f82d] Setting new tasks map. The map contains now 0 tasks 2017-01-25 16:55:30,200 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler3) [6e36f82d] Cleared all tasks of pool '5824bb37-0258-026d-0107-00000000004d'. 2017-01-25 16:55:30,470 INFO [org.ovirt.engine.core.bll.AddVmPoolWithVmsCommand] (default task-40) [12f23e68] Lock Acquired to object 'EngineLock:{exclusiveLocks='[devins-pool=<VM_POOL_NAME, ACTION_TYPE_FAILED_VM_POOL_IS_BEING_CREATED$VmPoolName devins-pool>]', sharedLocks='null'}' 2017-01-25 16:55:30,551 INFO [org.ovirt.engine.core.bll.AddVmPoolWithVmsCommand] (org.ovirt.thread.pool-6-thread-18) [12f23e68] Running command: AddVmPoolWithVmsCommand internal: false. Entities affected : ID: 5824bb37-00b7-031b-0035-0000000002b7 Type: ClusterAction group CREATE_VM_POOL with role type USER, ID: bca9dcd9-f00f-494d-8d5a-f2903eb8632a Type: VmTemplateAction group CREATE_VM with role type USER 2017-01-25 16:55:30,552 INFO [org.ovirt.engine.core.bll.AddVmPoolWithVmsCommand] (org.ovirt.thread.pool-6-thread-18) [12f23e68] Lock freed to object 'EngineLock:{exclusiveLocks='[devins-pool=<VM_POOL_NAME, ACTION_TYPE_FAILED_VM_POOL_IS_BEING_CREATED$VmPoolName devins-pool>]', sharedLocks='null'}' 2017-01-25 16:55:30,679 INFO [org.ovirt.engine.core.bll.AddVmCommand] (org.ovirt.thread.pool-6-thread-18) [424815a6] Lock Acquired to object 'EngineLock:{exclusiveLocks='[devins-pool-1=<VM_NAME, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='[bca9dcd9-f00f-494d-8d5a-f2903eb8632a=<TEMPLATE, ACTION_TYPE_FAILED_TEMPLATE_IS_USED_FOR_CREATE_VM$VmName devins-pool-1>, 612217c0-87e0-41f6-b06d-edb99f177da8=<VM_POOL, ACTION_TYPE_FAILED_VM_POOL_IS_USED_FOR_CREATE_VM$VmName devins-pool-1>, 1cd672e8-9f78-403e-9d8c-c61c2047b672=<DISK, ACTION_TYPE_FAILED_DISK_IS_USED_FOR_CREATE_VM$VmName devins-pool-1>]'}' 2017-01-25 16:55:30,726 INFO [org.ovirt.engine.core.bll.AddVmCommand] (org.ovirt.thread.pool-6-thread-18) [] Running command: AddVmCommand internal: true. Entities affected : ID: 5824bb37-00b7-031b-0035-0000000002b7 Type: ClusterAction group CREATE_VM with role type USER, ID: bca9dcd9-f00f-494d-8d5a-f2903eb8632a Type: VmTemplateAction group CREATE_VM with role type USER, ID: f8d5f826-bd95-4229-9e87-9005b79f2448 Type: StorageAction group CREATE_DISK with role type USER 2017-01-25 16:55:30,808 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, SetVmStatusVDSCommand( SetVmStatusVDSCommandParameters:{runAsync='true', vmId='0d203249-b120-4d04-9b8e-21d874957592', status='ImageLocked', exitStatus='Normal'}), log id: 7a39ac9f 2017-01-25 16:55:30,813 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, SetVmStatusVDSCommand, log id: 7a39ac9f 2017-01-25 16:55:30,819 INFO [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] (org.ovirt.thread.pool-6-thread-18) [6a0aa969] Running command: CreateSnapshotFromTemplateCommand internal: true. Entities affected : ID: f8d5f826-bd95-4229-9e87-9005b79f2448 Type: Storage 2017-01-25 16:55:30,834 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSCommand] (org.ovirt.thread.pool-6-thread-18) [6a0aa969] START, CreateSnapshotVDSCommand( CreateSnapshotVDSCommandParameters:{runAsync='true', storagePoolId='5824bb37-0258-026d-0107-00000000004d', ignoreFailoverLimit='false', storageDomainId='f8d5f826-bd95-4229-9e87-9005b79f2448', imageGroupId='3e53272f-180e-4517-9dac-75813951a56b', imageSizeInBytes='75161927680', volumeFormat='COW', newImageId='a9f169f8-6005-42f7-8437-4c6271db016d', newImageDescription='', imageInitialSizeInBytes='0', imageId='150f3171-e992-473e-a96d-aac599c9e556', sourceImageGroupId='1cd672e8-9f78-403e-9d8c-c61c2047b672'}), log id: 2f0c9075 2017-01-25 16:55:30,835 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSCommand] (org.ovirt.thread.pool-6-thread-18) [6a0aa969] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID 2017-01-25 16:55:31,885 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSCommand] (org.ovirt.thread.pool-6-thread-18) [6a0aa969] FINISH, CreateSnapshotVDSCommand, return: a9f169f8-6005-42f7-8437-4c6271db016d, log id: 2f0c9075 2017-01-25 16:55:31,889 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-18) [6a0aa969] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 'f4a67fb3-e3ca-4fe1-ab8e-8240bd85f260' 2017-01-25 16:55:31,889 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-6-thread-18) [6a0aa969] CommandMultiAsyncTasks::attachTask: Attaching task 'a22c4e91-dac8-40e4-ba47-42807173ea91' to command 'f4a67fb3-e3ca-4fe1-ab8e-8240bd85f260'. 2017-01-25 16:55:31,901 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-6-thread-18) [6a0aa969] Adding task 'a22c4e91-dac8-40e4-ba47-42807173ea91' (Parent Command 'CreateSnapshotFromTemplate', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters'), polling hasn't started yet.. 2017-01-25 16:55:32,178 INFO [org.ovirt.engine.core.bll.AddGraphicsDeviceCommand] (org.ovirt.thread.pool-6-thread-18) [25fb544d] Running command: AddGraphicsDeviceCommand internal: true. Entities affected : ID: 0d203249-b120-4d04-9b8e-21d874957592 Type: VMAction group EDIT_VM_PROPERTIES with role type USER 2017-01-25 16:55:32,237 INFO [org.ovirt.engine.core.bll.AddVmToPoolCommand] (org.ovirt.thread.pool-6-thread-18) [f2d0c64] Running command: AddVmToPoolCommand internal: true. Entities affected : ID: 612217c0-87e0-41f6-b06d-edb99f177da8 Type: VmPool 2017-01-25 16:55:32,249 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [f2d0c64] Correlation ID: 424815a6, Job ID: c6b9ef03-c542-45be-b659-8946450b79c3, Call Stack: null, Custom Event ID: -1, Message: VM devins-pool-1 creation was initiated by devin.acosta(a)lxi.domain.com-authz. 2017-01-25 16:55:32,249 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-18) [f2d0c64] BaseAsyncTask::startPollingTask: Starting to poll task 'a22c4e91-dac8-40e4-ba47-42807173ea91'. 2017-01-25 16:55:32,368 INFO [org.ovirt.engine.core.bll.AddVmCommand] (org.ovirt.thread.pool-6-thread-18) [6504b822] Lock Acquired to object 'EngineLock:{exclusiveLocks='[devins-pool-2=<VM_NAME, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='[bca9dcd9-f00f-494d-8d5a-f2903eb8632a=<TEMPLATE, ACTION_TYPE_FAILED_TEMPLATE_IS_USED_FOR_CREATE_VM$VmName devins-pool-2>, 612217c0-87e0-41f6-b06d-edb99f177da8=<VM_POOL, ACTION_TYPE_FAILED_VM_POOL_IS_USED_FOR_CREATE_VM$VmName devins-pool-2>, 1cd672e8-9f78-403e-9d8c-c61c2047b672=<DISK, ACTION_TYPE_FAILED_DISK_IS_USED_FOR_CREATE_VM$VmName devins-pool-2>]'}' 2017-01-25 16:55:32,415 INFO [org.ovirt.engine.core.bll.AddVmCommand] (org.ovirt.thread.pool-6-thread-18) [] Running command: AddVmCommand internal: true. Entities affected : ID: 5824bb37-00b7-031b-0035-0000000002b7 Type: ClusterAction group CREATE_VM with role type USER, ID: bca9dcd9-f00f-494d-8d5a-f2903eb8632a Type: VmTemplateAction group CREATE_VM with role type USER, ID: f8d5f826-bd95-4229-9e87-9005b79f2448 Type: StorageAction group CREATE_DISK with role type USER 2017-01-25 16:55:32,459 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] START, SetVmStatusVDSCommand( SetVmStatusVDSCommandParameters:{runAsync='true', vmId='4631557a-4aa7-48da-bd0b-e3ba8191cf61', status='ImageLocked', exitStatus='Normal'}), log id: 68fb2cd8 2017-01-25 16:55:32,463 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (org.ovirt.thread.pool-6-thread-18) [] FINISH, SetVmStatusVDSCommand, log id: 68fb2cd8 2017-01-25 16:55:32,470 INFO [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] (org.ovirt.thread.pool-6-thread-18) [57bd66ee] Running command: CreateSnapshotFromTemplateCommand internal: true. Entities affected : ID: f8d5f826-bd95-4229-9e87-9005b79f2448 Type: Storage 2017-01-25 16:55:32,482 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSCommand] (org.ovirt.thread.pool-6-thread-18) [57bd66ee] START, CreateSnapshotVDSCommand( CreateSnapshotVDSCommandParameters:{runAsync='true', storagePoolId='5824bb37-0258-026d-0107-00000000004d', ignoreFailoverLimit='false', storageDomainId='f8d5f826-bd95-4229-9e87-9005b79f2448', imageGroupId='b604b900-83f9-47f8-bc7f-d2267095586f', imageSizeInBytes='75161927680', volumeFormat='COW', newImageId='c1f0485d-b2b9-4931-8eb1-6fc5b5f0a106', newImageDescription='', imageInitialSizeInBytes='0', imageId='150f3171-e992-473e-a96d-aac599c9e556', sourceImageGroupId='1cd672e8-9f78-403e-9d8c-c61c2047b672'}), log id: 46e84547 2017-01-25 16:55:32,482 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSCommand] (org.ovirt.thread.pool-6-thread-18) [57bd66ee] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID 2017-01-25 16:55:32,591 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSCommand] (org.ovirt.thread.pool-6-thread-18) [57bd66ee] FINISH, CreateSnapshotVDSCommand, return: c1f0485d-b2b9-4931-8eb1-6fc5b5f0a106, log id: 46e84547 2017-01-25 16:55:32,595 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-18) [57bd66ee] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command '74f8e242-92fe-4f3a-bf4a-a63b482f89c4' 2017-01-25 16:55:32,595 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-6-thread-18) [57bd66ee] CommandMultiAsyncTasks::attachTask: Attaching task 'dbad59d1-711b-4807-af30-250ff7cfd1cd' to command '74f8e242-92fe-4f3a-bf4a-a63b482f89c4'. 2017-01-25 16:55:32,605 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-6-thread-18) [57bd66ee] Adding task 'dbad59d1-711b-4807-af30-250ff7cfd1cd' (Parent Command 'CreateSnapshotFromTemplate', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters'), polling hasn't started yet.. 2017-01-25 16:55:32,694 INFO [org.ovirt.engine.core.bll.AddGraphicsDeviceCommand] (org.ovirt.thread.pool-6-thread-18) [5266def0] Running command: AddGraphicsDeviceCommand internal: true. Entities affected : ID: 4631557a-4aa7-48da-bd0b-e3ba8191cf61 Type: VMAction group EDIT_VM_PROPERTIES with role type USER 2017-01-25 16:55:32,703 INFO [org.ovirt.engine.core.bll.AddVmToPoolCommand] (org.ovirt.thread.pool-6-thread-18) [15776f98] Running command: AddVmToPoolCommand internal: true. Entities affected : ID: 612217c0-87e0-41f6-b06d-edb99f177da8 Type: VmPool 2017-01-25 16:55:32,716 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [15776f98] Correlation ID: 6504b822, Job ID: c6b9ef03-c542-45be-b659-8946450b79c3, Call Stack: null, Custom Event ID: -1, Message: VM devins-pool-2 creation was initiated by devin.acosta(a)lxi.domain.com-authz. 2017-01-25 16:55:32,716 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-18) [15776f98] BaseAsyncTask::startPollingTask: Starting to poll task 'dbad59d1-711b-4807-af30-250ff7cfd1cd'. 2017-01-25 16:55:32,731 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [15776f98] Correlation ID: 12f23e68, Job ID: c6b9ef03-c542-45be-b659-8946450b79c3, Call Stack: null, Custom Event ID: -1, Message: VM Pool devins-pool (containing 2 VMs) was created by devin.acosta(a)lxi.domain.com-authz. 2017-01-25 16:55:32,896 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler7) [424815a6] Command 'AddVmPoolWithVms' (id: '36264463-029e-4457-85c4-1c73f6cda00d') waiting on child command id: 'fbe6e9e5-aabf-48d8-a9c0-266b2f420eb3' type:'AddVm' to complete 2017-01-25 16:55:32,921 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler7) [57bd66ee] Command 'AddVm' (id: '10a5e5ba-6024-4e5b-a2be-34f107829e80') waiting on child command id: '74f8e242-92fe-4f3a-bf4a-a63b482f89c4' type:'CreateSnapshotFromTemplate' to complete 2017-01-25 16:55:32,941 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler7) [6a0aa969] Command 'AddVm' (id: 'fbe6e9e5-aabf-48d8-a9c0-266b2f420eb3') waiting on child command id: 'f4a67fb3-e3ca-4fe1-ab8e-8240bd85f260' type:'CreateSnapshotFromTemplate' to complete 2017-01-25 16:55:34,982 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler6) [57bd66ee] Command 'AddVm' (id: '10a5e5ba-6024-4e5b-a2be-34f107829e80') waiting on child command id: '74f8e242-92fe-4f3a-bf4a-a63b482f89c4' type:'CreateSnapshotFromTemplate' to complete 2017-01-25 16:55:37,058 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler10) [424815a6] Command 'AddVmPoolWithVms' (id: '36264463-029e-4457-85c4-1c73f6cda00d') waiting on child command id: 'fbe6e9e5-aabf-48d8-a9c0-266b2f420eb3' type:'AddVm' to complete 2017-01-25 16:55:37,086 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler10) [6a0aa969] Command 'AddVm' (id: 'fbe6e9e5-aabf-48d8-a9c0-266b2f420eb3') waiting on child command id: 'f4a67fb3-e3ca-4fe1-ab8e-8240bd85f260' type:'CreateSnapshotFromTemplate' to complete 2017-01-25 16:55:37,644 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler5) [457a14b4] Polling and updating Async Tasks: 2 tasks, 2 tasks to poll now 2017-01-25 16:55:38,584 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [457a14b4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM dev01-002-001 command failed: Cannot get parent volume 2017-01-25 16:55:38,588 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [457a14b4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM dev01-002-001 command failed: Cannot get parent volume 2017-01-25 16:55:38,588 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler5) [457a14b4] SPMAsyncTask::PollTask: Polling task 'a22c4e91-dac8-40e4-ba47-42807173ea91' (Parent Command 'CreateSnapshotFromTemplate', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned status 'finished', result 'cleanSuccess'. 2017-01-25 16:55:38,590 ERROR [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler5) [457a14b4] BaseAsyncTask::logEndTaskFailure: Task 'a22c4e91-dac8-40e4-ba47-42807173ea91' (Parent Command 'CreateSnapshotFromTemplate', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with failure: -- Result: 'cleanSuccess' -- Message: 'VDSGenericException: VDSErrorException: Failed in vdscommand to HSMGetAllTasksStatusesVDS, error = Cannot get parent volume', -- Exception: 'VDSGenericException: VDSErrorException: Failed in vdscommand to HSMGetAllTasksStatusesVDS, error = Cannot get parent volume' 2017-01-25 16:55:38,592 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler5) [457a14b4] CommandAsyncTask::endActionIfNecessary: All tasks of command 'f4a67fb3-e3ca-4fe1-ab8e-8240bd85f260' has ended -> executing 'endAction' 2017-01-25 16:55:38,592 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler5) [457a14b4] CommandAsyncTask::endAction: Ending action for '1' tasks (command ID: 'f4a67fb3-e3ca-4fe1-ab8e-8240bd85f260'): calling endAction '. 2017-01-25 16:55:38,592 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler5) [457a14b4] SPMAsyncTask::PollTask: Polling task 'dbad59d1-711b-4807-af30-250ff7cfd1cd' (Parent Command 'CreateSnapshotFromTemplate', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned status 'finished', result 'cleanSuccess'. 2017-01-25 16:55:38,592 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-3) [457a14b4] CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction 'CreateSnapshotFromTemplate', executionIndex: '0' 2017-01-25 16:55:38,594 ERROR [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler5) [457a14b4] BaseAsyncTask::logEndTaskFailure: Task 'dbad59d1-711b-4807-af30-250ff7cfd1cd' (Parent Command 'CreateSnapshotFromTemplate', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with failure: -- Result: 'cleanSuccess' -- Message: 'VDSGenericException: VDSErrorException: Failed in vdscommand to HSMGetAllTasksStatusesVDS, error = Cannot get parent volume', -- Exception: 'VDSGenericException: VDSErrorException: Failed in vdscommand to HSMGetAllTasksStatusesVDS, error = Cannot get parent volume' 2017-01-25 16:55:38,597 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler5) [457a14b4] CommandAsyncTask::endActionIfNecessary: All tasks of command '74f8e242-92fe-4f3a-bf4a-a63b482f89c4' has ended -> executing 'endAction' 2017-01-25 16:55:38,597 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler5) [457a14b4] CommandAsyncTask::endAction: Ending action for '1' tasks (command ID: '74f8e242-92fe-4f3a-bf4a-a63b482f89c4'): calling endAction '. 2017-01-25 16:55:38,597 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-39) [457a14b4] CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction 'CreateSnapshotFromTemplate', executionIndex: '0' 2017-01-25 16:55:38,600 INFO [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] (org.ovirt.thread.pool-6-thread-3) [6a0aa969] Command [id=f4a67fb3-e3ca-4fe1-ab8e-8240bd85f260]: Updating status to 'FAILED', The command end method logic will be executed by one of its parent commands. 2017-01-25 16:55:38,600 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-3) [6a0aa969] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'CreateSnapshotFromTemplate' completed, handling the result. 2017-01-25 16:55:38,601 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-3) [6a0aa969] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'CreateSnapshotFromTemplate' succeeded, clearing tasks. 2017-01-25 16:55:38,601 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-3) [6a0aa969] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'a22c4e91-dac8-40e4-ba47-42807173ea91' 2017-01-25 16:55:38,601 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-3) [6a0aa969] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{runAsync='true', storagePoolId='5824bb37-0258-026d-0107-00000000004d', ignoreFailoverLimit='false', taskId='a22c4e91-dac8-40e4-ba47-42807173ea91'}), log id: 4cd9d13e 2017-01-25 16:55:38,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-3) [6a0aa969] START, HSMClearTaskVDSCommand(HostName = dev01-002-001, HSMTaskGuidBaseVDSCommandParameters:{runAsync='true', hostId='fb1dbd7b-c059-44f6-8da8-e1db4540d91c', taskId='a22c4e91-dac8-40e4-ba47-42807173ea91'}), log id: 61f4ae2a 2017-01-25 16:55:38,604 INFO [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] (org.ovirt.thread.pool-6-thread-39) [57bd66ee] Command [id=74f8e242-92fe-4f3a-bf4a-a63b482f89c4]: Updating status to 'FAILED', The command end method logic will be executed by one of its parent commands. 2017-01-25 16:55:38,604 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-39) [57bd66ee] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'CreateSnapshotFromTemplate' completed, handling the result. 2017-01-25 16:55:38,604 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-39) [57bd66ee] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'CreateSnapshotFromTemplate' succeeded, clearing tasks. 2017-01-25 16:55:38,604 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-39) [57bd66ee] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'dbad59d1-711b-4807-af30-250ff7cfd1cd' 2017-01-25 16:55:38,605 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-39) [57bd66ee] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{runAsync='true', storagePoolId='5824bb37-0258-026d-0107-00000000004d', ignoreFailoverLimit='false', taskId='dbad59d1-711b-4807-af30-250ff7cfd1cd'}), log id: 2d04146e 2017-01-25 16:55:38,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-3) [6a0aa969] FINISH, HSMClearTaskVDSCommand, log id: 61f4ae2a 2017-01-25 16:55:38,619 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-3) [6a0aa969] FINISH, SPMClearTaskVDSCommand, log id: 4cd9d13e 2017-01-25 16:55:38,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-39) [57bd66ee] START, HSMClearTaskVDSCommand(HostName = dev01-002-001, HSMTaskGuidBaseVDSCommandParameters:{runAsync='true', hostId='fb1dbd7b-c059-44f6-8da8-e1db4540d91c', taskId='dbad59d1-711b-4807-af30-250ff7cfd1cd'}), log id: 71c703c8 2017-01-25 16:55:38,622 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-3) [6a0aa969] BaseAsyncTask::removeTaskFromDB: Removed task 'a22c4e91-dac8-40e4-ba47-42807173ea91' from DataBase 2017-01-25 16:55:38,622 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-3) [6a0aa969] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 'f4a67fb3-e3ca-4fe1-ab8e-8240bd85f260' 2017-01-25 16:55:39,130 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler4) [57bd66ee] Command 'AddVm' id: '10a5e5ba-6024-4e5b-a2be-34f107829e80' child commands '[74f8e242-92fe-4f3a-bf4a-a63b482f89c4]' executions were completed, status 'FAILED' 2017-01-25 16:55:39,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-39) [57bd66ee] FINISH, HSMClearTaskVDSCommand, log id: 71c703c8 2017-01-25 16:55:39,438 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-39) [57bd66ee] FINISH, SPMClearTaskVDSCommand, log id: 2d04146e 2017-01-25 16:55:39,441 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-39) [57bd66ee] BaseAsyncTask::removeTaskFromDB: Removed task 'dbad59d1-711b-4807-af30-250ff7cfd1cd' from DataBase 2017-01-25 16:55:39,442 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-39) [57bd66ee] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity '74f8e242-92fe-4f3a-bf4a-a63b482f89c4' 2017-01-25 16:55:40,179 ERROR [org.ovirt.engine.core.bll.AddVmCommand] (DefaultQuartzScheduler9) [57bd66ee] Ending command 'org.ovirt.engine.core.bll.AddVmCommand' with failure. 2017-01-25 16:55:40,185 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] (DefaultQuartzScheduler9) [57bd66ee] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand' with failure. 2017-01-25 16:55:40,308 INFO [org.ovirt.engine.core.bll.AddVmCommand] (DefaultQuartzScheduler9) [] Lock freed to object 'EngineLock:{exclusiveLocks='[devins-pool-2=<VM_NAME, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='[bca9dcd9-f00f-494d-8d5a-f2903eb8632a=<TEMPLATE, ACTION_TYPE_FAILED_TEMPLATE_IS_USED_FOR_CREATE_VM$VmName devins-pool-2>, 612217c0-87e0-41f6-b06d-edb99f177da8=<VM_POOL, ACTION_TYPE_FAILED_VM_POOL_IS_USED_FOR_CREATE_VM$VmName devins-pool-2>, 1cd672e8-9f78-403e-9d8c-c61c2047b672=<DISK, ACTION_TYPE_FAILED_DISK_IS_USED_FOR_CREATE_VM$VmName devins-pool-2>]'}' 2017-01-25 16:55:40,319 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [] Correlation ID: 6504b822, Job ID: c6b9ef03-c542-45be-b659-8946450b79c3, Call Stack: null, Custom Event ID: -1, Message: Failed to complete VM devins-pool-2 creation. 2017-01-25 16:55:41,390 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler10) [424815a6] Command 'AddVmPoolWithVms' (id: '36264463-029e-4457-85c4-1c73f6cda00d') waiting on child command id: 'fbe6e9e5-aabf-48d8-a9c0-266b2f420eb3' type:'AddVm' to complete 2017-01-25 16:55:43,461 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler6) [424815a6] Command 'AddVmPoolWithVms' (id: '36264463-029e-4457-85c4-1c73f6cda00d') waiting on child command id: 'fbe6e9e5-aabf-48d8-a9c0-266b2f420eb3' type:'AddVm' to complete 2017-01-25 16:55:45,503 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler2) [6a0aa969] Command 'AddVm' id: 'fbe6e9e5-aabf-48d8-a9c0-266b2f420eb3' child commands '[f4a67fb3-e3ca-4fe1-ab8e-8240bd85f260]' executions were completed, status 'FAILED' 2017-01-25 16:55:46,551 ERROR [org.ovirt.engine.core.bll.AddVmCommand] (DefaultQuartzScheduler9) [6a0aa969] Ending command 'org.ovirt.engine.core.bll.AddVmCommand' with failure. 2017-01-25 16:55:46,554 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand] (DefaultQuartzScheduler9) [6a0aa969] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotFromTemplateCommand' with failure. 2017-01-25 16:55:46,643 INFO [org.ovirt.engine.core.bll.AddVmCommand] (DefaultQuartzScheduler9) [] Lock freed to object 'EngineLock:{exclusiveLocks='[devins-pool-1=<VM_NAME, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='[bca9dcd9-f00f-494d-8d5a-f2903eb8632a=<TEMPLATE, ACTION_TYPE_FAILED_TEMPLATE_IS_USED_FOR_CREATE_VM$VmName devins-pool-1>, 612217c0-87e0-41f6-b06d-edb99f177da8=<VM_POOL, ACTION_TYPE_FAILED_VM_POOL_IS_USED_FOR_CREATE_VM$VmName devins-pool-1>, 1cd672e8-9f78-403e-9d8c-c61c2047b672=<DISK, ACTION_TYPE_FAILED_DISK_IS_USED_FOR_CREATE_VM$VmName devins-pool-1>]'}' 2017-01-25 16:55:46,654 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [] Correlation ID: 424815a6, Job ID: c6b9ef03-c542-45be-b659-8946450b79c3, Call Stack: null, Custom Event ID: -1, Message: Failed to complete VM devins-pool-1 creation. 2017-01-25 16:55:47,781 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler2) [6504b822] Command 'AddVmPoolWithVms' id: '36264463-029e-4457-85c4-1c73f6cda00d' child commands '[fbe6e9e5-aabf-48d8-a9c0-266b2f420eb3, 10a5e5ba-6024-4e5b-a2be-34f107829e80]' executions were completed, status 'FAILED' 2017-01-25 16:55:48,881 ERROR [org.ovirt.engine.core.bll.AddVmPoolWithVmsCommand] (DefaultQuartzScheduler4) [6504b822] Ending command 'org.ovirt.engine.core.bll.AddVmPoolWithVmsCommand' with failure. 2017-01-25 16:55:48,888 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6504b822] Correlation ID: 12f23e68, Job ID: c6b9ef03-c542-45be-b659-8946450b79c3, Call Stack: null, Custom Event ID: -1, Message: VM Pool devins-pool (containing 2 VMs) was created by devin.acosta(a)lxi.domain.com-authz. 2017-01-25 16:56:02,640 INFO [org.ovirt.engine.core.sso.servlets.OAuthRevokeServlet] (default task-58) [] User devin.acosta(a)lxi.domain.com successfully logged out 2017-01-25 16:56:02,653 INFO [org.ovirt.engine.core.bll.aaa.TerminateSessionsForTokenCommand] (default task-61) [343a5bd4] Running command: TerminateSessionsForTokenCommand internal: true. 2017-01-25 16:56:42,271 INFO [org.ovirt.engine.core.bll.RemoveVmPoolCommand] (default task-25) [2d22ba57] Lock Acquired to object 'EngineLock:{exclusiveLocks='[612217c0-87e0-41f6-b06d-edb99f177da8=<VM_POOL, ACTION_TYPE_FAILED_VM_POOL_IS_BEING_REMOVED$VmPoolName devins-pool>]', sharedLocks='null'}' 2017-01-25 16:56:42,296 INFO [org.ovirt.engine.core.bll.RemoveVmPoolCommand] (org.ovirt.thread.pool-6-thread-5) [2d22ba57] Running command: RemoveVmPoolCommand internal: false. Entities affected : ID: 612217c0-87e0-41f6-b06d-edb99f177da8 Type: VmPoolAction group DELETE_VM_POOL with role type USER 2017-01-25 16:56:42,301 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [2d22ba57] Correlation ID: 2d22ba57, Job ID: d1b36387-a69d-425a-88ab-fc302ac20208, Call Stack: null, Custom Event ID: -1, Message: VM Pool devins-pool removal was initiated by devin.acosta(a)lxi.domain.com-authz. 2017-01-25 16:56:42,307 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [2d22ba57] Correlation ID: 2d22ba57, Job ID: d1b36387-a69d-425a-88ab-fc302ac20208, Call Stack: null, Custom Event ID: -1, Message: VM Pool devins-pool removal was initiated by devin.acosta(a)lxi.domain.com-authz. 2017-01-25 16:56:42,313 INFO [org.ovirt.engine.core.bll.RemoveVmPoolCommand] (org.ovirt.thread.pool-6-thread-5) [2d22ba57] Lock freed to object 'EngineLock:{exclusiveLocks='[612217c0-87e0-41f6-b06d-edb99f177da8=<VM_POOL, ACTION_TYPE_FAILED_VM_POOL_IS_BEING_REMOVED$VmPoolName devins-pool>]', sharedLocks='null'}' 2017-01-25 16:56:42,918 INFO [org.ovirt.engine.core.bll.RemoveVmPoolCommandCallback] (DefaultQuartzScheduler9) [] Command 'RemoveVmPool' id: '00ba7c5d-a30d-4ab9-977e-14c139305461' child commands '[]' executions were completed, status 'SUCCEEDED' -- Devin Acosta Red Hat Certified Architect, LinuxStack devin(a)linuxguru.co