2017-11-14 09:43:58,824-06 INFO [org.ovirt.engine.core.bll.storage.disk.MoveDisksCommand] (default task-61) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Running command: MoveDisksCommand internal: false. Entities affected : ID: 57b69fdf-93dd-444c-977b-8803fa83507b Type: DiskAction group CONFIGURE_DISK_STORAGE with role type USER 2017-11-14 09:43:58,911-06 INFO [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateVmDisksCommand] (default task-61) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[57b69fdf-93dd-444c-977b-8803fa83507b=DISK]', sharedLocks='[7c48c7f7-31d4-4627-a26f-6a239bf92f21=VM]'}' 2017-11-14 09:43:59,061-06 INFO [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateVmDisksCommand] (org.ovirt.thread.pool-6-thread-22) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Running command: LiveMigrateVmDisksCommand internal: false. Entities affected : ID: 57b69fdf-93dd-444c-977b-8803fa83507b Type: DiskAction group DISK_LIVE_STORAGE_MIGRATION with role type USER 2017-11-14 09:43:59,136-06 INFO [org.ovirt.engine.core.bll.snapshots.CreateAllSnapshotsFromVmCommand] (org.ovirt.thread.pool-6-thread-22) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Running command: CreateAllSnapshotsFromVmCommand internal: true. Entities affected : ID: 7c48c7f7-31d4-4627-a26f-6a239bf92f21 Type: VMAction group MANIPULATE_VM_SNAPSHOTS with role type USER 2017-11-14 09:43:59,162-06 INFO [org.ovirt.engine.core.bll.snapshots.CreateSnapshotCommand] (org.ovirt.thread.pool-6-thread-22) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Running command: CreateSnapshotCommand internal: true. Entities affected : ID: 00000000-0000-0000-0000-000000000000 Type: Storage 2017-11-14 09:43:59,187-06 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSCommand] (org.ovirt.thread.pool-6-thread-22) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] START, CreateSnapshotVDSCommand( CreateSnapshotVDSCommandParameters:{runAsync='true', storagePoolId='02abeab8-fac8-41a9-bc2d-7fb0eaac0e07', ignoreFailoverLimit='false', storageDomainId='0e201616-a896-4c4a-90e6-791933ea5393', imageGroupId='57b69fdf-93dd-444c-977b-8803fa83507b', imageSizeInBytes='107374182400', volumeFormat='COW', newImageId='0028c34f-1117-4b06-b9d9-83be9bf263cf', imageType='Sparse', newImageDescription='', imageInitialSizeInBytes='0', imageId='d2c2e7a9-ae5b-4481-9d8c-185ab5cd8c14', sourceImageGroupId='57b69fdf-93dd-444c-977b-8803fa83507b'}), log id: 7d8623d3 2017-11-14 09:43:59,187-06 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSCommand] (org.ovirt.thread.pool-6-thread-22) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] -- executeIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID 2017-11-14 09:44:00,274-06 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateSnapshotVDSCommand] (org.ovirt.thread.pool-6-thread-22) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] FINISH, CreateSnapshotVDSCommand, return: 0028c34f-1117-4b06-b9d9-83be9bf263cf, log id: 7d8623d3 2017-11-14 09:44:00,279-06 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-22) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 'edece02c-623b-482e-a5fa-7caad2d15a9f' 2017-11-14 09:44:00,280-06 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (org.ovirt.thread.pool-6-thread-22) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] CommandMultiAsyncTasks::attachTask: Attaching task 'ef8cc0ca-ef62-4a92-9564-457d0fa82584' to command 'edece02c-623b-482e-a5fa-7caad2d15a9f'. 2017-11-14 09:44:00,298-06 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-6-thread-22) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Adding task 'ef8cc0ca-ef62-4a92-9564-457d0fa82584' (Parent Command 'CreateSnapshot', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters'), polling hasn't started yet.. 2017-11-14 09:44:00,382-06 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] EVENT_ID: USER_CREATE_SNAPSHOT(45), Correlation ID: bae5243a-a7a1-4f13-8d5e-04132f98b35d, Job ID: b2c0c8f8-57d1-4012-ab4c-e0b7544036a7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Snapshot 'Auto-generated for Live Storage Migration' creation for VM 'ansible.altn.int' was initiated by admin@internal-authz. 2017-11-14 09:44:00,383-06 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-22) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] BaseAsyncTask::startPollingTask: Starting to poll task 'ef8cc0ca-ef62-4a92-9564-457d0fa82584'. 2017-11-14 09:44:01,622-06 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (DefaultQuartzScheduler4) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Command 'LiveMigrateVmDisks' (id: '086e6603-498c-44de-9ac4-8d9b1bbb71f2') waiting on child command id: 'db4fda08-af04-4526-a42f-c492fe3ed79c' type:'CreateAllSnapshotsFromVm' to complete 2017-11-14 09:44:01,632-06 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler4) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Command 'CreateAllSnapshotsFromVm' (id: 'db4fda08-af04-4526-a42f-c492fe3ed79c') waiting on child command id: 'edece02c-623b-482e-a5fa-7caad2d15a9f' type:'CreateSnapshot' to complete 2017-11-14 09:44:02,345-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler5) [5025e5a3] START, GlusterServersListVDSCommand(HostName = vm-idev-colo-1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='c05df7cb-fee4-43e5-b132-769f8743bd6e'}), log id: 1304e7a5 2017-11-14 09:44:03,575-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler5) [5025e5a3] FINISH, GlusterServersListVDSCommand, return: [10.20.101.185/24:CONNECTED, vm-host-colo-2-gluster.altn.int:CONNECTED, vm-host-colo-1-gluster.altn.int:CONNECTED], log id: 1304e7a5 2017-11-14 09:44:03,591-06 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler5) [5025e5a3] probe could not be done for server 'vm-idev-colo-1' as no alternate UP server found 2017-11-14 09:44:03,592-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler5) [5025e5a3] START, GlusterVolumesListVDSCommand(HostName = vm-idev-colo-1, GlusterVolumesListVDSParameters:{runAsync='true', hostId='c05df7cb-fee4-43e5-b132-769f8743bd6e'}), log id: 2b699fee 2017-11-14 09:44:04,180-06 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler8) [648c9499] Polling and updating Async Tasks: 2 tasks, 1 tasks to poll now 2017-11-14 09:44:04,189-06 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler5) [5025e5a3] Could not associate brick 'vm-host-colo-2-gluster.altn.int:/gluster/gl-vm12/brick1' of volume '65d7c61e-ea64-4ebf-a464-5639713c37ba' with correct network as no gluster network found in cluster '99b72a34-a21d-4930-88fc-331998a90417' 2017-11-14 09:44:04,192-06 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler5) [5025e5a3] Could not associate brick 'vm-idev-colo-1-gluster.altn.int:/gluster/gl-vm12' of volume '65d7c61e-ea64-4ebf-a464-5639713c37ba' with correct network as no gluster network found in cluster '99b72a34-a21d-4930-88fc-331998a90417' 2017-11-14 09:44:04,192-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler5) [5025e5a3] FINISH, GlusterVolumesListVDSCommand, return: {65d7c61e-ea64-4ebf-a464-5639713c37ba=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@667e9b07}, log id: 2b699fee 2017-11-14 09:44:04,248-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler5) [5025e5a3] START, GlusterServersListVDSCommand(HostName = vm-host-colo-2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='4c62f4ab-79fe-40e0-9ade-1d5403f2c659'}), log id: 5cef75a9 2017-11-14 09:44:04,249-06 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler8) [648c9499] SPMAsyncTask::PollTask: Polling task 'ef8cc0ca-ef62-4a92-9564-457d0fa82584' (Parent Command 'CreateSnapshot', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned status 'finished', result 'success'. 2017-11-14 09:44:04,254-06 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler8) [648c9499] BaseAsyncTask::onTaskEndSuccess: Task 'ef8cc0ca-ef62-4a92-9564-457d0fa82584' (Parent Command 'CreateSnapshot', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended successfully. 2017-11-14 09:44:04,254-06 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler8) [648c9499] CommandAsyncTask::endActionIfNecessary: All tasks of command 'edece02c-623b-482e-a5fa-7caad2d15a9f' has ended -> executing 'endAction' 2017-11-14 09:44:04,254-06 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler8) [648c9499] CommandAsyncTask::endAction: Ending action for '1' tasks (command ID: 'edece02c-623b-482e-a5fa-7caad2d15a9f'): calling endAction '. 2017-11-14 09:44:04,255-06 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-25) [648c9499] CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction 'CreateSnapshot', 2017-11-14 09:44:04,261-06 INFO [org.ovirt.engine.core.bll.snapshots.CreateSnapshotCommand] (org.ovirt.thread.pool-6-thread-25) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Command [id=edece02c-623b-482e-a5fa-7caad2d15a9f]: Updating status to 'SUCCEEDED', The command end method logic will be executed by one of its parent commands. 2017-11-14 09:44:04,261-06 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-25) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'CreateSnapshot' completed, handling the result. 2017-11-14 09:44:04,261-06 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-25) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'CreateSnapshot' succeeded, clearing tasks. 2017-11-14 09:44:04,261-06 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-25) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'ef8cc0ca-ef62-4a92-9564-457d0fa82584' 2017-11-14 09:44:04,261-06 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-25) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{runAsync='true', storagePoolId='02abeab8-fac8-41a9-bc2d-7fb0eaac0e07', ignoreFailoverLimit='false', taskId='ef8cc0ca-ef62-4a92-9564-457d0fa82584'}), log id: 5e94f136 2017-11-14 09:44:04,262-06 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-25) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] START, HSMClearTaskVDSCommand(HostName = vm-host-colo-2, HSMTaskGuidBaseVDSCommandParameters:{runAsync='true', hostId='4c62f4ab-79fe-40e0-9ade-1d5403f2c659', taskId='ef8cc0ca-ef62-4a92-9564-457d0fa82584'}), log id: 5724cae4 2017-11-14 09:44:04,633-06 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-25) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] FINISH, HSMClearTaskVDSCommand, log id: 5724cae4 2017-11-14 09:44:04,633-06 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-25) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] FINISH, SPMClearTaskVDSCommand, log id: 5e94f136 2017-11-14 09:44:04,637-06 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-25) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] BaseAsyncTask::removeTaskFromDB: Removed task 'ef8cc0ca-ef62-4a92-9564-457d0fa82584' from DataBase 2017-11-14 09:44:04,637-06 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-25) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 'edece02c-623b-482e-a5fa-7caad2d15a9f' 2017-11-14 09:44:05,643-06 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (DefaultQuartzScheduler4) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Command 'LiveMigrateVmDisks' (id: '086e6603-498c-44de-9ac4-8d9b1bbb71f2') waiting on child command id: 'db4fda08-af04-4526-a42f-c492fe3ed79c' type:'CreateAllSnapshotsFromVm' to complete 2017-11-14 09:44:05,653-06 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler4) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Command 'CreateAllSnapshotsFromVm' id: 'db4fda08-af04-4526-a42f-c492fe3ed79c' child commands '[edece02c-623b-482e-a5fa-7caad2d15a9f]' executions were completed, status 'SUCCEEDED' 2017-11-14 09:44:06,165-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler5) [5025e5a3] FINISH, GlusterServersListVDSCommand, return: [10.20.101.183/24:CONNECTED, vm-host-colo-1-gluster.altn.int:CONNECTED, vm-idev-colo-1-gluster.altn.int:CONNECTED], log id: 5cef75a9 2017-11-14 09:44:06,173-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler5) [5025e5a3] START, GlusterVolumesListVDSCommand(HostName = vm-host-colo-2, GlusterVolumesListVDSParameters:{runAsync='true', hostId='4c62f4ab-79fe-40e0-9ade-1d5403f2c659'}), log id: 5a1fa8fe 2017-11-14 09:44:06,405-06 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler5) [5025e5a3] Could not associate brick 'vm-host-colo-2-gluster.altn.int:/gluster/gl-vm12/brick1' of volume '65d7c61e-ea64-4ebf-a464-5639713c37ba' with correct network as no gluster network found in cluster '3f5f7109-420e-4bfa-a933-9b69970851dc' 2017-11-14 09:44:06,409-06 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler5) [5025e5a3] Could not associate brick 'vm-idev-colo-1-gluster.altn.int:/gluster/gl-vm12' of volume '65d7c61e-ea64-4ebf-a464-5639713c37ba' with correct network as no gluster network found in cluster '3f5f7109-420e-4bfa-a933-9b69970851dc' 2017-11-14 09:44:06,409-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler5) [5025e5a3] FINISH, GlusterVolumesListVDSCommand, return: {65d7c61e-ea64-4ebf-a464-5639713c37ba=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b3023f43}, log id: 5a1fa8fe 2017-11-14 09:44:06,716-06 INFO [org.ovirt.engine.core.bll.snapshots.CreateAllSnapshotsFromVmCommand] (DefaultQuartzScheduler6) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateAllSnapshotsFromVmCommand' successfully. 2017-11-14 09:44:06,723-06 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] (DefaultQuartzScheduler6) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] START, SnapshotVDSCommand(HostName = vm-host-colo-1, SnapshotVDSCommandParameters:{runAsync='true', hostId='c67e3248-23f1-4267-9903-29fa3ceea81f', vmId='7c48c7f7-31d4-4627-a26f-6a239bf92f21'}), log id: 7d2f9046 2017-11-14 09:44:07,308-06 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SnapshotVDSCommand] (DefaultQuartzScheduler6) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] FINISH, SnapshotVDSCommand, log id: 7d2f9046 2017-11-14 09:44:07,315-06 INFO [org.ovirt.engine.core.bll.snapshots.CreateSnapshotCommand] (DefaultQuartzScheduler6) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotCommand' successfully. 2017-11-14 09:44:07,317-06 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (DefaultQuartzScheduler6) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] START, GetImageInfoVDSCommand( GetImageInfoVDSCommandParameters:{runAsync='true', storagePoolId='02abeab8-fac8-41a9-bc2d-7fb0eaac0e07', ignoreFailoverLimit='false', storageDomainId='0e201616-a896-4c4a-90e6-791933ea5393', imageGroupId='57b69fdf-93dd-444c-977b-8803fa83507b', imageId='0028c34f-1117-4b06-b9d9-83be9bf263cf'}), log id: 8339005 2017-11-14 09:44:07,317-06 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] (DefaultQuartzScheduler6) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] START, GetVolumeInfoVDSCommand(HostName = vm-host-colo-2, GetVolumeInfoVDSCommandParameters:{runAsync='true', hostId='4c62f4ab-79fe-40e0-9ade-1d5403f2c659', storagePoolId='02abeab8-fac8-41a9-bc2d-7fb0eaac0e07', storageDomainId='0e201616-a896-4c4a-90e6-791933ea5393', imageGroupId='57b69fdf-93dd-444c-977b-8803fa83507b', imageId='0028c34f-1117-4b06-b9d9-83be9bf263cf'}), log id: ba7440f 2017-11-14 09:44:08,363-06 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] (DefaultQuartzScheduler6) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] FINISH, GetVolumeInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.storage.DiskImage@bc0b007f, log id: ba7440f 2017-11-14 09:44:08,363-06 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (DefaultQuartzScheduler6) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.storage.DiskImage@bc0b007f, log id: 8339005 2017-11-14 09:44:08,369-06 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetQemuImageInfoVDSCommand] (DefaultQuartzScheduler6) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] START, GetQemuImageInfoVDSCommand(HostName = vm-host-colo-1, GetVolumeInfoVDSCommandParameters:{runAsync='true', hostId='c67e3248-23f1-4267-9903-29fa3ceea81f', storagePoolId='02abeab8-fac8-41a9-bc2d-7fb0eaac0e07', storageDomainId='0e201616-a896-4c4a-90e6-791933ea5393', imageGroupId='57b69fdf-93dd-444c-977b-8803fa83507b', imageId='0028c34f-1117-4b06-b9d9-83be9bf263cf'}), log id: 6077503f 2017-11-14 09:44:09,471-06 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetQemuImageInfoVDSCommand] (DefaultQuartzScheduler6) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] FINISH, GetQemuImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.storage.QemuImageInfo@35c8a8a5, log id: 6077503f 2017-11-14 09:44:09,508-06 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [] EVENT_ID: USER_CREATE_SNAPSHOT_FINISHED_SUCCESS(68), Correlation ID: bae5243a-a7a1-4f13-8d5e-04132f98b35d, Job ID: b2c0c8f8-57d1-4012-ab4c-e0b7544036a7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Snapshot 'Auto-generated for Live Storage Migration' creation for VM 'ansible.altn.int' has been completed. 2017-11-14 09:44:10,823-06 INFO [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand] (DefaultQuartzScheduler5) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Running command: LiveMigrateDiskCommand internal: true. Entities affected : ID: 57b69fdf-93dd-444c-977b-8803fa83507b Type: DiskAction group CONFIGURE_DISK_STORAGE with role type USER, ID: 8b925ab7-f38d-40a8-ad72-17bd045137e2 Type: StorageAction group CREATE_DISK with role type USER 2017-11-14 09:44:10,857-06 INFO [org.ovirt.engine.core.bll.storage.lsm.CreateImagePlaceholderCommand] (DefaultQuartzScheduler5) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Running command: CreateImagePlaceholderCommand internal: true. 2017-11-14 09:44:10,873-06 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CloneImageGroupStructureVDSCommand] (DefaultQuartzScheduler5) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] START, CloneImageGroupStructureVDSCommand( TargetDomainImageGroupVDSCommandParameters:{runAsync='true', storagePoolId='02abeab8-fac8-41a9-bc2d-7fb0eaac0e07', ignoreFailoverLimit='false', storageDomainId='0e201616-a896-4c4a-90e6-791933ea5393', imageGroupId='57b69fdf-93dd-444c-977b-8803fa83507b', dstDomainId='8b925ab7-f38d-40a8-ad72-17bd045137e2'}), log id: 421c4ae4 2017-11-14 09:44:11,443-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler4) [6fc04592] START, GlusterServersListVDSCommand(HostName = vm-idev-colo-1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='c05df7cb-fee4-43e5-b132-769f8743bd6e'}), log id: 754e52ac 2017-11-14 09:44:11,444-06 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CloneImageGroupStructureVDSCommand] (DefaultQuartzScheduler5) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] FINISH, CloneImageGroupStructureVDSCommand, log id: 421c4ae4 2017-11-14 09:44:11,448-06 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler5) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command '325a3696-bcaf-4ff7-87e9-e42c7015fea6' 2017-11-14 09:44:11,448-06 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler5) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] CommandMultiAsyncTasks::attachTask: Attaching task '2e5b3734-faab-4a54-8719-9e82d7048f59' to command '325a3696-bcaf-4ff7-87e9-e42c7015fea6'. 2017-11-14 09:44:11,464-06 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler5) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Adding task '2e5b3734-faab-4a54-8719-9e82d7048f59' (Parent Command 'CreateImagePlaceholder', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters'), polling hasn't started yet.. 2017-11-14 09:44:11,476-06 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler5) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] BaseAsyncTask::startPollingTask: Starting to poll task '2e5b3734-faab-4a54-8719-9e82d7048f59'. 2017-11-14 09:44:11,510-06 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] EVENT_ID: USER_MOVED_DISK(2,008), Correlation ID: bae5243a-a7a1-4f13-8d5e-04132f98b35d, Job ID: eb576b8f-4fef-48c0-b66b-67c609fb6502, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: User admin@internal-authz moving disk ansible.altn.int_Disk1 to domain gl-vm-12. 2017-11-14 09:44:12,158-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler4) [6fc04592] FINISH, GlusterServersListVDSCommand, return: [10.20.101.185/24:CONNECTED, vm-host-colo-2-gluster.altn.int:CONNECTED, vm-host-colo-1-gluster.altn.int:CONNECTED], log id: 754e52ac 2017-11-14 09:44:12,174-06 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler4) [6fc04592] probe could not be done for server 'vm-idev-colo-1' as no alternate UP server found 2017-11-14 09:44:12,175-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler4) [6fc04592] START, GlusterVolumesListVDSCommand(HostName = vm-idev-colo-1, GlusterVolumesListVDSParameters:{runAsync='true', hostId='c05df7cb-fee4-43e5-b132-769f8743bd6e'}), log id: 3e781b0 2017-11-14 09:44:12,341-06 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler4) [6fc04592] Could not associate brick 'vm-host-colo-2-gluster.altn.int:/gluster/gl-vm12/brick1' of volume '65d7c61e-ea64-4ebf-a464-5639713c37ba' with correct network as no gluster network found in cluster '99b72a34-a21d-4930-88fc-331998a90417' 2017-11-14 09:44:12,344-06 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler4) [6fc04592] Could not associate brick 'vm-idev-colo-1-gluster.altn.int:/gluster/gl-vm12' of volume '65d7c61e-ea64-4ebf-a464-5639713c37ba' with correct network as no gluster network found in cluster '99b72a34-a21d-4930-88fc-331998a90417' 2017-11-14 09:44:12,344-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler4) [6fc04592] FINISH, GlusterVolumesListVDSCommand, return: {65d7c61e-ea64-4ebf-a464-5639713c37ba=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@667e9b07}, log id: 3e781b0 2017-11-14 09:44:12,401-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler4) [6fc04592] START, GlusterServersListVDSCommand(HostName = vm-host-colo-2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='4c62f4ab-79fe-40e0-9ade-1d5403f2c659'}), log id: ce75234 2017-11-14 09:44:12,405-06 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler10) [42078816] START, FullListVDSCommand(HostName = vm-host-colo-1, FullListVDSCommandParameters:{runAsync='true', hostId='c67e3248-23f1-4267-9903-29fa3ceea81f', vmIds='[7c48c7f7-31d4-4627-a26f-6a239bf92f21]'}), log id: 5e43f489 2017-11-14 09:44:12,511-06 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler10) [42078816] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, afterMigrationStatus=, pid=58847, guestDiskMapping={0QEMU_QEMU_HARDDISK_57b69fdf-93dd-444c-9={name=/dev/sda}, QEMU_DVD-ROM_QM00003={name=/dev/sr0}}, transparentHugePages=true, timeOffset=-21600, cpuType=Broadwell, smp=1, guestNumaNodes=[Ljava.lang.Object;@7028e313, display=qxl, custom={device_de1c50ba-da58-47a9-bc85-cf695ca97f77device_19ddb462-2ef5-4078-87b7-f7762b069446=VmDevice:{id='VmDeviceId:{deviceId='19ddb462-2ef5-4078-87b7-f7762b069446', vmId='7c48c7f7-31d4-4627-a26f-6a239bf92f21'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_de1c50ba-da58-47a9-bc85-cf695ca97f77=VmDevice:{id='VmDeviceId:{deviceId='de1c50ba-da58-47a9-bc85-cf695ca97f77', vmId='7c48c7f7-31d4-4627-a26f-6a239bf92f21'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_de1c50ba-da58-47a9-bc85-cf695ca97f77device_19ddb462-2ef5-4078-87b7-f7762b069446device_ab254618-f4ef-48a9-971a-295b316a6131device_485be8eb-6dc6-49d1-be49-c330775945fb=VmDevice:{id='VmDeviceId:{deviceId='485be8eb-6dc6-49d1-be49-c330775945fb', vmId='7c48c7f7-31d4-4627-a26f-6a239bf92f21'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_de1c50ba-da58-47a9-bc85-cf695ca97f77device_19ddb462-2ef5-4078-87b7-f7762b069446device_ab254618-f4ef-48a9-971a-295b316a6131=VmDevice:{id='VmDeviceId:{deviceId='ab254618-f4ef-48a9-971a-295b316a6131', vmId='7c48c7f7-31d4-4627-a26f-6a239bf92f21'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=2048, smpCoresPerSocket=1, vmName=ansible.altn.int, nice=0, status=Up, maxMemSize=8192, bootMenuEnable=false, vmId=7c48c7f7-31d4-4627-a26f-6a239bf92f21, smpThreadsPerCore=1, smartcardEnable=false, kvmEnable=true, pitReinjection=false, displayNetwork=ovirtmgmt, devices=[Ljava.lang.Object;@e252c04, memGuaranteedSize=2048, maxVCpus=16, clientIp=, statusTime=4356856730, maxMemSlots=16}], log id: 5e43f489 2017-11-14 09:44:12,512-06 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler10) [42078816] Received a memballoon Device without an address when processing VM 7c48c7f7-31d4-4627-a26f-6a239bf92f21 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, target=2097152, alias=balloon0} 2017-11-14 09:44:12,512-06 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler10) [42078816] Received a spice Device without an address when processing VM 7c48c7f7-31d4-4627-a26f-6a239bf92f21 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true, displayIp=10.20.101.181, displayNetwork=ovirtmgmt}, type=graphics, deviceId=6936b80c-d82d-4782-b3f2-71566d74be64, tlsPort=5902} 2017-11-14 09:44:12,586-06 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (DefaultQuartzScheduler5) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Command 'LiveMigrateDisk' (id: 'eace3cfc-32f5-4a98-a286-c0c7c0c13b06') waiting on child command id: '325a3696-bcaf-4ff7-87e9-e42c7015fea6' type:'CreateImagePlaceholder' to complete 2017-11-14 09:44:13,543-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler4) [6fc04592] FINISH, GlusterServersListVDSCommand, return: [10.20.101.183/24:CONNECTED, vm-host-colo-1-gluster.altn.int:CONNECTED, vm-idev-colo-1-gluster.altn.int:CONNECTED], log id: ce75234 2017-11-14 09:44:13,551-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler4) [6fc04592] START, GlusterVolumesListVDSCommand(HostName = vm-host-colo-2, GlusterVolumesListVDSParameters:{runAsync='true', hostId='4c62f4ab-79fe-40e0-9ade-1d5403f2c659'}), log id: 9101143 2017-11-14 09:44:13,602-06 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler3) [68ce5beb] Fetched 19 VMs from VDS '4c62f4ab-79fe-40e0-9ade-1d5403f2c659' 2017-11-14 09:44:13,705-06 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (DefaultQuartzScheduler9) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Command 'LiveMigrateVmDisks' (id: '086e6603-498c-44de-9ac4-8d9b1bbb71f2') waiting on child command id: 'eace3cfc-32f5-4a98-a286-c0c7c0c13b06' type:'LiveMigrateDisk' to complete 2017-11-14 09:44:14,255-06 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler6) [32cbf953] Polling and updating Async Tasks: 3 tasks, 1 tasks to poll now 2017-11-14 09:44:14,265-06 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler4) [6fc04592] Could not associate brick 'vm-host-colo-2-gluster.altn.int:/gluster/gl-vm12/brick1' of volume '65d7c61e-ea64-4ebf-a464-5639713c37ba' with correct network as no gluster network found in cluster '3f5f7109-420e-4bfa-a933-9b69970851dc' 2017-11-14 09:44:14,268-06 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler4) [6fc04592] Could not associate brick 'vm-idev-colo-1-gluster.altn.int:/gluster/gl-vm12' of volume '65d7c61e-ea64-4ebf-a464-5639713c37ba' with correct network as no gluster network found in cluster '3f5f7109-420e-4bfa-a933-9b69970851dc' 2017-11-14 09:44:14,268-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler4) [6fc04592] FINISH, GlusterVolumesListVDSCommand, return: {65d7c61e-ea64-4ebf-a464-5639713c37ba=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b3023f43}, log id: 9101143 2017-11-14 09:44:14,772-06 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (DefaultQuartzScheduler8) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Command 'LiveMigrateDisk' (id: 'eace3cfc-32f5-4a98-a286-c0c7c0c13b06') waiting on child command id: '325a3696-bcaf-4ff7-87e9-e42c7015fea6' type:'CreateImagePlaceholder' to complete 2017-11-14 09:44:15,275-06 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [32cbf953] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM vm-host-colo-2 command HSMGetAllTasksStatusesVDS failed: Volume already exists: ('d2c2e7a9-ae5b-4481-9d8c-185ab5cd8c14',) 2017-11-14 09:44:15,275-06 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler6) [32cbf953] SPMAsyncTask::PollTask: Polling task '2e5b3734-faab-4a54-8719-9e82d7048f59' (Parent Command 'CreateImagePlaceholder', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned status 'finished', result 'cleanSuccess'. 2017-11-14 09:44:15,279-06 ERROR [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler6) [32cbf953] BaseAsyncTask::logEndTaskFailure: Task '2e5b3734-faab-4a54-8719-9e82d7048f59' (Parent Command 'CreateImagePlaceholder', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with failure: -- Result: 'cleanSuccess' -- Message: 'VDSGenericException: VDSErrorException: Failed in vdscommand to HSMGetAllTasksStatusesVDS, error = Volume already exists: ('d2c2e7a9-ae5b-4481-9d8c-185ab5cd8c14',)', -- Exception: 'VDSGenericException: VDSErrorException: Failed in vdscommand to HSMGetAllTasksStatusesVDS, error = Volume already exists: ('d2c2e7a9-ae5b-4481-9d8c-185ab5cd8c14',)' 2017-11-14 09:44:15,280-06 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler6) [32cbf953] CommandAsyncTask::endActionIfNecessary: All tasks of command '325a3696-bcaf-4ff7-87e9-e42c7015fea6' has ended -> executing 'endAction' 2017-11-14 09:44:15,280-06 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler6) [32cbf953] CommandAsyncTask::endAction: Ending action for '1' tasks (command ID: '325a3696-bcaf-4ff7-87e9-e42c7015fea6'): calling endAction '. 2017-11-14 09:44:15,280-06 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-32) [32cbf953] CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction 'CreateImagePlaceholder', 2017-11-14 09:44:15,288-06 ERROR [org.ovirt.engine.core.bll.storage.lsm.CreateImagePlaceholderCommand] (org.ovirt.thread.pool-6-thread-32) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Ending command 'org.ovirt.engine.core.bll.storage.lsm.CreateImagePlaceholderCommand' with failure. 2017-11-14 09:44:15,297-06 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-32) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'CreateImagePlaceholder' completed, handling the result. 2017-11-14 09:44:15,297-06 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-32) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'CreateImagePlaceholder' succeeded, clearing tasks. 2017-11-14 09:44:15,297-06 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-32) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] SPMAsyncTask::ClearAsyncTask: Attempting to clear task '2e5b3734-faab-4a54-8719-9e82d7048f59' 2017-11-14 09:44:15,298-06 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-32) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{runAsync='true', storagePoolId='02abeab8-fac8-41a9-bc2d-7fb0eaac0e07', ignoreFailoverLimit='false', taskId='2e5b3734-faab-4a54-8719-9e82d7048f59'}), log id: 2b114026 2017-11-14 09:44:15,298-06 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-32) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] START, HSMClearTaskVDSCommand(HostName = vm-host-colo-2, HSMTaskGuidBaseVDSCommandParameters:{runAsync='true', hostId='4c62f4ab-79fe-40e0-9ade-1d5403f2c659', taskId='2e5b3734-faab-4a54-8719-9e82d7048f59'}), log id: 380667d8 2017-11-14 09:44:16,041-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler5) [4cfc3f0e] START, GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = vm-host-colo-2, GlusterVolumeAdvancedDetailsVDSParameters:{runAsync='true', hostId='4c62f4ab-79fe-40e0-9ade-1d5403f2c659', volumeName='gl-vm12'}), log id: 5aed93ad 2017-11-14 09:44:16,041-06 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-32) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] FINISH, HSMClearTaskVDSCommand, log id: 380667d8 2017-11-14 09:44:16,041-06 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-32) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] FINISH, SPMClearTaskVDSCommand, log id: 2b114026 2017-11-14 09:44:16,046-06 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-32) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] BaseAsyncTask::removeTaskFromDB: Removed task '2e5b3734-faab-4a54-8719-9e82d7048f59' from DataBase 2017-11-14 09:44:16,046-06 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-32) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity '325a3696-bcaf-4ff7-87e9-e42c7015fea6' 2017-11-14 09:44:17,881-06 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (DefaultQuartzScheduler9) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Command 'LiveMigrateVmDisks' (id: '086e6603-498c-44de-9ac4-8d9b1bbb71f2') waiting on child command id: 'eace3cfc-32f5-4a98-a286-c0c7c0c13b06' type:'LiveMigrateDisk' to complete 2017-11-14 09:44:18,946-06 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (DefaultQuartzScheduler3) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Command 'LiveMigrateDisk' id: 'eace3cfc-32f5-4a98-a286-c0c7c0c13b06' child commands '[325a3696-bcaf-4ff7-87e9-e42c7015fea6]' executions were completed, status 'FAILED' 2017-11-14 09:44:19,303-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler4) [33eb448b] START, GlusterServersListVDSCommand(HostName = vm-idev-colo-1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='c05df7cb-fee4-43e5-b132-769f8743bd6e'}), log id: 553894d9 2017-11-14 09:44:19,970-06 ERROR [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand] (DefaultQuartzScheduler8) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Ending command 'org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand' with failure. 2017-11-14 09:44:20,004-06 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] EVENT_ID: USER_MOVED_DISK_FINISHED_FAILURE(2,011), Correlation ID: bae5243a-a7a1-4f13-8d5e-04132f98b35d, Job ID: eb576b8f-4fef-48c0-b66b-67c609fb6502, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: User admin@internal-authz have failed to move disk ansible.altn.int_Disk1 to domain gl-vm-12. 2017-11-14 09:44:20,532-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler4) [33eb448b] FINISH, GlusterServersListVDSCommand, return: [10.20.101.185/24:CONNECTED, vm-host-colo-2-gluster.altn.int:CONNECTED, vm-host-colo-1-gluster.altn.int:CONNECTED], log id: 553894d9 2017-11-14 09:44:20,549-06 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler4) [33eb448b] probe could not be done for server 'vm-idev-colo-1' as no alternate UP server found 2017-11-14 09:44:20,550-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler4) [33eb448b] START, GlusterVolumesListVDSCommand(HostName = vm-idev-colo-1, GlusterVolumesListVDSParameters:{runAsync='true', hostId='c05df7cb-fee4-43e5-b132-769f8743bd6e'}), log id: 6f5b59a6 2017-11-14 09:44:21,233-06 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (DefaultQuartzScheduler1) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Running command: RemoveSnapshotCommand internal: true. Entities affected : ID: 7c48c7f7-31d4-4627-a26f-6a239bf92f21 Type: VMAction group MANIPULATE_VM_SNAPSHOTS with role type USER 2017-11-14 09:44:21,277-06 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (pool-5-thread-7) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Running command: RemoveSnapshotSingleDiskLiveCommand internal: true. Entities affected : ID: 0e201616-a896-4c4a-90e6-791933ea5393 Type: Storage 2017-11-14 09:44:21,298-06 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [] EVENT_ID: USER_REMOVE_SNAPSHOT(342), Correlation ID: bae5243a-a7a1-4f13-8d5e-04132f98b35d, Job ID: 20bba8e6-e9c5-4ced-bae5-1e8936518c19, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Snapshot 'Auto-generated for Live Storage Migration' deletion for VM 'ansible.altn.int' was initiated by admin@internal-authz. 2017-11-14 09:44:21,677-06 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler4) [33eb448b] Could not associate brick 'vm-host-colo-2-gluster.altn.int:/gluster/gl-vm12/brick1' of volume '65d7c61e-ea64-4ebf-a464-5639713c37ba' with correct network as no gluster network found in cluster '99b72a34-a21d-4930-88fc-331998a90417' 2017-11-14 09:44:21,681-06 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler4) [33eb448b] Could not associate brick 'vm-idev-colo-1-gluster.altn.int:/gluster/gl-vm12' of volume '65d7c61e-ea64-4ebf-a464-5639713c37ba' with correct network as no gluster network found in cluster '99b72a34-a21d-4930-88fc-331998a90417' 2017-11-14 09:44:21,681-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler4) [33eb448b] FINISH, GlusterVolumesListVDSCommand, return: {65d7c61e-ea64-4ebf-a464-5639713c37ba=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@667e9b07}, log id: 6f5b59a6 2017-11-14 09:44:21,760-06 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler4) [33eb448b] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[3f5f7109-420e-4bfa-a933-9b69970851dc=GLUSTER]', sharedLocks=''}' 2017-11-14 09:44:21,851-06 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler4) [33eb448b] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[3f5f7109-420e-4bfa-a933-9b69970851dc=GLUSTER]', sharedLocks=''}' 2017-11-14 09:44:22,312-06 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler1) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Command 'RemoveSnapshot' (id: '66b0705e-cc95-480a-b2c7-19f74257ad21') waiting on child command id: '26032c3d-1701-47a7-8eab-53d9700962ad' type:'RemoveSnapshotSingleDiskLive' to complete 2017-11-14 09:44:22,316-06 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler1) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Executing Live Merge command step 'EXTEND' 2017-11-14 09:44:22,348-06 INFO [org.ovirt.engine.core.bll.MergeExtendCommand] (pool-5-thread-8) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Running command: MergeExtendCommand internal: true. Entities affected : ID: 0e201616-a896-4c4a-90e6-791933ea5393 Type: Storage 2017-11-14 09:44:22,349-06 INFO [org.ovirt.engine.core.bll.MergeExtendCommand] (pool-5-thread-8) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Base and top image sizes are the same; no image size update required 2017-11-14 09:44:22,452-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler5) [4cfc3f0e] FINISH, GetGlusterVolumeAdvancedDetailsVDSCommand, return: org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeAdvancedDetails@c5e1ed8a, log id: 5aed93ad 2017-11-14 09:44:22,458-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler4) [33eb448b] START, GlusterServersListVDSCommand(HostName = vm-host-colo-2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='4c62f4ab-79fe-40e0-9ade-1d5403f2c659'}), log id: 2a9e0967 2017-11-14 09:44:23,355-06 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (DefaultQuartzScheduler3) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Command 'LiveMigrateVmDisks' (id: '086e6603-498c-44de-9ac4-8d9b1bbb71f2') waiting on child command id: '66b0705e-cc95-480a-b2c7-19f74257ad21' type:'RemoveSnapshot' to complete 2017-11-14 09:44:23,360-06 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler3) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Executing Live Merge command step 'MERGE' 2017-11-14 09:44:23,398-06 INFO [org.ovirt.engine.core.bll.MergeCommand] (pool-5-thread-9) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Running command: MergeCommand internal: true. Entities affected : ID: 0e201616-a896-4c4a-90e6-791933ea5393 Type: Storage 2017-11-14 09:44:23,398-06 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-5-thread-9) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] START, MergeVDSCommand(HostName = vm-host-colo-1, MergeVDSCommandParameters:{runAsync='true', hostId='c67e3248-23f1-4267-9903-29fa3ceea81f', vmId='7c48c7f7-31d4-4627-a26f-6a239bf92f21', storagePoolId='02abeab8-fac8-41a9-bc2d-7fb0eaac0e07', storageDomainId='0e201616-a896-4c4a-90e6-791933ea5393', imageGroupId='57b69fdf-93dd-444c-977b-8803fa83507b', imageId='0028c34f-1117-4b06-b9d9-83be9bf263cf', baseImageId='d2c2e7a9-ae5b-4481-9d8c-185ab5cd8c14', topImageId='0028c34f-1117-4b06-b9d9-83be9bf263cf', bandwidth='0'}), log id: 2078d686 2017-11-14 09:44:23,399-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler4) [33eb448b] FINISH, GlusterServersListVDSCommand, return: [10.20.101.183/24:CONNECTED, vm-host-colo-1-gluster.altn.int:CONNECTED, vm-idev-colo-1-gluster.altn.int:CONNECTED], log id: 2a9e0967 2017-11-14 09:44:23,406-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler4) [33eb448b] START, GlusterVolumesListVDSCommand(HostName = vm-host-colo-2, GlusterVolumesListVDSParameters:{runAsync='true', hostId='4c62f4ab-79fe-40e0-9ade-1d5403f2c659'}), log id: 1b66cd4a 2017-11-14 09:44:23,604-06 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-5-thread-9) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] FINISH, MergeVDSCommand, return: c2925d8b-5bf1-4eeb-849d-70b0859134d4, log id: 2078d686 2017-11-14 09:44:23,606-06 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmJobsMonitoring] (pool-5-thread-9) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Stored placeholder for VM Job [c2925d8b-5bf1-4eeb-849d-70b0859134d4] 2017-11-14 09:44:24,381-06 INFO [org.ovirt.engine.core.bll.MergeCommandCallback] (DefaultQuartzScheduler7) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Waiting on merge command to complete (jobId = c2925d8b-5bf1-4eeb-849d-70b0859134d4) 2017-11-14 09:44:24,395-06 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler7) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Command 'RemoveSnapshot' (id: '66b0705e-cc95-480a-b2c7-19f74257ad21') waiting on child command id: '26032c3d-1701-47a7-8eab-53d9700962ad' type:'RemoveSnapshotSingleDiskLive' to complete 2017-11-14 09:44:24,622-06 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler4) [33eb448b] Could not associate brick 'vm-host-colo-2-gluster.altn.int:/gluster/gl-vm12/brick1' of volume '65d7c61e-ea64-4ebf-a464-5639713c37ba' with correct network as no gluster network found in cluster '3f5f7109-420e-4bfa-a933-9b69970851dc' 2017-11-14 09:44:24,624-06 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler4) [33eb448b] Could not associate brick 'vm-idev-colo-1-gluster.altn.int:/gluster/gl-vm12' of volume '65d7c61e-ea64-4ebf-a464-5639713c37ba' with correct network as no gluster network found in cluster '3f5f7109-420e-4bfa-a933-9b69970851dc' 2017-11-14 09:44:24,625-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler4) [33eb448b] FINISH, GlusterVolumesListVDSCommand, return: {65d7c61e-ea64-4ebf-a464-5639713c37ba=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b3023f43}, log id: 1b66cd4a 2017-11-14 09:44:25,410-06 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (DefaultQuartzScheduler1) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Command 'RemoveSnapshotSingleDiskLive' (id: '26032c3d-1701-47a7-8eab-53d9700962ad') waiting on child command id: '590cc8c5-0496-4cbd-affe-5964cf4f25d7' type:'Merge' to complete 2017-11-14 09:44:26,413-06 INFO [org.ovirt.engine.core.bll.MergeCommandCallback] (DefaultQuartzScheduler10) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Waiting on merge command to complete (jobId = c2925d8b-5bf1-4eeb-849d-70b0859134d4) 2017-11-14 09:44:27,550-06 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (DefaultQuartzScheduler2) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Command 'LiveMigrateVmDisks' (id: '086e6603-498c-44de-9ac4-8d9b1bbb71f2') waiting on child command id: '66b0705e-cc95-480a-b2c7-19f74257ad21' type:'RemoveSnapshot' to complete 2017-11-14 09:44:27,584-06 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmJobsMonitoring] (DefaultQuartzScheduler6) [4b64b307] VM Job [c2925d8b-5bf1-4eeb-849d-70b0859134d4]: Deleting 2017-11-14 09:44:27,587-06 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler6) [4b64b307] START, FullListVDSCommand(HostName = vm-host-colo-1, FullListVDSCommandParameters:{runAsync='true', hostId='c67e3248-23f1-4267-9903-29fa3ceea81f', vmIds='[7c48c7f7-31d4-4627-a26f-6a239bf92f21]'}), log id: 178d909f 2017-11-14 09:44:27,904-06 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler6) [4b64b307] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, afterMigrationStatus=, pid=58847, guestDiskMapping={0QEMU_QEMU_HARDDISK_57b69fdf-93dd-444c-9={name=/dev/sda}, QEMU_DVD-ROM_QM00003={name=/dev/sr0}}, transparentHugePages=true, timeOffset=-21600, cpuType=Broadwell, smp=1, guestNumaNodes=[Ljava.lang.Object;@6f84a919, display=qxl, custom={device_de1c50ba-da58-47a9-bc85-cf695ca97f77device_19ddb462-2ef5-4078-87b7-f7762b069446=VmDevice:{id='VmDeviceId:{deviceId='19ddb462-2ef5-4078-87b7-f7762b069446', vmId='7c48c7f7-31d4-4627-a26f-6a239bf92f21'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_de1c50ba-da58-47a9-bc85-cf695ca97f77=VmDevice:{id='VmDeviceId:{deviceId='de1c50ba-da58-47a9-bc85-cf695ca97f77', vmId='7c48c7f7-31d4-4627-a26f-6a239bf92f21'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_de1c50ba-da58-47a9-bc85-cf695ca97f77device_19ddb462-2ef5-4078-87b7-f7762b069446device_ab254618-f4ef-48a9-971a-295b316a6131device_485be8eb-6dc6-49d1-be49-c330775945fb=VmDevice:{id='VmDeviceId:{deviceId='485be8eb-6dc6-49d1-be49-c330775945fb', vmId='7c48c7f7-31d4-4627-a26f-6a239bf92f21'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_de1c50ba-da58-47a9-bc85-cf695ca97f77device_19ddb462-2ef5-4078-87b7-f7762b069446device_ab254618-f4ef-48a9-971a-295b316a6131=VmDevice:{id='VmDeviceId:{deviceId='ab254618-f4ef-48a9-971a-295b316a6131', vmId='7c48c7f7-31d4-4627-a26f-6a239bf92f21'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=2048, smpCoresPerSocket=1, vmName=ansible.altn.int, nice=0, status=Up, maxMemSize=8192, bootMenuEnable=false, vmId=7c48c7f7-31d4-4627-a26f-6a239bf92f21, smpThreadsPerCore=1, smartcardEnable=false, kvmEnable=true, pitReinjection=false, displayNetwork=ovirtmgmt, devices=[Ljava.lang.Object;@1669a3f5, memGuaranteedSize=2048, maxVCpus=16, clientIp=, statusTime=4356871910, maxMemSlots=16}], log id: 178d909f 2017-11-14 09:44:27,907-06 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler6) [4b64b307] Received a memballoon Device without an address when processing VM 7c48c7f7-31d4-4627-a26f-6a239bf92f21 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, target=2097152, alias=balloon0} 2017-11-14 09:44:27,907-06 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler6) [4b64b307] Received a spice Device without an address when processing VM 7c48c7f7-31d4-4627-a26f-6a239bf92f21 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true, displayIp=10.20.101.181, displayNetwork=ovirtmgmt}, type=graphics, deviceId=6936b80c-d82d-4782-b3f2-71566d74be64, tlsPort=5902} 2017-11-14 09:44:28,467-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler9) [a3d4e8] START, GlusterTasksListVDSCommand(HostName = vm-idev-colo-1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='c05df7cb-fee4-43e5-b132-769f8743bd6e'}), log id: ed97d7e 2017-11-14 09:44:28,555-06 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler4) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Command 'RemoveSnapshot' (id: '66b0705e-cc95-480a-b2c7-19f74257ad21') waiting on child command id: '26032c3d-1701-47a7-8eab-53d9700962ad' type:'RemoveSnapshotSingleDiskLive' to complete 2017-11-14 09:44:28,629-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler9) [a3d4e8] FINISH, GlusterTasksListVDSCommand, return: [], log id: ed97d7e 2017-11-14 09:44:28,656-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler9) [a3d4e8] START, GlusterTasksListVDSCommand(HostName = vm-host-colo-2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='4c62f4ab-79fe-40e0-9ade-1d5403f2c659'}), log id: 1a355321 2017-11-14 09:44:28,887-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler9) [a3d4e8] FINISH, GlusterTasksListVDSCommand, return: [], log id: 1a355321 2017-11-14 09:44:29,562-06 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (DefaultQuartzScheduler6) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Command 'RemoveSnapshotSingleDiskLive' (id: '26032c3d-1701-47a7-8eab-53d9700962ad') waiting on child command id: '590cc8c5-0496-4cbd-affe-5964cf4f25d7' type:'Merge' to complete 2017-11-14 09:44:29,680-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler4) [7bac08b7] START, GlusterServersListVDSCommand(HostName = vm-idev-colo-1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='c05df7cb-fee4-43e5-b132-769f8743bd6e'}), log id: 20a0f07b 2017-11-14 09:44:30,575-06 INFO [org.ovirt.engine.core.bll.MergeCommandCallback] (DefaultQuartzScheduler2) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Merge command (jobId = c2925d8b-5bf1-4eeb-849d-70b0859134d4) has completed for images 'd2c2e7a9-ae5b-4481-9d8c-185ab5cd8c14'..'0028c34f-1117-4b06-b9d9-83be9bf263cf' 2017-11-14 09:44:30,908-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler4) [7bac08b7] FINISH, GlusterServersListVDSCommand, return: [10.20.101.185/24:CONNECTED, vm-host-colo-2-gluster.altn.int:CONNECTED, vm-host-colo-1-gluster.altn.int:CONNECTED], log id: 20a0f07b 2017-11-14 09:44:30,925-06 WARN [org.ovirt.engine.core.bll.gluster.GlusterSyncJob] (DefaultQuartzScheduler4) [7bac08b7] probe could not be done for server 'vm-idev-colo-1' as no alternate UP server found 2017-11-14 09:44:30,926-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler4) [7bac08b7] START, GlusterVolumesListVDSCommand(HostName = vm-idev-colo-1, GlusterVolumesListVDSParameters:{runAsync='true', hostId='c05df7cb-fee4-43e5-b132-769f8743bd6e'}), log id: 13b58e4f 2017-11-14 09:44:31,585-06 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler8) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Executing Live Merge command step 'MERGE_STATUS' 2017-11-14 09:44:31,617-06 INFO [org.ovirt.engine.core.bll.MergeStatusCommand] (pool-5-thread-10) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Running command: MergeStatusCommand internal: true. Entities affected : ID: 0e201616-a896-4c4a-90e6-791933ea5393 Type: Storage 2017-11-14 09:44:31,619-06 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (pool-5-thread-10) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] START, FullListVDSCommand(HostName = vm-host-colo-1, FullListVDSCommandParameters:{runAsync='true', hostId='c67e3248-23f1-4267-9903-29fa3ceea81f', vmIds='[7c48c7f7-31d4-4627-a26f-6a239bf92f21]'}), log id: 1844111b 2017-11-14 09:44:31,628-06 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler4) [7bac08b7] Could not associate brick 'vm-host-colo-2-gluster.altn.int:/gluster/gl-vm12/brick1' of volume '65d7c61e-ea64-4ebf-a464-5639713c37ba' with correct network as no gluster network found in cluster '99b72a34-a21d-4930-88fc-331998a90417' 2017-11-14 09:44:31,631-06 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler4) [7bac08b7] Could not associate brick 'vm-idev-colo-1-gluster.altn.int:/gluster/gl-vm12' of volume '65d7c61e-ea64-4ebf-a464-5639713c37ba' with correct network as no gluster network found in cluster '99b72a34-a21d-4930-88fc-331998a90417' 2017-11-14 09:44:31,631-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler4) [7bac08b7] FINISH, GlusterVolumesListVDSCommand, return: {65d7c61e-ea64-4ebf-a464-5639713c37ba=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@667e9b07}, log id: 13b58e4f 2017-11-14 09:44:31,688-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler4) [7bac08b7] START, GlusterServersListVDSCommand(HostName = vm-host-colo-2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='4c62f4ab-79fe-40e0-9ade-1d5403f2c659'}), log id: 1753ae3e 2017-11-14 09:44:31,689-06 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (pool-5-thread-10) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, afterMigrationStatus=, pid=58847, guestDiskMapping={0QEMU_QEMU_HARDDISK_57b69fdf-93dd-444c-9={name=/dev/sda}, QEMU_DVD-ROM_QM00003={name=/dev/sr0}}, transparentHugePages=true, timeOffset=-21600, cpuType=Broadwell, smp=1, guestNumaNodes=[Ljava.lang.Object;@66bf2fab, display=qxl, custom={device_de1c50ba-da58-47a9-bc85-cf695ca97f77device_19ddb462-2ef5-4078-87b7-f7762b069446=VmDevice:{id='VmDeviceId:{deviceId='19ddb462-2ef5-4078-87b7-f7762b069446', vmId='7c48c7f7-31d4-4627-a26f-6a239bf92f21'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_de1c50ba-da58-47a9-bc85-cf695ca97f77=VmDevice:{id='VmDeviceId:{deviceId='de1c50ba-da58-47a9-bc85-cf695ca97f77', vmId='7c48c7f7-31d4-4627-a26f-6a239bf92f21'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_de1c50ba-da58-47a9-bc85-cf695ca97f77device_19ddb462-2ef5-4078-87b7-f7762b069446device_ab254618-f4ef-48a9-971a-295b316a6131device_485be8eb-6dc6-49d1-be49-c330775945fb=VmDevice:{id='VmDeviceId:{deviceId='485be8eb-6dc6-49d1-be49-c330775945fb', vmId='7c48c7f7-31d4-4627-a26f-6a239bf92f21'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_de1c50ba-da58-47a9-bc85-cf695ca97f77device_19ddb462-2ef5-4078-87b7-f7762b069446device_ab254618-f4ef-48a9-971a-295b316a6131=VmDevice:{id='VmDeviceId:{deviceId='ab254618-f4ef-48a9-971a-295b316a6131', vmId='7c48c7f7-31d4-4627-a26f-6a239bf92f21'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=2048, smpCoresPerSocket=1, vmName=ansible.altn.int, nice=0, status=Up, maxMemSize=8192, bootMenuEnable=false, vmId=7c48c7f7-31d4-4627-a26f-6a239bf92f21, smpThreadsPerCore=1, smartcardEnable=false, kvmEnable=true, pitReinjection=false, displayNetwork=ovirtmgmt, devices=[Ljava.lang.Object;@25ccdd20, memGuaranteedSize=2048, maxVCpus=16, clientIp=, statusTime=4356875950, maxMemSlots=16}], log id: 1844111b 2017-11-14 09:44:31,690-06 INFO [org.ovirt.engine.core.bll.MergeStatusCommand] (pool-5-thread-10) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Successfully removed volume(s): [0028c34f-1117-4b06-b9d9-83be9bf263cf] 2017-11-14 09:44:31,690-06 INFO [org.ovirt.engine.core.bll.MergeStatusCommand] (pool-5-thread-10) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Volume merge type 'COMMIT' 2017-11-14 09:44:32,612-06 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler3) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Executing Live Merge command step 'DESTROY_IMAGE' 2017-11-14 09:44:32,642-06 INFO [org.ovirt.engine.core.bll.storage.disk.image.DestroyImageCommand] (pool-5-thread-1) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Running command: DestroyImageCommand internal: true. Entities affected : ID: 0e201616-a896-4c4a-90e6-791933ea5393 Type: Storage 2017-11-14 09:44:32,659-06 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DestroyImageVDSCommand] (pool-5-thread-1) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] START, DestroyImageVDSCommand( DestroyImageVDSCommandParameters:{runAsync='true', storagePoolId='02abeab8-fac8-41a9-bc2d-7fb0eaac0e07', ignoreFailoverLimit='false', storageDomainId='0e201616-a896-4c4a-90e6-791933ea5393', imageGroupId='57b69fdf-93dd-444c-977b-8803fa83507b', imageId='00000000-0000-0000-0000-000000000000', imageList='[0028c34f-1117-4b06-b9d9-83be9bf263cf]', postZero='false', force='false'}), log id: 49feb3f2 2017-11-14 09:44:32,660-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler4) [7bac08b7] FINISH, GlusterServersListVDSCommand, return: [10.20.101.183/24:CONNECTED, vm-host-colo-1-gluster.altn.int:CONNECTED, vm-idev-colo-1-gluster.altn.int:CONNECTED], log id: 1753ae3e 2017-11-14 09:44:32,668-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler4) [7bac08b7] START, GlusterVolumesListVDSCommand(HostName = vm-host-colo-2, GlusterVolumesListVDSParameters:{runAsync='true', hostId='4c62f4ab-79fe-40e0-9ade-1d5403f2c659'}), log id: 5b80a332 2017-11-14 09:44:32,866-06 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DestroyImageVDSCommand] (pool-5-thread-1) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] FINISH, DestroyImageVDSCommand, log id: 49feb3f2 2017-11-14 09:44:32,881-06 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (pool-5-thread-1) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 'df91564b-854a-4a6c-89e8-b3c6cc150b74' 2017-11-14 09:44:32,882-06 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (pool-5-thread-1) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] CommandMultiAsyncTasks::attachTask: Attaching task '357f6361-14de-4e45-b3b2-2c75e902c66e' to command 'df91564b-854a-4a6c-89e8-b3c6cc150b74'. 2017-11-14 09:44:32,897-06 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (pool-5-thread-1) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Adding task '357f6361-14de-4e45-b3b2-2c75e902c66e' (Parent Command 'DestroyImage', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters'), polling hasn't started yet.. 2017-11-14 09:44:32,897-06 INFO [org.ovirt.engine.core.bll.storage.disk.image.DestroyImageCommand] (pool-5-thread-1) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Successfully started task to remove orphaned volumes resulting from live merge 2017-11-14 09:44:32,909-06 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (pool-5-thread-1) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] BaseAsyncTask::startPollingTask: Starting to poll task '357f6361-14de-4e45-b3b2-2c75e902c66e'. 2017-11-14 09:44:33,877-06 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler4) [7bac08b7] Could not associate brick 'vm-host-colo-2-gluster.altn.int:/gluster/gl-vm12/brick1' of volume '65d7c61e-ea64-4ebf-a464-5639713c37ba' with correct network as no gluster network found in cluster '3f5f7109-420e-4bfa-a933-9b69970851dc' 2017-11-14 09:44:33,880-06 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler4) [7bac08b7] Could not associate brick 'vm-idev-colo-1-gluster.altn.int:/gluster/gl-vm12' of volume '65d7c61e-ea64-4ebf-a464-5639713c37ba' with correct network as no gluster network found in cluster '3f5f7109-420e-4bfa-a933-9b69970851dc' 2017-11-14 09:44:33,881-06 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler4) [7bac08b7] FINISH, GlusterVolumesListVDSCommand, return: {65d7c61e-ea64-4ebf-a464-5639713c37ba=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b3023f43}, log id: 5b80a332 2017-11-14 09:44:34,634-06 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler10) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Executing Live Merge command step 'DESTROY_IMAGE_CHECK' 2017-11-14 09:44:34,663-06 INFO [org.ovirt.engine.core.bll.DestroyImageCheckCommand] (pool-5-thread-2) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Running command: DestroyImageCheckCommand internal: true. 2017-11-14 09:44:34,663-06 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetVolumeInfoVDSCommand] (pool-5-thread-2) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] START, SPMGetVolumeInfoVDSCommand( SPMGetVolumeInfoVDSCommandParameters:{expectedEngineErrors='[VolumeDoesNotExist]', runAsync='true', storagePoolId='02abeab8-fac8-41a9-bc2d-7fb0eaac0e07', ignoreFailoverLimit='false', storageDomainId='0e201616-a896-4c4a-90e6-791933ea5393', imageGroupId='57b69fdf-93dd-444c-977b-8803fa83507b', imageId='0028c34f-1117-4b06-b9d9-83be9bf263cf'}), log id: 4ed745d5 2017-11-14 09:44:34,663-06 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetVolumeInfoVDSCommand] (pool-5-thread-2) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Executing GetVolumeInfo using the current SPM 2017-11-14 09:44:34,663-06 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] (pool-5-thread-2) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] START, GetVolumeInfoVDSCommand(HostName = vm-host-colo-2, GetVolumeInfoVDSCommandParameters:{expectedEngineErrors='[VolumeDoesNotExist]', runAsync='true', hostId='4c62f4ab-79fe-40e0-9ade-1d5403f2c659', storagePoolId='02abeab8-fac8-41a9-bc2d-7fb0eaac0e07', storageDomainId='0e201616-a896-4c4a-90e6-791933ea5393', imageGroupId='57b69fdf-93dd-444c-977b-8803fa83507b', imageId='0028c34f-1117-4b06-b9d9-83be9bf263cf'}), log id: 4e2b64d1 2017-11-14 09:44:35,281-06 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler8) [37582e18] Polling and updating Async Tasks: 4 tasks, 1 tasks to poll now 2017-11-14 09:44:35,664-06 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (DefaultQuartzScheduler5) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Command 'LiveMigrateVmDisks' (id: '086e6603-498c-44de-9ac4-8d9b1bbb71f2') waiting on child command id: '66b0705e-cc95-480a-b2c7-19f74257ad21' type:'RemoveSnapshot' to complete 2017-11-14 09:44:35,677-06 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] (pool-5-thread-2) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Failed in 'GetVolumeInfoVDS' method 2017-11-14 09:44:35,678-06 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] (pool-5-thread-2) [bae5243a-a7a1-4f13-8d5e-04132f98b35d] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand' return value ' VolumeInfoReturn:{status='Status [code=201, message=Volume does not exist: (u'0028c34f-1117-4b06-b9d9-83be9bf263cf',)]'} '