Re: [ovirt-users] Unable to shutdown VM's

Hi, After upgrading to 3.4.2 rc I want to reinstall the nodes. But there are some benchmark VM's that I'm unable to shutdown to put the nodes in maintenance. How can I kill these VM's without having to kill the power to the nodes manually? Kind regards, Jorick Astrego Netbulae B.V. 2014-Jun-05, 12:24 Failed to switch Host node1.test.now to Maintenance mode. 2014-Jun-05, 12:24 Host node1.test.now cannot change into maintenance mode - not all Vms have been migrated successfully. Consider manual intervention: stopping/migrating Vms: Bench1 (User: admin). 2014-Jun-05, 12:05 Shutdown of VM Bench2 failed. 2014-Jun-05, 12:05 Shutdown of VM Bench4 failed. 2014-Jun-05, 12:05 Shutdown of VM Bench3 failed. 2014-Jun-05, 12:05 Shutdown of VM Bench1 failed. 2014-Jun-05, 12:00 VM shutdown initiated by admin on VM Bench2 (Host: node2.test.now). 2014-Jun-05, 12:00 VM shutdown initiated by admin on VM Bench4 (Host: node4.test.now). 2014-Jun-05, 12:00 VM shutdown initiated by admin on VM Bench3 (Host: node3.test.now). 2014-Jun-05, 12:00 VM shutdown initiated by admin on VM Bench1 (Host: node1.test.now). 2014-06-05 12:00:33,127 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-6-thread-8) [7c4b08f7] Start running CanDoAction for command number 1/3 (Command type: ShutdownVm) 2014-06-05 12:00:33,130 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-6-thread-14) [2239cfe1] Start running CanDoAction for command number 2/3 (Command type: ShutdownVm) 2014-06-05 12:00:33,134 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-6-thread-50) [17ece55] Start running CanDoAction for command number 3/3 (Command type: ShutdownVm) 2014-06-05 12:00:33,225 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-6-thread-50) [17ece55] Finish handling CanDoAction for command number 3/3 (Command type: ShutdownVm) 2014-06-05 12:00:33,229 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-6-thread-8) [7c4b08f7] Finish handling CanDoAction for command number 1/3 (Command type: ShutdownVm) 2014-06-05 12:00:33,234 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (org.ovirt.thread.pool-6-thread-14) [2239cfe1] Finish handling CanDoAction for command number 2/3 (Command type: ShutdownVm) 2014-06-05 12:00:33,710 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (org.ovirt.thread.pool-6-thread-3) [7c4b08f7] Running command: ShutdownVmCommand internal: false. Entities affected : ID: 6709eaa1-163f-4dc5-9101-e46870438f38 Type: VM 2014-06-05 12:00:33,728 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (org.ovirt.thread.pool-6-thread-3) [7c4b08f7] Entered (VM Bench3). 2014-06-05 12:00:33,728 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (org.ovirt.thread.pool-6-thread-3) [7c4b08f7] Sending shutdown command for VM Bench3. 2014-06-05 12:00:33,764 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-3) [7c4b08f7] START, DestroyVmVDSCommand(HostName = node3.test.now, HostId = 7415506c-cda7-4018-804d-5f6d3beddbfb, vmId=6709eaa1-163f-4dc5-9101-e46870438f38, force=false, secondsToWait=30, gracefully=true), log id: 5eddc358 2014-06-05 12:00:33,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-3) [7c4b08f7] START, DestroyVDSCommand(HostName = node3.test.now, HostId = 7415506c-cda7-4018-804d-5f6d3beddbfb, vmId=6709eaa1-163f-4dc5-9101-e46870438f38, force=false, secondsToWait=30, gracefully=true), log id: 19f6f7c8 2014-06-05 12:00:33,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-3) [7c4b08f7] FINISH, DestroyVDSCommand, log id: 19f6f7c8 2014-06-05 12:00:33,843 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-3) [7c4b08f7] FINISH, DestroyVmVDSCommand, return: PoweringDown, log id: 5eddc358 2014-06-05 12:00:33,855 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [7c4b08f7] Correlation ID: 7c4b08f7, Job ID: 406c481e-102b-4488-9286-c9b38197ef36, Call Stack: null, Custom Event ID: -1, Message: VM shutdown initiated by admin on VM Bench3 (Host: node3.test.now). 2014-06-05 12:00:33,921 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (org.ovirt.thread.pool-6-thread-3) [2239cfe1] Running command: ShutdownVmCommand internal: false. Entities affected : ID: 8b305f06-2c82-4a13-99f2-5beab5a056ea Type: VM 2014-06-05 12:00:33,967 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (org.ovirt.thread.pool-6-thread-3) [2239cfe1] Entered (VM Bench4). 2014-06-05 12:00:33,969 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (org.ovirt.thread.pool-6-thread-3) [2239cfe1] Sending shutdown command for VM Bench4. 2014-06-05 12:00:34,001 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-3) [2239cfe1] START, DestroyVmVDSCommand(HostName = node4.test.now, HostId = bb13752e-85cb-4945-822b-48ab2a7b1329, vmId=8b305f06-2c82-4a13-99f2-5beab5a056ea, force=false, secondsToWait=30, gracefully=true), log id: 23611c0e 2014-06-05 12:00:34,041 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-3) [2239cfe1] START, DestroyVDSCommand(HostName = node4.test.now, HostId = bb13752e-85cb-4945-822b-48ab2a7b1329, vmId=8b305f06-2c82-4a13-99f2-5beab5a056ea, force=false, secondsToWait=30, gracefully=true), log id: 4f1663eb 2014-06-05 12:00:34,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-3) [2239cfe1] FINISH, DestroyVDSCommand, log id: 4f1663eb 2014-06-05 12:00:34,257 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-3) [2239cfe1] FINISH, DestroyVmVDSCommand, return: PoweringDown, log id: 23611c0e 2014-06-05 12:00:34,455 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [2239cfe1] Correlation ID: 2239cfe1, Job ID: 00fb00a7-88b0-4fb1-b14a-fa099e3d6409, Call Stack: null, Custom Event ID: -1, Message: VM shutdown initiated by admin on VM Bench4 (Host: node4.test.now). 2014-06-05 12:00:34,677 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (org.ovirt.thread.pool-6-thread-3) [17ece55] Running command: ShutdownVmCommand internal: false. Entities affected : ID: 764b60fd-c255-479b-9082-3f4f04b95cb2 Type: VM 2014-06-05 12:00:34,695 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (org.ovirt.thread.pool-6-thread-3) [17ece55] Entered (VM Bench2). 2014-06-05 12:00:34,696 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (org.ovirt.thread.pool-6-thread-3) [17ece55] Sending shutdown command for VM Bench2. 2014-06-05 12:00:34,718 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-3) [17ece55] START, DestroyVmVDSCommand(HostName = node2.test.now, HostId = 28fdcb5d-7acd-410e-8b65-0b4f483cb575, vmId=764b60fd-c255-479b-9082-3f4f04b95cb2, force=false, secondsToWait=30, gracefully=true), log id: 215e8d55 2014-06-05 12:00:34,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-3) [17ece55] START, DestroyVDSCommand(HostName = node2.test.now, HostId = 28fdcb5d-7acd-410e-8b65-0b4f483cb575, vmId=764b60fd-c255-479b-9082-3f4f04b95cb2, force=false, secondsToWait=30, gracefully=true), log id: 6a5599de 2014-06-05 12:00:34,755 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-3) [17ece55] FINISH, DestroyVDSCommand, log id: 6a5599de 2014-06-05 12:00:34,777 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-3) [17ece55] FINISH, DestroyVmVDSCommand, return: PoweringDown, log id: 215e8d55 2014-06-05 12:00:34,833 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [17ece55] Correlation ID: 17ece55, Job ID: d92b0c9a-fb0f-4ca6-bc51-8d363aaf378c, Call Stack: null, Custom Event ID: -1, Message: VM shutdown initiated by admin on VM Bench2 (Host: node2.test.now). 2014-06-05 12:01:00,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-45) [22ac5fb6] FINISH, ConnectStorageServerVDSCommand, return: {f47e9f99-9989-4297-a84f-4f75338eeced=0}, log id: 7045fa3b 2014-06-05 12:01:00,650 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-48) [4fbc37fa] FINISH, ConnectStorageServerVDSCommand, return: {f47e9f99-9989-4297-a84f-4f75338eeced=0}, log id: 306b54b0 2014-06-05 12:01:00,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-25) [760d647e] FINISH, ConnectStorageServerVDSCommand, return: {f47e9f99-9989-4297-a84f-4f75338eeced=0}, log id: 7f7a769d 2014-06-05 12:01:00,661 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-44) [6949bbc8] FINISH, ConnectStorageServerVDSCommand, return: {f47e9f99-9989-4297-a84f-4f75338eeced=0}, log id: 610dcd45 2014-06-05 12:01:00,663 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-94) [21aa560a] ConnectDomainToStorage. After Connect all hosts to pool. Time:6/5/14 12:01 PM 2014-06-05 12:02:12,172 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-50) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:02:15,245 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-37) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:02:30,697 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-9) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:02:30,724 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-2) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:03:07,860 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-39) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:04:27,836 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-19) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:04:27,841 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-50) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:05:00,296 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-28) [10a45e4f] Autorecovering 1 storage domains 2014-06-05 12:05:00,296 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-28) [10a45e4f] Autorecovering storage domains id: 9923a5a1-61e0-4edb-a04d-22c962190c53 2014-06-05 12:05:00,299 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-28) [9bf7039] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 9923a5a1-61e0-4edb-a04d-22c962190c53 Type: Storage 2014-06-05 12:05:00,304 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-28) [9bf7039] ConnectDomainToStorage. Before Connect all hosts to pool. Time:6/5/14 12:05 PM 2014-06-05 12:05:00,441 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-35) [3164009c] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2014-06-05 12:05:00,446 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-29) [1a51a7f5] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2014-06-05 12:05:00,450 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-5) [19f33713] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2014-06-05 12:05:00,454 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-10) [3dd2802] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2014-06-05 12:05:00,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-35) [3164009c] START, ConnectStorageServerVDSCommand(HostName = node4.test.now, HostId = bb13752e-85cb-4945-822b-48ab2a7b1329, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f47e9f99-9989-4297-a84f-4f75338eeced, connection: 10.100.100.105:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7ea7832a 2014-06-05 12:05:00,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-29) [1a51a7f5] START, ConnectStorageServerVDSCommand(HostName = node3.test.now, HostId = 7415506c-cda7-4018-804d-5f6d3beddbfb, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f47e9f99-9989-4297-a84f-4f75338eeced, connection: 10.100.100.105:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 143f44d4 2014-06-05 12:05:00,580 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-10) [3dd2802] START, ConnectStorageServerVDSCommand(HostName = node1.test.now, HostId = e17a740c-47ba-4a81-99ab-6c386b572f26, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f47e9f99-9989-4297-a84f-4f75338eeced, connection: 10.100.100.105:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4058bc78 2014-06-05 12:05:00,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-5) [19f33713] START, ConnectStorageServerVDSCommand(HostName = node2.test.now, HostId = 28fdcb5d-7acd-410e-8b65-0b4f483cb575, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f47e9f99-9989-4297-a84f-4f75338eeced, connection: 10.100.100.105:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6eb8491e 2014-06-05 12:05:23,071 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) [77f10397] VM Bench1 63b1fd02-1fb5-44d1-b1cc-54cb5c0fdb0e moved from PoweringDown --> Up 2014-06-05 12:05:23,153 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [77f10397] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Shutdown of VM Bench1 failed. 2014-06-05 12:05:34,341 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) [604c5b5d] VM Bench3 6709eaa1-163f-4dc5-9101-e46870438f38 moved from PoweringDown --> Up 2014-06-05 12:05:34,411 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) [604c5b5d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Shutdown of VM Bench3 failed. 2014-06-05 12:05:35,312 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) [3bbcf288] VM Bench4 8b305f06-2c82-4a13-99f2-5beab5a056ea moved from PoweringDown --> Up 2014-06-05 12:05:35,439 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-38) [3bbcf288] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Shutdown of VM Bench4 failed. 2014-06-05 12:05:35,663 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) [40fb7b62] VM Bench2 764b60fd-c255-479b-9082-3f4f04b95cb2 moved from PoweringDown --> Up 2014-06-05 12:05:36,101 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [40fb7b62] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Shutdown of VM Bench2 failed. 2014-06-05 12:06:00,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-10) [3dd2802] FINISH, ConnectStorageServerVDSCommand, return: {f47e9f99-9989-4297-a84f-4f75338eeced=0}, log id: 4058bc78 2014-06-05 12:06:00,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-29) [1a51a7f5] FINISH, ConnectStorageServerVDSCommand, return: {f47e9f99-9989-4297-a84f-4f75338eeced=0}, log id: 143f44d4 2014-06-05 12:06:00,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-35) [3164009c] FINISH, ConnectStorageServerVDSCommand, return: {f47e9f99-9989-4297-a84f-4f75338eeced=0}, log id: 7ea7832a 2014-06-05 12:06:00,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-5) [19f33713] FINISH, ConnectStorageServerVDSCommand, return: {f47e9f99-9989-4297-a84f-4f75338eeced=0}, log id: 6eb8491e 2014-06-05 12:06:00,647 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-28) [9bf7039] ConnectDomainToStorage. After Connect all hosts to pool. Time:6/5/14 12:06 PM 2014-06-05 12:07:17,243 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-11) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:08:16,532 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-1) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:08:25,687 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-46) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:10:00,442 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-77) [7cace458] Autorecovering 1 storage domains 2014-06-05 12:10:00,442 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-77) [7cace458] Autorecovering storage domains id: 9923a5a1-61e0-4edb-a04d-22c962190c53 2014-06-05 12:10:00,443 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-77) [77f9bcd3] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 9923a5a1-61e0-4edb-a04d-22c962190c53 Type: Storage 2014-06-05 12:10:00,444 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-77) [77f9bcd3] ConnectDomainToStorage. Before Connect all hosts to pool. Time:6/5/14 12:10 PM 2014-06-05 12:10:00,597 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-37) Executing a command: java.util.concurrent.FutureTask , but note that there are 3 tasks in the queue. 2014-06-05 12:10:00,598 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-10) Executing a command: java.util.concurrent.FutureTask , but note that there are 2 tasks in the queue. 2014-06-05 12:10:00,599 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-35) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:10:00,600 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-5) [7ecb4630] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2014-06-05 12:10:00,602 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-35) [25233142] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2014-06-05 12:10:00,606 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-37) [175926fc] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2014-06-05 12:10:00,607 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-10) [8d05e48] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2014-06-05 12:10:00,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-35) [25233142] START, ConnectStorageServerVDSCommand(HostName = node2.test.now, HostId = 28fdcb5d-7acd-410e-8b65-0b4f483cb575, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f47e9f99-9989-4297-a84f-4f75338eeced, connection: 10.100.100.105:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 16b62d67 2014-06-05 12:10:00,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-5) [7ecb4630] START, ConnectStorageServerVDSCommand(HostName = node1.test.now, HostId = e17a740c-47ba-4a81-99ab-6c386b572f26, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f47e9f99-9989-4297-a84f-4f75338eeced, connection: 10.100.100.105:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2bc2c88 2014-06-05 12:10:00,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-10) [8d05e48] START, ConnectStorageServerVDSCommand(HostName = node3.test.now, HostId = 7415506c-cda7-4018-804d-5f6d3beddbfb, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f47e9f99-9989-4297-a84f-4f75338eeced, connection: 10.100.100.105:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4cb7e97c 2014-06-05 12:10:00,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-37) [175926fc] START, ConnectStorageServerVDSCommand(HostName = node4.test.now, HostId = bb13752e-85cb-4945-822b-48ab2a7b1329, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f47e9f99-9989-4297-a84f-4f75338eeced, connection: 10.100.100.105:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 442ca304 2014-06-05 12:10:51,626 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-38) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:10:54,669 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-14) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:11:00,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-10) [8d05e48] FINISH, ConnectStorageServerVDSCommand, return: {f47e9f99-9989-4297-a84f-4f75338eeced=0}, log id: 4cb7e97c 2014-06-05 12:11:00,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-37) [175926fc] FINISH, ConnectStorageServerVDSCommand, return: {f47e9f99-9989-4297-a84f-4f75338eeced=0}, log id: 442ca304 2014-06-05 12:11:00,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-5) [7ecb4630] FINISH, ConnectStorageServerVDSCommand, return: {f47e9f99-9989-4297-a84f-4f75338eeced=0}, log id: 2bc2c88 2014-06-05 12:11:00,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-35) [25233142] FINISH, ConnectStorageServerVDSCommand, return: {f47e9f99-9989-4297-a84f-4f75338eeced=0}, log id: 16b62d67 2014-06-05 12:11:00,854 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-77) [77f9bcd3] ConnectDomainToStorage. After Connect all hosts to pool. Time:6/5/14 12:11 PM 2014-06-05 12:11:40,654 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-25) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:12:14,657 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-31) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:12:27,462 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-37) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:13:59,781 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-22) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:14:15,341 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-10) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:14:21,429 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-35) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:14:21,455 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-4) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:14:40,159 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-17) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:14:52,383 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-15) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:14:52,412 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-27) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:14:55,658 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-42) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:15:00,031 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-65) [604c5b5d] Autorecovering 1 storage domains 2014-06-05 12:15:00,031 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-65) [604c5b5d] Autorecovering storage domains id: 9923a5a1-61e0-4edb-a04d-22c962190c53 2014-06-05 12:15:00,032 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-65) [7d4ef42] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 9923a5a1-61e0-4edb-a04d-22c962190c53 Type: Storage 2014-06-05 12:15:00,035 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-65) [7d4ef42] ConnectDomainToStorage. Before Connect all hosts to pool. Time:6/5/14 12:15 PM 2014-06-05 12:15:00,109 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-45) [6c18413] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2014-06-05 12:15:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-5) [d3ad154] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2014-06-05 12:15:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-1) [32d6d63f] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2014-06-05 12:15:00,122 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-24) [718f19c7] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2014-06-05 12:15:00,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-24) [718f19c7] START, ConnectStorageServerVDSCommand(HostName = node1.test.now, HostId = e17a740c-47ba-4a81-99ab-6c386b572f26, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f47e9f99-9989-4297-a84f-4f75338eeced, connection: 10.100.100.105:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4bba2a9a 2014-06-05 12:15:00,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-5) [d3ad154] START, ConnectStorageServerVDSCommand(HostName = node3.test.now, HostId = 7415506c-cda7-4018-804d-5f6d3beddbfb, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f47e9f99-9989-4297-a84f-4f75338eeced, connection: 10.100.100.105:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4c27ad04 2014-06-05 12:15:00,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-1) [32d6d63f] START, ConnectStorageServerVDSCommand(HostName = node2.test.now, HostId = 28fdcb5d-7acd-410e-8b65-0b4f483cb575, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f47e9f99-9989-4297-a84f-4f75338eeced, connection: 10.100.100.105:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1d382d00 2014-06-05 12:15:00,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-45) [6c18413] START, ConnectStorageServerVDSCommand(HostName = node4.test.now, HostId = bb13752e-85cb-4945-822b-48ab2a7b1329, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f47e9f99-9989-4297-a84f-4f75338eeced, connection: 10.100.100.105:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7daf7955 2014-06-05 12:16:00,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-5) [d3ad154] FINISH, ConnectStorageServerVDSCommand, return: {f47e9f99-9989-4297-a84f-4f75338eeced=0}, log id: 4c27ad04 2014-06-05 12:16:00,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-24) [718f19c7] FINISH, ConnectStorageServerVDSCommand, return: {f47e9f99-9989-4297-a84f-4f75338eeced=0}, log id: 4bba2a9a 2014-06-05 12:16:00,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-45) [6c18413] FINISH, ConnectStorageServerVDSCommand, return: {f47e9f99-9989-4297-a84f-4f75338eeced=0}, log id: 7daf7955 2014-06-05 12:16:00,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-1) [32d6d63f] FINISH, ConnectStorageServerVDSCommand, return: {f47e9f99-9989-4297-a84f-4f75338eeced=0}, log id: 1d382d00 2014-06-05 12:16:00,424 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-65) [7d4ef42] ConnectDomainToStorage. After Connect all hosts to pool. Time:6/5/14 12:16 PM 2014-06-05 12:18:27,725 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-30) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:18:43,043 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-48) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:18:43,064 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-9) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:18:55,383 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-40) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:18:58,450 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-31) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:19:02,314 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-6) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:20:00,088 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-58) [2e5174b2] Autorecovering 1 storage domains 2014-06-05 12:20:00,088 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-58) [2e5174b2] Autorecovering storage domains id: 9923a5a1-61e0-4edb-a04d-22c962190c53 2014-06-05 12:20:00,089 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-58) [106f43a5] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 9923a5a1-61e0-4edb-a04d-22c962190c53 Type: Storage 2014-06-05 12:20:00,090 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-58) [106f43a5] ConnectDomainToStorage. Before Connect all hosts to pool. Time:6/5/14 12:20 PM 2014-06-05 12:20:00,175 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-23) [4233af79] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2014-06-05 12:20:00,180 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-11) [4aa18a96] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2014-06-05 12:20:00,184 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-40) [48fedac6] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2014-06-05 12:20:00,188 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-26) [4b39df17] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2014-06-05 12:20:00,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-40) [48fedac6] START, ConnectStorageServerVDSCommand(HostName = node2.test.now, HostId = 28fdcb5d-7acd-410e-8b65-0b4f483cb575, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f47e9f99-9989-4297-a84f-4f75338eeced, connection: 10.100.100.105:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2eb75130 2014-06-05 12:20:00,321 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-26) [4b39df17] START, ConnectStorageServerVDSCommand(HostName = node1.test.now, HostId = e17a740c-47ba-4a81-99ab-6c386b572f26, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f47e9f99-9989-4297-a84f-4f75338eeced, connection: 10.100.100.105:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 957b133 2014-06-05 12:20:00,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-11) [4aa18a96] START, ConnectStorageServerVDSCommand(HostName = node3.test.now, HostId = 7415506c-cda7-4018-804d-5f6d3beddbfb, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f47e9f99-9989-4297-a84f-4f75338eeced, connection: 10.100.100.105:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 28e7a51e 2014-06-05 12:20:00,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-23) [4233af79] START, ConnectStorageServerVDSCommand(HostName = node4.test.now, HostId = bb13752e-85cb-4945-822b-48ab2a7b1329, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f47e9f99-9989-4297-a84f-4f75338eeced, connection: 10.100.100.105:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1e3eb653 2014-06-05 12:21:00,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-40) [48fedac6] FINISH, ConnectStorageServerVDSCommand, return: {f47e9f99-9989-4297-a84f-4f75338eeced=0}, log id: 2eb75130 2014-06-05 12:21:00,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-11) [4aa18a96] FINISH, ConnectStorageServerVDSCommand, return: {f47e9f99-9989-4297-a84f-4f75338eeced=0}, log id: 28e7a51e 2014-06-05 12:21:00,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-26) [4b39df17] FINISH, ConnectStorageServerVDSCommand, return: {f47e9f99-9989-4297-a84f-4f75338eeced=0}, log id: 957b133 2014-06-05 12:21:00,400 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-23) [4233af79] FINISH, ConnectStorageServerVDSCommand, return: {f47e9f99-9989-4297-a84f-4f75338eeced=0}, log id: 1e3eb653 2014-06-05 12:21:00,403 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-58) [106f43a5] ConnectDomainToStorage. After Connect all hosts to pool. Time:6/5/14 12:21 PM 2014-06-05 12:23:10,334 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-23) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:23:34,686 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) [74449d64] Attempting to update VMs/Templates Ovf. 2014-06-05 12:23:34,687 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) [74449d64] Attempting to update VM OVFs in Data Center Default 2014-06-05 12:23:34,692 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) [74449d64] Successfully updated VM OVFs in Data Center Default 2014-06-05 12:23:34,693 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) [74449d64] Attempting to update template OVFs in Data Center Default 2014-06-05 12:23:34,695 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) [74449d64] Successfully updated templates OVFs in Data Center Default 2014-06-05 12:23:34,695 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) [74449d64] Attempting to remove unneeded template/vm OVFs in Data Center Default 2014-06-05 12:23:34,697 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) [74449d64] Successfully removed unneeded template/vm OVFs in Data Center Default 2014-06-05 12:23:38,356 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-29) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:23:44,441 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-16) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:24:00,113 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (org.ovirt.thread.pool-6-thread-44) [43b031c4] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: e17a740c-47ba-4a81-99ab-6c386b572f26 Type: VDS 2014-06-05 12:24:00,146 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-44) [43b031c4] START, SetVdsStatusVDSCommand(HostName = node1.test.now, HostId = e17a740c-47ba-4a81-99ab-6c386b572f26, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 1d0d3a10 2014-06-05 12:24:00,171 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-44) [43b031c4] FINISH, SetVdsStatusVDSCommand, log id: 1d0d3a10 2014-06-05 12:24:00,460 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-6-thread-44) [43b031c4] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: e17a740c-47ba-4a81-99ab-6c386b572f26 Type: VDS 2014-06-05 12:24:00,539 INFO [org.ovirt.engine.core.bll.InternalMigrateVmCommand] (org.ovirt.thread.pool-6-thread-44) [155d3855] Lock Acquired to object EngineLock [exclusiveLocks= key: 63b1fd02-1fb5-44d1-b1cc-54cb5c0fdb0e value: VM , sharedLocks= ] 2014-06-05 12:24:00,622 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (DefaultQuartzScheduler_Worker-69) [2b68c51d] Running command: MaintenanceNumberOfVdssCommand internal: true. Entities affected : ID: e17a740c-47ba-4a81-99ab-6c386b572f26 Type: VDS 2014-06-05 12:24:00,779 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (org.ovirt.thread.pool-6-thread-44) [155d3855] Candidate host node2.test.now (28fdcb5d-7acd-410e-8b65-0b4f483cb575) was filtered out by VAR__FILTERTYPE__INTERNAL filter Memory 2014-06-05 12:24:00,808 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (org.ovirt.thread.pool-6-thread-44) [155d3855] Candidate host node3.test.now (7415506c-cda7-4018-804d-5f6d3beddbfb) was filtered out by VAR__FILTERTYPE__INTERNAL filter Memory 2014-06-05 12:24:00,809 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (org.ovirt.thread.pool-6-thread-44) [155d3855] Candidate host node4.test.now (bb13752e-85cb-4945-822b-48ab2a7b1329) was filtered out by VAR__FILTERTYPE__INTERNAL filter Memory 2014-06-05 12:24:00,813 WARN [org.ovirt.engine.core.bll.InternalMigrateVmCommand] (org.ovirt.thread.pool-6-thread-44) [155d3855] CanDoAction of action InternalMigrateVm failed. Reasons:VAR__ACTION__MIGRATE,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_MEMORY,ACTION_TYPE_FAILED_VDS_VM_MEMORY,ACTION_TYPE_FAILED_VDS_VM_MEMORY,SCHEDULING_ALL_HOSTS_FILTERED_OUT,VAR__FILTERTYPE__INTERNAL,$hostName node2.test.now,$filterName Memory,SCHEDULING_HOST_FILTERED_REASON,VAR__FILTERTYPE__INTERNAL,$hostName node4.test.now,$filterName Memory,SCHEDULING_HOST_FILTERED_REASON,VAR__FILTERTYPE__INTERNAL,$hostName node3.test.now,$filterName Memory,SCHEDULING_HOST_FILTERED_REASON 2014-06-05 12:24:00,825 INFO [org.ovirt.engine.core.bll.InternalMigrateVmCommand] (org.ovirt.thread.pool-6-thread-44) [155d3855] Lock freed to object EngineLock [exclusiveLocks= key: 63b1fd02-1fb5-44d1-b1cc-54cb5c0fdb0e value: VM , sharedLocks= ] 2014-06-05 12:24:00,860 ERROR [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (org.ovirt.thread.pool-6-thread-44) [155d3855] ResourceManager::vdsMaintenance - Failed migrating desktop Bench1 2014-06-05 12:24:00,883 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [155d3855] Correlation ID: 43b031c4, Job ID: 38a05481-0ced-4285-b3ee-6574ce39eb77, Call Stack: null, Custom Event ID: -1, Message: Host node1.test.now cannot change into maintenance mode - not all Vms have been migrated successfully. Consider manual intervention: stopping/migrating Vms: Bench1 (User: admin). 2014-06-05 12:24:00,894 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (DefaultQuartzScheduler_Worker-69) [2b68c51d] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: e17a740c-47ba-4a81-99ab-6c386b572f26 Type: VDS 2014-06-05 12:24:01,005 INFO [org.ovirt.engine.core.bll.InternalMigrateVmCommand] (DefaultQuartzScheduler_Worker-69) [36b0c4c7] Lock Acquired to object EngineLock [exclusiveLocks= key: 63b1fd02-1fb5-44d1-b1cc-54cb5c0fdb0e value: VM , sharedLocks= ] 2014-06-05 12:24:01,147 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (DefaultQuartzScheduler_Worker-69) [36b0c4c7] Candidate host node2.test.now (28fdcb5d-7acd-410e-8b65-0b4f483cb575) was filtered out by VAR__FILTERTYPE__INTERNAL filter Memory 2014-06-05 12:24:01,155 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (DefaultQuartzScheduler_Worker-69) [36b0c4c7] Candidate host node3.test.now (7415506c-cda7-4018-804d-5f6d3beddbfb) was filtered out by VAR__FILTERTYPE__INTERNAL filter Memory 2014-06-05 12:24:01,166 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (DefaultQuartzScheduler_Worker-69) [36b0c4c7] Candidate host node4.test.now (bb13752e-85cb-4945-822b-48ab2a7b1329) was filtered out by VAR__FILTERTYPE__INTERNAL filter Memory 2014-06-05 12:24:01,174 WARN [org.ovirt.engine.core.bll.InternalMigrateVmCommand] (DefaultQuartzScheduler_Worker-69) [36b0c4c7] CanDoAction of action InternalMigrateVm failed. Reasons:VAR__ACTION__MIGRATE,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_MEMORY,ACTION_TYPE_FAILED_VDS_VM_MEMORY,ACTION_TYPE_FAILED_VDS_VM_MEMORY,SCHEDULING_ALL_HOSTS_FILTERED_OUT,VAR__FILTERTYPE__INTERNAL,$hostName node2.test.now,$filterName Memory,SCHEDULING_HOST_FILTERED_REASON,VAR__FILTERTYPE__INTERNAL,$hostName node4.test.now,$filterName Memory,SCHEDULING_HOST_FILTERED_REASON,VAR__FILTERTYPE__INTERNAL,$hostName node3.test.now,$filterName Memory,SCHEDULING_HOST_FILTERED_REASON 2014-06-05 12:24:01,187 INFO [org.ovirt.engine.core.bll.InternalMigrateVmCommand] (DefaultQuartzScheduler_Worker-69) [36b0c4c7] Lock freed to object EngineLock [exclusiveLocks= key: 63b1fd02-1fb5-44d1-b1cc-54cb5c0fdb0e value: VM , sharedLocks= ] 2014-06-05 12:24:01,228 ERROR [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (DefaultQuartzScheduler_Worker-69) [36b0c4c7] ResourceManager::vdsMaintenance - Failed migrating desktop Bench1 2014-06-05 12:24:01,238 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [36b0c4c7] Correlation ID: 2b68c51d, Job ID: 3a956cf5-ec2e-45b5-a2bc-40632496fec6, Call Stack: null, Custom Event ID: -1, Message: Failed to switch Host node1.test.now to Maintenance mode. 2014-06-05 12:24:40,507 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-4) [32d3da21] Lock Acquired to object EngineLock [exclusiveLocks= key: e17a740c-47ba-4a81-99ab-6c386b572f26 value: VDS_FENCE , sharedLocks= ] 2014-06-05 12:24:40,798 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host node3.test.now from cluster Default was chosen as a proxy to execute Status command on Host node1.test.now. 2014-06-05 12:24:40,799 INFO [org.ovirt.engine.core.bll.FenceExecutor] (ajp--127.0.0.1-8702-4) Using Host node3.test.now from cluster Default as proxy to execute Status command on Host node1.test.now 2014-06-05 12:24:40,800 WARN [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-4) CanDoAction of action StopVds failed. Reasons:VAR__ACTION__STOP,VDS_STATUS_NOT_VALID_FOR_STOP 2014-06-05 12:24:40,811 INFO [org.ovirt.engine.core.bll.StopVdsCommand] (ajp--127.0.0.1-8702-4) Lock freed to object EngineLock [exclusiveLocks= key: e17a740c-47ba-4a81-99ab-6c386b572f26 value: VDS_FENCE , sharedLocks= ] 2014-06-05 12:25:00,021 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-27) [70486be2] Autorecovering 1 storage domains 2014-06-05 12:25:00,021 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-27) [70486be2] Autorecovering storage domains id: 9923a5a1-61e0-4edb-a04d-22c962190c53 2014-06-05 12:25:00,022 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-27) [70d58101] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 9923a5a1-61e0-4edb-a04d-22c962190c53 Type: Storage 2014-06-05 12:25:00,023 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-27) [70d58101] ConnectDomainToStorage. Before Connect all hosts to pool. Time:6/5/14 12:25 PM 2014-06-05 12:25:00,088 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-6) [362a5fe3] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2014-06-05 12:25:00,093 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-47) [683e82b0] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2014-06-05 12:25:00,100 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-14) [6ed26267] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2014-06-05 12:25:00,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-6) [362a5fe3] START, ConnectStorageServerVDSCommand(HostName = node4.test.now, HostId = bb13752e-85cb-4945-822b-48ab2a7b1329, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f47e9f99-9989-4297-a84f-4f75338eeced, connection: 10.100.100.105:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4591e85a 2014-06-05 12:25:00,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-14) [6ed26267] START, ConnectStorageServerVDSCommand(HostName = node2.test.now, HostId = 28fdcb5d-7acd-410e-8b65-0b4f483cb575, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f47e9f99-9989-4297-a84f-4f75338eeced, connection: 10.100.100.105:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 17ef6900 2014-06-05 12:25:00,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-47) [683e82b0] START, ConnectStorageServerVDSCommand(HostName = node3.test.now, HostId = 7415506c-cda7-4018-804d-5f6d3beddbfb, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f47e9f99-9989-4297-a84f-4f75338eeced, connection: 10.100.100.105:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: db1af1a 2014-06-05 12:25:00,207 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-2) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:25:19,989 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-42) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:26:00,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-47) [683e82b0] FINISH, ConnectStorageServerVDSCommand, return: {f47e9f99-9989-4297-a84f-4f75338eeced=0}, log id: db1af1a 2014-06-05 12:26:00,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-6) [362a5fe3] FINISH, ConnectStorageServerVDSCommand, return: {f47e9f99-9989-4297-a84f-4f75338eeced=0}, log id: 4591e85a 2014-06-05 12:26:00,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-14) [6ed26267] FINISH, ConnectStorageServerVDSCommand, return: {f47e9f99-9989-4297-a84f-4f75338eeced=0}, log id: 17ef6900 2014-06-05 12:26:00,265 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-27) [70d58101] ConnectDomainToStorage. After Connect all hosts to pool. Time:6/5/14 12:26 PM 2014-06-05 12:28:15,399 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-17) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:28:18,443 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-8) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:28:55,121 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-45) Executing a command: java.util.concurrent.FutureTask , but note that there are 1 tasks in the queue. 2014-06-05 12:29:04,927 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (DefaultQuartzScheduler_Worker-51) [51b730a8] Running command: MaintenanceNumberOfVdssCommand internal: true. Entities affected : ID: e17a740c-47ba-4a81-99ab-6c386b572f26 Type: VDS 2014-06-05 12:29:05,029 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (DefaultQuartzScheduler_Worker-51) [51b730a8] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: e17a740c-47ba-4a81-99ab-6c386b572f26 Type: VDS 2014-06-05 12:29:05,073 INFO [org.ovirt.engine.core.bll.InternalMigrateVmCommand] (DefaultQuartzScheduler_Worker-51) [63736865] Lock Acquired to object EngineLock [exclusiveLocks= key: 63b1fd02-1fb5-44d1-b1cc-54cb5c0fdb0e value: VM , sharedLocks= ] 2014-06-05 12:29:05,164 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (DefaultQuartzScheduler_Worker-51) [63736865] Candidate host node2.test.now (28fdcb5d-7acd-410e-8b65-0b4f483cb575) was filtered out by VAR__FILTERTYPE__INTERNAL filter Memory 2014-06-05 12:29:05,167 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (DefaultQuartzScheduler_Worker-51) [63736865] Candidate host node3.test.now (7415506c-cda7-4018-804d-5f6d3beddbfb) was filtered out by VAR__FILTERTYPE__INTERNAL filter Memory 2014-06-05 12:29:05,170 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (DefaultQuartzScheduler_Worker-51) [63736865] Candidate host node4.test.now (bb13752e-85cb-4945-822b-48ab2a7b1329) was filtered out by VAR__FILTERTYPE__INTERNAL filter Memory 2014-06-05 12:29:05,173 WARN [org.ovirt.engine.core.bll.InternalMigrateVmCommand] (DefaultQuartzScheduler_Worker-51) [63736865] CanDoAction of action InternalMigrateVm failed. Reasons:VAR__ACTION__MIGRATE,VAR__TYPE__VM,ACTION_TYPE_FAILED_VDS_VM_MEMORY,ACTION_TYPE_FAILED_VDS_VM_MEMORY,ACTION_TYPE_FAILED_VDS_VM_MEMORY,SCHEDULING_ALL_HOSTS_FILTERED_OUT,VAR__FILTERTYPE__INTERNAL,$hostName node2.test.now,$filterName Memory,SCHEDULING_HOST_FILTERED_REASON,VAR__FILTERTYPE__INTERNAL,$hostName node4.test.now,$filterName Memory,SCHEDULING_HOST_FILTERED_REASON,VAR__FILTERTYPE__INTERNAL,$hostName node3.test.now,$filterName Memory,SCHEDULING_HOST_FILTERED_REASON 2014-06-05 12:29:05,181 INFO [org.ovirt.engine.core.bll.InternalMigrateVmCommand] (DefaultQuartzScheduler_Worker-51) [63736865] Lock freed to object EngineLock [exclusiveLocks= key: 63b1fd02-1fb5-44d1-b1cc-54cb5c0fdb0e value: VM , sharedLocks= ] 2014-06-05 12:29:05,207 ERROR [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (DefaultQuartzScheduler_Worker-51) [63736865] ResourceManager::vdsMaintenance - Failed migrating desktop Bench1 2014-06-05 12:29:05,218 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [63736865] Correlation ID: 51b730a8, Job ID: b8441378-051c-4af0-8802-e2efb9e5d4a4, Call Stack: null, Custom Event ID: -1, Message: Failed to switch Host node1.test.now to Maintenance mode. 2014-06-05 12:30:00,018 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-52) [2e3993e6] Autorecovering 1 storage domains 2014-06-05 12:30:00,018 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-52) [2e3993e6] Autorecovering storage domains id: 9923a5a1-61e0-4edb-a04d-22c962190c53 2014-06-05 12:30:00,021 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-52) [1585ee27] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 9923a5a1-61e0-4edb-a04d-22c962190c53 Type: Storage 2014-06-05 12:30:00,024 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-52) [1585ee27] ConnectDomainToStorage. Before Connect all hosts to pool. Time:6/5/14 12:30 PM 2014-06-05 12:30:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-17) [6c0a8af9] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2014-06-05 12:30:00,121 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-26) [10056f73] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2014-06-05 12:30:00,134 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-24) [750ef0f4] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2014-06-05 12:30:00,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-17) [6c0a8af9] START, ConnectStorageServerVDSCommand(HostName = node4.test.now, HostId = bb13752e-85cb-4945-822b-48ab2a7b1329, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f47e9f99-9989-4297-a84f-4f75338eeced, connection: 10.100.100.105:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1c35901d 2014-06-05 12:30:00,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-24) [750ef0f4] START, ConnectStorageServerVDSCommand(HostName = node2.test.now, HostId = 28fdcb5d-7acd-410e-8b65-0b4f483cb575, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f47e9f99-9989-4297-a84f-4f75338eeced, connection: 10.100.100.105:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1a4d78a1 2014-06-05 12:30:00,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-26) [10056f73] START, ConnectStorageServerVDSCommand(HostName = node3.test.now, HostId = 7415506c-cda7-4018-804d-5f6d3beddbfb, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f47e9f99-9989-4297-a84f-4f75338eeced, connection: 10.100.100.105:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 60464b56 _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users