- Users - oVirt List Archives

ERROR 'no free file handlers in pool' while creating VM from template
by Tiemen Ruiten 07 Jan '15

07 Jan '15

Hello, I ran into a nasty problem today when creating a new, cloned VM from a template (one virtual 20 GBdisk) on our two-node oVirt cluster: on the node where I started a VM creation job, load skyrocketed and some VMs stopped responding until and after the job failed. Everything recovered without intervention, but this obviously shouldn't happen. I have attached the relevant vdsm log file. The button to create the VM was pressed around 11:17, the first error in the vdsm log is at 11:23:58. The ISO domain is a gluster volume exposed via NFS, the storage domain for the VM's is also a gluster volume. The underlying filesystem is ZFS. The hypervisor nodes are full CentOS 6 installs. I'm guessing the 'no free file handlers in pool' in the vdsm log file is key here. What can I do to prevent this from happening again? Apart from not creating new VMs of course :) Tiemen

3 12

Re: [ovirt-users] LDAP Certificate Location?
by Donny Davis 06 Jan '15

06 Jan '15

1 0

Re: [ovirt-users] Can't remove a storage domain related to a broken hardware
by Olivier Navas 06 Jan '15

06 Jan '15

Parts of engine.log follow. - Part 1 repeats every 5 minutes, all the time - Part 2 is when I try to detach storage domain from the GUI (mixed with some logs as in 1) plus logs related to a VM start initiated by one of my colleagues). You will notice that engine reports some VM as "not responding". It happens during the process of removing the defective storage domain, but in fact VMs are fine and their state reverts back to normal when the task of removing the storage domain finally fails. Thank you for your help. ===== part 1 ===== 2015-01-06 03:50:00,016 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-81) [5ca5ae35] Autorecovering 1 storage domains 2015-01-06 03:50:00,017 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-81) [5ca5ae35] Autorecovering storage domains id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 2015-01-06 03:50:00,020 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-81) [789ace0] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 Type: Storage 2015-01-06 03:50:00,022 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-81) [789ace0] ConnectDomainToStorage. Before Connect all hosts to pool. Time:1/6/15 3:50 AM 2015-01-06 03:50:00,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-12) [789ace0] START, ConnectStorageServerVDSCommand(HostName = n2orna, HostId = 38893029-0eb8-4d19-a28f-07680d8d6868, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 65ce62da 2015-01-06 03:50:00,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-37) [789ace0] START, ConnectStorageServerVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 50bc0a6d 2015-01-06 03:50:00,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-29) [789ace0] START, ConnectStorageServerVDSCommand(HostName = n3orna, HostId = ab744426-2294-4c0b-aaf5-08ebb162f542, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: e40ca91 2015-01-06 03:50:00,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-32) [789ace0] START, ConnectStorageServerVDSCommand(HostName = n1orna, HostId = 60d8c75b-38f1-4cd0-b162-729285eadefd, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5ddb16e 2015-01-06 03:53:00,216 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-12) [789ace0] Command ConnectStorageServerVDSCommand(HostName = n2orna, HostId = 38893029-0eb8-4d19-a28f-07680d8d6868, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 03:53:00,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-12) [789ace0] FINISH, ConnectStorageServerVDSCommand, log id: 65ce62da 2015-01-06 03:53:00,223 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-37) [789ace0] Command ConnectStorageServerVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 03:53:00,224 ERROR [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (org.ovirt.thread.pool-8-thread-12) [789ace0] Failed to connect host n2orna to storage domain (name: fujitsu_backup_rsync, id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7). Exception: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:58) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:31) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.connectStorageToDomainByVdsId(ISCSIStorageHelper.java:227) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:336) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:331) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:112) [utils.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71] 2015-01-06 03:53:00,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-37) [789ace0] FINISH, ConnectStorageServerVDSCommand, log id: 50bc0a6d 2015-01-06 03:53:00,245 ERROR [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (org.ovirt.thread.pool-8-thread-37) [789ace0] Failed to connect host n4orna to storage domain (name: fujitsu_backup_rsync, id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7). Exception: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:58) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:31) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.connectStorageToDomainByVdsId(ISCSIStorageHelper.java:227) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:336) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:331) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:112) [utils.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71] 2015-01-06 03:53:00,249 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-29) [789ace0] Command ConnectStorageServerVDSCommand(HostName = n3orna, HostId = ab744426-2294-4c0b-aaf5-08ebb162f542, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 03:53:00,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-29) [789ace0] FINISH, ConnectStorageServerVDSCommand, log id: e40ca91 2015-01-06 03:53:00,265 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-32) [789ace0] Command ConnectStorageServerVDSCommand(HostName = n1orna, HostId = 60d8c75b-38f1-4cd0-b162-729285eadefd, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 03:53:00,266 ERROR [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (org.ovirt.thread.pool-8-thread-29) [789ace0] Failed to connect host n3orna to storage domain (name: fujitsu_backup_rsync, id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7). Exception: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:58) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:31) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.connectStorageToDomainByVdsId(ISCSIStorageHelper.java:227) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:336) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:331) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:112) [utils.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71] 2015-01-06 03:53:00,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-32) [789ace0] FINISH, ConnectStorageServerVDSCommand, log id: 5ddb16e 2015-01-06 03:53:00,287 ERROR [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (org.ovirt.thread.pool-8-thread-32) [789ace0] Failed to connect host n1orna to storage domain (name: fujitsu_backup_rsync, id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7). Exception: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:58) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:31) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.connectStorageToDomainByVdsId(ISCSIStorageHelper.java:227) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:336) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:331) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:112) [utils.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71] 2015-01-06 03:53:00,301 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-81) [789ace0] ConnectDomainToStorage. After Connect all hosts to pool. Time:1/6/15 3:53 AM ===== part 2 ===== 2015-01-06 09:55:00,009 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-9) [71cf84f9] Autorecovering 1 storage domains 2015-01-06 09:55:00,011 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-9) [71cf84f9] Autorecovering storage domains id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 2015-01-06 09:55:00,013 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-9) [6ce74644] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 Type: Storage 2015-01-06 09:55:00,016 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-9) [6ce74644] ConnectDomainToStorage. Before Connect all hosts to pool. Time:1/6/15 9:55 AM 2015-01-06 09:55:00,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-49) [6ce74644] START, ConnectStorageServerVDSCommand(HostName = n2orna, HostId = 38893029-0eb8-4d19-a28f-07680d8d6868, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2bb163a4 2015-01-06 09:55:00,203 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-18) [6ce74644] START, ConnectStorageServerVDSCommand(HostName = n1orna, HostId = 60d8c75b-38f1-4cd0-b162-729285eadefd, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 49562818 2015-01-06 09:55:00,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-39) [6ce74644] START, ConnectStorageServerVDSCommand(HostName = n3orna, HostId = ab744426-2294-4c0b-aaf5-08ebb162f542, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4cb88aac 2015-01-06 09:55:00,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-7) [6ce74644] START, ConnectStorageServerVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 23eae7f9 2015-01-06 09:55:57,598 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp--127.0.0.1-8702-6) Invalid locale found in configuration: 2015-01-06 09:55:57,600 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp--127.0.0.1-8702-6) Invalid locale found in configuration: 2015-01-06 09:56:00,437 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp--127.0.0.1-8702-11) Invalid locale found in configuration: 2015-01-06 09:56:00,438 INFO [org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper] (ajp--127.0.0.1-8702-11) Invalid locale found in configuration: 2015-01-06 09:56:10,268 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-3) Running command: LoginAdminUserCommand internal: false. 2015-01-06 09:56:10,276 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin logged in. 2015-01-06 09:56:10,388 INFO [org.ovirt.engine.core.bll.aaa.LoginUserCommand] (ajp--127.0.0.1-8702-9) Running command: LoginUserCommand internal: false. 2015-01-06 09:56:24,013 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp--127.0.0.1-8702-13) [c2fb019] Lock Acquired to object EngineLock [exclusiveLocks= key: e2abb477-f807-470f-ae20-a7205e690638 value: VM , sharedLocks= ] 2015-01-06 09:56:24,134 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-13) [c2fb019] START, IsVmDuringInitiatingVDSCommand( vmId = e2abb477-f807-470f-ae20-a7205e690638), log id: 5c2ceac9 2015-01-06 09:56:24,136 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp--127.0.0.1-8702-13) [c2fb019] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 5c2ceac9 2015-01-06 09:56:24,357 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-9) [c2fb019] Running command: RunVmCommand internal: false. Entities affected : ID: e2abb477-f807-470f-ae20-a7205e690638 Type: VMAction group VM_BASIC_OPERATIONS with role type USER 2015-01-06 09:56:24,500 INFO [org.ovirt.engine.core.bll.scheduling.policyunits.HaReservationWeightPolicyUnit] (org.ovirt.thread.pool-8-thread-9) [c2fb019] Started HA reservation scoring method 2015-01-06 09:56:24,561 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-9) [c2fb019] START, UpdateVmDynamicDataVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, vmDynamic=org.ovirt.engine.core.common.businessentities.VmDynamic@57214812), log id: 63eaa18e 2015-01-06 09:56:24,573 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-8-thread-9) [c2fb019] FINISH, UpdateVmDynamicDataVDSCommand, log id: 63eaa18e 2015-01-06 09:56:24,612 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-9) [c2fb019] START, CreateVmVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, vmId=e2abb477-f807-470f-ae20-a7205e690638, vm=VM [ciril-recette]), log id: 5c6b51df 2015-01-06 09:56:24,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-9) [c2fb019] START, CreateVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, vmId=e2abb477-f807-470f-ae20-a7205e690638, vm=VM [ciril-recette]), log id: 6bcd4ecc 2015-01-06 09:56:24,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-9) [c2fb019] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand kvmEnable=true,keyboardLayout=en-us,nice=0,pitReinjection=false,displayNetwork=ovirtmgmt,copyPasteEnable=true,timeOffset=0,transparentHugePages=true,vmId=e2abb477-f807-470f-ae20-a7205e690638,acpiEnable=true,custom={device_5cc74d49-61c4-4887-8cf3-41135fbdee99device_ee0562ba-7750-4857-8151-f422de37a388device_41d6489a-07ab-481d-b35e-a14e007df6cd=VmDevice {vmId=e2abb477-f807-470f-ae20-a7205e690638, deviceId=41d6489a-07ab-481d-b35e-a14e007df6cd, device=spicevmc, type=CHANNEL, bootOrder=0, specParams={}, address={port=3, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel2, customProperties={}, snapshotId=null}, device_5cc74d49-61c4-4887-8cf3-41135fbdee99device_ee0562ba-7750-4857-8151-f422de37a388device_41d6489a-07ab-481d-b35e-a14e007df6cddevice_e3fa7b29-5b4f-462d-bcf4-298916996aaf=VmDevice {vmId=e2abb477-f807-470f-ae20-a7205e690638, deviceId=e3fa7b29-5b4f-462d-bcf4-298916996aaf, device=ide, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x01, function=0x1}, managed=false, plugged=true, readOnly=false, deviceAlias=ide0, customProperties={}, snapshotId=null}, device_5cc74d49-61c4-4887-8cf3-41135fbdee99=VmDevice {vmId=e2abb477-f807-470f-ae20-a7205e690638, deviceId=5cc74d49-61c4-4887-8cf3-41135fbdee99, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=1, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel0, customProperties={}, snapshotId=null}, device_5cc74d49-61c4-4887-8cf3-41135fbdee99device_ee0562ba-7750-4857-8151-f422de37a388=VmDevice {vmId=e2abb477-f807-470f-ae20-a7205e690638, deviceId=ee0562ba-7750-4857-8151-f422de37a388, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=2, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel1, customProperties={}, snapshotId=null}},spiceSslCipherSuite=DEFAULT,memSize=6000,smp=4,emulatedMachine=rhel6.5.0,vmType=kvm,memGuaranteedSize=1500,display=qxl,smartcardEnable=false,bootMenuEnable=false,numaTune={mode=interleave, nodeset=0,1},spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,smpCoresPerSocket=1,maxVCpus=16,devices=[{address={bus=0x00, domain=0x0000, slot=0x02, type=pci, function=0x0}, specParams={ram=65536, vram=32768, heads=1}, device=qxl, type=video, deviceId=061ed0c0-22f5-4099-8bee-44a3dfa19956}, {shared=false, iface=ide, index=2, address={unit=0, bus=1, target=0, controller=0, type=drive}, specParams={path=}, path=, device=cdrom, type=disk, readonly=true, deviceId=dd759f21-7cca-4d56-8f0f-348d61492714}, {shared=false, index=0, volumeID=82528148-a4db-4892-87e6-b41c7b2f3ab8, propagateErrors=off, format=raw, type=disk, iface=virtio, bootOrder=1, address={bus=0x00, domain=0x0000, slot=0x06, type=pci, function=0x0}, domainID=05d4e406-de6d-47dc-8414-bdc7381b6d4a, imageID=f09ba312-5078-4070-801c-4a8c7e28d0aa, specParams={}, optional=false, device=disk, poolID=00000002-0002-0002-0002-00000000000f, readonly=false, deviceId=f09ba312-5078-4070-801c-4a8c7e28d0aa}, {nicModel=pv, address={bus=0x00, domain=0x0000, slot=0x03, type=pci, function=0x0}, specParams={outbound={}, inbound={}}, macAddr=00:1a:4a:70:41:a3, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=Cluster_VM, deviceId=b2e197f0-00f4-44f4-bd5c-32824d872c95}, {address={bus=0x00, domain=0x0000, slot=0x07, type=pci, function=0x0}, specParams={model=virtio}, device=memballoon, type=balloon, deviceId=e0a803e7-eb97-4285-a49f-6e99a5df974c}, {index=0, model=virtio-scsi, address={bus=0x00, domain=0x0000, slot=0x04, type=pci, function=0x0}, specParams={}, device=scsi, type=controller, deviceId=bd1cf13f-212b-4ceb-96c7-ee4950f7a7d3}, {address={bus=0x00, domain=0x0000, slot=0x05, type=pci, function=0x0}, specParams={}, device=virtio-serial, type=controller, deviceId=a2df4f4f-2920-4243-ba9a-26206057de6e}],vmName=ciril-recette,cpuType=Westmere,fileTransferEnable=true 2015-01-06 09:56:24,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-8-thread-9) [c2fb019] FINISH, CreateVDSCommand, log id: 6bcd4ecc 2015-01-06 09:56:24,852 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-8-thread-9) [c2fb019] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 5c6b51df 2015-01-06 09:56:24,854 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-8-thread-9) [c2fb019] Lock freed to object EngineLock [exclusiveLocks= key: e2abb477-f807-470f-ae20-a7205e690638 value: VM , sharedLocks= ] 2015-01-06 09:56:24,873 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-9) [c2fb019] Correlation ID: c2fb019, Job ID: 42d86800-fc18-42d6-98a6-0cf205a85a2c, Call Stack: null, Custom Event ID: -1, Message: VM ciril-recette was started by admin (Host: n4orna). 2015-01-06 09:56:28,296 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) [128f72d7] VM ciril-recette e2abb477-f807-470f-ae20-a7205e690638 moved from WaitForLaunch --> PoweringUp 2015-01-06 09:56:28,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-73) [128f72d7] START, FullListVdsCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, vds=Host[n4orna,920bf64c-62f5-4b12-a69e-eef9936576c5], vmIds=[e2abb477-f807-470f-ae20-a7205e690638]), log id: 79d27c1c 2015-01-06 09:56:28,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-73) [128f72d7] FINISH, FullListVdsCommand, return: [{displaySecurePort=5909, kvmEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.99.23.4, pauseCode=NOERR, pitReinjection=false, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true, vmId=e2abb477-f807-470f-ae20-a7205e690638, acpiEnable=true, custom={device_5cc74d49-61c4-4887-8cf3-41135fbdee99device_ee0562ba-7750-4857-8151-f422de37a388device_41d6489a-07ab-481d-b35e-a14e007df6cd=VmDevice {vmId=e2abb477-f807-470f-ae20-a7205e690638, deviceId=41d6489a-07ab-481d-b35e-a14e007df6cd, device=spicevmc, type=CHANNEL, bootOrder=0, specParams={}, address={port=3, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel2, customProperties={}, snapshotId=null}, device_5cc74d49-61c4-4887-8cf3-41135fbdee99device_ee0562ba-7750-4857-8151-f422de37a388device_41d6489a-07ab-481d-b35e-a14e007df6cddevice_e3fa7b29-5b4f-462d-bcf4-298916996aaf=VmDevice {vmId=e2abb477-f807-470f-ae20-a7205e690638, deviceId=e3fa7b29-5b4f-462d-bcf4-298916996aaf, device=ide, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x01, function=0x1}, managed=false, plugged=true, readOnly=false, deviceAlias=ide0, customProperties={}, snapshotId=null}, device_5cc74d49-61c4-4887-8cf3-41135fbdee99=VmDevice {vmId=e2abb477-f807-470f-ae20-a7205e690638, deviceId=5cc74d49-61c4-4887-8cf3-41135fbdee99, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=1, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel0, customProperties={}, snapshotId=null}, device_5cc74d49-61c4-4887-8cf3-41135fbdee99device_ee0562ba-7750-4857-8151-f422de37a388=VmDevice {vmId=e2abb477-f807-470f-ae20-a7205e690638, deviceId=ee0562ba-7750-4857-8151-f422de37a388, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=2, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel1, customProperties={}, snapshotId=null}}, spiceSslCipherSuite=DEFAULT, memSize=6000, smp=4, displayPort=-1, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up, memGuaranteedSize=1500, display=qxl, pid=23762, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numaTune={nodeset=0,1, mode=interleave}, smpCoresPerSocket=1, maxVCpus=16, clientIp=, devices=[Ljava.lang.Object;@69d250bf, vmName=ciril-recette, fileTransferEnable=true, cpuType=Westmere}], log id: 79d27c1c 2015-01-06 09:56:28,328 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) [128f72d7] Received a spice Device without an address when processing VM e2abb477-f807-470f-ae20-a7205e690638 devices, skipping device: {specParams={spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayNetwork=ovirtmgmt, copyPasteEnable=true, displayIp=10.99.23.4}, device=spice, tlsPort=5909, type=graphics} 2015-01-06 09:56:46,936 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) [27e4f6a6] VM ciril-recette e2abb477-f807-470f-ae20-a7205e690638 moved from PoweringUp --> Up 2015-01-06 09:56:47,065 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [27e4f6a6] Correlation ID: c2fb019, Job ID: 42d86800-fc18-42d6-98a6-0cf205a85a2c, Call Stack: null, Custom Event ID: -1, Message: VM ciril-recette started on Host n4orna 2015-01-06 09:57:57,112 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (ajp--127.0.0.1-8702-6) [631b7fb5] Lock Acquired to object EngineLock [exclusiveLocks= key: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 value: STORAGE , sharedLocks= ] 2015-01-06 09:57:57,517 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-17) [631b7fb5] Running command: DetachStorageDomainFromPoolCommand internal: false. Entities affected : ID: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2015-01-06 09:57:57,520 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-17) [631b7fb5] Start detach storage domain 2015-01-06 09:57:57,530 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-17) [631b7fb5] Detach storage domain: before connect 2015-01-06 09:57:57,720 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-50) [631b7fb5] START, ConnectStorageServerVDSCommand(HostName = n2orna, HostId = 38893029-0eb8-4d19-a28f-07680d8d6868, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 13e1b92c 2015-01-06 09:57:57,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-21) [631b7fb5] START, ConnectStorageServerVDSCommand(HostName = n1orna, HostId = 60d8c75b-38f1-4cd0-b162-729285eadefd, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7b9a83cf 2015-01-06 09:57:57,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-37) [631b7fb5] START, ConnectStorageServerVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2a3e2925 2015-01-06 09:57:57,759 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-16) [631b7fb5] START, ConnectStorageServerVDSCommand(HostName = n3orna, HostId = ab744426-2294-4c0b-aaf5-08ebb162f542, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6386df71 2015-01-06 09:58:00,209 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-49) [6ce74644] Command ConnectStorageServerVDSCommand(HostName = n2orna, HostId = 38893029-0eb8-4d19-a28f-07680d8d6868, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 09:58:00,214 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-18) [6ce74644] Command ConnectStorageServerVDSCommand(HostName = n1orna, HostId = 60d8c75b-38f1-4cd0-b162-729285eadefd, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 09:58:00,220 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-39) [6ce74644] Command ConnectStorageServerVDSCommand(HostName = n3orna, HostId = ab744426-2294-4c0b-aaf5-08ebb162f542, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 09:58:00,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-49) [6ce74644] FINISH, ConnectStorageServerVDSCommand, log id: 2bb163a4 2015-01-06 09:58:00,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-39) [6ce74644] FINISH, ConnectStorageServerVDSCommand, log id: 4cb88aac 2015-01-06 09:58:00,225 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-7) [6ce74644] Command ConnectStorageServerVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 09:58:00,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-18) [6ce74644] FINISH, ConnectStorageServerVDSCommand, log id: 49562818 2015-01-06 09:58:00,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-7) [6ce74644] FINISH, ConnectStorageServerVDSCommand, log id: 23eae7f9 2015-01-06 09:58:00,230 ERROR [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (org.ovirt.thread.pool-8-thread-39) [6ce74644] Failed to connect host n3orna to storage domain (name: fujitsu_backup_rsync, id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7). Exception: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:58) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:31) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.connectStorageToDomainByVdsId(ISCSIStorageHelper.java:227) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:336) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:331) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:112) [utils.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71] 2015-01-06 09:58:00,228 ERROR [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (org.ovirt.thread.pool-8-thread-49) [6ce74644] Failed to connect host n2orna to storage domain (name: fujitsu_backup_rsync, id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7). Exception: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:58) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:31) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.connectStorageToDomainByVdsId(ISCSIStorageHelper.java:227) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:336) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:331) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:112) [utils.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71] 2015-01-06 09:58:00,239 ERROR [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (org.ovirt.thread.pool-8-thread-7) [6ce74644] Failed to connect host n4orna to storage domain (name: fujitsu_backup_rsync, id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7). Exception: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:58) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:31) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.connectStorageToDomainByVdsId(ISCSIStorageHelper.java:227) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:336) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:331) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:112) [utils.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71] 2015-01-06 09:58:00,237 ERROR [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (org.ovirt.thread.pool-8-thread-18) [6ce74644] Failed to connect host n1orna to storage domain (name: fujitsu_backup_rsync, id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7). Exception: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:58) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:31) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.connectStorageToDomainByVdsId(ISCSIStorageHelper.java:227) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:336) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:331) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:112) [utils.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71] 2015-01-06 09:58:00,294 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-9) [6ce74644] ConnectDomainToStorage. After Connect all hosts to pool. Time:1/6/15 9:58 AM 2015-01-06 09:58:03,032 INFO [org.ovirt.engine.core.bll.storage.SyncLunsInfoForBlockStorageDomainCommand] (org.ovirt.thread.pool-8-thread-6) [4205431d] Running command: SyncLunsInfoForBlockStorageDomainCommand internal: true. Entities affected : ID: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 Type: Storage 2015-01-06 09:58:03,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-8-thread-6) [4205431d] START, GetVGInfoVDSCommand(HostName = n2orna, HostId = 38893029-0eb8-4d19-a28f-07680d8d6868, VGID=Zqoys8-LKeb-QJIG-TRDR-dnkP-0RWd-ftbOom), log id: a46928a 2015-01-06 09:58:03,225 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-8-thread-6) [4205431d] Failed in GetVGInfoVDS method 2015-01-06 09:58:03,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-8-thread-6) [4205431d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand return value OneVGReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=506, mMessage=Volume Group does not exist: ('vg_uuid: Zqoys8-LKeb-QJIG-TRDR-dnkP-0RWd-ftbOom',)]] 2015-01-06 09:58:03,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-8-thread-6) [4205431d] HostName = n2orna 2015-01-06 09:58:03,231 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-8-thread-6) [4205431d] Command GetVGInfoVDSCommand(HostName = n2orna, HostId = 38893029-0eb8-4d19-a28f-07680d8d6868, VGID=Zqoys8-LKeb-QJIG-TRDR-dnkP-0RWd-ftbOom) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetVGInfoVDS, error = Volume Group does not exist: ('vg_uuid: Zqoys8-LKeb-QJIG-TRDR-dnkP-0RWd-ftbOom',), code = 506 2015-01-06 09:58:03,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-8-thread-6) [4205431d] FINISH, GetVGInfoVDSCommand, log id: a46928a 2015-01-06 09:58:03,236 ERROR [org.ovirt.engine.core.bll.storage.SyncLunsInfoForBlockStorageDomainCommand] (org.ovirt.thread.pool-8-thread-6) [4205431d] Command org.ovirt.engine.core.bll.storage.SyncLunsInfoForBlockStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetVGInfoVDS, error = Volume Group does not exist: ('vg_uuid: Zqoys8-LKeb-QJIG-TRDR-dnkP-0RWd-ftbOom',), code = 506 (Failed with error VolumeGroupDoesNotExist and code 506) 2015-01-06 09:58:06,526 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) domain 34c95a44-db7f-4d0f-ba13-5f06a7feefe7:fujitsu_backup_rsync in problem. vds: n1orna 2015-01-06 09:58:06,529 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) Host 60d8c75b-38f1-4cd0-b162-729285eadefd has reported new storage access problem to the following domains 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 marking it for storage connections and pool metadata refresh (report id: 6a081ad8-201f-48e4-ba56-5a581fac4475) 2015-01-06 09:58:06,559 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-49) Host 38893029-0eb8-4d19-a28f-07680d8d6868 has reported new storage access problem to the following domains 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 marking it for storage connections and pool metadata refresh (report id: 922d1075-c975-4d9f-87d1-db8d5aa4059d) 2015-01-06 09:58:06,644 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-57) [6e3156c6] Running storage connections refresh for hosts [38893029-0eb8-4d19-a28f-07680d8d6868, 60d8c75b-38f1-4cd0-b162-729285eadefd] 2015-01-06 09:58:06,648 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (org.ovirt.thread.pool-8-thread-1) [265f9df9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-00000000000f Type: StoragePool 2015-01-06 09:58:06,648 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (org.ovirt.thread.pool-8-thread-42) [4f7c5a1d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-00000000000f Type: StoragePool 2015-01-06 09:58:06,700 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-24) Host 920bf64c-62f5-4b12-a69e-eef9936576c5 has reported new storage access problem to the following domains 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 marking it for storage connections and pool metadata refresh (report id: 12dfe3fe-966a-486e-8eda-89575648545f) 2015-01-06 09:58:06,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-1) [265f9df9] START, ConnectStorageServerVDSCommand(HostName = n1orna, HostId = 60d8c75b-38f1-4cd0-b162-729285eadefd, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 70a8393b 2015-01-06 09:58:06,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-42) [4f7c5a1d] START, ConnectStorageServerVDSCommand(HostName = n2orna, HostId = 38893029-0eb8-4d19-a28f-07680d8d6868, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 396e347 2015-01-06 09:58:15,975 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-31) Host ab744426-2294-4c0b-aaf5-08ebb162f542 has reported new storage access problem to the following domains 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 marking it for storage connections and pool metadata refresh (report id: 18fffe35-403c-413e-acb1-8d173aa447da) 2015-01-06 09:58:52,862 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-1) [7face9e1] VM ciril-recette e2abb477-f807-470f-ae20-a7205e690638 moved from Up --> RebootInProgress 2015-01-06 09:59:23,829 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) [4c461177] VM ciril-recette e2abb477-f807-470f-ae20-a7205e690638 moved from RebootInProgress --> Up 2015-01-06 09:59:48,453 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) domain 05d4e406-de6d-47dc-8414-bdc7381b6d4a:MSA1-O2-vd01 in problem. vds: n3orna 2015-01-06 10:00:34,503 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) [289a4e02] VM ldap-recette ad20808d-b62c-4087-a41a-31ed7672e364 moved from Up --> NotResponding 2015-01-06 10:00:34,550 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) [289a4e02] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM ldap-recette is not responding. 2015-01-06 10:00:34,572 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) [289a4e02] VM ciril 627fbbe4-4812-4218-a512-cc1ed26124fd moved from Up --> NotResponding 2015-01-06 10:00:34,618 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) [289a4e02] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM ciril is not responding. 2015-01-06 10:00:50,138 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) [50e93ebc] VM antivirus2 1437b20a-d3cb-4537-ace5-2dee650f561f moved from Up --> NotResponding 2015-01-06 10:00:50,183 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [50e93ebc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM antivirus2 is not responding. 2015-01-06 10:00:50,188 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) [50e93ebc] VM rsst-formation 064f2b1f-f245-4f34-9747-27c28ce388df moved from Up --> NotResponding 2015-01-06 10:00:50,257 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [50e93ebc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM rsst-formation is not responding. 2015-01-06 10:00:50,262 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) [50e93ebc] VM forum c94e5f1f-1f10-47e5-b4bb-87d1ef31caf9 moved from Up --> NotResponding 2015-01-06 10:00:50,303 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [50e93ebc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM forum is not responding. 2015-01-06 10:00:57,729 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-50) [631b7fb5] Command ConnectStorageServerVDSCommand(HostName = n2orna, HostId = 38893029-0eb8-4d19-a28f-07680d8d6868, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 10:00:57,737 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-21) [631b7fb5] Command ConnectStorageServerVDSCommand(HostName = n1orna, HostId = 60d8c75b-38f1-4cd0-b162-729285eadefd, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 10:00:57,745 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-37) [631b7fb5] Command ConnectStorageServerVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 10:00:57,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-50) [631b7fb5] FINISH, ConnectStorageServerVDSCommand, log id: 13e1b92c 2015-01-06 10:00:57,760 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-37) [631b7fb5] FINISH, ConnectStorageServerVDSCommand, log id: 2a3e2925 2015-01-06 10:00:57,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-21) [631b7fb5] FINISH, ConnectStorageServerVDSCommand, log id: 7b9a83cf 2015-01-06 10:00:57,764 ERROR [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-37) [631b7fb5] Failed to connect host n4orna to storage domain (name: fujitsu_backup_rsync, id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7). Exception: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:58) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:31) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.connectStorageToDomainByVdsId(ISCSIStorageHelper.java:227) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:336) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:331) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:112) [utils.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71] 2015-01-06 10:00:57,762 ERROR [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-50) [631b7fb5] Failed to connect host n2orna to storage domain (name: fujitsu_backup_rsync, id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7). Exception: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:58) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:31) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.connectStorageToDomainByVdsId(ISCSIStorageHelper.java:227) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:336) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:331) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:112) [utils.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71] 2015-01-06 10:00:57,767 ERROR [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-21) [631b7fb5] Failed to connect host n1orna to storage domain (name: fujitsu_backup_rsync, id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7). Exception: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:58) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:31) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.connectStorageToDomainByVdsId(ISCSIStorageHelper.java:227) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:336) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:331) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:112) [utils.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71] 2015-01-06 10:00:57,767 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-16) [631b7fb5] Command ConnectStorageServerVDSCommand(HostName = n3orna, HostId = ab744426-2294-4c0b-aaf5-08ebb162f542, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 10:00:57,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-16) [631b7fb5] FINISH, ConnectStorageServerVDSCommand, log id: 6386df71 2015-01-06 10:00:57,871 ERROR [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-16) [631b7fb5] Failed to connect host n3orna to storage domain (name: fujitsu_backup_rsync, id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7). Exception: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:58) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.runConnectionStorageToDomain(ISCSIStorageHelper.java:31) [bll.jar:] at org.ovirt.engine.core.bll.storage.ISCSIStorageHelper.connectStorageToDomainByVdsId(ISCSIStorageHelper.java:227) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:336) [bll.jar:] at org.ovirt.engine.core.bll.storage.StorageDomainCommandBase$3.call(StorageDomainCommandBase.java:331) [bll.jar:] at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:112) [utils.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71] 2015-01-06 10:00:57,902 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-17) [631b7fb5] Detach storage domain: after connect 2015-01-06 10:00:57,906 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-17) [631b7fb5] START, DetachStorageDomainVDSCommand( storagePoolId = 00000002-0002-0002-0002-00000000000f, ignoreFailoverLimit = false, storageDomainId = 34c95a44-db7f-4d0f-ba13-5f06a7feefe7, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1, force = false), log id: 7e14d5fa 2015-01-06 10:01:05,877 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) [3c054f3d] VM voip-admin 34d7033b-72bd-4c61-9a24-a73c5541955c moved from Up --> NotResponding 2015-01-06 10:01:05,925 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [3c054f3d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM voip-admin is not responding. 2015-01-06 10:01:06,770 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-1) [265f9df9] Command ConnectStorageServerVDSCommand(HostName = n1orna, HostId = 60d8c75b-38f1-4cd0-b162-729285eadefd, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 10:01:06,776 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-42) [4f7c5a1d] Command ConnectStorageServerVDSCommand(HostName = n2orna, HostId = 38893029-0eb8-4d19-a28f-07680d8d6868, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]) execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException 2015-01-06 10:01:06,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-1) [265f9df9] FINISH, ConnectStorageServerVDSCommand, log id: 70a8393b 2015-01-06 10:01:06,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-42) [4f7c5a1d] FINISH, ConnectStorageServerVDSCommand, log id: 396e347 2015-01-06 10:01:06,799 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (org.ovirt.thread.pool-8-thread-1) [265f9df9] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) 2015-01-06 10:01:06,801 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (org.ovirt.thread.pool-8-thread-42) [4f7c5a1d] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) 2015-01-06 10:01:06,813 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-57) [6e3156c6] Submitting to the event queue pool refresh for hosts [38893029-0eb8-4d19-a28f-07680d8d6868, 60d8c75b-38f1-4cd0-b162-729285eadefd] 2015-01-06 10:01:06,816 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-37) Running storage pool metadata refresh for hosts {1} 2015-01-06 10:01:06,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-47) START, ConnectStoragePoolVDSCommand(HostName = n2orna, HostId = 38893029-0eb8-4d19-a28f-07680d8d6868, vdsId = 38893029-0eb8-4d19-a28f-07680d8d6868, storagePoolId = 00000002-0002-0002-0002-00000000000f, masterVersion = 1), log id: 377e2518 2015-01-06 10:01:06,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-26) START, ConnectStoragePoolVDSCommand(HostName = n1orna, HostId = 60d8c75b-38f1-4cd0-b162-729285eadefd, vdsId = 60d8c75b-38f1-4cd0-b162-729285eadefd, storagePoolId = 00000002-0002-0002-0002-00000000000f, masterVersion = 1), log id: 2d074200 2015-01-06 10:01:15,895 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-17) [631b7fb5] Failed in DetachStorageDomainVDS method 2015-01-06 10:01:15,898 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-8-thread-17) [631b7fb5] IrsBroker::Failed::DetachStorageDomainVDS due to: IRSErrorException: IRSGenericException: IRSErrorException: Failed to DetachStorageDomainVDS, error = Storage domain does not exist: ('34c95a44-db7f-4d0f-ba13-5f06a7feefe7',), code = 358 2015-01-06 10:01:21,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-26) FINISH, ConnectStoragePoolVDSCommand, log id: 2d074200 2015-01-06 10:01:22,578 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) [2ba9b406] VM packetfence adef4f71-d322-4307-b1d1-c0988bd4efcb moved from Up --> NotResponding 2015-01-06 10:01:22,624 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [2ba9b406] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM packetfence is not responding. 2015-01-06 10:01:22,629 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) [2ba9b406] VM afs1 2256fed4-ff9e-4520-9e89-fe0bca30ec1b moved from Up --> NotResponding 2015-01-06 10:01:22,672 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [2ba9b406] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM afs1 is not responding. 2015-01-06 10:01:22,676 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) [2ba9b406] VM piros e22277e0-723f-40b6-a46d-ca8f821807e7 moved from Up --> NotResponding 2015-01-06 10:01:22,719 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [2ba9b406] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM piros is not responding. 2015-01-06 10:03:21,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-17) [631b7fb5] START, SpmStopVDSCommand(HostName = n2orna, HostId = 38893029-0eb8-4d19-a28f-07680d8d6868, storagePoolId = 00000002-0002-0002-0002-00000000000f), log id: 15cd9c64 2015-01-06 10:03:22,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-17) [631b7fb5] SpmStopVDSCommand::Stopping SPM on vds n2orna, pool id 00000002-0002-0002-0002-00000000000f 2015-01-06 10:03:22,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 377e2518 2015-01-06 10:03:22,037 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-3) domain 75a3e25a-2ea6-4b95-9d9f-9fe791e38e97:MSA2-O2 in problem. vds: n2orna 2015-01-06 10:03:22,047 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) starting processDomainRecovery for domain 34c95a44-db7f-4d0f-ba13-5f06a7feefe7:fujitsu_backup_rsync 2015-01-06 10:03:22,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-17) [631b7fb5] FINISH, SpmStopVDSCommand, log id: 15cd9c64 2015-01-06 10:03:22,181 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [631b7fb5] Irs placed on server 38893029-0eb8-4d19-a28f-07680d8d6868 failed. Proceed Failover 2015-01-06 10:03:22,219 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) Domain 34c95a44-db7f-4d0f-ba13-5f06a7feefe7:fujitsu_backup_rsync was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. 2015-01-06 10:03:22,251 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-46) [74a0283] Failed to Acquire Lock to object EngineLock [exclusiveLocks= key: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 value: STORAGE , sharedLocks= key: 00000002-0002-0002-0002-00000000000f value: POOL ] 2015-01-06 10:03:22,258 WARN [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-8-thread-46) [74a0283] CanDoAction of action DeactivateStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__DEACTIVATE,ACTION_TYPE_FAILED_OBJECT_LOCKED 2015-01-06 10:03:22,279 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [74a0283] Removing vds [] from the domain in maintenance cache 2015-01-06 10:03:22,282 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-46) [74a0283] Removing host(s) [] from hosts unseen domain report cache 2015-01-06 10:03:22,300 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (org.ovirt.thread.pool-8-thread-17) [422d495e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-00000000000f Type: StoragePool 2015-01-06 10:03:22,306 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (org.ovirt.thread.pool-8-thread-17) [422d495e] Storage Pool 00000002-0002-0002-0002-00000000000f - Updating Storage Domain 05d4e406-de6d-47dc-8414-bdc7381b6d4a status from Active to Unknown, reason : null 2015-01-06 10:03:22,311 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (org.ovirt.thread.pool-8-thread-17) [422d495e] Storage Pool 00000002-0002-0002-0002-00000000000f - Updating Storage Domain 75a3e25a-2ea6-4b95-9d9f-9fe791e38e97 status from Active to Unknown, reason : null 2015-01-06 10:03:22,315 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (org.ovirt.thread.pool-8-thread-17) [422d495e] Storage Pool 00000002-0002-0002-0002-00000000000f - Updating Storage Domain 5a8e48d4-25f0-46f4-9682-e92785e9057a status from Active to Unknown, reason : null 2015-01-06 10:03:22,319 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (org.ovirt.thread.pool-8-thread-17) [422d495e] Storage Pool 00000002-0002-0002-0002-00000000000f - Updating Storage Domain 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 status from Active to Unknown, reason : null 2015-01-06 10:03:22,363 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-17) [422d495e] Correlation ID: 422d495e, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete. 2015-01-06 10:03:22,381 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [422d495e] hostFromVds::selectedVds - n3orna, spmStatus Free, storage pool Default 2015-01-06 10:03:22,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [422d495e] starting spm on vds n3orna, storage pool Default, prevId -1, LVER -1 2015-01-06 10:03:22,418 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (org.ovirt.thread.pool-8-thread-17) [422d495e] START, SpmStartVDSCommand(HostName = n3orna, HostId = ab744426-2294-4c0b-aaf5-08ebb162f542, storagePoolId = 00000002-0002-0002-0002-00000000000f, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 51aadbb5 2015-01-06 10:03:22,446 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (org.ovirt.thread.pool-8-thread-17) [422d495e] spmStart polling started: taskId = 58d4fff4-d230-4c6b-aae3-62b7ff427098 2015-01-06 10:03:26,697 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) [7ad552fa] VM ldap-recette ad20808d-b62c-4087-a41a-31ed7672e364 moved from NotResponding --> Up 2015-01-06 10:03:26,699 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) [7ad552fa] VM voip-admin 34d7033b-72bd-4c61-9a24-a73c5541955c moved from NotResponding --> Up 2015-01-06 10:03:26,702 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) [7ad552fa] VM packetfence adef4f71-d322-4307-b1d1-c0988bd4efcb moved from NotResponding --> Up 2015-01-06 10:03:26,704 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) [7ad552fa] VM afs1 2256fed4-ff9e-4520-9e89-fe0bca30ec1b moved from NotResponding --> Up 2015-01-06 10:03:26,707 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) [7ad552fa] VM piros e22277e0-723f-40b6-a46d-ca8f821807e7 moved from NotResponding --> Up 2015-01-06 10:03:26,709 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) [7ad552fa] VM ciril 627fbbe4-4812-4218-a512-cc1ed26124fd moved from NotResponding --> Up 2015-01-06 10:03:26,712 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) [7ad552fa] VM antivirus2 1437b20a-d3cb-4537-ace5-2dee650f561f moved from NotResponding --> Up 2015-01-06 10:03:26,714 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) [7ad552fa] VM rsst-formation 064f2b1f-f245-4f34-9747-27c28ce388df moved from NotResponding --> Up 2015-01-06 10:03:26,716 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) [7ad552fa] VM forum c94e5f1f-1f10-47e5-b4bb-87d1ef31caf9 moved from NotResponding --> Up 2015-01-06 10:03:28,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (org.ovirt.thread.pool-8-thread-17) [422d495e] spmStart polling ended: taskId = 58d4fff4-d230-4c6b-aae3-62b7ff427098 task status = finished 2015-01-06 10:03:28,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (org.ovirt.thread.pool-8-thread-17) [422d495e] spmStart polling ended, spm status: SPM 2015-01-06 10:03:28,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-17) [422d495e] START, HSMClearTaskVDSCommand(HostName = n3orna, HostId = ab744426-2294-4c0b-aaf5-08ebb162f542, taskId=58d4fff4-d230-4c6b-aae3-62b7ff427098), log id: 3c3c7472 2015-01-06 10:03:28,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-17) [422d495e] FINISH, HSMClearTaskVDSCommand, log id: 3c3c7472 2015-01-06 10:03:28,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (org.ovirt.thread.pool-8-thread-17) [422d495e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@146b8db3, log id: 51aadbb5 2015-01-06 10:03:28,212 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [422d495e] Initialize Irs proxy from vds: 10.99.23.3 2015-01-06 10:03:28,248 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-17) [422d495e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host n3orna (Address: 10.99.23.3). 2015-01-06 10:03:28,273 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) [422d495e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-00000000000f, ignoreFailoverLimit = false), log id: 5be8ceb1 2015-01-06 10:03:28,530 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-17) [422d495e] Failed in DetachStorageDomainVDS method 2015-01-06 10:03:28,533 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-8-thread-17) [422d495e] IrsBroker::Failed::DetachStorageDomainVDS due to: IRSErrorException: IRSGenericException: IRSErrorException: Failed to DetachStorageDomainVDS, error = Storage domain does not exist: ('34c95a44-db7f-4d0f-ba13-5f06a7feefe7',), code = 358 2015-01-06 10:03:28,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-17) [422d495e] START, SpmStopVDSCommand(HostName = n3orna, HostId = ab744426-2294-4c0b-aaf5-08ebb162f542, storagePoolId = 00000002-0002-0002-0002-00000000000f), log id: 430bdc38 2015-01-06 10:03:28,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-17) [422d495e] SpmStopVDSCommand::Stopping SPM on vds n3orna, pool id 00000002-0002-0002-0002-00000000000f 2015-01-06 10:03:28,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-17) [422d495e] FINISH, SpmStopVDSCommand, log id: 430bdc38 2015-01-06 10:03:28,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [422d495e] Irs placed on server ab744426-2294-4c0b-aaf5-08ebb162f542 failed. Proceed Failover 2015-01-06 10:03:29,052 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-00000000000f Type: StoragePool 2015-01-06 10:03:29,099 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-17) [18526185] Correlation ID: 18526185, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete. 2015-01-06 10:03:29,113 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [18526185] hostFromVds::selectedVds - n4orna, spmStatus Free, storage pool Default 2015-01-06 10:03:29,123 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [18526185] starting spm on vds n4orna, storage pool Default, prevId -1, LVER -1 2015-01-06 10:03:29,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] START, SpmStartVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, storagePoolId = 00000002-0002-0002-0002-00000000000f, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3992a29c 2015-01-06 10:03:29,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] spmStart polling started: taskId = d2c1864d-c71c-4378-a4a5-97d0afafd775 2015-01-06 10:03:31,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] spmStart polling ended: taskId = d2c1864d-c71c-4378-a4a5-97d0afafd775 task status = finished 2015-01-06 10:03:31,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] spmStart polling ended, spm status: SPM 2015-01-06 10:03:31,518 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] START, HSMClearTaskVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, taskId=d2c1864d-c71c-4378-a4a5-97d0afafd775), log id: 29266dd2 2015-01-06 10:03:31,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] FINISH, HSMClearTaskVDSCommand, log id: 29266dd2 2015-01-06 10:03:31,528 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@59cd7a77, log id: 3992a29c 2015-01-06 10:03:31,539 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) [18526185] Initialize Irs proxy from vds: 10.99.23.4 2015-01-06 10:03:31,575 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-17) [18526185] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host n4orna (Address: 10.99.23.4). 2015-01-06 10:03:31,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [18526185] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 00000002-0002-0002-0002-00000000000f, ignoreFailoverLimit = false), log id: 5e2b03ab 2015-01-06 10:03:31,711 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] Failed in DetachStorageDomainVDS method 2015-01-06 10:03:31,714 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] IrsBroker::Failed::DetachStorageDomainVDS due to: IRSErrorException: IRSGenericException: IRSErrorException: Failed to DetachStorageDomainVDS, error = Storage domain does not exist: ('34c95a44-db7f-4d0f-ba13-5f06a7feefe7',), code = 358 2015-01-06 10:03:31,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] FINISH, DetachStorageDomainVDSCommand, log id: 7e14d5fa 2015-01-06 10:03:31,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [18526185] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-00000000000f 2015-01-06 10:03:31,721 ERROR [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] Command org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException: IRSErrorException: Failed to DetachStorageDomainVDS, error = Storage domain does not exist: ('34c95a44-db7f-4d0f-ba13-5f06a7feefe7',), code = 358 (Failed with error StorageDomainDoesNotExist and code 358) 2015-01-06 10:03:31,741 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] Command [id=3b4688b2-b626-470c-9ac9-4a65b0ef0864]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: EntityStatusSnapshot [id=storagePoolId = 00000002-0002-0002-0002-00000000000f, storageId = 34c95a44-db7f-4d0f-ba13-5f06a7feefe7, status=Inactive]. 2015-01-06 10:03:31,761 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-17) [18526185] Correlation ID: 631b7fb5, Job ID: 780fb013-3a4c-4cc7-9bfd-a755989a6eeb, Call Stack: null, Custom Event ID: -1, Message: Failed to detach Storage Domain fujitsu_backup_rsync to Data Center Default. (User: admin) 2015-01-06 10:03:31,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [18526185] START, HSMGetAllTasksInfoVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5), log id: 32a82db7 2015-01-06 10:03:31,773 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (org.ovirt.thread.pool-8-thread-17) [18526185] Lock freed to object EngineLock [exclusiveLocks= key: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 value: STORAGE , sharedLocks= ] 2015-01-06 10:03:31,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [18526185] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 32a82db7 2015-01-06 10:03:31,782 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-13) [18526185] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5e2b03ab 2015-01-06 10:03:31,782 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) [422d495e] -- executeIrsBrokerCommand: Attempting on storage pool 00000002-0002-0002-0002-00000000000f 2015-01-06 10:03:31,785 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-13) [18526185] Discovered no tasks on Storage Pool Default 2015-01-06 10:03:31,815 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) [422d495e] START, HSMGetAllTasksInfoVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5), log id: 4eb41b3a 2015-01-06 10:03:31,825 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) [422d495e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4eb41b3a 2015-01-06 10:03:31,829 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-8-thread-26) [422d495e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 5be8ceb1 2015-01-06 10:03:31,831 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-8-thread-26) [422d495e] Discovered no tasks on Storage Pool Default 2015-01-06 10:03:32,008 INFO [org.ovirt.engine.core.bll.storage.SyncLunsInfoForBlockStorageDomainCommand] (org.ovirt.thread.pool-8-thread-32) [afbbe0a] Running command: SyncLunsInfoForBlockStorageDomainCommand internal: true. Entities affected : ID: 05d4e406-de6d-47dc-8414-bdc7381b6d4a Type: Storage 2015-01-06 10:03:32,023 INFO [org.ovirt.engine.core.bll.storage.SyncLunsInfoForBlockStorageDomainCommand] (org.ovirt.thread.pool-8-thread-7) [73793c2b] Running command: SyncLunsInfoForBlockStorageDomainCommand internal: true. Entities affected : ID: 75a3e25a-2ea6-4b95-9d9f-9fe791e38e97 Type: Storage 2015-01-06 10:03:32,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) [afbbe0a] START, GetVGInfoVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, VGID=CtELsP-UYZb-zgRH-bVz7-lijD-Elfy-frzEJp), log id: 19fe6181 2015-01-06 10:03:32,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-8-thread-7) [73793c2b] START, GetVGInfoVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, VGID=dEB6XY-cNBu-hsoz-avXf-m6Vp-4jOq-eDMCBK), log id: 14fc74a1 2015-01-06 10:03:32,102 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-55) [27e4f6a6] Running storage connections refresh for hosts [ab744426-2294-4c0b-aaf5-08ebb162f542, 920bf64c-62f5-4b12-a69e-eef9936576c5] 2015-01-06 10:03:32,107 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (org.ovirt.thread.pool-8-thread-27) [4bd57497] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-00000000000f Type: StoragePool 2015-01-06 10:03:32,107 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (org.ovirt.thread.pool-8-thread-15) [2ad98c66] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-00000000000f Type: StoragePool 2015-01-06 10:03:32,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-27) [4bd57497] START, ConnectStorageServerVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = NFS, connectionList = [{ id: 79bae336-2d76-4986-bf1e-50e350cde9f7, connection: cluster.33sdis.fr:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1acd4935 2015-01-06 10:03:32,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-15) [2ad98c66] START, ConnectStorageServerVDSCommand(HostName = n3orna, HostId = ab744426-2294-4c0b-aaf5-08ebb162f542, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = NFS, connectionList = [{ id: 79bae336-2d76-4986-bf1e-50e350cde9f7, connection: cluster.33sdis.fr:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 10087d1 2015-01-06 10:03:32,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-15) [2ad98c66] FINISH, ConnectStorageServerVDSCommand, return: {79bae336-2d76-4986-bf1e-50e350cde9f7=0}, log id: 10087d1 2015-01-06 10:03:32,199 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (org.ovirt.thread.pool-8-thread-15) [2ad98c66] Host n3orna storage connection was succeeded 2015-01-06 10:03:32,643 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-8-thread-32) [afbbe0a] FINISH, GetVGInfoVDSCommand, return: [LUNs [id=3600c0ff000149e9d22c0105401000000, physicalVolumeId=OMa7pu-EyxG-mdNM-9EP5-eP2f-2GPV-tWiexe, volumeGroupId=CtELsP-UYZb-zgRH-bVz7-lijD-Elfy-frzEJp, serial=SHP_P2000_G3_FC_00c0ff149e9d000022c0105401000000, lunMapping=0, vendorId=HP, productId=P2000 G3 FC, _lunConnections=[], deviceSize=5583, vendorName=HP, pathsDictionary={sdb=true, sdd=true}, lunType=UNKNOWN, status=null, diskId=null, diskAlias=null, storageDomainId=05d4e406-de6d-47dc-8414-bdc7381b6d4a, storageDomainName=null]], log id: 19fe6181 2015-01-06 10:03:32,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-8-thread-7) [73793c2b] FINISH, GetVGInfoVDSCommand, return: [LUNs [id=3600c0ff00015986a35d98a5401000000, physicalVolumeId=oWeQjL-dOc0-G9dH-wu1g-8pCd-jGjG-rzL0ZM, volumeGroupId=dEB6XY-cNBu-hsoz-avXf-m6Vp-4jOq-eDMCBK, serial=SHP_P2000_G3_FC_00c0ff15986a000035d98a5401000000, lunMapping=0, vendorId=HP, productId=P2000 G3 FC, _lunConnections=[], deviceSize=5583, vendorName=HP, pathsDictionary={sdc=true, sde=true}, lunType=UNKNOWN, status=null, diskId=null, diskAlias=null, storageDomainId=75a3e25a-2ea6-4b95-9d9f-9fe791e38e97, storageDomainName=null]], log id: 14fc74a1 2015-01-06 10:03:32,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-27) [4bd57497] FINISH, ConnectStorageServerVDSCommand, return: {79bae336-2d76-4986-bf1e-50e350cde9f7=0}, log id: 1acd4935 2015-01-06 10:03:32,668 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (org.ovirt.thread.pool-8-thread-27) [4bd57497] Host n4orna storage connection was succeeded 2015-01-06 10:03:32,671 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-55) [27e4f6a6] Submitting to the event queue pool refresh for hosts [ab744426-2294-4c0b-aaf5-08ebb162f542, 920bf64c-62f5-4b12-a69e-eef9936576c5] 2015-01-06 10:03:32,675 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-35) Running storage pool metadata refresh for hosts {1} 2015-01-06 10:03:32,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-10) START, ConnectStoragePoolVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, vdsId = 920bf64c-62f5-4b12-a69e-eef9936576c5, storagePoolId = 00000002-0002-0002-0002-00000000000f, masterVersion = 1), log id: 66dc8feb 2015-01-06 10:03:32,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-41) START, ConnectStoragePoolVDSCommand(HostName = n3orna, HostId = ab744426-2294-4c0b-aaf5-08ebb162f542, vdsId = ab744426-2294-4c0b-aaf5-08ebb162f542, storagePoolId = 00000002-0002-0002-0002-00000000000f, masterVersion = 1), log id: 1a6cc20f 2015-01-06 10:03:34,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-10) FINISH, ConnectStoragePoolVDSCommand, log id: 66dc8feb 2015-01-06 10:03:34,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 1a6cc20f 2015-01-06 10:03:37,386 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) Host 60d8c75b-38f1-4cd0-b162-729285eadefd no longer storage access problem to any relevant domain clearing its report (report id: 6a081ad8-201f-48e4-ba56-5a581fac4475) 2015-01-06 10:03:37,392 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-47) Domain 75a3e25a-2ea6-4b95-9d9f-9fe791e38e97:MSA2-O2 recovered from problem. vds: n1orna 2015-01-06 10:03:37,465 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-34) Host 38893029-0eb8-4d19-a28f-07680d8d6868 no longer storage access problem to any relevant domain clearing its report (report id: 922d1075-c975-4d9f-87d1-db8d5aa4059d) 2015-01-06 10:03:37,471 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-34) Domain 75a3e25a-2ea6-4b95-9d9f-9fe791e38e97:MSA2-O2 recovered from problem. vds: n2orna 2015-01-06 10:03:37,475 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-34) Domain 75a3e25a-2ea6-4b95-9d9f-9fe791e38e97:MSA2-O2 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. 2015-01-06 10:03:37,634 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-11) Host 920bf64c-62f5-4b12-a69e-eef9936576c5 no longer storage access problem to any relevant domain clearing its report (report id: 12dfe3fe-966a-486e-8eda-89575648545f) 2015-01-06 10:03:39,263 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) Host ab744426-2294-4c0b-aaf5-08ebb162f542 no longer storage access problem to any relevant domain clearing its report (report id: 18fffe35-403c-413e-acb1-8d173aa447da) 2015-01-06 10:03:39,268 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) Domain 05d4e406-de6d-47dc-8414-bdc7381b6d4a:MSA1-O2-vd01 recovered from problem. vds: n3orna 2015-01-06 10:03:39,270 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-17) Domain 05d4e406-de6d-47dc-8414-bdc7381b6d4a:MSA1-O2-vd01 has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. 2015-01-06 10:03:44,310 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-45) [5770eda1] No hosts has reported storage access problem to domains, clearing the handled hosts reports map 2015-01-06 10:05:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-29) [1cbf3978] Autorecovering 1 storage domains 2015-01-06 10:05:00,009 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-29) [1cbf3978] Autorecovering storage domains id: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 2015-01-06 10:05:00,014 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-29) [344e83b5] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 34c95a44-db7f-4d0f-ba13-5f06a7feefe7 Type: Storage 2015-01-06 10:05:00,019 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-29) [344e83b5] ConnectDomainToStorage. Before Connect all hosts to pool. Time:1/6/15 10:05 AM 2015-01-06 10:05:00,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-37) [344e83b5] START, ConnectStorageServerVDSCommand(HostName = n4orna, HostId = 920bf64c-62f5-4b12-a69e-eef9936576c5, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1d6d563c 2015-01-06 10:05:00,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-28) [344e83b5] START, ConnectStorageServerVDSCommand(HostName = n1orna, HostId = 60d8c75b-38f1-4cd0-b162-729285eadefd, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 12a471c7 2015-01-06 10:05:00,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-34) [344e83b5] START, ConnectStorageServerVDSCommand(HostName = n3orna, HostId = ab744426-2294-4c0b-aaf5-08ebb162f542, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 70547eab 2015-01-06 10:05:00,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-8-thread-19) [344e83b5] START, ConnectStorageServerVDSCommand(HostName = n2orna, HostId = 38893029-0eb8-4d19-a28f-07680d8d6868, storagePoolId = 00000002-0002-0002-0002-00000000000f, storageType = ISCSI, connectionList = [{ id: 1b9f3167-3236-431e-93c2-ab5ee18eba04, connection: 10.23.2.199, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea5971f8-e1a0-42e3-826d-b95e9031ce53, connection: 10.23.2.198, iqn: iqn.1999-06.com.fujitsu-siemens:0907d7d0e3.a, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6438f79b 2015-01-06 10:06:11,746 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-83) [453f31bd] VM ciril-recette e2abb477-f807-470f-ae20-a7205e690638 moved from Up --> RebootInProgress 2015-01-06 10:06:39,611 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-84) [537321db] VM ciril-recette e2abb477-f807-470f-ae20-a7205e690638 moved from RebootInProgress --> Up -- Ce courriel et tous les fichiers attachés qu'il contient sont confidentiels et destinés exclusivement à la personne à laquelle ils sont adressés. Si vous avez reçu ce courriel par erreur, merci de le retourner à son expéditeur et de le détruire. Il est rappelé que tout message électronique est susceptible d'alteration au cours de son acheminement sur internet. Seuls les documents officiels du SDIS sont de nature à engager sa responsabilité. Les idées ou opinions présentées dans ce courriel sont celles de son auteur et ne représentent pas nécessairement celles du SDIS de la Gironde.

1 0

Re: [ovirt-users] 3.5 live merge findings and mysteries [was Re: Simple way to activate live merge in FC20 cluster]
by Gianluca Cecchi 06 Jan '15

06 Jan '15

On Fri, Dec 12, 2014 at 4:32 PM, Itamar Heim <iheim(a)redhat.com> wrote: > > On 11/21/2014 09:53 AM, Gianluca Cecchi wrote: > >> So the official statement is this one at: >> http://www.ovirt.org/OVirt_3.5_Release_Notes >> >> Live Merge >> If an image has one or more snapshots, oVirt 3.5's merge command will >> combine the data of one volume into another. Live merges can be >> performed with data is pulled from one snapshot into another snapshot. >> The engine can merge multiple disks at the same time and each merge can >> independently fail or succeed in each operation. >> >> I think we should remove the part above, or at least have some of the >> developers to better clarify it. >> The feature in my opinion is very important and crucial for oVirt/RHEV >> because it is able to almost-fill the gap with VMware, especially in >> development environments, where having flexibility on snapshots >> management is very important and could be a starting point to have >> greater user base familiarize with the product and adopt it. >> >> So these are my findings and all combinations and none of them was able >> to provide live merge.... >> Could anyone tell me where I'm failing? Or correct release notes? >> >> 1) Environment with All-In-One F20 >> >> installing oVirt AIO on F20 automatically gives the virt-preview repo >> through the ovirt-3.5-dependencies.repo file, but only for libvirt* >> packages: >> >> the same server is the engine and the hypervisor >> >> [root@tekkaman qemu]# rpm -q libvirt >> libvirt-1.2.9.1-1.fc20.x86_64 >> >> [root@tekkaman qemu]# vdsClient -s 0 getVdsCaps | grep -i merge >> liveMerge = 'true' >> >> but >> [root@tekkaman qemu]# rpm -q qemu >> qemu-1.6.2-10.fc20.x86_64 >> >> So that tryng live merge it initially start but you get this in vdsm.log: >> libvirtError: unsupported configuration: active commit not supported >> with this QEMU binary >> >> >> 2) Another seaparate environment with a dedicated 3.5 engine f20 engine >> and 4 test cases tried >> >> a) ovirt node installed and put in a dedicated cluster >> latest available seems ovirt-node-iso-3.5.0.ovirt35.20140912.el6.iso >> from 3.5 rc test days >> >> At the end of oVirt Node install and activation in engine: >> [root@ovnode01 ~]# vdsClient -s 0 getVdsCaps | grep -i merge >> liveMerge = 'false' >> >> [root@ovnode01 ~]# rpm -qa libvirt* qemu* >> libvirt-0.10.2-29.el6_5.12.x86_64 >> libvirt-lock-sanlock-0.10.2-29.el6_5.12.x86_64 >> libvirt-python-0.10.2-29.el6_5.12.x86_64 >> qemu-kvm-tools-0.12.1.2-2.415.el6_5.14.x86_64 >> qemu-kvm-rhev-0.12.1.2-2.415.el6_5.14.x86_64 >> libvirt-client-0.10.2-29.el6_5.12.x86_64 >> qemu-img-rhev-0.12.1.2-2.415.el6_5.14.x86_64 >> >> >> >> b) f20 + latest updates host installed as OS and then installed from >> webadmin in another cluster >> virt-preview on host it is not enabled so that libvirt/qemu are not ready >> >> >> [root@ovnode02 network-scripts]# vdsClient -s 0 getVdsCaps | grep -i >> merge >> liveMerge = 'false' >> >> [root@ovnode02 network-scripts]# rpm -qa libvirt* qemu* >> libvirt-daemon-1.1.3.6-2.fc20.x86_64 >> libvirt-python-1.1.3.6-2.fc20.x86_64 >> libvirt-daemon-config-nwfilter-1.1.3.6-2.fc20.x86_64 >> qemu-kvm-1.6.2-10.fc20.x86_64 >> qemu-common-1.6.2-10.fc20.x86_64 >> libvirt-client-1.1.3.6-2.fc20.x86_64 >> libvirt-daemon-driver-network-1.1.3.6-2.fc20.x86_64 >> libvirt-daemon-driver-nwfilter-1.1.3.6-2.fc20.x86_64 >> libvirt-daemon-driver-interface-1.1.3.6-2.fc20.x86_64 >> libvirt-daemon-driver-nodedev-1.1.3.6-2.fc20.x86_64 >> libvirt-daemon-driver-secret-1.1.3.6-2.fc20.x86_64 >> qemu-system-x86-1.6.2-10.fc20.x86_64 >> libvirt-daemon-kvm-1.1.3.6-2.fc20.x86_64 >> qemu-kvm-tools-1.6.2-10.fc20.x86_64 >> qemu-img-1.6.2-10.fc20.x86_64 >> libvirt-daemon-driver-qemu-1.1.3.6-2.fc20.x86_64 >> libvirt-daemon-driver-storage-1.1.3.6-2.fc20.x86_64 >> libvirt-lock-sanlock-1.1.3.6-2.fc20.x86_64 >> >> >> >> c) CentOS 6.6 host + latest updates installed as OS and then installed >> from webadmin in another cluster >> >> [root@ovnode03 ~]# vdsClient -s 0 getVdsCaps | grep -i merge >> liveMerge = 'false' >> >> [root@ovnode03 ~]# rpm -qa libvirt* qemu* >> libvirt-python-0.10.2-46.el6_6.2.x86_64 >> qemu-kvm-rhev-tools-0.12.1.2-2.415.el6_5.14.x86_64 >> qemu-img-rhev-0.12.1.2-2.415.el6_5.14.x86_64 >> libvirt-client-0.10.2-46.el6_6.2.x86_64 >> libvirt-0.10.2-46.el6_6.2.x86_64 >> qemu-kvm-rhev-0.12.1.2-2.415.el6_5.14.x86_64 >> libvirt-lock-sanlock-0.10.2-46.el6_6.2.x86_64 >> >> >> >> d) CentOS 7.0 host + latest updates installed as OS and then installed >> from webadmin in another cluster >> >> [root@ovnode04 ~]# vdsClient -s 0 getVdsCaps | grep -i merge >> liveMerge = 'false' >> >> [root@ovnode04 ~]# rpm -qa qemu* libvirt* >> qemu-img-rhev-1.5.3-60.el7_0.2.x86_64 >> libvirt-daemon-1.1.1-29.el7_0.3.x86_64 >> libvirt-daemon-driver-storage-1.1.1-29.el7_0.3.x86_64 >> libvirt-daemon-driver-nodedev-1.1.1-29.el7_0.3.x86_64 >> libvirt-daemon-kvm-1.1.1-29.el7_0.3.x86_64 >> qemu-kvm-tools-rhev-1.5.3-60.el7_0.2.x86_64 >> qemu-kvm-common-rhev-1.5.3-60.el7_0.2.x86_64 >> libvirt-client-1.1.1-29.el7_0.3.x86_64 >> libvirt-daemon-driver-nwfilter-1.1.1-29.el7_0.3.x86_64 >> libvirt-daemon-driver-interface-1.1.1-29.el7_0.3.x86_64 >> libvirt-lock-sanlock-1.1.1-29.el7_0.3.x86_64 >> libvirt-daemon-config-nwfilter-1.1.1-29.el7_0.3.x86_64 >> libvirt-daemon-driver-network-1.1.1-29.el7_0.3.x86_64 >> qemu-kvm-rhev-1.5.3-60.el7_0.2.x86_64 >> libvirt-python-1.1.1-29.el7_0.3.x86_64 >> libvirt-daemon-driver-secret-1.1.1-29.el7_0.3.x86_64 >> libvirt-daemon-driver-qemu-1.1.1-29.el7_0.3.x86_64 >> >> >> Thanks in advance, >> Gianluca >> >> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> >> > was this resolved? > In my opinion it was not resolved and what written into release notes doesn't correspond to features set. See my test cases and eventually describe other test cases where live merge is usable See also other findings about live merge on active layer here: http://lists.ovirt.org/pipermail/users/2014-November/029450.html Gianluca

4 7

Debug Environment for RHEVM
by Xie, Chao 06 Jan '15

06 Jan '15

--_000_EE4D679B9474414187D2E27D8B6890F694E491G08CNEXMBPEKD03g0_ Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: base64 SEksDQoNCkkgZm91bmQgdGhlcmUgaXMgYSBkZWJ1ZyBlbnZpcm9ubWVudCBmb3Igb1ZpcnQ6IGh0 dHA6Ly93aWtpLm92aXJ0Lm9yZy9PVmlydF9FbmdpbmVfRGV2ZWxvcG1lbnRfRW52aXJvbm1lbnQN CklzIGl0IGFsc28gdXNlZnVsIGZvciBSSEVWTSBzb3VyY2UgY29kZT8NCg0KQmVzdCBSZWdhcmRz LA0KWGllDQo= --_000_EE4D679B9474414187D2E27D8B6890F694E491G08CNEXMBPEKD03g0_ Content-Type: text/html; charset="gb2312" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr= osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:= //www.w3.org/TR/REC-html40"> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dgb2312"> <meta name=3D"Generator" content=3D"Microsoft Word 14 (filtered medium)"> <style></style> </head> <body lang=3D"ZH-CN" link=3D"blue" vlink=3D"purple" style=3D"text-justify-t= rim:punctuation"> <div class=3D"WordSection1"> <p class=3D"MsoNormal"><span lang=3D"EN-US">HI, <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">I found there is a debug enviro= nment for oVirt: <a href=3D"http://wiki.ovirt.org/OVirt_Engine_Development_Environment">http= ://wiki.ovirt.org/OVirt_Engine_Development_Environment</a><o:p></o:p></span= ></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">Is it also useful for RHEVM sou= rce code?<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US">Best Regards,<o:p></o:p></span>= </p> <p class=3D"MsoNormal"><span lang=3D"EN-US">Xie<o:p></o:p></span></p> </div> </body> </html> --_000_EE4D679B9474414187D2E27D8B6890F694E491G08CNEXMBPEKD03g0_--

3 2

templates and freeipa
by Jim Kinney 06 Jan '15

06 Jan '15

Ovirt 3.5 is running well for me and I have freeIPA controlling access to the user portal. I would like to provide templates of various linux setups that all have freeipa for user authentication in the VM for my developers to be able to create a new VM from and then log in using their freeIPA access and sudo control. I'm wanting to group developers by project and use freeIPA to set sudo commands as needed (group A get oracle, group B get postgresql, etc). Wanting to maximize developer ability while minimizing my clean up time :-) They will be able to delete VMs they create. It's possible to do a kickstart deploy with freeIPA registration but a template from that will be a problem as it will have the same keys for all VMs. Is there a post-creation scripting process I can attach to in ovirt or should I look at a default root user and script that personalizes the new VM? -- -- James P. Kinney III Every time you stop a school, you will have to build a jail. What you gain at one end you lose at the other. It's like feeding a dog on his own tail. It won't fatten the dog. - Speech 11/23/1900 Mark Twain *http://heretothereideas.blogspot.com/ <http://heretothereideas.blogspot.com/>*

3 2

Ovirt Engine Clear All tasks
by Donny Davis 06 Jan '15

06 Jan '15

I tried to migrate disks from one storage domain to another, and it is taking an unreasonable amount of time to complete. The disks have been migrating for 6 hours, and is bringing my system to it's knees. I have used the taskcleaner utility when the engine was stopped, and when I start the engine, it starts trying to migrate the disks again. How can I fix this Thanks -- Donny Davis CloudSpin.me

2 1

Can't remove a storage domain related to a broken hardware
by Olivier Navas 06 Jan '15

06 Jan '15

------=_Part_520073_53340046.1420536522128 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hello Ovirt users ! I experienced an hardware failure on a ISCSI storage making it unrecoverable, and I would like to remove it from storage domains. There was 1 disk on this storage domain, and this disk isn't attached to any VM anymore, but I still can't detach this storage domain from cluster. The storage domain is in "inactive" status and if I try to "detach" it from data center, ovirt tries to activate it. Obviously it can't activate it since hardware is broken, and it fails after several minutes with the event "Failed to detach Storage Domain my_storage_domain to Data Center Default. (User: admin)". I can post my engine.log if useful. I need a way to force removal of this storage domain. Any trick would be greatly appreciated. Perhaps does Ovirt miss some sort of "force detach, i know what i'm doing" button ? I am running an Ovirt 3.5 cluster (engine with CentOS 6.6, 4 nodes with CentOS 7) with FC and ISCSI storage domains. Thanks for your help. -- Ce courriel et tous les fichiers attachés qu'il contient sont confidentiels et destinés exclusivement à la personne à laquelle ils sont adressés. Si vous avez reçu ce courriel par erreur, merci de le retourner à son expéditeur et de le détruire. Il est rappelé que tout message électronique est susceptible d'alteration au cours de son acheminement sur internet. Seuls les documents officiels du SDIS sont de nature à engager sa responsabilité. Les idées ou opinions présentées dans ce courriel sont celles de son auteur et ne représentent pas nécessairement celles du SDIS de la Gironde. ------=_Part_520073_53340046.1420536522128 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><body><div style=3D"font-family: times new roman, new york, times, se= rif; font-size: 12pt; color: #000000"><div><br></div><div>Hello Ovirt users= !<br></div><div><br></div><div>I experienced an hardware failure on a ISCS= I storage making it unrecoverable, and I would like to remove it from stora= ge domains.<br></div><div><br></div><div>There was 1 disk on this storage d= omain, and this disk isn't attached to any VM anymore, but I still can't de= tach this storage domain from cluster.<br></div><div><br></div><div>The sto= rage domain is in "inactive" status and if I try to "detach" it from data c= enter, ovirt tries to activate it. Obviously it can't activate it since har= dware is broken, and it fails after several minutes with the event "Failed = to detach Storage Domain my_storage_domain to Data Center Default. (User: a= dmin)". I can post my engine.log if useful.<br></div><div><br></div><div>I = need a way to force removal of this storage domain. Any trick would be grea= tly appreciated. <br></div><div><br></div><div>Perhaps does Ovirt miss some= sort of "force detach, i know what i'm doing" button ? <br></div><div><br>= </div><div>I am running an Ovirt 3.5 cluster (engine with CentOS 6.6, 4 nod= es with CentOS 7) with FC and ISCSI storage domains.<br><div><br></div><div= >Thanks for your help.<br></div><div><br></div><div><br></div></div></div><= /body> <br>= =0A<hr><font size=3D"-2"><em>=0ACe courriel et tous les fichiers attach=C3= =A9s qu'il contient sont confidentiels et destin=C3=A9s exclusivement =C3=A0= la personne =C3=A0 laquelle ils sont adress=C3=A9s. Si vous avez re=C3=A7u = ce courriel par erreur, merci de le retourner =C3=A0 son exp=C3=A9diteur et = de le d=C3=A9truire. Il est rappel=C3=A9 que tout message =C3=A9lectronique = est susceptible d'alteration au cours de son acheminement sur internet. Seul= s les documents officiels du SDIS sont de nature =C3=A0 engager sa responsab= ilit=C3=A9. Les id=C3=A9es ou opinions pr=C3=A9sent=C3=A9es dans ce courriel= sont celles de son auteur et ne repr=C3=A9sentent pas n=C3=A9cessairement c= elles du SDIS de la Gironde.=0A</font></em>=0A <br>= </html> ------=_Part_520073_53340046.1420536522128--

1 0

Affinity
by Koen Vanoppen 06 Jan '15

06 Jan '15

Hi All, First of all, let me say a Happy New Year with all the best wishes!! Now, let's get down to business :-). I would lik eto implement the Affinity option in our datacenter. I already activated it, put the 2 vms in it and set in on negative so they won't run together on the same hypervisor. Now, the question... Is there a way I can force the VM's so they will run on a hypervisor that is not in the same datacenter? We have this; Hyp1 -> Datacenter1 Hyp2 -> Datacenter2 Hyp3 -> Datacenter1 Hyp4 -> Datacenter2 For the moment they run on a different Hypervisor, but they are in the same Datacenter. I can manually now move them to the other one, but I would like this that oVirt manages this... Is this possible...? Kind regards, Koen

3 6

Re: [ovirt-users] How can I add usernames in ovirt? i.e., is admin account
by Sandvik Agustin 06 Jan '15

06 Jan '15

Hi, Thanks guys for the quick reply and I really appreciate it, I'll look upon your suggestions right now. By the way, I forgot to mention that I'm using oVirt Engine Version: 3.5.0.1-1.el6. Thanks Again, I'll update you guys about my progress. Thanks Again. On Tue, Jan 6, 2015 at 2:39 AM, Donny Davis <donny(a)cloudspin.me> wrote: > Ensure you don't try to install freeipa to the manager machine, there will > be conflicts. > > 389ds works and is really easy to setup > > Regards > DonnyOn Jan 5, 2015 11:36 AM, Donny Davis <donny(a)cloudspin.me> wrote: > > > > I did a write up on AAA LDAP. > > > > https://cloudspin.me/ovirt-simple-ldap-aaa/ > > > > Hope its helpful > > > > DonnyOn Jan 5, 2015 11:26 AM, Alon Bar-Lev <alonbl(a)redhat.com> wrote: > > > > > > Hello, > > > > > > For now you need to use somekind of LDAP with > ovirt-engine-extension-ldap[1][2] package. > > > In future we will support database based repository. > > > > > > Until someone from infra will have the time to publish the latest > version of the package, please download it directly from here[3], please > note that until engine-3.5.1 is out you will need to specify full path in > config.profile.file.1 variable at authn and authz extension configuration. > > > > > > Regards, > > > Alon > > > > > > [1] > http://gerrit.ovirt.org/gitweb?p=ovirt-engine-extension-aaa-ldap.git;a=blob… > > > [2] http://www.ovirt.org/Features/AAA > > > [3] > http://jenkins.ovirt.org/job/ovirt-engine-extension-aaa-ldap_any_create-rpm… > > > > > > ----- Original Message ----- > > > > From: "Sandvik Agustin" <agustinsandvik(a)gmail.com> > > > > To: users(a)ovirt.org > > > > Sent: Monday, January 5, 2015 8:14:27 PM > > > > Subject: [ovirt-users] How can I add usernames in ovirt? i.e., is > admin account > > > > > > > > Hi guys, > > > > > > > > Good day, I just want to know if how can I add usernames in ovirt? > i.e., is > > > > admin account is already exist and I want to create another account > i.e., > > > > users or clients account. > > > > > > > > > > > > TIA > > > > > > > > _______________________________________________ > > > > Users mailing list > > > > Users(a)ovirt.org > > > > http://lists.ovirt.org/mailman/listinfo/users > > > > > > > _______________________________________________ > > > Users mailing list > > > Users(a)ovirt.org > > > http://lists.ovirt.org/mailman/listinfo/users > > _______________________________________________ > > Users mailing list > > Users(a)ovirt.org > > http://lists.ovirt.org/mailman/listinfo/users >

2 1