[Users] Reattach storage domain
Alexandr
shuriku at shurik.kiev.ua
Sat Feb 1 17:38:13 UTC 2014
Hello!
Unfortunately my master storage domain (gluster) is dead. I setup
another gluster-storage and attach it to the ovirt. Hostname, path and
volume name are same as old ones. Then I restored from tar archive all
files. But I cannot activate master domain, operation failed and domain
status remains "inactive". I see - it mounted on nodes:
vms02.lis.ua:STORAGE on /rhev/data-center/mnt/glusterSD/vms02.lis.ua:STORAGE
I attached engine.log, can someone provide me recovery steps?
P.S. Sorry for my english
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20140201/7911941e/attachment-0001.html>
-------------- next part --------------
2014-02-01 19:00:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-15) [10581b8] Autorecovering 1 storage domains
2014-02-01 19:00:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-15) [10581b8] Autorecovering storage domains id: 7f9709c1-3ab6-4af8-9e58-955ef7c9452e
2014-02-01 19:00:00,004 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-15) [1c9d0819] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 7f9709c1-3ab6-4af8-9e58-955ef7c9452e Type: Storage
2014-02-01 19:00:00,005 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-15) [1c9d0819] ConnectDomainToStorage. Before Connect all hosts to pool. Time:2/1/14 7:00 PM
2014-02-01 19:00:00,010 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-6-thread-44) [3fc57c01] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System
2014-02-01 19:00:00,010 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-6-thread-49) [4277b006] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System
2014-02-01 19:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-44) [3fc57c01] START, ConnectStorageServerVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = GLUSTERFS, connectionList = [{ id: 4a1d8b07-f393-4134-86cc-4f46145cca2b, connection: vms02.lis.ua:STORAGE, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 257cdd99
2014-02-01 19:00:00,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-49) [4277b006] START, ConnectStorageServerVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = GLUSTERFS, connectionList = [{ id: 4a1d8b07-f393-4134-86cc-4f46145cca2b, connection: vms02.lis.ua:STORAGE, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6e12cbf6
2014-02-01 19:00:00,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-49) [4277b006] FINISH, ConnectStorageServerVDSCommand, return: {4a1d8b07-f393-4134-86cc-4f46145cca2b=0}, log id: 6e12cbf6
2014-02-01 19:00:02,221 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-74) [97b4f50] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 285f605c
2014-02-01 19:00:02,253 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-74) [97b4f50] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at a53a00d1}, log id: 285f605c
2014-02-01 19:00:04,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-44) [3fc57c01] FINISH, ConnectStorageServerVDSCommand, return: {4a1d8b07-f393-4134-86cc-4f46145cca2b=0}, log id: 257cdd99
2014-02-01 19:00:04,046 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-15) [1c9d0819] ConnectDomainToStorage. After Connect all hosts to pool. Time:2/1/14 7:00 PM
2014-02-01 19:00:07,261 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-12) [41c61dd1] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 1554591b
2014-02-01 19:00:07,295 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-12) [41c61dd1] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at ad4663ff}, log id: 1554591b
2014-02-01 19:00:08,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:00:08,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] HostName = vms02.lis.ua
2014-02-01 19:00:08,439 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:00:10,453 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:00:10,456 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 67fed0ff
2014-02-01 19:00:12,303 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 320273fd
2014-02-01 19:00:15,856 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] FINISH, ConnectStoragePoolVDSCommand, log id: 67fed0ff
2014-02-01 19:00:15,857 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:00:20,356 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:00:20,379 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:00:20,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 8857b76
2014-02-01 19:00:22,893 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 33c86750}, log id: 320273fd
2014-02-01 19:00:23,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] FINISH, ConnectStoragePoolVDSCommand, log id: 8857b76
2014-02-01 19:00:23,754 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:00:27,901 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-100) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 55c23f77
2014-02-01 19:00:29,936 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-100) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 56a0e3ae}, log id: 55c23f77
2014-02-01 19:00:33,774 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:00:33,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-44) HostName = vms02.lis.ua
2014-02-01 19:00:33,776 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-44) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:00:33,788 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-44) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:00:33,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-44) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 3a2d1ad3
2014-02-01 19:00:34,946 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-19) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 408edaba
2014-02-01 19:00:34,980 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-19) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at d54f38ad}, log id: 408edaba
2014-02-01 19:00:38,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, ConnectStoragePoolVDSCommand, log id: 3a2d1ad3
2014-02-01 19:00:38,155 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-44) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:00:38,166 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-44) Irs placed on server 0072ab64-05da-4dde-9493-a0af6aa97402 failed. Proceed Failover
2014-02-01 19:00:39,988 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-86) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 6dad3376
2014-02-01 19:00:42,200 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-44) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:00:42,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-44) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 5880b792
2014-02-01 19:00:44,669 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-86) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at fe723f8d}, log id: 6dad3376
2014-02-01 19:00:47,608 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, ConnectStoragePoolVDSCommand, log id: 5880b792
2014-02-01 19:00:47,609 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-44) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:00:49,678 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-57) [774cb764] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 4bd18b95
2014-02-01 19:00:49,709 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-57) [774cb764] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at c4a3fe97}, log id: 4bd18b95
2014-02-01 19:00:54,729 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-84) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 596edf66
2014-02-01 19:00:56,763 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-84) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at d11d1971}, log id: 596edf66
2014-02-01 19:00:58,403 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-52) START, GlusterTasksListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 283beaf
2014-02-01 19:00:58,448 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-52) Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:00:58,449 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-52) FINISH, GlusterTasksListVDSCommand, log id: 283beaf
2014-02-01 19:00:58,450 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-52) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:01:01,773 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-13) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 6e414442
2014-02-01 19:01:01,807 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-13) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 802b79f5}, log id: 6e414442
2014-02-01 19:01:06,668 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-42) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:01:06,670 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-42) HostName = vms01.lis.ua
2014-02-01 19:01:06,670 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-42) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:01:06,817 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-66) [40858177] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 15c56eda
2014-02-01 19:01:06,848 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-66) [40858177] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 199bd667}, log id: 15c56eda
2014-02-01 19:01:11,920 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-54) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 5085306b
2014-02-01 19:01:14,674 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-42) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:01:14,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-42) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 7d8fc01b
2014-02-01 19:01:16,700 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-54) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 4153eafd}, log id: 5085306b
2014-02-01 19:01:21,709 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-92) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 77bd6921
2014-02-01 19:01:23,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-42) FINISH, ConnectStoragePoolVDSCommand, log id: 7d8fc01b
2014-02-01 19:01:23,200 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-42) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:01:26,687 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-42) Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:01:26,688 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-92) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 99d33cf6}, log id: 77bd6921
2014-02-01 19:01:26,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-42) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:01:26,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-42) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 5be3180f
2014-02-01 19:01:30,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-42) FINISH, ConnectStoragePoolVDSCommand, log id: 5be3180f
2014-02-01 19:01:30,064 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-42) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:01:31,698 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-98) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 6888e490
2014-02-01 19:01:35,684 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-98) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 22f0ea22}, log id: 6888e490
2014-02-01 19:01:40,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-38) [120e5b00] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:01:40,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-38) [120e5b00] HostName = vms02.lis.ua
2014-02-01 19:01:40,086 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-38) [120e5b00] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:01:40,694 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-29) [288044b9] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 39d6b7b4
2014-02-01 19:01:48,110 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-38) [120e5b00] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:01:48,112 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-38) [120e5b00] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 1bf392f1
2014-02-01 19:01:51,123 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-29) [288044b9] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at acd0f01b}, log id: 39d6b7b4
2014-02-01 19:01:53,625 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-38) [120e5b00] FINISH, ConnectStoragePoolVDSCommand, log id: 1bf392f1
2014-02-01 19:01:53,626 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-38) [120e5b00] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:01:56,154 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-17) [6dd7fbdf] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 50dc4f11
2014-02-01 19:01:58,460 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] START, GlusterTasksListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 4814fb9
2014-02-01 19:02:00,184 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-17) [6dd7fbdf] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 612a570e}, log id: 50dc4f11
2014-02-01 19:02:00,646 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-38) [120e5b00] Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:02:00,657 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-38) [120e5b00] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:02:00,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-38) [120e5b00] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 79c49c39
2014-02-01 19:02:02,724 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:02:02,725 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] FINISH, GlusterTasksListVDSCommand, log id: 4814fb9
2014-02-01 19:02:02,726 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-91) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:02:05,193 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-33) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 25494b83
2014-02-01 19:02:05,225 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-33) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 9f27df4e}, log id: 25494b83
2014-02-01 19:02:06,030 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-38) [120e5b00] FINISH, ConnectStoragePoolVDSCommand, log id: 79c49c39
2014-02-01 19:02:06,030 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-38) [120e5b00] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:02:10,233 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-30) [3ea5fd22] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 5c80db57
2014-02-01 19:02:12,702 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-30) [3ea5fd22] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at b774edb0}, log id: 5c80db57
2014-02-01 19:02:16,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-68) [584ec02a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:02:16,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-68) [584ec02a] HostName = vms02.lis.ua
2014-02-01 19:02:16,052 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-68) [584ec02a] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:02:17,711 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-11) [4ad2e5fd] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 145cda85
2014-02-01 19:02:17,743 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-11) [4ad2e5fd] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 65d6aef3}, log id: 145cda85
2014-02-01 19:02:20,058 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-68) [584ec02a] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:02:20,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-68) [584ec02a] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 1aa7f720
2014-02-01 19:02:22,753 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-8) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 740289df
2014-02-01 19:02:22,786 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-8) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 64393d19}, log id: 740289df
2014-02-01 19:02:25,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-68) [584ec02a] FINISH, ConnectStoragePoolVDSCommand, log id: 1aa7f720
2014-02-01 19:02:25,465 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-68) [584ec02a] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:02:27,796 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-96) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 10b37782
2014-02-01 19:02:27,830 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at bfd38e41}, log id: 10b37782
2014-02-01 19:02:29,966 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-68) [584ec02a] Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:02:29,978 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-68) [584ec02a] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:02:29,980 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-68) [584ec02a] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 31a5789c
2014-02-01 19:02:32,840 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-84) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 2f442a37
2014-02-01 19:02:33,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-68) [584ec02a] FINISH, ConnectStoragePoolVDSCommand, log id: 31a5789c
2014-02-01 19:02:33,332 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-68) [584ec02a] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:02:34,967 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-84) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 9f0082c5}, log id: 2f442a37
2014-02-01 19:02:39,975 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 2b54b036
2014-02-01 19:02:40,007 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at e6873a29}, log id: 2b54b036
2014-02-01 19:02:43,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-63) [5dcfa174] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:02:43,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-63) [5dcfa174] HostName = vms02.lis.ua
2014-02-01 19:02:43,358 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-63) [5dcfa174] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:02:45,016 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-85) [22829429] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 6972f5a
2014-02-01 19:02:45,048 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-85) [22829429] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 49985ccf}, log id: 6972f5a
2014-02-01 19:02:45,373 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-63) [5dcfa174] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:02:45,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-63) [5dcfa174] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 3d5913ff
2014-02-01 19:02:50,056 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-58) [2d2a3fc9] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 742c65e9
2014-02-01 19:02:50,088 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-58) [2d2a3fc9] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 6b7dae7c}, log id: 742c65e9
2014-02-01 19:02:50,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-63) [5dcfa174] FINISH, ConnectStoragePoolVDSCommand, log id: 3d5913ff
2014-02-01 19:02:50,387 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-63) [5dcfa174] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:02:50,501 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler_Worker-54) START, GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 7e7bfa2a
2014-02-01 19:02:54,888 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-63) [5dcfa174] Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:02:54,899 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-63) [5dcfa174] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:02:54,902 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-63) [5dcfa174] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 51b67f18
2014-02-01 19:02:55,103 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-27) Failed to acquire lock and wait lock EngineLock [exclusiveLocks= key: 99408929-82cf-4dc7-a532-9d998063fa95 value: GLUSTER
2014-02-01 19:02:55,825 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-27) Failed to acquire lock and wait lock EngineLock [exclusiveLocks= key: 99408929-82cf-4dc7-a532-9d998063fa95 value: GLUSTER
2014-02-01 19:02:58,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-63) [5dcfa174] FINISH, ConnectStoragePoolVDSCommand, log id: 51b67f18
2014-02-01 19:02:58,291 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-63) [5dcfa174] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:02:58,835 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-27) Failed to acquire lock and wait lock EngineLock [exclusiveLocks= key: 99408929-82cf-4dc7-a532-9d998063fa95 value: GLUSTER
2014-02-01 19:02:59,999 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-27) Failed to acquire lock and wait lock EngineLock [exclusiveLocks= key: 99408929-82cf-4dc7-a532-9d998063fa95 value: GLUSTER
2014-02-01 19:03:01,843 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-27) Failed to acquire lock and wait lock EngineLock [exclusiveLocks= key: 99408929-82cf-4dc7-a532-9d998063fa95 value: GLUSTER
2014-02-01 19:03:01,926 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler_Worker-54) FINISH, GetGlusterVolumeAdvancedDetailsVDSCommand, return: org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeAdvancedDetails at 92075eb, log id: 7e7bfa2a
2014-02-01 19:03:01,928 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-27) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 321d9bfb
2014-02-01 19:03:02,735 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-67) [1c47cc59] START, GlusterTasksListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 51fb9521
2014-02-01 19:03:02,781 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-67) [1c47cc59] Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:03:02,782 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-67) [1c47cc59] FINISH, GlusterTasksListVDSCommand, log id: 51fb9521
2014-02-01 19:03:02,782 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-67) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:03:04,997 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-27) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at d5e4a593}, log id: 321d9bfb
2014-02-01 19:03:08,311 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-69) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:03:08,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-69) HostName = vms02.lis.ua
2014-02-01 19:03:08,313 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-69) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:03:10,006 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-53) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 53dc6dc3
2014-02-01 19:03:10,037 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-53) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 4eecd60c}, log id: 53dc6dc3
2014-02-01 19:03:10,325 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-69) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:03:10,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-69) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 36082963
2014-02-01 19:03:15,046 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-36) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 35cb2b45
2014-02-01 19:03:15,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-69) FINISH, ConnectStoragePoolVDSCommand, log id: 36082963
2014-02-01 19:03:15,748 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-69) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:03:22,783 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-69) Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:03:22,784 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-36) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at f34d3a4f}, log id: 35cb2b45
2014-02-01 19:03:22,797 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-69) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:03:22,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-69) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 384e4af3
2014-02-01 19:03:26,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-69) FINISH, ConnectStoragePoolVDSCommand, log id: 384e4af3
2014-02-01 19:03:26,128 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-69) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:03:27,793 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 26a6095d
2014-02-01 19:03:27,829 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at f935cd7}, log id: 26a6095d
2014-02-01 19:03:32,839 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 340746fd
2014-02-01 19:03:36,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-56) [21ac4370] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:03:36,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-56) [21ac4370] HostName = vms02.lis.ua
2014-02-01 19:03:36,173 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-56) [21ac4370] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:03:36,184 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-56) [21ac4370] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:03:36,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-56) [21ac4370] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 29e8a7f3
2014-02-01 19:03:36,843 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at b58895e4}, log id: 340746fd
2014-02-01 19:03:40,530 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-56) [21ac4370] FINISH, ConnectStoragePoolVDSCommand, log id: 29e8a7f3
2014-02-01 19:03:40,530 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-56) [21ac4370] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:03:40,542 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-56) [21ac4370] Irs placed on server 0072ab64-05da-4dde-9493-a0af6aa97402 failed. Proceed Failover
2014-02-01 19:03:41,852 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-55) [2783e7c5] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 2c7be70b
2014-02-01 19:03:41,885 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-55) [2783e7c5] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at b1c7de2d}, log id: 2c7be70b
2014-02-01 19:03:42,544 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-56) [21ac4370] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:03:42,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-56) [21ac4370] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 30b934c3
2014-02-01 19:03:46,895 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-70) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 3787ce09
2014-02-01 19:03:46,926 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-70) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 5e05e6e4}, log id: 3787ce09
2014-02-01 19:03:47,969 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-56) [21ac4370] FINISH, ConnectStoragePoolVDSCommand, log id: 30b934c3
2014-02-01 19:03:47,970 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-56) [21ac4370] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:03:51,935 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-65) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 2797b193
2014-02-01 19:03:53,967 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-65) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at f9b5da63}, log id: 2797b193
2014-02-01 19:03:58,977 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-96) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 1b08d5d8
2014-02-01 19:03:59,012 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at c37a75ed}, log id: 1b08d5d8
2014-02-01 19:04:02,793 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-71) START, GlusterTasksListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 1fa4ba56
2014-02-01 19:04:04,022 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-68) [584ec02a] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 49f6a213
2014-02-01 19:04:04,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-57) [774cb764] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:04:04,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-57) [774cb764] HostName = vms01.lis.ua
2014-02-01 19:04:04,485 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-57) [774cb764] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:04:08,983 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-68) [584ec02a] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at fbac03e2}, log id: 49f6a213
2014-02-01 19:04:09,021 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-71) Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:04:09,022 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-71) FINISH, GlusterTasksListVDSCommand, log id: 1fa4ba56
2014-02-01 19:04:09,022 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-71) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:04:13,482 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-57) [774cb764] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:04:13,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-57) [774cb764] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 45c87f79
2014-02-01 19:04:13,992 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-14) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 658dff85
2014-02-01 19:04:14,041 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 2ed3342b}, log id: 658dff85
2014-02-01 19:04:19,050 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-83) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: dd11fac
2014-02-01 19:04:21,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-57) [774cb764] FINISH, ConnectStoragePoolVDSCommand, log id: 45c87f79
2014-02-01 19:04:21,622 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-57) [774cb764] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:04:23,623 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-57) [774cb764] Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:04:23,624 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-83) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 63e52502}, log id: dd11fac
2014-02-01 19:04:23,634 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-57) [774cb764] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:04:23,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-57) [774cb764] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 4ceae16a
2014-02-01 19:04:26,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-57) [774cb764] FINISH, ConnectStoragePoolVDSCommand, log id: 4ceae16a
2014-02-01 19:04:26,999 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-57) [774cb764] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:04:28,633 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-78) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 72621f33
2014-02-01 19:04:28,667 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-78) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at e338465e}, log id: 72621f33
2014-02-01 19:04:33,676 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-92) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 7177ce65
2014-02-01 19:04:33,711 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-92) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 1ab5d6b}, log id: 7177ce65
2014-02-01 19:04:37,020 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-77) [6a32be82] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:04:37,022 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-77) [6a32be82] HostName = vms02.lis.ua
2014-02-01 19:04:37,022 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-77) [6a32be82] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:04:37,033 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-77) [6a32be82] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:04:37,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-77) [6a32be82] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 3d55c56c
2014-02-01 19:04:38,721 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-34) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 30ef57ef
2014-02-01 19:04:40,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-77) [6a32be82] FINISH, ConnectStoragePoolVDSCommand, log id: 3d55c56c
2014-02-01 19:04:40,374 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-77) [6a32be82] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:04:40,385 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-77) [6a32be82] Irs placed on server 0072ab64-05da-4dde-9493-a0af6aa97402 failed. Proceed Failover
2014-02-01 19:04:40,747 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-34) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 9049c375}, log id: 30ef57ef
2014-02-01 19:04:45,757 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-98) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 42047524
2014-02-01 19:04:47,755 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-77) [6a32be82] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:04:47,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-77) [6a32be82] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: e559d6a
2014-02-01 19:04:51,794 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-98) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 4d71b73d}, log id: 42047524
2014-02-01 19:04:56,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-77) [6a32be82] FINISH, ConnectStoragePoolVDSCommand, log id: e559d6a
2014-02-01 19:04:56,295 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-77) [6a32be82] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:04:56,803 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-36) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 7eca5ce5
2014-02-01 19:04:56,833 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-36) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 4711d212}, log id: 7eca5ce5
2014-02-01 19:05:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-48) Autorecovering 1 storage domains
2014-02-01 19:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-48) Autorecovering storage domains id: 7f9709c1-3ab6-4af8-9e58-955ef7c9452e
2014-02-01 19:05:00,005 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-48) [5544b7d0] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 7f9709c1-3ab6-4af8-9e58-955ef7c9452e Type: Storage
2014-02-01 19:05:00,005 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-48) [5544b7d0] ConnectDomainToStorage. Before Connect all hosts to pool. Time:2/1/14 7:05 PM
2014-02-01 19:05:00,010 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-6-thread-45) [4b83c62b] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System
2014-02-01 19:05:00,010 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-6-thread-47) [da40f72] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System
2014-02-01 19:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-45) [4b83c62b] START, ConnectStorageServerVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = GLUSTERFS, connectionList = [{ id: 4a1d8b07-f393-4134-86cc-4f46145cca2b, connection: vms02.lis.ua:STORAGE, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7cdc8c8a
2014-02-01 19:05:00,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-47) [da40f72] START, ConnectStorageServerVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = GLUSTERFS, connectionList = [{ id: 4a1d8b07-f393-4134-86cc-4f46145cca2b, connection: vms02.lis.ua:STORAGE, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6165ae2d
2014-02-01 19:05:00,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-47) [da40f72] FINISH, ConnectStorageServerVDSCommand, return: {4a1d8b07-f393-4134-86cc-4f46145cca2b=0}, log id: 6165ae2d
2014-02-01 19:05:01,843 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-69) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 50fc3a4e
2014-02-01 19:05:04,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-45) [4b83c62b] FINISH, ConnectStorageServerVDSCommand, return: {4a1d8b07-f393-4134-86cc-4f46145cca2b=0}, log id: 7cdc8c8a
2014-02-01 19:05:04,528 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-48) [5544b7d0] ConnectDomainToStorage. After Connect all hosts to pool. Time:2/1/14 7:05 PM
2014-02-01 19:05:06,559 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-69) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 2bf934e6}, log id: 50fc3a4e
2014-02-01 19:05:09,033 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] START, GlusterTasksListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 492cc4cb
2014-02-01 19:05:11,565 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-100) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 3b737909
2014-02-01 19:05:16,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-33) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:05:16,034 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:05:16,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-33) HostName = vms01.lis.ua
2014-02-01 19:05:16,035 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] FINISH, GlusterTasksListVDSCommand, log id: 492cc4cb
2014-02-01 19:05:16,036 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-33) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:05:16,036 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-91) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:05:20,062 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-100) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at b364e047}, log id: 3b737909
2014-02-01 19:05:22,558 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-33) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:05:22,561 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-33) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 82a9328
2014-02-01 19:05:25,071 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-86) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 310eb7c7
2014-02-01 19:05:30,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-33) FINISH, ConnectStoragePoolVDSCommand, log id: 82a9328
2014-02-01 19:05:30,606 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-33) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:05:33,092 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-86) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 8a42f1a4}, log id: 310eb7c7
2014-02-01 19:05:35,090 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-33) Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:05:35,107 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-33) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:05:35,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-33) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 24f21e9c
2014-02-01 19:05:37,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-33) FINISH, ConnectStoragePoolVDSCommand, log id: 24f21e9c
2014-02-01 19:05:37,493 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-33) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:05:38,102 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-45) [4b2b1c98] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 17f6a615
2014-02-01 19:05:42,638 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-45) [4b2b1c98] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 49e443c3}, log id: 17f6a615
2014-02-01 19:05:47,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-5) [7688e091] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:05:47,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-5) [7688e091] HostName = vms02.lis.ua
2014-02-01 19:05:47,516 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-5) [7688e091] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:05:47,526 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-5) [7688e091] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:05:47,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-5) [7688e091] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 287d0112
2014-02-01 19:05:47,644 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-71) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 19034dd8
2014-02-01 19:05:47,680 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-71) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at f6fc74df}, log id: 19034dd8
2014-02-01 19:05:50,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-5) [7688e091] FINISH, ConnectStoragePoolVDSCommand, log id: 287d0112
2014-02-01 19:05:50,853 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-5) [7688e091] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:05:50,864 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-5) [7688e091] Irs placed on server 0072ab64-05da-4dde-9493-a0af6aa97402 failed. Proceed Failover
2014-02-01 19:05:52,689 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-88) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 6c5c17db
2014-02-01 19:05:54,654 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-5) [7688e091] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:05:54,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-5) [7688e091] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 27795853
2014-02-01 19:06:01,706 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-88) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 6597052d}, log id: 6c5c17db
2014-02-01 19:06:03,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-5) [7688e091] FINISH, ConnectStoragePoolVDSCommand, log id: 27795853
2014-02-01 19:06:03,592 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-5) [7688e091] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:06:05,592 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp--127.0.0.1-8702-7) [3ede9bba] Running command: LogoutUserCommand internal: false.
2014-02-01 19:06:05,634 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [3ede9bba] Correlation ID: 3ede9bba, Call Stack: null, Custom Event ID: -1, Message: User support at lis.ua logged out.
2014-02-01 19:06:06,715 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-43) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 51ba8f98
2014-02-01 19:06:10,607 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-43) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 5bcbb2c9}, log id: 51ba8f98
2014-02-01 19:06:15,616 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-89) [e0b5c73] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 7ba43e00
2014-02-01 19:06:16,046 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-74) [97b4f50] START, GlusterTasksListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 207f3c80
2014-02-01 19:06:16,091 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-74) [97b4f50] Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:06:16,092 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-74) [97b4f50] FINISH, GlusterTasksListVDSCommand, log id: 207f3c80
2014-02-01 19:06:16,092 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-74) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:06:22,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-90) [5555a419] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:06:22,645 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-89) [e0b5c73] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at dfb4b12c}, log id: 7ba43e00
2014-02-01 19:06:22,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-90) [5555a419] HostName = vms01.lis.ua
2014-02-01 19:06:22,647 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-90) [5555a419] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:06:22,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-90) [5555a419] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:06:22,661 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-90) [5555a419] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 26076724
2014-02-01 19:06:27,659 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-18) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 629d217e
2014-02-01 19:06:27,692 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-18) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at a4e55187}, log id: 629d217e
2014-02-01 19:06:28,030 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-90) [5555a419] FINISH, ConnectStoragePoolVDSCommand, log id: 26076724
2014-02-01 19:06:28,031 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-90) [5555a419] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:06:28,044 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-90) [5555a419] Irs placed on server 0072ab64-05da-4dde-9493-a0af6aa97402 failed. Proceed Failover
2014-02-01 19:06:32,044 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-90) [5555a419] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:06:32,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-90) [5555a419] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 3a23de22
2014-02-01 19:06:32,701 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-20) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 47e5d908
2014-02-01 19:06:32,735 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-20) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at da4103d}, log id: 47e5d908
2014-02-01 19:06:37,745 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-17) [6dd7fbdf] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 63ac1e9a
2014-02-01 19:06:39,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-90) [5555a419] FINISH, ConnectStoragePoolVDSCommand, log id: 3a23de22
2014-02-01 19:06:39,779 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-90) [5555a419] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:06:39,780 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-17) [6dd7fbdf] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 12eca59c}, log id: 63ac1e9a
2014-02-01 19:06:44,789 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-41) [52947046] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 25123e2c
2014-02-01 19:06:44,820 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-41) [52947046] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at adaf6b73}, log id: 25123e2c
2014-02-01 19:06:49,829 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-59) [42d3ce47] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 53e7e118
2014-02-01 19:06:49,861 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-59) [42d3ce47] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at a9ae0f2}, log id: 53e7e118
2014-02-01 19:06:54,870 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-4) [5c2c5123] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 18ef7e1b
2014-02-01 19:06:54,902 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-4) [5c2c5123] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 48cea67b}, log id: 18ef7e1b
2014-02-01 19:06:59,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-19) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:06:59,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-19) HostName = vms01.lis.ua
2014-02-01 19:06:59,584 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-19) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:06:59,594 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-19) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:06:59,597 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-19) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 3f2e9436
2014-02-01 19:06:59,913 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-11) [4ad2e5fd] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 24cd1007
2014-02-01 19:06:59,947 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-11) [4ad2e5fd] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 269b61a2}, log id: 24cd1007
2014-02-01 19:07:02,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-19) FINISH, ConnectStoragePoolVDSCommand, log id: 3f2e9436
2014-02-01 19:07:02,976 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-19) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:07:02,988 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-19) Irs placed on server 0072ab64-05da-4dde-9493-a0af6aa97402 failed. Proceed Failover
2014-02-01 19:07:04,956 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-87) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 5877a9aa
2014-02-01 19:07:04,989 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-87) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 33752c61}, log id: 5877a9aa
2014-02-01 19:07:04,998 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-19) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:07:05,001 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-19) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 32b80901
2014-02-01 19:07:09,997 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-51) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 7bc310a4
2014-02-01 19:07:10,029 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-51) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at c09bb753}, log id: 7bc310a4
2014-02-01 19:07:10,466 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-19) FINISH, ConnectStoragePoolVDSCommand, log id: 32b80901
2014-02-01 19:07:10,466 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-19) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:07:15,038 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-9) [173dbe9d] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 1dbc2a74
2014-02-01 19:07:16,102 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-26) START, GlusterTasksListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 7607e16a
2014-02-01 19:07:19,560 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-9) [173dbe9d] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 4b44c869}, log id: 1dbc2a74
2014-02-01 19:07:21,559 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-26) Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:07:21,560 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-26) FINISH, GlusterTasksListVDSCommand, log id: 7607e16a
2014-02-01 19:07:21,561 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-26) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:07:24,570 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-66) [40858177] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 1f226ecc
2014-02-01 19:07:31,571 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp--127.0.0.1-8702-1) Running command: LoginAdminUserCommand internal: false.
2014-02-01 19:07:31,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-85) [22829429] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:07:31,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-85) [22829429] HostName = vms01.lis.ua
2014-02-01 19:07:31,586 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-85) [22829429] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:07:31,598 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-85) [22829429] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:07:31,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-85) [22829429] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 69a050f0
2014-02-01 19:07:31,605 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User support logged in.
2014-02-01 19:07:31,611 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-66) [40858177] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at ef9085e3}, log id: 1f226ecc
2014-02-01 19:07:31,647 INFO [org.ovirt.engine.api.restapi.security.auth.LoginValidator] (ajp--127.0.0.1-8702-9) Validating session failed, reason: Session does not exist.
2014-02-01 19:07:35,935 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-85) [22829429] FINISH, ConnectStoragePoolVDSCommand, log id: 69a050f0
2014-02-01 19:07:35,936 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-85) [22829429] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:07:35,947 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-85) [22829429] Irs placed on server 0072ab64-05da-4dde-9493-a0af6aa97402 failed. Proceed Failover
2014-02-01 19:07:36,622 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-78) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 3ca9aa3a
2014-02-01 19:07:36,655 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-78) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 26afba5e}, log id: 3ca9aa3a
2014-02-01 19:07:40,090 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-85) [22829429] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:07:40,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-85) [22829429] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 78abbc57
2014-02-01 19:07:41,684 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-58) [2d2a3fc9] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 1ef1b498
2014-02-01 19:07:45,608 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-85) [22829429] FINISH, ConnectStoragePoolVDSCommand, log id: 78abbc57
2014-02-01 19:07:45,609 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-58) [2d2a3fc9] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at a1c61619}, log id: 1ef1b498
2014-02-01 19:07:45,609 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-85) [22829429] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:07:50,619 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-54) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 41123b1e
2014-02-01 19:07:50,650 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-54) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at e9b7d409}, log id: 41123b1e
2014-02-01 19:07:55,659 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-12) [41c61dd1] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 1ed51fae
2014-02-01 19:08:01,932 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-18) Failed to acquire lock and wait lock EngineLock [exclusiveLocks= key: 99408929-82cf-4dc7-a532-9d998063fa95 value: GLUSTER
2014-02-01 19:08:04,237 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-18) Failed to acquire lock and wait lock EngineLock [exclusiveLocks= key: 99408929-82cf-4dc7-a532-9d998063fa95 value: GLUSTER
2014-02-01 19:08:04,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:08:04,660 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-12) [41c61dd1] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 4ca7a093}, log id: 1ed51fae
2014-02-01 19:08:04,661 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-80) HostName = vms01.lis.ua
2014-02-01 19:08:04,662 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-80) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:08:04,667 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler_Worker-18) START, GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 4836acd7
2014-02-01 19:08:04,706 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler_Worker-18) FINISH, GetGlusterVolumeAdvancedDetailsVDSCommand, return: org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeAdvancedDetails at 480137e3, log id: 4836acd7
2014-02-01 19:08:06,668 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-80) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:08:06,672 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-80) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 7da116b2
2014-02-01 19:08:09,670 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-48) [5544b7d0] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 7f2fa8b8
2014-02-01 19:08:12,171 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-48) [5544b7d0] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at c368c343}, log id: 7f2fa8b8
2014-02-01 19:08:14,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-80) FINISH, ConnectStoragePoolVDSCommand, log id: 7da116b2
2014-02-01 19:08:14,169 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-80) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:08:16,668 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-80) Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:08:16,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-80) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:08:16,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-80) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: ba7f11d
2014-02-01 19:08:17,180 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-69) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 4fb047a
2014-02-01 19:08:17,214 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-69) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 72ab4f87}, log id: 4fb047a
2014-02-01 19:08:19,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-80) FINISH, ConnectStoragePoolVDSCommand, log id: ba7f11d
2014-02-01 19:08:19,050 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-80) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:08:21,570 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-38) [120e5b00] START, GlusterTasksListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 6e276b78
2014-02-01 19:08:22,223 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-55) [2783e7c5] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 4f78cbf6
2014-02-01 19:08:23,577 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-38) [120e5b00] Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:08:23,578 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-38) [120e5b00] FINISH, GlusterTasksListVDSCommand, log id: 6e276b78
2014-02-01 19:08:23,578 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-38) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:08:29,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-4) [5c2c5123] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:08:29,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-4) [5c2c5123] HostName = vms02.lis.ua
2014-02-01 19:08:29,075 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-4) [5c2c5123] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:08:29,107 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-4) [5c2c5123] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:08:29,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-4) [5c2c5123] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 94c7b88
2014-02-01 19:08:30,615 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-55) [2783e7c5] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 1bdd112b}, log id: 4f78cbf6
2014-02-01 19:08:32,545 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-4) [5c2c5123] FINISH, ConnectStoragePoolVDSCommand, log id: 94c7b88
2014-02-01 19:08:32,545 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-4) [5c2c5123] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:08:32,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-4) [5c2c5123] Irs placed on server 0072ab64-05da-4dde-9493-a0af6aa97402 failed. Proceed Failover
2014-02-01 19:08:35,625 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-21) [24a0a9c9] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 5ad6f8f4
2014-02-01 19:08:35,733 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-4) [5c2c5123] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:08:35,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-4) [5c2c5123] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 58f1ae25
2014-02-01 19:08:42,768 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-21) [24a0a9c9] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at c2beca43}, log id: 5ad6f8f4
2014-02-01 19:08:43,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-4) [5c2c5123] FINISH, ConnectStoragePoolVDSCommand, log id: 58f1ae25
2014-02-01 19:08:43,653 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-4) [5c2c5123] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:08:47,777 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-52) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 54698f87
2014-02-01 19:08:47,810 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-52) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at ffbb1a7b}, log id: 54698f87
2014-02-01 19:08:52,819 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-2) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 291e2194
2014-02-01 19:08:55,670 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-2) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 25ab70dd}, log id: 291e2194
2014-02-01 19:09:00,679 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-6) [727a775a] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 4c3e9094
2014-02-01 19:09:00,711 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-6) [727a775a] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 1ee9caf4}, log id: 4c3e9094
2014-02-01 19:09:05,721 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-5) [7688e091] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 2ce8f61
2014-02-01 19:09:05,753 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-5) [7688e091] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at de16e6d5}, log id: 2ce8f61
2014-02-01 19:09:07,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:09:07,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-71) HostName = vms01.lis.ua
2014-02-01 19:09:07,187 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-71) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:09:07,199 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-71) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:09:07,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-71) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: c7dd625
2014-02-01 19:09:10,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-71) FINISH, ConnectStoragePoolVDSCommand, log id: c7dd625
2014-02-01 19:09:10,592 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-71) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:09:10,603 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-71) Irs placed on server 0072ab64-05da-4dde-9493-a0af6aa97402 failed. Proceed Failover
2014-02-01 19:09:10,761 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-78) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 34693b60
2014-02-01 19:09:12,717 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-71) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:09:12,720 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-71) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 4381ec82
2014-02-01 19:09:19,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-71) FINISH, ConnectStoragePoolVDSCommand, log id: 4381ec82
2014-02-01 19:09:19,216 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-71) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:09:19,752 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-78) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 60002e54}, log id: 34693b60
2014-02-01 19:09:23,605 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-72) START, GlusterTasksListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 4566ea7b
2014-02-01 19:09:23,650 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-72) Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:09:23,651 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-72) FINISH, GlusterTasksListVDSCommand, log id: 4566ea7b
2014-02-01 19:09:23,651 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-72) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:09:24,761 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-3) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 5fec9609
2014-02-01 19:09:26,796 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 60ea7a9f}, log id: 5fec9609
2014-02-01 19:09:31,805 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-89) [e0b5c73] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 72760165
2014-02-01 19:09:38,850 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-15) [1c9d0819] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:09:38,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-15) [1c9d0819] HostName = vms01.lis.ua
2014-02-01 19:09:38,852 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-89) [e0b5c73] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at cc02cb05}, log id: 72760165
2014-02-01 19:09:38,852 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-15) [1c9d0819] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:09:38,865 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-15) [1c9d0819] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:09:38,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-15) [1c9d0819] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 2b46e2f5
2014-02-01 19:09:42,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-15) [1c9d0819] FINISH, ConnectStoragePoolVDSCommand, log id: 2b46e2f5
2014-02-01 19:09:42,179 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-15) [1c9d0819] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:09:42,190 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-15) [1c9d0819] Irs placed on server 0072ab64-05da-4dde-9493-a0af6aa97402 failed. Proceed Failover
2014-02-01 19:09:43,871 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-75) [6968161f] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 14f0cf13
2014-02-01 19:09:45,355 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-15) [1c9d0819] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:09:45,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-15) [1c9d0819] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 13568df1
2014-02-01 19:09:49,856 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-75) [6968161f] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 8f572f7d}, log id: 14f0cf13
2014-02-01 19:09:54,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-15) [1c9d0819] FINISH, ConnectStoragePoolVDSCommand, log id: 13568df1
2014-02-01 19:09:54,360 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-15) [1c9d0819] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:09:54,891 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-36) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 31ebfe90
2014-02-01 19:09:54,922 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-36) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at f8869786}, log id: 31ebfe90
2014-02-01 19:09:59,932 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-76) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 5a10e4a1
2014-02-01 19:09:59,963 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-76) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at ce2c4b48}, log id: 5a10e4a1
2014-02-01 19:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-38) [120e5b00] Autorecovering 1 storage domains
2014-02-01 19:10:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-38) [120e5b00] Autorecovering storage domains id: 7f9709c1-3ab6-4af8-9e58-955ef7c9452e
2014-02-01 19:10:00,004 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-38) [2f5cea56] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 7f9709c1-3ab6-4af8-9e58-955ef7c9452e Type: Storage
2014-02-01 19:10:00,004 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-38) [2f5cea56] ConnectDomainToStorage. Before Connect all hosts to pool. Time:2/1/14 7:10 PM
2014-02-01 19:10:00,009 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-6-thread-44) [64c949e3] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System
2014-02-01 19:10:00,009 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-6-thread-49) [7d3fd4df] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System
2014-02-01 19:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-44) [64c949e3] START, ConnectStorageServerVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = GLUSTERFS, connectionList = [{ id: 4a1d8b07-f393-4134-86cc-4f46145cca2b, connection: vms02.lis.ua:STORAGE, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2e530d42
2014-02-01 19:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-49) [7d3fd4df] START, ConnectStorageServerVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = GLUSTERFS, connectionList = [{ id: 4a1d8b07-f393-4134-86cc-4f46145cca2b, connection: vms02.lis.ua:STORAGE, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 67286cad
2014-02-01 19:10:00,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-49) [7d3fd4df] FINISH, ConnectStorageServerVDSCommand, return: {4a1d8b07-f393-4134-86cc-4f46145cca2b=0}, log id: 67286cad
2014-02-01 19:10:04,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-44) [64c949e3] FINISH, ConnectStorageServerVDSCommand, return: {4a1d8b07-f393-4134-86cc-4f46145cca2b=0}, log id: 2e530d42
2014-02-01 19:10:04,375 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-38) [2f5cea56] ConnectDomainToStorage. After Connect all hosts to pool. Time:2/1/14 7:10 PM
2014-02-01 19:10:04,972 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-55) [2783e7c5] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 9712dbf
2014-02-01 19:10:05,003 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-55) [2783e7c5] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 933c3c73}, log id: 9712dbf
2014-02-01 19:10:10,012 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-8) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 4f4e462
2014-02-01 19:10:10,044 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-8) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 16bc4de7}, log id: 4f4e462
2014-02-01 19:10:14,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-70) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:10:14,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-70) HostName = vms01.lis.ua
2014-02-01 19:10:14,394 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-70) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:10:14,405 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-70) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:10:14,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-70) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: e6ae0b0
2014-02-01 19:10:15,054 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-4) [5c2c5123] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 30ce812a
2014-02-01 19:10:15,086 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-4) [5c2c5123] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 9a65b9a5}, log id: 30ce812a
2014-02-01 19:10:17,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-70) FINISH, ConnectStoragePoolVDSCommand, log id: e6ae0b0
2014-02-01 19:10:17,767 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-70) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:10:17,777 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-70) Irs placed on server 0072ab64-05da-4dde-9493-a0af6aa97402 failed. Proceed Failover
2014-02-01 19:10:19,778 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-70) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:10:19,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-70) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 3de00677
2014-02-01 19:10:20,095 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-52) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 699d6a2a
2014-02-01 19:10:20,128 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-52) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at faa3a62e}, log id: 699d6a2a
2014-02-01 19:10:23,662 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-19) START, GlusterTasksListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 59199950
2014-02-01 19:10:25,137 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-84) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 17bf253b
2014-02-01 19:10:25,169 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-84) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at f447bf99}, log id: 17bf253b
2014-02-01 19:10:25,670 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-70) FINISH, ConnectStoragePoolVDSCommand, log id: 3de00677
2014-02-01 19:10:25,671 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-70) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:10:25,708 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-19) Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:10:25,709 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-19) FINISH, GlusterTasksListVDSCommand, log id: 59199950
2014-02-01 19:10:25,710 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-19) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:10:30,179 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-9) [173dbe9d] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 3c33abaf
2014-02-01 19:10:30,211 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-9) [173dbe9d] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at b87f4636}, log id: 3c33abaf
2014-02-01 19:10:35,221 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-14) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 584c6f8b
2014-02-01 19:10:35,254 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 8fc2f3bd}, log id: 584c6f8b
2014-02-01 19:10:40,264 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-68) [584ec02a] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 35af9d6d
2014-02-01 19:10:40,295 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-68) [584ec02a] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 96b75725}, log id: 35af9d6d
2014-02-01 19:10:42,685 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-88) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:10:42,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-88) HostName = vms01.lis.ua
2014-02-01 19:10:42,687 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-88) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:10:45,304 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-63) [5dcfa174] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 6db42ba9
2014-02-01 19:10:45,335 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-63) [5dcfa174] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at a561c98e}, log id: 6db42ba9
2014-02-01 19:10:47,684 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-88) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:10:47,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-88) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 42b8de2c
2014-02-01 19:10:50,344 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-43) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 2a081ce6
2014-02-01 19:10:50,384 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-43) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 49c6060}, log id: 2a081ce6
2014-02-01 19:10:52,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-88) FINISH, ConnectStoragePoolVDSCommand, log id: 42b8de2c
2014-02-01 19:10:52,701 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-88) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:10:55,393 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-72) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 3d8934ee
2014-02-01 19:10:55,425 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-72) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 111d38a7}, log id: 3d8934ee
2014-02-01 19:10:59,722 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-88) Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:10:59,756 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-88) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:10:59,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-88) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 7e22365d
2014-02-01 19:11:00,435 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-3) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 48768322
2014-02-01 19:11:02,465 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 2857c7a6}, log id: 48768322
2014-02-01 19:11:03,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-88) FINISH, ConnectStoragePoolVDSCommand, log id: 7e22365d
2014-02-01 19:11:03,111 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-88) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:11:07,475 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-89) [e0b5c73] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 2307222f
2014-02-01 19:11:07,507 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-89) [e0b5c73] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at fdee2d71}, log id: 2307222f
2014-02-01 19:11:12,518 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-40) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 39e814cb
2014-02-01 19:11:12,550 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-40) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 54d3a188}, log id: 39e814cb
2014-02-01 19:11:13,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:11:13,142 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] HostName = vms02.lis.ua
2014-02-01 19:11:13,143 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:11:16,965 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:11:16,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 4e2a47be
2014-02-01 19:11:17,559 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-95) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 3524715d
2014-02-01 19:11:21,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] FINISH, ConnectStoragePoolVDSCommand, log id: 4e2a47be
2014-02-01 19:11:21,465 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:11:24,596 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-95) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 3a7ec060}, log id: 3524715d
2014-02-01 19:11:25,722 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-80) START, GlusterTasksListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 25925992
2014-02-01 19:11:25,766 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:11:25,767 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-80) FINISH, GlusterTasksListVDSCommand, log id: 25925992
2014-02-01 19:11:25,767 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-80) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:11:26,605 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:11:26,616 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:11:26,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 359ee79d
2014-02-01 19:11:29,605 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-76) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 23eed525
2014-02-01 19:11:31,638 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-76) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at c9d288b6}, log id: 23eed525
2014-02-01 19:11:31,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] FINISH, ConnectStoragePoolVDSCommand, log id: 359ee79d
2014-02-01 19:11:31,978 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:11:36,648 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 23665b4a
2014-02-01 19:11:41,170 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at b824f6df}, log id: 23665b4a
2014-02-01 19:11:41,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-64) [2fb23e30] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:11:41,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-64) [2fb23e30] HostName = vms02.lis.ua
2014-02-01 19:11:41,999 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-64) [2fb23e30] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:11:42,010 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) [2fb23e30] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:11:42,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-64) [2fb23e30] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 59d37838
2014-02-01 19:11:45,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-64) [2fb23e30] FINISH, ConnectStoragePoolVDSCommand, log id: 59d37838
2014-02-01 19:11:45,385 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) [2fb23e30] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:11:45,426 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) [2fb23e30] Irs placed on server 0072ab64-05da-4dde-9493-a0af6aa97402 failed. Proceed Failover
2014-02-01 19:11:46,180 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-96) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 3424a31f
2014-02-01 19:11:46,212 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 8d21e6cc}, log id: 3424a31f
2014-02-01 19:11:48,590 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) [2fb23e30] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:11:48,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-64) [2fb23e30] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 6139c5cd
2014-02-01 19:11:51,222 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-52) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: fa2cd01
2014-02-01 19:11:54,080 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-52) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 80fa00d3}, log id: fa2cd01
2014-02-01 19:11:56,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-64) [2fb23e30] FINISH, ConnectStoragePoolVDSCommand, log id: 6139c5cd
2014-02-01 19:11:56,559 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) [2fb23e30] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:11:59,089 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 16b7c4c5
2014-02-01 19:12:03,575 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 14eb95a4}, log id: 16b7c4c5
2014-02-01 19:12:08,592 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-73) [667071b9] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 73f59a1f
2014-02-01 19:12:15,631 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:12:15,631 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-73) [667071b9] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 5ebed583}, log id: 73f59a1f
2014-02-01 19:12:15,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-47) HostName = vms01.lis.ua
2014-02-01 19:12:15,633 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:12:17,669 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-47) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:12:17,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-47) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 51aaf504
2014-02-01 19:12:20,642 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-1) [3fe67e5e] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 216256e
2014-02-01 19:12:20,675 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-1) [3fe67e5e] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 3792bb73}, log id: 216256e
2014-02-01 19:12:25,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-47) FINISH, ConnectStoragePoolVDSCommand, log id: 51aaf504
2014-02-01 19:12:25,185 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-47) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:12:25,684 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-78) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 5541ce67
2014-02-01 19:12:25,717 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-78) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 2dc186b9}, log id: 5541ce67
2014-02-01 19:12:25,778 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-97) START, GlusterTasksListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 698027e
2014-02-01 19:12:25,822 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-97) Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:12:25,823 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-97) FINISH, GlusterTasksListVDSCommand, log id: 698027e
2014-02-01 19:12:25,824 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-97) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:12:27,685 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-47) Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:12:27,697 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-47) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:12:27,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-47) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 4fa13360
2014-02-01 19:12:30,726 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-54) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 654dca21
2014-02-01 19:12:30,789 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-54) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 64df666b}, log id: 654dca21
2014-02-01 19:12:31,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-47) FINISH, ConnectStoragePoolVDSCommand, log id: 4fa13360
2014-02-01 19:12:31,117 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-47) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:12:35,810 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-88) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 79bb64d8
2014-02-01 19:12:35,842 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-88) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at ead965bf}, log id: 79bb64d8
2014-02-01 19:12:40,851 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-28) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 285c4f
2014-02-01 19:12:41,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-12) [41c61dd1] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:12:41,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-12) [41c61dd1] HostName = vms02.lis.ua
2014-02-01 19:12:41,141 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-12) [41c61dd1] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:12:43,153 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-28) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 4db1e676}, log id: 285c4f
2014-02-01 19:12:45,152 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-12) [41c61dd1] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:12:45,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-12) [41c61dd1] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 7273291e
2014-02-01 19:12:48,188 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-50) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 2afab50
2014-02-01 19:12:50,625 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-50) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at a9dc3768}, log id: 2afab50
2014-02-01 19:12:50,625 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-12) [41c61dd1] FINISH, ConnectStoragePoolVDSCommand, log id: 7273291e
2014-02-01 19:12:50,627 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-12) [41c61dd1] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:12:55,634 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-90) [5555a419] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 4b2bda35
2014-02-01 19:12:57,658 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-12) [41c61dd1] Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:12:57,669 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-12) [41c61dd1] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:12:57,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-12) [41c61dd1] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 2a78f6c8
2014-02-01 19:12:59,696 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-90) [5555a419] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 46253ec0}, log id: 4b2bda35
2014-02-01 19:13:01,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-12) [41c61dd1] FINISH, ConnectStoragePoolVDSCommand, log id: 2a78f6c8
2014-02-01 19:13:01,077 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-12) [41c61dd1] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:13:04,706 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-80) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 347276f0
2014-02-01 19:13:04,713 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-49) Failed to acquire lock and wait lock EngineLock [exclusiveLocks= key: 99408929-82cf-4dc7-a532-9d998063fa95 value: GLUSTER
2014-02-01 19:13:06,676 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-49) Failed to acquire lock and wait lock EngineLock [exclusiveLocks= key: 99408929-82cf-4dc7-a532-9d998063fa95 value: GLUSTER
2014-02-01 19:13:09,241 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-80) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at fd5d967a}, log id: 347276f0
2014-02-01 19:13:09,247 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler_Worker-49) START, GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 5078bbd5
2014-02-01 19:13:09,287 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler_Worker-49) FINISH, GetGlusterVolumeAdvancedDetailsVDSCommand, return: org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeAdvancedDetails at 2ca3bf6c, log id: 5078bbd5
2014-02-01 19:13:11,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:13:11,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-24) HostName = vms02.lis.ua
2014-02-01 19:13:11,101 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-24) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:13:14,168 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-24) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:13:14,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-24) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 50be8b3
2014-02-01 19:13:14,250 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-11) [4ad2e5fd] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 44185ffa
2014-02-01 19:13:17,607 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-11) [4ad2e5fd] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 3277bfea}, log id: 44185ffa
2014-02-01 19:13:22,617 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-25) [2f977190] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 4e7cba15
2014-02-01 19:13:24,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-24) FINISH, ConnectStoragePoolVDSCommand, log id: 50be8b3
2014-02-01 19:13:24,646 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-24) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:13:25,833 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-33) START, GlusterTasksListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 2b2a9b30
2014-02-01 19:13:25,877 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-33) Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:13:25,878 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-33) FINISH, GlusterTasksListVDSCommand, log id: 2b2a9b30
2014-02-01 19:13:25,879 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-33) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:13:26,668 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-24) Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:13:26,670 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-25) [2f977190] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 585a0df8}, log id: 4e7cba15
2014-02-01 19:13:26,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-24) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:13:26,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-24) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 4ab7c89
2014-02-01 19:13:31,023 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-24) FINISH, ConnectStoragePoolVDSCommand, log id: 4ab7c89
2014-02-01 19:13:31,023 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-24) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:13:31,679 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-64) [2fb23e30] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 605f1589
2014-02-01 19:13:31,711 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-64) [2fb23e30] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 9c2f5699}, log id: 605f1589
2014-02-01 19:13:36,721 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-13) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: f6eda0d
2014-02-01 19:13:36,755 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-13) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 12b897b4}, log id: f6eda0d
2014-02-01 19:13:41,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-51) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:13:41,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-51) HostName = vms02.lis.ua
2014-02-01 19:13:41,068 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-51) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:13:41,764 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-60) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 77ae2008
2014-02-01 19:13:46,184 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-60) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 83cb6dd}, log id: 77ae2008
2014-02-01 19:13:49,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-51) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:13:49,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-51) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 2fe96461
2014-02-01 19:13:51,193 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-73) [667071b9] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 3960a375
2014-02-01 19:13:55,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-51) FINISH, ConnectStoragePoolVDSCommand, log id: 2fe96461
2014-02-01 19:13:55,592 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-51) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:13:55,592 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-73) [667071b9] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 4b2b8a2d}, log id: 3960a375
2014-02-01 19:14:00,602 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-71) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 7e180ea7
2014-02-01 19:14:00,635 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-71) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at c6187124}, log id: 7e180ea7
2014-02-01 19:14:02,611 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-51) Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:14:02,633 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-51) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:14:02,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-51) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 782d128f
2014-02-01 19:14:05,645 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-72) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 2ae437b7
2014-02-01 19:14:06,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-51) FINISH, ConnectStoragePoolVDSCommand, log id: 782d128f
2014-02-01 19:14:06,032 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-51) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:14:09,686 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-72) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at f2e33584}, log id: 2ae437b7
2014-02-01 19:14:14,694 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-88) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 710097e1
2014-02-01 19:14:14,725 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-88) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 92a1a1ea}, log id: 710097e1
2014-02-01 19:14:16,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-62) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:14:16,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-62) HostName = vms02.lis.ua
2014-02-01 19:14:16,057 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-62) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:14:19,685 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-62) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:14:19,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-62) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 6899856a
2014-02-01 19:14:19,733 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-20) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 31c39b27
2014-02-01 19:14:25,890 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-75) [6968161f] START, GlusterTasksListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 198d5082
2014-02-01 19:14:26,768 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-20) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 6a980012}, log id: 31c39b27
2014-02-01 19:14:27,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-62) FINISH, ConnectStoragePoolVDSCommand, log id: 6899856a
2014-02-01 19:14:27,591 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-62) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:14:29,629 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-75) [6968161f] Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:14:29,630 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-75) [6968161f] FINISH, GlusterTasksListVDSCommand, log id: 198d5082
2014-02-01 19:14:29,630 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-75) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:14:31,778 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-36) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 290534ba
2014-02-01 19:14:31,811 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-36) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at f15ea18b}, log id: 290534ba
2014-02-01 19:14:36,595 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-62) Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:14:36,609 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-62) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:14:36,611 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-62) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 208f3d89
2014-02-01 19:14:36,822 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-29) [288044b9] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 7ff63634
2014-02-01 19:14:39,001 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-62) FINISH, ConnectStoragePoolVDSCommand, log id: 208f3d89
2014-02-01 19:14:39,002 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-62) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:14:41,107 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-29) [288044b9] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 71df361a}, log id: 7ff63634
2014-02-01 19:14:46,116 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-49) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 789b7bd3
2014-02-01 19:14:49,020 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-41) [52947046] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:14:49,022 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-41) [52947046] HostName = vms02.lis.ua
2014-02-01 19:14:49,022 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-41) [52947046] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:14:50,637 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-49) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 142c1d42}, log id: 789b7bd3
2014-02-01 19:14:53,136 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-41) [52947046] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:14:53,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-41) [52947046] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 5c43e7a7
2014-02-01 19:14:55,647 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-87) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 762b9806
2014-02-01 19:14:57,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-41) [52947046] FINISH, ConnectStoragePoolVDSCommand, log id: 5c43e7a7
2014-02-01 19:14:57,672 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-41) [52947046] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:14:57,673 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-87) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 7b69a3ce}, log id: 762b9806
2014-02-01 19:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-35) [39f0155c] Autorecovering 1 storage domains
2014-02-01 19:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-35) [39f0155c] Autorecovering storage domains id: 7f9709c1-3ab6-4af8-9e58-955ef7c9452e
2014-02-01 19:15:00,005 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-35) [7769700f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 7f9709c1-3ab6-4af8-9e58-955ef7c9452e Type: Storage
2014-02-01 19:15:00,005 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-35) [7769700f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:2/1/14 7:15 PM
2014-02-01 19:15:00,010 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-6-thread-44) [4c811145] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System
2014-02-01 19:15:00,011 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-6-thread-49) [3239447] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System
2014-02-01 19:15:00,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-44) [4c811145] START, ConnectStorageServerVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = GLUSTERFS, connectionList = [{ id: 4a1d8b07-f393-4134-86cc-4f46145cca2b, connection: vms02.lis.ua:STORAGE, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 71cd4bbf
2014-02-01 19:15:00,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-49) [3239447] START, ConnectStorageServerVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = GLUSTERFS, connectionList = [{ id: 4a1d8b07-f393-4134-86cc-4f46145cca2b, connection: vms02.lis.ua:STORAGE, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 39643fa
2014-02-01 19:15:00,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-49) [3239447] FINISH, ConnectStorageServerVDSCommand, return: {4a1d8b07-f393-4134-86cc-4f46145cca2b=0}, log id: 39643fa
2014-02-01 19:15:02,682 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-33) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 531bee7a
2014-02-01 19:15:02,715 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-33) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 50018f27}, log id: 531bee7a
2014-02-01 19:15:05,552 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-41) [52947046] Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:15:05,562 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-41) [52947046] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:15:05,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-41) [52947046] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 1a8f0d83
2014-02-01 19:15:07,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-44) [4c811145] FINISH, ConnectStorageServerVDSCommand, return: {4a1d8b07-f393-4134-86cc-4f46145cca2b=0}, log id: 71cd4bbf
2014-02-01 19:15:07,559 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-35) [7769700f] ConnectDomainToStorage. After Connect all hosts to pool. Time:2/1/14 7:15 PM
2014-02-01 19:15:07,724 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-84) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 3e55fa29
2014-02-01 19:15:09,936 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-41) [52947046] FINISH, ConnectStoragePoolVDSCommand, log id: 1a8f0d83
2014-02-01 19:15:09,936 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-41) [52947046] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:15:15,693 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-84) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at ec5b7176}, log id: 3e55fa29
2014-02-01 19:15:19,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-66) [40858177] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:15:19,958 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-66) [40858177] HostName = vms02.lis.ua
2014-02-01 19:15:19,958 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-66) [40858177] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:15:19,968 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-66) [40858177] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:15:19,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-66) [40858177] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 133cf439
2014-02-01 19:15:20,701 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-60) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 3133087e
2014-02-01 19:15:20,734 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-60) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 3183cee5}, log id: 3133087e
2014-02-01 19:15:23,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-66) [40858177] FINISH, ConnectStoragePoolVDSCommand, log id: 133cf439
2014-02-01 19:15:23,316 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-66) [40858177] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:15:23,327 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-66) [40858177] Irs placed on server 0072ab64-05da-4dde-9493-a0af6aa97402 failed. Proceed Failover
2014-02-01 19:15:25,342 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-66) [40858177] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:15:25,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-66) [40858177] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 55eccb7a
2014-02-01 19:15:25,778 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-93) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 4c46c1e7
2014-02-01 19:15:29,641 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-1) [3fe67e5e] START, GlusterTasksListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 11358573
2014-02-01 19:15:30,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-66) [40858177] FINISH, ConnectStoragePoolVDSCommand, log id: 55eccb7a
2014-02-01 19:15:30,842 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-66) [40858177] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:15:32,815 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-93) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at e57d3b6f}, log id: 4c46c1e7
2014-02-01 19:15:35,342 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-1) [3fe67e5e] Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:15:35,343 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-1) [3fe67e5e] FINISH, GlusterTasksListVDSCommand, log id: 11358573
2014-02-01 19:15:35,344 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-1) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:15:37,826 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-23) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 264e500f
2014-02-01 19:15:37,859 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-23) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 8936b886}, log id: 264e500f
2014-02-01 19:15:42,869 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-54) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 18e9c488
2014-02-01 19:15:47,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-92) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:15:47,356 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-54) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at b973f547}, log id: 18e9c488
2014-02-01 19:15:47,357 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-92) HostName = vms01.lis.ua
2014-02-01 19:15:47,358 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-92) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:15:51,859 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-92) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:15:51,862 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-92) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 6deca58d
2014-02-01 19:15:52,367 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-27) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 10dee6ca
2014-02-01 19:15:57,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-92) FINISH, ConnectStoragePoolVDSCommand, log id: 6deca58d
2014-02-01 19:15:57,329 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-27) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 9ce97d3b}, log id: 10dee6ca
2014-02-01 19:15:57,329 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-92) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:16:01,826 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-92) Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:16:01,838 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-92) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:16:01,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-92) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 6bd8092
2014-02-01 19:16:02,339 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-82) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 7d582b11
2014-02-01 19:16:06,843 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-82) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 516bc516}, log id: 7d582b11
2014-02-01 19:16:07,321 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-92) FINISH, ConnectStoragePoolVDSCommand, log id: 6bd8092
2014-02-01 19:16:07,321 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-92) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:16:11,879 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-10) [18711601] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 7e4212f1
2014-02-01 19:16:11,910 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-10) [18711601] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at ea939ca7}, log id: 7e4212f1
2014-02-01 19:16:16,920 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-37) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 7af852ef
2014-02-01 19:16:16,953 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-37) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at b90097ad}, log id: 7af852ef
2014-02-01 19:16:17,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-76) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:16:17,412 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-76) HostName = vms02.lis.ua
2014-02-01 19:16:17,412 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-76) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:16:21,408 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:16:21,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-76) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 77c1fc81
2014-02-01 19:16:21,963 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-16) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 4448b84c
2014-02-01 19:16:21,994 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-16) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 2740b45c}, log id: 4448b84c
2014-02-01 19:16:26,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-76) FINISH, ConnectStoragePoolVDSCommand, log id: 77c1fc81
2014-02-01 19:16:26,800 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:16:27,004 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-44) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 12b46cd1
2014-02-01 19:16:27,038 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at b2bfc83f}, log id: 12b46cd1
2014-02-01 19:16:31,810 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:16:31,821 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:16:31,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-76) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 200be74a
2014-02-01 19:16:32,047 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-4) [5c2c5123] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 2148d94c
2014-02-01 19:16:35,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-76) FINISH, ConnectStoragePoolVDSCommand, log id: 200be74a
2014-02-01 19:16:35,255 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:16:35,355 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-56) [21ac4370] START, GlusterTasksListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 4c5455f3
2014-02-01 19:16:36,829 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-4) [5c2c5123] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 1dcddbe0}, log id: 2148d94c
2014-02-01 19:16:41,365 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-56) [21ac4370] Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:16:41,365 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-56) [21ac4370] FINISH, GlusterTasksListVDSCommand, log id: 4c5455f3
2014-02-01 19:16:41,366 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-56) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:16:41,867 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-35) [7769700f] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 6daad205
2014-02-01 19:16:45,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-19) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:16:45,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-19) HostName = vms02.lis.ua
2014-02-01 19:16:45,279 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-19) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:16:46,374 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-35) [7769700f] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 36842fed}, log id: 6daad205
2014-02-01 19:16:51,383 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-14) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 3ca3b91
2014-02-01 19:16:53,377 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-19) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:16:53,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-19) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 207f0965
2014-02-01 19:16:55,904 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at f2b1a4e}, log id: 3ca3b91
2014-02-01 19:17:00,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-19) FINISH, ConnectStoragePoolVDSCommand, log id: 207f0965
2014-02-01 19:17:00,404 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-19) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:17:00,913 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-63) [5dcfa174] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 25a91387
2014-02-01 19:17:00,944 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-63) [5dcfa174] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at ba6aff9a}, log id: 25a91387
2014-02-01 19:17:05,403 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-19) Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:17:05,413 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-19) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:17:05,415 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-19) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 2ebb3a68
2014-02-01 19:17:05,953 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-1) [3fe67e5e] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 271c480c
2014-02-01 19:17:05,987 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-1) [3fe67e5e] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 574c60e1}, log id: 271c480c
2014-02-01 19:17:08,760 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-19) FINISH, ConnectStoragePoolVDSCommand, log id: 2ebb3a68
2014-02-01 19:17:08,760 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-19) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:17:10,996 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-51) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 6b75695f
2014-02-01 19:17:11,029 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-51) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at edf124c8}, log id: 6b75695f
2014-02-01 19:17:16,037 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-85) [22829429] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 6f73ee14
2014-02-01 19:17:16,068 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-85) [22829429] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at b28034f}, log id: 6f73ee14
2014-02-01 19:17:18,782 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-3) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:17:18,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-3) HostName = vms02.lis.ua
2014-02-01 19:17:18,784 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-3) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:17:18,795 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-3) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:17:18,797 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-3) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 34d0c101
2014-02-01 19:17:21,078 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-79) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 3f93448e
2014-02-01 19:17:21,110 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-79) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at c68d8a50}, log id: 3f93448e
2014-02-01 19:17:23,133 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, ConnectStoragePoolVDSCommand, log id: 34d0c101
2014-02-01 19:17:23,133 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-3) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:17:23,144 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-3) Irs placed on server 0072ab64-05da-4dde-9493-a0af6aa97402 failed. Proceed Failover
2014-02-01 19:17:26,119 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-53) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 5a7bffde
2014-02-01 19:17:26,149 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-53) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at af0f2489}, log id: 5a7bffde
2014-02-01 19:17:27,152 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-3) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:17:27,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-3) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 3f677385
2014-02-01 19:17:31,158 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-98) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 10ff9b75
2014-02-01 19:17:31,191 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-98) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at b9e94ae2}, log id: 10ff9b75
2014-02-01 19:17:32,576 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, ConnectStoragePoolVDSCommand, log id: 3f677385
2014-02-01 19:17:32,577 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-3) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:17:36,200 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-90) [5555a419] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: eccf263
2014-02-01 19:17:40,733 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-90) [5555a419] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at e7e3c92c}, log id: eccf263
2014-02-01 19:17:41,376 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-48) [5544b7d0] START, GlusterTasksListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 46b8cd4c
2014-02-01 19:17:43,372 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-48) [5544b7d0] Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:17:43,373 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-48) [5544b7d0] FINISH, GlusterTasksListVDSCommand, log id: 46b8cd4c
2014-02-01 19:17:43,373 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-48) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:17:45,741 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-59) [42d3ce47] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 7d36a726
2014-02-01 19:17:52,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-10) [18711601] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:17:52,374 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-59) [42d3ce47] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 253b0854}, log id: 7d36a726
2014-02-01 19:17:52,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-10) [18711601] HostName = vms01.lis.ua
2014-02-01 19:17:52,383 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-10) [18711601] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:17:52,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-10) [18711601] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:17:52,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-10) [18711601] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 339d615b
2014-02-01 19:17:55,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-10) [18711601] FINISH, ConnectStoragePoolVDSCommand, log id: 339d615b
2014-02-01 19:17:55,770 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-10) [18711601] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:17:55,780 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-10) [18711601] Irs placed on server 0072ab64-05da-4dde-9493-a0af6aa97402 failed. Proceed Failover
2014-02-01 19:17:57,383 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-11) [4ad2e5fd] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 49584616
2014-02-01 19:17:57,416 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-11) [4ad2e5fd] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 39411487}, log id: 49584616
2014-02-01 19:17:57,793 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-10) [18711601] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:17:57,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-10) [18711601] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 53b036d2
2014-02-01 19:18:02,424 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-87) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 179eb5d7
2014-02-01 19:18:02,459 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-87) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at a3dabce6}, log id: 179eb5d7
2014-02-01 19:18:04,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-10) [18711601] FINISH, ConnectStoragePoolVDSCommand, log id: 53b036d2
2014-02-01 19:18:04,388 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-10) [18711601] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:18:07,468 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-96) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 6f9fc1eb
2014-02-01 19:18:07,499 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 93b50005}, log id: 6f9fc1eb
2014-02-01 19:18:09,312 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler_Worker-64) [2fb23e30] START, GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 1ec15019
2014-02-01 19:18:09,349 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler_Worker-64) [2fb23e30] FINISH, GetGlusterVolumeAdvancedDetailsVDSCommand, return: org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeAdvancedDetails at 31d58d95, log id: 1ec15019
2014-02-01 19:18:12,508 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-41) [52947046] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 2ddedd68
2014-02-01 19:18:14,544 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-41) [52947046] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 89dba5e2}, log id: 2ddedd68
2014-02-01 19:18:19,563 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-83) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 7a15ee23
2014-02-01 19:18:19,600 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-83) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 760566}, log id: 7a15ee23
2014-02-01 19:18:24,620 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-68) [584ec02a] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 2f97494c
2014-02-01 19:18:26,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:18:26,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] HostName = vms01.lis.ua
2014-02-01 19:18:26,061 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:18:26,071 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:18:26,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 67d6a4ee
2014-02-01 19:18:28,560 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-68) [584ec02a] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 9e4df983}, log id: 2f97494c
2014-02-01 19:18:30,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] FINISH, ConnectStoragePoolVDSCommand, log id: 67d6a4ee
2014-02-01 19:18:30,399 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:18:30,411 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] Irs placed on server 0072ab64-05da-4dde-9493-a0af6aa97402 failed. Proceed Failover
2014-02-01 19:18:33,571 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-7) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 7fbd659c
2014-02-01 19:18:35,058 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:18:35,061 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 7c7a3700
2014-02-01 19:18:37,559 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-7) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at f73b1d7f}, log id: 7fbd659c
2014-02-01 19:18:42,567 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-19) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 4a29f102
2014-02-01 19:18:42,600 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-19) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 94d6fe01}, log id: 4a29f102
2014-02-01 19:18:43,383 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-58) [2d2a3fc9] START, GlusterTasksListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 6969fd3d
2014-02-01 19:18:43,428 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-58) [2d2a3fc9] Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:18:43,429 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-58) [2d2a3fc9] FINISH, GlusterTasksListVDSCommand, log id: 6969fd3d
2014-02-01 19:18:43,429 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-58) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:18:45,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] FINISH, ConnectStoragePoolVDSCommand, log id: 7c7a3700
2014-02-01 19:18:45,591 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:18:47,609 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-34) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 696934d8
2014-02-01 19:18:49,639 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-34) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 9dd66e57}, log id: 696934d8
2014-02-01 19:18:54,649 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-54) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 3780650f
2014-02-01 19:19:02,613 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-54) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 21c74261}, log id: 3780650f
2014-02-01 19:19:04,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-27) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:19:04,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-27) HostName = vms01.lis.ua
2014-02-01 19:19:04,624 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-27) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:19:07,621 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-15) [1c9d0819] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 6a66cd7a
2014-02-01 19:19:07,650 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-15) [1c9d0819] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 352fb7f0}, log id: 6a66cd7a
2014-02-01 19:19:11,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-27) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:19:11,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-27) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: a3d7e72
2014-02-01 19:19:12,660 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-82) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 5a386cd1
2014-02-01 19:19:12,691 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-82) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at a8815e4d}, log id: 5a386cd1
2014-02-01 19:19:17,700 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-30) [3ea5fd22] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 6585816b
2014-02-01 19:19:17,733 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-30) [3ea5fd22] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at daa208de}, log id: 6585816b
2014-02-01 19:19:19,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-27) FINISH, ConnectStoragePoolVDSCommand, log id: a3d7e72
2014-02-01 19:19:19,105 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-27) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:19:22,105 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-27) Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:19:22,116 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-27) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:19:22,118 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-27) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: ed494a1
2014-02-01 19:19:22,743 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 24fba8cf
2014-02-01 19:19:22,775 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at ec54578a}, log id: 24fba8cf
2014-02-01 19:19:25,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-27) FINISH, ConnectStoragePoolVDSCommand, log id: ed494a1
2014-02-01 19:19:25,468 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-27) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:19:27,785 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-38) [2f5cea56] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 552d60c1
2014-02-01 19:19:27,817 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-38) [2f5cea56] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 93b83ef1}, log id: 552d60c1
2014-02-01 19:19:32,826 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-49) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 2acfee60
2014-02-01 19:19:32,859 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-49) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at cd99061b}, log id: 2acfee60
2014-02-01 19:19:35,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-86) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:19:35,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-86) HostName = vms02.lis.ua
2014-02-01 19:19:35,492 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-86) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:19:37,498 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-86) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:19:37,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-86) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 5b0cffec
2014-02-01 19:19:37,869 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-52) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 7fe5d347
2014-02-01 19:19:42,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-86) FINISH, ConnectStoragePoolVDSCommand, log id: 5b0cffec
2014-02-01 19:19:42,935 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-86) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:19:43,440 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-96) START, GlusterTasksListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 7d079cea
2014-02-01 19:19:44,909 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-52) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 557457a5}, log id: 7fe5d347
2014-02-01 19:19:47,434 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-86) Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:19:47,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-86) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:19:47,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-86) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 425e69ce
2014-02-01 19:19:49,947 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-70) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: ebf22ed
2014-02-01 19:19:49,980 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-70) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 32ecb57d}, log id: ebf22ed
2014-02-01 19:19:50,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-86) FINISH, ConnectStoragePoolVDSCommand, log id: 425e69ce
2014-02-01 19:19:50,847 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-86) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:19:51,975 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-96) Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:19:51,976 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, GlusterTasksListVDSCommand, log id: 7d079cea
2014-02-01 19:19:51,976 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-96) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:19:54,990 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-9) [173dbe9d] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 70def3e7
2014-02-01 19:19:59,013 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-9) [173dbe9d] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 8198ed13}, log id: 70def3e7
2014-02-01 19:20:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-26) Autorecovering 1 storage domains
2014-02-01 19:20:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-26) Autorecovering storage domains id: 7f9709c1-3ab6-4af8-9e58-955ef7c9452e
2014-02-01 19:20:00,007 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-26) [335e2869] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 7f9709c1-3ab6-4af8-9e58-955ef7c9452e Type: Storage
2014-02-01 19:20:00,008 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-26) [335e2869] ConnectDomainToStorage. Before Connect all hosts to pool. Time:2/1/14 7:20 PM
2014-02-01 19:20:00,012 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-6-thread-47) [929cebe] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System
2014-02-01 19:20:00,013 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-6-thread-49) [19449375] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System
2014-02-01 19:20:00,015 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-47) [929cebe] START, ConnectStorageServerVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = GLUSTERFS, connectionList = [{ id: 4a1d8b07-f393-4134-86cc-4f46145cca2b, connection: vms02.lis.ua:STORAGE, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7c20a081
2014-02-01 19:20:00,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-49) [19449375] START, ConnectStorageServerVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = GLUSTERFS, connectionList = [{ id: 4a1d8b07-f393-4134-86cc-4f46145cca2b, connection: vms02.lis.ua:STORAGE, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1a204439
2014-02-01 19:20:00,040 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-49) [19449375] FINISH, ConnectStorageServerVDSCommand, return: {4a1d8b07-f393-4134-86cc-4f46145cca2b=0}, log id: 1a204439
2014-02-01 19:20:00,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:20:00,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-46) HostName = vms02.lis.ua
2014-02-01 19:20:00,869 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-46) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:20:00,879 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-46) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:20:00,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-46) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 29413871
2014-02-01 19:20:03,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-46) FINISH, ConnectStoragePoolVDSCommand, log id: 29413871
2014-02-01 19:20:03,360 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-46) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:20:03,393 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-46) Irs placed on server 0072ab64-05da-4dde-9493-a0af6aa97402 failed. Proceed Failover
2014-02-01 19:20:04,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-47) [929cebe] FINISH, ConnectStorageServerVDSCommand, return: {4a1d8b07-f393-4134-86cc-4f46145cca2b=0}, log id: 7c20a081
2014-02-01 19:20:04,013 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-26) [335e2869] ConnectDomainToStorage. After Connect all hosts to pool. Time:2/1/14 7:20 PM
2014-02-01 19:20:04,022 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-68) [584ec02a] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 751a6c9c
2014-02-01 19:20:08,512 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-46) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:20:08,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-46) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 7783a7a0
2014-02-01 19:20:10,513 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-68) [584ec02a] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at a9b7efc5}, log id: 751a6c9c
2014-02-01 19:20:15,522 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-97) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 1cf0e502
2014-02-01 19:20:16,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-46) FINISH, ConnectStoragePoolVDSCommand, log id: 7783a7a0
2014-02-01 19:20:16,512 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-46) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:20:19,039 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-97) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at d9c2cb14}, log id: 1cf0e502
2014-02-01 19:20:24,048 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-85) [22829429] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 11c2d7a1
2014-02-01 19:20:24,080 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-85) [22829429] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at cb9c129c}, log id: 11c2d7a1
2014-02-01 19:20:29,090 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-79) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 290c61dd
2014-02-01 19:20:33,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-74) [97b4f50] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:20:33,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-74) [97b4f50] HostName = vms01.lis.ua
2014-02-01 19:20:33,017 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-74) [97b4f50] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:20:37,514 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-74) [97b4f50] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:20:37,515 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-79) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 70f226c6}, log id: 290c61dd
2014-02-01 19:20:37,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-74) [97b4f50] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: c70e04c
2014-02-01 19:20:42,525 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-75) [6968161f] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 13e6c24a
2014-02-01 19:20:42,556 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-75) [6968161f] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 732fcb6d}, log id: 13e6c24a
2014-02-01 19:20:42,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-74) [97b4f50] FINISH, ConnectStoragePoolVDSCommand, log id: c70e04c
2014-02-01 19:20:42,997 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-74) [97b4f50] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:20:47,496 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-74) [97b4f50] Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:20:47,508 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-74) [97b4f50] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:20:47,510 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-74) [97b4f50] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 1ac65219
2014-02-01 19:20:47,566 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-82) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 2a2439af
2014-02-01 19:20:47,599 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-82) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 16261574}, log id: 2a2439af
2014-02-01 19:20:50,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-74) [97b4f50] FINISH, ConnectStoragePoolVDSCommand, log id: 1ac65219
2014-02-01 19:20:50,881 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-74) [97b4f50] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:20:51,986 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-48) [5544b7d0] START, GlusterTasksListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 709b8fcb
2014-02-01 19:20:52,608 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-95) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 2d809d70
2014-02-01 19:20:53,982 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-48) [5544b7d0] Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:20:53,983 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-48) [5544b7d0] FINISH, GlusterTasksListVDSCommand, log id: 709b8fcb
2014-02-01 19:20:53,984 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-48) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:21:00,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-100) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:21:00,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-100) HostName = vms02.lis.ua
2014-02-01 19:21:00,907 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-100) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:21:01,033 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-95) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 1864d9b9}, log id: 2d809d70
2014-02-01 19:21:04,545 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-100) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:21:04,548 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-100) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 6190c228
2014-02-01 19:21:06,049 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-49) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 7bb0fcfb
2014-02-01 19:21:09,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-100) FINISH, ConnectStoragePoolVDSCommand, log id: 6190c228
2014-02-01 19:21:09,985 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-49) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 584c0ab3}, log id: 7bb0fcfb
2014-02-01 19:21:09,985 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-100) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:21:14,481 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-100) Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:21:14,492 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-100) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:21:14,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-100) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 1a14941d
2014-02-01 19:21:14,995 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-24) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 6790e4ca
2014-02-01 19:21:18,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-100) FINISH, ConnectStoragePoolVDSCommand, log id: 1a14941d
2014-02-01 19:21:18,852 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-100) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:21:18,985 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-24) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at c8dd5a75}, log id: 6790e4ca
2014-02-01 19:21:23,994 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-41) [52947046] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 6e5fc3aa
2014-02-01 19:21:24,025 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-41) [52947046] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at f70ccff2}, log id: 6e5fc3aa
2014-02-01 19:21:28,898 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-45) [4b2b1c98] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:21:28,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-45) [4b2b1c98] HostName = vms02.lis.ua
2014-02-01 19:21:28,901 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-45) [4b2b1c98] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:21:29,035 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-35) [7769700f] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 16d38374
2014-02-01 19:21:29,067 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-35) [7769700f] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at c505ac17}, log id: 16d38374
2014-02-01 19:21:31,043 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-45) [4b2b1c98] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:21:31,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-45) [4b2b1c98] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 760d601a
2014-02-01 19:21:34,076 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-84) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 5d335d46
2014-02-01 19:21:34,109 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-84) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at d4c93326}, log id: 5d335d46
2014-02-01 19:21:38,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-45) [4b2b1c98] FINISH, ConnectStoragePoolVDSCommand, log id: 760d601a
2014-02-01 19:21:38,043 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-45) [4b2b1c98] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:21:39,119 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-93) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 1c788112
2014-02-01 19:21:39,152 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-93) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at de366685}, log id: 1c788112
2014-02-01 19:21:42,543 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-45) [4b2b1c98] Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:21:42,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-45) [4b2b1c98] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:21:42,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-45) [4b2b1c98] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 3240cc2
2014-02-01 19:21:44,162 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-73) [667071b9] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 4e7687de
2014-02-01 19:21:44,196 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-73) [667071b9] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 20c93c99}, log id: 4e7687de
2014-02-01 19:21:46,917 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-45) [4b2b1c98] FINISH, ConnectStoragePoolVDSCommand, log id: 3240cc2
2014-02-01 19:21:46,918 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-45) [4b2b1c98] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:21:49,206 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-32) [30a69804] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 705f3b3f
2014-02-01 19:21:53,741 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-32) [30a69804] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at a684740e}, log id: 705f3b3f
2014-02-01 19:21:53,993 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-97) START, GlusterTasksListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 63338006
2014-02-01 19:21:54,038 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-97) Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:21:54,038 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-97) FINISH, GlusterTasksListVDSCommand, log id: 63338006
2014-02-01 19:21:54,039 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-97) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:21:56,939 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-58) [2d2a3fc9] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:21:56,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-58) [2d2a3fc9] HostName = vms02.lis.ua
2014-02-01 19:21:56,941 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-58) [2d2a3fc9] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:21:56,953 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-58) [2d2a3fc9] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:21:56,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-58) [2d2a3fc9] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 69968cdc
2014-02-01 19:21:58,758 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-72) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 160e323e
2014-02-01 19:21:58,791 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-72) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at efca0144}, log id: 160e323e
2014-02-01 19:22:00,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-58) [2d2a3fc9] FINISH, ConnectStoragePoolVDSCommand, log id: 69968cdc
2014-02-01 19:22:00,361 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-58) [2d2a3fc9] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:22:00,372 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-58) [2d2a3fc9] Irs placed on server 0072ab64-05da-4dde-9493-a0af6aa97402 failed. Proceed Failover
2014-02-01 19:22:02,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-58) [2d2a3fc9] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:22:02,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-58) [2d2a3fc9] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 6d425262
2014-02-01 19:22:03,827 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-22) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 3848a2a7
2014-02-01 19:22:07,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-58) [2d2a3fc9] FINISH, ConnectStoragePoolVDSCommand, log id: 6d425262
2014-02-01 19:22:07,810 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-58) [2d2a3fc9] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:22:07,810 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-22) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at c0adc7e8}, log id: 3848a2a7
2014-02-01 19:22:12,820 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-3) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 6d53f055
2014-02-01 19:22:17,310 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 4642aa87}, log id: 6d53f055
2014-02-01 19:22:22,317 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-74) [97b4f50] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 64a7061d
2014-02-01 19:22:22,350 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-74) [97b4f50] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 970902a4}, log id: 64a7061d
2014-02-01 19:22:27,359 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-12) [41c61dd1] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 306ba932
2014-02-01 19:22:29,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:22:29,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-36) HostName = vms01.lis.ua
2014-02-01 19:22:29,323 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:22:29,334 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-36) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:22:29,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-36) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 4f4a8c6d
2014-02-01 19:22:31,825 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-12) [41c61dd1] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 77840040}, log id: 306ba932
2014-02-01 19:22:33,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-36) FINISH, ConnectStoragePoolVDSCommand, log id: 4f4a8c6d
2014-02-01 19:22:33,750 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-36) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:22:33,762 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-36) Irs placed on server 0072ab64-05da-4dde-9493-a0af6aa97402 failed. Proceed Failover
2014-02-01 19:22:36,324 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-36) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:22:36,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-36) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 92ddc23
2014-02-01 19:22:36,834 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-38) [2f5cea56] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 79189c14
2014-02-01 19:22:36,866 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-38) [2f5cea56] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 94f6123d}, log id: 79189c14
2014-02-01 19:22:41,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-36) FINISH, ConnectStoragePoolVDSCommand, log id: 92ddc23
2014-02-01 19:22:41,762 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-36) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:22:41,875 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-55) [2783e7c5] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 12783492
2014-02-01 19:22:41,905 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-55) [2783e7c5] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 4d100081}, log id: 12783492
2014-02-01 19:22:46,914 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-33) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 20744660
2014-02-01 19:22:46,945 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-33) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 9360db2d}, log id: 20744660
2014-02-01 19:22:51,953 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-52) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 31ed7a1d
2014-02-01 19:22:54,050 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-25) [2f977190] START, GlusterTasksListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 37161f57
2014-02-01 19:22:59,007 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-52) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at b90ba71}, log id: 31ed7a1d
2014-02-01 19:23:01,021 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-25) [2f977190] Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:23:01,022 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-25) [2f977190] FINISH, GlusterTasksListVDSCommand, log id: 37161f57
2014-02-01 19:23:01,023 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-25) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:23:04,017 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-60) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 10a76a55
2014-02-01 19:23:04,048 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-60) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at a3031a90}, log id: 10a76a55
2014-02-01 19:23:05,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:23:05,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] HostName = vms01.lis.ua
2014-02-01 19:23:05,998 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:23:06,008 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:23:06,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 66f5b4d6
2014-02-01 19:23:09,064 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-26) [335e2869] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 5067a23
2014-02-01 19:23:09,356 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-66) [40858177] Failed to acquire lock and wait lock EngineLock [exclusiveLocks= key: 99408929-82cf-4dc7-a532-9d998063fa95 value: GLUSTER
2014-02-01 19:23:09,602 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-66) [40858177] Failed to acquire lock and wait lock EngineLock [exclusiveLocks= key: 99408929-82cf-4dc7-a532-9d998063fa95 value: GLUSTER
2014-02-01 19:23:10,412 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] FINISH, ConnectStoragePoolVDSCommand, log id: 66f5b4d6
2014-02-01 19:23:10,413 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:23:10,424 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] Irs placed on server 0072ab64-05da-4dde-9493-a0af6aa97402 failed. Proceed Failover
2014-02-01 19:23:12,612 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-66) [40858177] Failed to acquire lock and wait lock EngineLock [exclusiveLocks= key: 99408929-82cf-4dc7-a532-9d998063fa95 value: GLUSTER
2014-02-01 19:23:13,075 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-66) [40858177] Failed to acquire lock and wait lock EngineLock [exclusiveLocks= key: 99408929-82cf-4dc7-a532-9d998063fa95 value: GLUSTER
2014-02-01 19:23:13,592 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-26) [335e2869] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 629e03ec}, log id: 5067a23
2014-02-01 19:23:13,598 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler_Worker-66) [40858177] START, GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 4773485b
2014-02-01 19:23:13,635 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler_Worker-66) [40858177] FINISH, GetGlusterVolumeAdvancedDetailsVDSCommand, return: org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeAdvancedDetails at 84d1f96, log id: 4773485b
2014-02-01 19:23:15,590 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:23:15,592 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: f5b1210
2014-02-01 19:23:18,602 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-68) [584ec02a] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: b402392
2014-02-01 19:23:19,966 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] FINISH, ConnectStoragePoolVDSCommand, log id: f5b1210
2014-02-01 19:23:19,966 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:23:24,965 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-68) [584ec02a] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 6009d63}, log id: b402392
2014-02-01 19:23:29,974 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-43) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 35ad1bf2
2014-02-01 19:23:30,007 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-43) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at a241e19b}, log id: 35ad1bf2
2014-02-01 19:23:35,016 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-88) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 553d6ae0
2014-02-01 19:23:39,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:23:39,470 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-88) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at df1a973e}, log id: 553d6ae0
2014-02-01 19:23:39,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] HostName = vms01.lis.ua
2014-02-01 19:23:39,472 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:23:39,485 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:23:39,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 53f84c83
2014-02-01 19:23:43,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] FINISH, ConnectStoragePoolVDSCommand, log id: 53f84c83
2014-02-01 19:23:43,818 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:23:43,829 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] Irs placed on server 0072ab64-05da-4dde-9493-a0af6aa97402 failed. Proceed Failover
2014-02-01 19:23:44,481 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-22) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 70928c1a
2014-02-01 19:23:44,532 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-22) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 1c41f71f}, log id: 70928c1a
2014-02-01 19:23:45,839 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:23:45,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: b97f34a
2014-02-01 19:23:49,541 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-75) [6968161f] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 41b6916e
2014-02-01 19:23:51,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] FINISH, ConnectStoragePoolVDSCommand, log id: b97f34a
2014-02-01 19:23:51,357 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-61) [29e59621] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:23:58,393 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-75) [6968161f] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 95b62e3}, log id: 41b6916e
2014-02-01 19:24:01,033 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-30) [3ea5fd22] START, GlusterTasksListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 4fb06a21
2014-02-01 19:24:03,402 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-37) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 3274201c
2014-02-01 19:24:05,527 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-30) [3ea5fd22] Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:24:05,528 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-30) [3ea5fd22] FINISH, GlusterTasksListVDSCommand, log id: 4fb06a21
2014-02-01 19:24:05,529 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-30) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:24:09,532 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-37) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at b13a35f3}, log id: 3274201c
2014-02-01 19:24:14,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-29) [288044b9] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:24:14,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-29) [288044b9] HostName = vms01.lis.ua
2014-02-01 19:24:14,535 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-29) [288044b9] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:24:14,542 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-11) [4ad2e5fd] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 156668e7
2014-02-01 19:24:14,560 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-29) [288044b9] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:24:14,563 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-29) [288044b9] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 541d5fb4
2014-02-01 19:24:14,586 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-11) [4ad2e5fd] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at a74fa4b7}, log id: 156668e7
2014-02-01 19:24:17,966 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-29) [288044b9] FINISH, ConnectStoragePoolVDSCommand, log id: 541d5fb4
2014-02-01 19:24:17,967 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-29) [288044b9] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:24:17,978 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-29) [288044b9] Irs placed on server 0072ab64-05da-4dde-9493-a0af6aa97402 failed. Proceed Failover
2014-02-01 19:24:19,596 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-44) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 7b92d56f
2014-02-01 19:24:19,982 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-29) [288044b9] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:24:19,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-29) [288044b9] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 613f6898
2014-02-01 19:24:24,486 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 9f3bb825}, log id: 7b92d56f
2014-02-01 19:24:29,496 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-41) [52947046] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 425de30d
2014-02-01 19:24:32,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-29) [288044b9] FINISH, ConnectStoragePoolVDSCommand, log id: 613f6898
2014-02-01 19:24:32,043 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-29) [288044b9] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:24:34,543 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-41) [52947046] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 29c4c23d}, log id: 425de30d
2014-02-01 19:24:39,552 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-6) [727a775a] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 12c87bf9
2014-02-01 19:24:39,587 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-6) [727a775a] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 170c92e8}, log id: 12c87bf9
2014-02-01 19:24:44,596 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-93) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 28879769
2014-02-01 19:24:53,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-9) [173dbe9d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:24:53,615 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-93) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at e0c22702}, log id: 28879769
2014-02-01 19:24:53,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-9) [173dbe9d] HostName = vms01.lis.ua
2014-02-01 19:24:53,618 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-9) [173dbe9d] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:24:53,635 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-9) [173dbe9d] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:24:53,639 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-9) [173dbe9d] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 2b7006c4
2014-02-01 19:24:56,504 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-9) [173dbe9d] FINISH, ConnectStoragePoolVDSCommand, log id: 2b7006c4
2014-02-01 19:24:56,505 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-9) [173dbe9d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:24:56,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-9) [173dbe9d] Irs placed on server 0072ab64-05da-4dde-9493-a0af6aa97402 failed. Proceed Failover
2014-02-01 19:24:58,627 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-78) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 4614ad44
2014-02-01 19:24:58,658 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-78) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at a8db2ad}, log id: 4614ad44
2014-02-01 19:25:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-19) Autorecovering 1 storage domains
2014-02-01 19:25:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-19) Autorecovering storage domains id: 7f9709c1-3ab6-4af8-9e58-955ef7c9452e
2014-02-01 19:25:00,004 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-19) [4c471182] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 7f9709c1-3ab6-4af8-9e58-955ef7c9452e Type: Storage
2014-02-01 19:25:00,005 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-19) [4c471182] ConnectDomainToStorage. Before Connect all hosts to pool. Time:2/1/14 7:25 PM
2014-02-01 19:25:00,010 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-6-thread-45) [58b87e52] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System
2014-02-01 19:25:00,010 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-6-thread-49) [16850c4a] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System
2014-02-01 19:25:00,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-45) [58b87e52] START, ConnectStorageServerVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = GLUSTERFS, connectionList = [{ id: 4a1d8b07-f393-4134-86cc-4f46145cca2b, connection: vms02.lis.ua:STORAGE, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6eec7b77
2014-02-01 19:25:00,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-49) [16850c4a] START, ConnectStorageServerVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = GLUSTERFS, connectionList = [{ id: 4a1d8b07-f393-4134-86cc-4f46145cca2b, connection: vms02.lis.ua:STORAGE, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 59387a93
2014-02-01 19:25:00,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-45) [58b87e52] FINISH, ConnectStorageServerVDSCommand, return: {4a1d8b07-f393-4134-86cc-4f46145cca2b=0}, log id: 6eec7b77
2014-02-01 19:25:03,536 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-9) [173dbe9d] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:25:03,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-9) [173dbe9d] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 6a8ce60a
2014-02-01 19:25:03,666 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-68) [584ec02a] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 53a18218
2014-02-01 19:25:05,541 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-43) START, GlusterTasksListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 683efc10
2014-02-01 19:25:05,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-49) [16850c4a] FINISH, ConnectStorageServerVDSCommand, return: {4a1d8b07-f393-4134-86cc-4f46145cca2b=0}, log id: 59387a93
2014-02-01 19:25:05,591 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-19) [4c471182] ConnectDomainToStorage. After Connect all hosts to pool. Time:2/1/14 7:25 PM
2014-02-01 19:25:13,633 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-68) [584ec02a] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at a20605bf}, log id: 53a18218
2014-02-01 19:25:14,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-9) [173dbe9d] FINISH, ConnectStoragePoolVDSCommand, log id: 6a8ce60a
2014-02-01 19:25:14,018 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-9) [173dbe9d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:25:16,012 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-43) Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:25:16,013 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-43) FINISH, GlusterTasksListVDSCommand, log id: 683efc10
2014-02-01 19:25:16,013 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-43) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:25:18,643 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-50) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 7adab26c
2014-02-01 19:25:18,675 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-50) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at df141e9c}, log id: 7adab26c
2014-02-01 19:25:23,712 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-53) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 10de2225
2014-02-01 19:25:23,743 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-53) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 57a60e75}, log id: 10de2225
2014-02-01 19:25:28,752 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-92) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: df15110
2014-02-01 19:25:28,784 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-92) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 19c28637}, log id: df15110
2014-02-01 19:25:30,528 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-17) [6dd7fbdf] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:25:30,530 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-17) [6dd7fbdf] HostName = vms01.lis.ua
2014-02-01 19:25:30,530 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-17) [6dd7fbdf] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:25:33,794 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-75) [6968161f] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 71022729
2014-02-01 19:25:33,827 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-75) [6968161f] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 8d55e68e}, log id: 71022729
2014-02-01 19:25:35,527 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-17) [6dd7fbdf] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:25:35,530 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-17) [6dd7fbdf] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 306f5437
2014-02-01 19:25:38,837 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-94) [1e005dd3] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 75f68584
2014-02-01 19:25:38,882 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-94) [1e005dd3] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 648d7f95}, log id: 75f68584
2014-02-01 19:25:43,892 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-27) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 1f1bea77
2014-02-01 19:25:44,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-17) [6dd7fbdf] FINISH, ConnectStoragePoolVDSCommand, log id: 306f5437
2014-02-01 19:25:44,513 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-17) [6dd7fbdf] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:25:48,513 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-17) [6dd7fbdf] Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:25:48,514 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-27) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 6465f412}, log id: 1f1bea77
2014-02-01 19:25:48,524 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-17) [6dd7fbdf] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:25:48,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-17) [6dd7fbdf] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 61f55901
2014-02-01 19:25:51,889 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-17) [6dd7fbdf] FINISH, ConnectStoragePoolVDSCommand, log id: 61f55901
2014-02-01 19:25:51,890 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-17) [6dd7fbdf] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:25:53,524 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-55) [2783e7c5] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 66183df9
2014-02-01 19:25:58,028 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-55) [2783e7c5] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 68f35392}, log id: 66183df9
2014-02-01 19:26:01,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-10) [18711601] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:26:01,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-10) [18711601] HostName = vms02.lis.ua
2014-02-01 19:26:01,913 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-10) [18711601] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:26:03,037 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-100) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 516fc102
2014-02-01 19:26:05,059 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-10) [18711601] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:26:05,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-10) [18711601] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 644f1475
2014-02-01 19:26:07,559 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-100) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at a44e171f}, log id: 516fc102
2014-02-01 19:26:12,567 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-6) [727a775a] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 4470d490
2014-02-01 19:26:12,601 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-6) [727a775a] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 5bfcf0d9}, log id: 4470d490
2014-02-01 19:26:14,513 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-10) [18711601] FINISH, ConnectStoragePoolVDSCommand, log id: 644f1475
2014-02-01 19:26:14,513 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-10) [18711601] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:26:16,023 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-14) START, GlusterTasksListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 365f472f
2014-02-01 19:26:16,067 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-14) Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:26:16,068 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, GlusterTasksListVDSCommand, log id: 365f472f
2014-02-01 19:26:16,069 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-14) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:26:16,512 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-10) [18711601] Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:26:16,523 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-10) [18711601] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:26:16,525 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-10) [18711601] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 4ba2d666
2014-02-01 19:26:17,610 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-63) [5dcfa174] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 7dbf668c
2014-02-01 19:26:19,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-10) [18711601] FINISH, ConnectStoragePoolVDSCommand, log id: 4ba2d666
2014-02-01 19:26:19,892 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-10) [18711601] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:26:21,528 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-63) [5dcfa174] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 955cc9a}, log id: 7dbf668c
2014-02-01 19:26:26,537 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-45) [4b2b1c98] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 21adb743
2014-02-01 19:26:26,569 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-45) [4b2b1c98] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 650e11e0}, log id: 21adb743
2014-02-01 19:26:29,916 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:26:29,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] HostName = vms02.lis.ua
2014-02-01 19:26:29,918 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:26:31,579 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-46) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 5a33914a
2014-02-01 19:26:33,606 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:26:33,607 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-46) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 63fba3a9}, log id: 5a33914a
2014-02-01 19:26:33,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 63a80c4d
2014-02-01 19:26:38,618 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-19) [4c471182] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 2c02b666
2014-02-01 19:26:41,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] FINISH, ConnectStoragePoolVDSCommand, log id: 63a80c4d
2014-02-01 19:26:41,638 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:26:43,637 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:26:43,638 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-19) [4c471182] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 9eb3c9bc}, log id: 2c02b666
2014-02-01 19:26:43,648 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:26:43,651 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 795f2e58
2014-02-01 19:26:47,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] FINISH, ConnectStoragePoolVDSCommand, log id: 795f2e58
2014-02-01 19:26:47,989 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-91) [254caed0] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:26:48,649 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-47) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 790b8bce
2014-02-01 19:26:52,637 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-47) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at d5d3e7f7}, log id: 790b8bce
2014-02-01 19:26:57,647 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-40) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 2b56a011
2014-02-01 19:26:57,679 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-40) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 5444d35c}, log id: 2b56a011
2014-02-01 19:26:58,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-53) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:26:58,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-53) HostName = vms02.lis.ua
2014-02-01 19:26:58,010 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-53) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:26:58,021 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-53) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:26:58,023 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-53) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 20cf4c81
2014-02-01 19:27:01,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-53) FINISH, ConnectStoragePoolVDSCommand, log id: 20cf4c81
2014-02-01 19:27:01,402 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-53) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:27:01,413 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-53) Irs placed on server 0072ab64-05da-4dde-9493-a0af6aa97402 failed. Proceed Failover
2014-02-01 19:27:02,705 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-92) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 1b6728e4
2014-02-01 19:27:02,737 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-92) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 7b6cfcbf}, log id: 1b6728e4
2014-02-01 19:27:05,683 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-53) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:27:05,685 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-53) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 15c40c55
2014-02-01 19:27:07,747 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 18a52ee2
2014-02-01 19:27:07,779 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-81) [2c8b4d8e] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 9caa22b9}, log id: 18a52ee2
2014-02-01 19:27:12,789 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-89) [e0b5c73] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 19ef780b
2014-02-01 19:27:13,730 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-53) FINISH, ConnectStoragePoolVDSCommand, log id: 15c40c55
2014-02-01 19:27:13,731 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-53) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:27:15,732 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-89) [e0b5c73] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 4f2aa9a3}, log id: 19ef780b
2014-02-01 19:27:16,079 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-12) [41c61dd1] START, GlusterTasksListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 6721c634
2014-02-01 19:27:16,124 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-12) [41c61dd1] Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:27:16,124 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-12) [41c61dd1] FINISH, GlusterTasksListVDSCommand, log id: 6721c634
2014-02-01 19:27:16,125 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-12) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:27:20,740 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-31) [4c8a23f1] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 4fb10a2d
2014-02-01 19:27:20,772 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-31) [4c8a23f1] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at acfad127}, log id: 4fb10a2d
2014-02-01 19:27:25,781 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-33) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: b806309
2014-02-01 19:27:32,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:27:32,815 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-33) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 11a84ae2}, log id: b806309
2014-02-01 19:27:32,816 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-36) HostName = vms01.lis.ua
2014-02-01 19:27:32,817 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:27:34,829 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-36) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:27:34,832 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-36) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 5d57c035
2014-02-01 19:27:37,825 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-29) [288044b9] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 343e37c1
2014-02-01 19:27:37,857 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-29) [288044b9] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 4c8094d}, log id: 343e37c1
2014-02-01 19:27:42,866 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-65) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 1e321d2c
2014-02-01 19:27:43,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-36) FINISH, ConnectStoragePoolVDSCommand, log id: 5d57c035
2014-02-01 19:27:43,777 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-36) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:27:45,781 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-36) Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:27:45,782 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-65) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 47d8d2b8}, log id: 1e321d2c
2014-02-01 19:27:45,793 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-36) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:27:45,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-36) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 68608a49
2014-02-01 19:27:49,142 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-36) FINISH, ConnectStoragePoolVDSCommand, log id: 68608a49
2014-02-01 19:27:49,143 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-36) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:27:50,792 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-14) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 319bb4e3
2014-02-01 19:27:50,823 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-14) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 85376875}, log id: 319bb4e3
2014-02-01 19:27:55,832 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-26) [335e2869] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 5aecad26
2014-02-01 19:27:59,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-73) [667071b9] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:27:59,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-73) [667071b9] HostName = vms02.lis.ua
2014-02-01 19:27:59,164 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-73) [667071b9] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:27:59,843 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-26) [335e2869] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at dcb3dcc4}, log id: 5aecad26
2014-02-01 19:28:04,852 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-7) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 17f6094d
2014-02-01 19:28:04,859 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-73) [667071b9] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:28:04,862 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-73) [667071b9] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 17b20585
2014-02-01 19:28:10,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-73) [667071b9] FINISH, ConnectStoragePoolVDSCommand, log id: 17b20585
2014-02-01 19:28:10,357 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-73) [667071b9] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:28:11,909 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-7) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 52aed10d}, log id: 17f6094d
2014-02-01 19:28:13,641 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler_Worker-1) [3fe67e5e] START, GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 652f059c
2014-02-01 19:28:14,855 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-73) [667071b9] Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:28:14,868 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-73) [667071b9] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:28:14,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-73) [667071b9] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 52acc7d4
2014-02-01 19:28:16,136 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-68) [584ec02a] START, GlusterTasksListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 6fe06230
2014-02-01 19:28:16,917 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-77) [6a32be82] Failed to acquire lock and wait lock EngineLock [exclusiveLocks= key: 99408929-82cf-4dc7-a532-9d998063fa95 value: GLUSTER
2014-02-01 19:28:18,107 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-77) [6a32be82] Failed to acquire lock and wait lock EngineLock [exclusiveLocks= key: 99408929-82cf-4dc7-a532-9d998063fa95 value: GLUSTER
2014-02-01 19:28:18,262 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-73) [667071b9] FINISH, ConnectStoragePoolVDSCommand, log id: 52acc7d4
2014-02-01 19:28:18,263 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-73) [667071b9] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:28:19,396 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-68) [584ec02a] Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:28:19,397 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-68) [584ec02a] FINISH, GlusterTasksListVDSCommand, log id: 6fe06230
2014-02-01 19:28:19,397 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-68) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:28:21,115 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-77) [6a32be82] Failed to acquire lock and wait lock EngineLock [exclusiveLocks= key: 99408929-82cf-4dc7-a532-9d998063fa95 value: GLUSTER
2014-02-01 19:28:21,894 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler_Worker-1) [3fe67e5e] FINISH, GetGlusterVolumeAdvancedDetailsVDSCommand, return: org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeAdvancedDetails at d863066, log id: 652f059c
2014-02-01 19:28:21,896 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-77) [6a32be82] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 64d37f79
2014-02-01 19:28:21,926 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-77) [6a32be82] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at a493ca09}, log id: 64d37f79
2014-02-01 19:28:26,935 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-22) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 4cf0860f
2014-02-01 19:28:26,968 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-22) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at f8c2232b}, log id: 4cf0860f
2014-02-01 19:28:28,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:28:28,286 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-79) HostName = vms02.lis.ua
2014-02-01 19:28:28,286 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:28:31,976 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-98) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 3875b954
2014-02-01 19:28:32,014 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-98) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 57d81b84}, log id: 3875b954
2014-02-01 19:28:32,792 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-79) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:28:32,795 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-79) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 5eada41b
2014-02-01 19:28:34,542 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-10) [170a729b] Lock Acquired to object EngineLock [exclusiveLocks= key: 7f9709c1-3ab6-4af8-9e58-955ef7c9452e value: STORAGE
2014-02-01 19:28:34,601 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-6-thread-47) [170a729b] Running command: ActivateStorageDomainCommand internal: false. Entities affected : ID: 7f9709c1-3ab6-4af8-9e58-955ef7c9452e Type: Storage
2014-02-01 19:28:34,618 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-6-thread-47) [170a729b] Lock freed to object EngineLock [exclusiveLocks= key: 7f9709c1-3ab6-4af8-9e58-955ef7c9452e value: STORAGE
2014-02-01 19:28:34,619 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-6-thread-47) [170a729b] ActivateStorage Domain. Before Connect all hosts to pool. Time:2/1/14 7:28 PM
2014-02-01 19:28:34,622 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-6-thread-50) [1af053d5] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System
2014-02-01 19:28:34,623 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-6-thread-46) [2fa81265] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System
2014-02-01 19:28:34,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-46) [2fa81265] START, ConnectStorageServerVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = GLUSTERFS, connectionList = [{ id: 4a1d8b07-f393-4134-86cc-4f46145cca2b, connection: vms02.lis.ua:STORAGE, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 341afffb
2014-02-01 19:28:34,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-50) [1af053d5] START, ConnectStorageServerVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = GLUSTERFS, connectionList = [{ id: 4a1d8b07-f393-4134-86cc-4f46145cca2b, connection: vms02.lis.ua:STORAGE, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 42cab24d
2014-02-01 19:28:34,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-50) [1af053d5] FINISH, ConnectStorageServerVDSCommand, return: {4a1d8b07-f393-4134-86cc-4f46145cca2b=0}, log id: 42cab24d
2014-02-01 19:28:37,025 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-3) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 2e4e26fc
2014-02-01 19:28:38,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-79) FINISH, ConnectStoragePoolVDSCommand, log id: 5eada41b
2014-02-01 19:28:38,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-46) [2fa81265] FINISH, ConnectStorageServerVDSCommand, return: {4a1d8b07-f393-4134-86cc-4f46145cca2b=0}, log id: 341afffb
2014-02-01 19:28:38,373 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-79) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:28:38,375 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (pool-6-thread-47) [170a729b] START, ActivateStorageDomainVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, storageDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e), log id: 3992a52
2014-02-01 19:28:42,873 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at fa8d9ee8}, log id: 2e4e26fc
2014-02-01 19:28:47,872 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-79) Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:28:47,882 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-82) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 47a33c6b
2014-02-01 19:28:47,886 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-79) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:28:47,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-79) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 30b4176f
2014-02-01 19:28:47,918 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-82) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at bcc02aa}, log id: 47a33c6b
2014-02-01 19:28:52,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-79) FINISH, ConnectStoragePoolVDSCommand, log id: 30b4176f
2014-02-01 19:28:52,317 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-79) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:28:52,926 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-12) [41c61dd1] START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 5f4f2255
2014-02-01 19:28:52,958 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-12) [41c61dd1] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 4551960d}, log id: 5f4f2255
2014-02-01 19:28:54,326 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-6-thread-47) [170a729b] hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:28:54,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-6-thread-47) [170a729b] START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 4b098d1a
2014-02-01 19:28:57,968 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-31) [4c8a23f1] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 5d4e26fd
2014-02-01 19:29:00,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-6-thread-47) [170a729b] FINISH, ConnectStoragePoolVDSCommand, log id: 4b098d1a
2014-02-01 19:29:00,814 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-6-thread-47) [170a729b] IrsBroker::Failed::ActivateStorageDomainVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:29:05,308 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-6-thread-47) [170a729b] Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:29:05,309 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-31) [4c8a23f1] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 578f658b}, log id: 5d4e26fd
2014-02-01 19:29:05,320 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-6-thread-47) [170a729b] hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:29:05,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-6-thread-47) [170a729b] START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 7690b7e0
2014-02-01 19:29:08,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-6-thread-47) [170a729b] FINISH, ConnectStoragePoolVDSCommand, log id: 7690b7e0
2014-02-01 19:29:08,697 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-6-thread-47) [170a729b] IrsBroker::Failed::ActivateStorageDomainVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:29:08,721 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (pool-6-thread-47) [170a729b] FINISH, ActivateStorageDomainVDSCommand, log id: 3992a52
2014-02-01 19:29:08,722 ERROR [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-6-thread-47) [170a729b] Command org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) (Failed with error ENGINE and code 5001)
2014-02-01 19:29:08,724 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-6-thread-47) [170a729b] Command [id=02c0f6b3-7c4b-409e-93b9-e95e2aa0012c]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: EntityStatusSnapshot [id=storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, storageId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, status=InActive].
2014-02-01 19:29:08,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:29:08,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-49) HostName = vms02.lis.ua
2014-02-01 19:29:08,742 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-49) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:29:08,784 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-6-thread-47) [170a729b] Correlation ID: 170a729b, Job ID: 549b9462-8a04-417c-9b24-4be5ba38bb02, Call Stack: null, Custom Event ID: -1, Message: Failed to activate Storage Domain STORAGE (Data Center Default) by support
2014-02-01 19:29:10,369 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-86) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 1b6dedfd
2014-02-01 19:29:10,745 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-49) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:29:10,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-49) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 33fe7b87
2014-02-01 19:29:17,730 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-49) FINISH, ConnectStoragePoolVDSCommand, log id: 33fe7b87
2014-02-01 19:29:17,731 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-49) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:29:17,782 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-86) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 9dda6980}, log id: 1b6dedfd
2014-02-01 19:29:19,407 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-25) [2f977190] START, GlusterTasksListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 561bb211
2014-02-01 19:29:22,791 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-2) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 6298ee62
2014-02-01 19:29:22,845 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-2) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 95aec05e}, log id: 6298ee62
2014-02-01 19:29:24,803 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-49) Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:29:24,814 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-49) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:29:24,816 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-49) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 68357e17
2014-02-01 19:29:26,809 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-25) [2f977190] Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:29:26,810 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-25) [2f977190] FINISH, GlusterTasksListVDSCommand, log id: 561bb211
2014-02-01 19:29:26,811 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-25) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:29:27,853 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-42) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 3876a45c
2014-02-01 19:29:28,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-49) FINISH, ConnectStoragePoolVDSCommand, log id: 68357e17
2014-02-01 19:29:28,222 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-49) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:29:31,310 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-42) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at df77fae6}, log id: 3876a45c
2014-02-01 19:29:36,319 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-93) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 610dcd66
2014-02-01 19:29:36,350 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-93) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at e835ef3f}, log id: 610dcd66
2014-02-01 19:29:38,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-83) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:29:38,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-83) HostName = vms02.lis.ua
2014-02-01 19:29:38,248 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-83) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:29:41,310 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-83) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:29:41,312 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-83) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 4f2238f6
2014-02-01 19:29:41,359 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-46) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 4aee95a8
2014-02-01 19:29:46,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-83) FINISH, ConnectStoragePoolVDSCommand, log id: 4f2238f6
2014-02-01 19:29:46,762 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-83) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:29:48,393 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-46) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at de379054}, log id: 4aee95a8
2014-02-01 19:29:53,262 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-83) Irs placed on server bf3edcf0-b44a-47e7-8526-8cf4be85d1bb failed. Proceed Failover
2014-02-01 19:29:53,275 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-83) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:29:53,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-83) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 44c2809e
2014-02-01 19:29:53,402 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-18) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 25f28433
2014-02-01 19:29:55,763 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-18) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at f9e83a69}, log id: 25f28433
2014-02-01 19:29:57,661 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-83) FINISH, ConnectStoragePoolVDSCommand, log id: 44c2809e
2014-02-01 19:29:57,662 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-83) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:30:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-54) Autorecovering 1 storage domains
2014-02-01 19:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-54) Autorecovering storage domains id: 7f9709c1-3ab6-4af8-9e58-955ef7c9452e
2014-02-01 19:30:00,005 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-54) [2ceb639c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 7f9709c1-3ab6-4af8-9e58-955ef7c9452e Type: Storage
2014-02-01 19:30:00,005 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-54) [2ceb639c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:2/1/14 7:30 PM
2014-02-01 19:30:00,010 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-6-thread-47) [abb0956] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System
2014-02-01 19:30:00,010 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-6-thread-49) [4dd4bf9d] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System
2014-02-01 19:30:00,015 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-47) [abb0956] START, ConnectStorageServerVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = GLUSTERFS, connectionList = [{ id: 4a1d8b07-f393-4134-86cc-4f46145cca2b, connection: vms02.lis.ua:STORAGE, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: e19d836
2014-02-01 19:30:00,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-49) [4dd4bf9d] START, ConnectStorageServerVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = GLUSTERFS, connectionList = [{ id: 4a1d8b07-f393-4134-86cc-4f46145cca2b, connection: vms02.lis.ua:STORAGE, iqn: null, vfsType: glusterfs, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 64879a64
2014-02-01 19:30:00,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-47) [abb0956] FINISH, ConnectStorageServerVDSCommand, return: {4a1d8b07-f393-4134-86cc-4f46145cca2b=0}, log id: e19d836
2014-02-01 19:30:00,773 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-77) [6a32be82] START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 34b137e0
2014-02-01 19:30:04,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-6-thread-49) [4dd4bf9d] FINISH, ConnectStorageServerVDSCommand, return: {4a1d8b07-f393-4134-86cc-4f46145cca2b=0}, log id: 64879a64
2014-02-01 19:30:04,529 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-54) [2ceb639c] ConnectDomainToStorage. After Connect all hosts to pool. Time:2/1/14 7:30 PM
2014-02-01 19:30:06,529 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-77) [6a32be82] FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 4364b04f}, log id: 34b137e0
2014-02-01 19:30:07,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-98) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:30:07,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-98) HostName = vms02.lis.ua
2014-02-01 19:30:07,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-98) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:30:07,700 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-98) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:30:07,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-98) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 398f4b7d
2014-02-01 19:30:11,537 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-92) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 69f6bd0d
2014-02-01 19:30:12,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-98) FINISH, ConnectStoragePoolVDSCommand, log id: 398f4b7d
2014-02-01 19:30:12,033 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-98) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:30:12,045 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-98) Irs placed on server 0072ab64-05da-4dde-9493-a0af6aa97402 failed. Proceed Failover
2014-02-01 19:30:19,048 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-98) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:30:19,049 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-92) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at b229e96e}, log id: 69f6bd0d
2014-02-01 19:30:19,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-98) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 4cb9a82d
2014-02-01 19:30:24,058 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-80) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 619786dc
2014-02-01 19:30:26,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-98) FINISH, ConnectStoragePoolVDSCommand, log id: 4cb9a82d
2014-02-01 19:30:26,090 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-98) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:30:26,821 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-59) [42d3ce47] START, GlusterTasksListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 46883fdf
2014-02-01 19:30:28,592 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-80) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at ba449c02}, log id: 619786dc
2014-02-01 19:30:30,632 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-59) [42d3ce47] Command GlusterTasksListVDS execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported
2014-02-01 19:30:30,633 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler_Worker-59) [42d3ce47] FINISH, GlusterTasksListVDSCommand, log id: 46883fdf
2014-02-01 19:30:30,633 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-59) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException
2014-02-01 19:30:33,601 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-95) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 45c54279
2014-02-01 19:30:33,635 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-95) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 37f776a3}, log id: 45c54279
2014-02-01 19:30:38,644 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-44) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 443c5fb4
2014-02-01 19:30:47,658 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-87) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
2014-02-01 19:30:47,659 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 2b72a703}, log id: 443c5fb4
2014-02-01 19:30:47,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-87) HostName = vms01.lis.ua
2014-02-01 19:30:47,661 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-87) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-02-01 19:30:47,672 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-87) hostFromVds::selectedVds - vms02.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:30:47,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-87) START, ConnectStoragePoolVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 307c6650
2014-02-01 19:30:51,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-87) FINISH, ConnectStoragePoolVDSCommand, log id: 307c6650
2014-02-01 19:30:51,034 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-87) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:30:51,045 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-87) Irs placed on server 0072ab64-05da-4dde-9493-a0af6aa97402 failed. Proceed Failover
2014-02-01 19:30:52,669 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-96) START, GlusterVolumesListVDSCommand(HostName = vms02.lis.ua, HostId = 0072ab64-05da-4dde-9493-a0af6aa97402), log id: 17b2d1fb
2014-02-01 19:30:52,701 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at fc16512d}, log id: 17b2d1fb
2014-02-01 19:30:54,700 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-87) hostFromVds::selectedVds - vms01.lis.ua, spmStatus Unknown_Pool, storage pool Default
2014-02-01 19:30:54,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-87) START, ConnectStoragePoolVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 7f9709c1-3ab6-4af8-9e58-955ef7c9452e, masterVersion = 1), log id: 6c6b805c
2014-02-01 19:30:57,711 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-65) START, GlusterVolumesListVDSCommand(HostName = vms01.lis.ua, HostId = bf3edcf0-b44a-47e7-8526-8cf4be85d1bb), log id: 11250ece
2014-02-01 19:31:02,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-87) FINISH, ConnectStoragePoolVDSCommand, log id: 6c6b805c
2014-02-01 19:31:02,763 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-87) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-02-01 19:31:04,794 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-65) FINISH, GlusterVolumesListVDSCommand, return: {d71a56fc-2e0d-41bd-901b-79e6646e98ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity at 31750bab}, log id: 11250ece
More information about the Users
mailing list