[ovirt-users] Datacenter in no more responsive
Elad Ben Aharon
ebenahar at redhat.com
Wed May 14 13:08:31 UTC 2014
Allon, could it be related to https://bugzilla.redhat.com/show_bug.cgi?id=1083476 ?
----- Original Message -----
From: "VONDRA Alain" <AVONDRA at unicef.fr>
To: users at ovirt.org
Sent: Wednesday, May 14, 2014 1:29:13 PM
Subject: [ovirt-users] Datacenter in no more responsive
Hi,
I have a very weird issue with the last version of oVirt 3.4.1, I’ve upgraded because I already had this issue.
It came when I wanted to re-create a new export domain on my first hypervisor larger than the first one.
After this point, the datacenter came in a non responsive state, so I’ve upgraded oVirt to 3.4.1, but the issue is still there.
I join the tail of the engine.log and the vdsm.log of one of the two hypervisors.
Tell me if you want more logs.
Thank you in advance for your help J
Engine.log :
TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]]
2014-05-14 12:26:34,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-82) HostName = unc-srv-hyp2
2014-05-14 12:26:34,199 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-82) Command HSMGetAllTasksStatusesVDSCommand(HostName = unc-srv-hyp2, HostId = 2d8722cc-5041-427d-964d-8980f40c5aa6) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-05-14 12:26:34,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) hostFromVds::selectedVds - unc-srv-hyp2, spmStatus Unknown_Pool, storage pool UNICEF
2014-05-14 12:26:34,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-82) START, ConnectStoragePoolVDSCommand(HostName = unc-srv-hyp2, HostId = 2d8722cc-5041-427d-964d-8980f40c5aa6, storagePoolId = 4eeccf64-715d-4ebe-a44c-eeca94a09a05, vds_spm_id = 2, masterDomainId = 9bf9ed01-43ab-4372-acdd-3500645f3bd0, masterVersion = 1), log id: aaf1e60
2014-05-14 12:26:35,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-82) FINISH, ConnectStoragePoolVDSCommand, log id: aaf1e60
2014-05-14 12:26:35,998 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-05-14 12:26:36,082 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) Irs placed on server 2d8722cc-5041-427d-964d-8980f40c5aa6 failed. Proceed Failover
2014-05-14 12:26:36,126 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) hostFromVds::selectedVds - unc-srv-hyp1, spmStatus Unknown_Pool, storage pool UNICEF
2014-05-14 12:26:36,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-82) START, ConnectStoragePoolVDSCommand(HostName = unc-srv-hyp1, HostId = 4987bc7d-82b2-444f-ad6c-8289da5e4fb9, storagePoolId = 4eeccf64-715d-4ebe-a44c-eeca94a09a05, vds_spm_id = 1, masterDomainId = 9bf9ed01-43ab-4372-acdd-3500645f3bd0, masterVersion = 1), log id: 3b569aaf
2014-05-14 12:26:37,177 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-82) FINISH, ConnectStoragePoolVDSCommand, log id: 3b569aaf
2014-05-14 12:26:37,182 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-05-14 12:26:47,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-13) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]]
2014-05-14 12:26:47,321 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-13) HostName = unc-srv-hyp1
2014-05-14 12:26:47,325 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-13) Command HSMGetAllTasksStatusesVDSCommand(HostName = unc-srv-hyp1, HostId = 4987bc7d-82b2-444f-ad6c-8289da5e4fb9) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2014-05-14 12:26:47,375 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-13) hostFromVds::selectedVds - unc-srv-hyp2, spmStatus Unknown_Pool, storage pool UNICEF
2014-05-14 12:26:47,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-13) START, ConnectStoragePoolVDSCommand(HostName = unc-srv-hyp2, HostId = 2d8722cc-5041-427d-964d-8980f40c5aa6, storagePoolId = 4eeccf64-715d-4ebe-a44c-eeca94a09a05, vds_spm_id = 2, masterDomainId = 9bf9ed01-43ab-4372-acdd-3500645f3bd0, masterVersion = 1), log id: 2fce851e
2014-05-14 12:26:49,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-13) FINISH, ConnectStoragePoolVDSCommand, log id: 2fce851e
2014-05-14 12:26:49,042 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-13) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
Vdsm.log :
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 873, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 1009, in connectStoragePool
spUUID, hostID, msdUUID, masterVersion, domainsMap)
File "/usr/share/vdsm/storage/hsm.py", line 1080, in _connectStoragePool
res = pool.connect(hostID, msdUUID, masterVersion)
File "/usr/share/vdsm/storage/sp.py", line 638, in connect
self.__createMailboxMonitor()
File "/usr/share/vdsm/storage/sp.py", line 459, in __createMailboxMonitor
self.masterDomain.supportsMailbox):
File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
return getattr(self.getRealDomain(), attrName)
AttributeError: BlockStorageDomain instance has no attribute 'supportsMailbox'
Thread-4535::DEBUG::2014-05-14 12:27:52,895::task::885::TaskManager.Task::(_run) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::Task._run: 2096f0da-3c8a-4837-9d0d-2e4b8abed77a ('4eeccf64-715d-4ebe-a44c-eeca94a09a05', 1, '9bf9ed01-43ab-4372-acdd-3500645f3bd0', 1, None) {} failed - stopping task
Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::1211::TaskManager.Task::(stop) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::stopping in state preparing (force False)
Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::990::TaskManager.Task::(_decref) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::ref 1 aborting True
Thread-4535::INFO::2014-05-14 12:27:52,896::task::1168::TaskManager.Task::(prepare) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::aborting: Task is aborted: u"BlockStorageDomain instance has no attribute 'supportsMailbox'" - code 100
Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::1173::TaskManager.Task::(prepare) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::Prepare: aborted: BlockStorageDomain instance has no attribute 'supportsMailbox'
Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::990::TaskManager.Task::(_decref) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::ref 0 aborting True
Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::925::TaskManager.Task::(_doAbort) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::Task._doAbort: force False
Thread-4535::DEBUG::2014-05-14 12:27:52,896::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-4535::DEBUG::2014-05-14 12:27:52,897::task::595::TaskManager.Task::(_updateState) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::moving from state preparing -> state aborting
Thread-4535::DEBUG::2014-05-14 12:27:52,897::task::550::TaskManager.Task::(__state_aborting) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::_aborting: recover policy none
Thread-4535::DEBUG::2014-05-14 12:27:52,897::task::595::TaskManager.Task::(_updateState) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::moving from state aborting -> state failed
Thread-4535::DEBUG::2014-05-14 12:27:52,897::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-4535::DEBUG::2014-05-14 12:27:52,897::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-4535::ERROR::2014-05-14 12:27:52,897::dispatcher::68::Storage.Dispatcher.Protect::(run) BlockStorageDomain instance has no attribute 'supportsMailbox'
Traceback (most recent call last):
File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
result = ctask.prepare(self.func, *args, **kwargs)
File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
return m(self, *a, **kw)
File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
raise self.error
AttributeError: BlockStorageDomain instance has no attribute 'supportsMailbox'
Thread-16::DEBUG::2014-05-14 12:27:55,639::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'a|/dev/mapper/3690b11c0005592a9000004855344bf5c|\', \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 9bf9ed01-43ab-4372-acdd-3500645f3bd0' (cwd None)
Thread-16::DEBUG::2014-05-14 12:27:55,765::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0
Thread-16::DEBUG::2014-05-14 12:27:55,768::blockSD::605::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/dev/9bf9ed01-43ab-4372-acdd-3500645f3bd0/metadata bs=4096 count=1' (cwd None)
Thread-16::DEBUG::2014-05-14 12:27:55,773::blockSD::605::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0003455 s, 11.9 MB/s\n'; <rc> = 0
Thread-4535::DEBUG::2014-05-14 12:27:57,995::task::595::TaskManager.Task::(_updateState) Task=`89ef3bd0-f222-470b-b678-6228d5254831`::moving from state init -> state preparing
Thread-4535::INFO::2014-05-14 12:27:57,995::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-4535::INFO::2014-05-14 12:27:57,996::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'9bf9ed01-43ab-4372-acdd-3500645f3bd0': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0003455', 'lastCheck': '2.2', 'valid': True}}
Thread-4535::DEBUG::2014-05-14 12:27:57,996::task::1185::TaskManager.Task::(prepare) Task=`89ef3bd0-f222-470b-b678-6228d5254831`::finished: {'9bf9ed01-43ab-4372-acdd-3500645f3bd0': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0003455', 'lastCheck': '2.2', 'valid': True}}
Thread-4535::DEBUG::2014-05-14 12:27:57,996::task::595::TaskManager.Task::(_updateState) Task=`89ef3bd0-f222-470b-b678-6228d5254831`::moving from state preparing -> state finished
Thread-4535::DEBUG::2014-05-14 12:27:57,996::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-4535::DEBUG::2014-05-14 12:27:57,996::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-4535::DEBUG::2014-05-14 12:27:57,996::task::990::TaskManager.Task::(_decref) Task=`89ef3bd0-f222-470b-b678-6228d5254831`::ref 0 aborting False
Alain VONDRA
Chargé d'exploitation des Systèmes d'Information
Direction Administrative et Financière
+33 1 44 39 77 76
UNICEF France
3 rue Duguay Trouin 75006 PARIS
www.unicef.fr
_______________________________________________
Users mailing list
Users at ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
More information about the Users
mailing list