Re: [ovirt-users] Datacenter in no more responsive

Wednesday, 14 May 2014

Allon, could it be related to https://bugzilla.redhat.com/show_bug.cgi?id=1083476 ?

----- Original Message -----
From: "VONDRA Alain" <AVONDRA(a)unicef.fr&gt;
To: users(a)ovirt.org
Sent: Wednesday, May 14, 2014 1:29:13 PM
Subject: [ovirt-users] Datacenter in no more responsive

Hi, 

I have a very weird issue with the last version of oVirt 3.4.1, I’ve upgraded because I
already had this issue. 

It came when I wanted to re-create a new export domain on my first hypervisor larger than
the first one. 

After this point, the datacenter came in a non responsive state, so I’ve upgraded oVirt to
3.4.1, but the issue is still there. 

I join the tail of the engine.log and the vdsm.log of one of the two hypervisors. 

Tell me if you want more logs. 

Thank you in advance for your help J 

Engine.log : 

TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]]

2014-05-14 12:26:34,195 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-82) HostName = unc-srv-hyp2 

2014-05-14 12:26:34,199 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-82) Command HSMGetAllTasksStatusesVDSCommand(HostName =
unc-srv-hyp2, HostId = 2d8722cc-5041-427d-964d-8980f40c5aa6) execution failed. Exception:
IRSNonOperationalException: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Not SPM: () 

2014-05-14 12:26:34,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-82) hostFromVds::selectedVds - unc-srv-hyp2, spmStatus
Unknown_Pool, storage pool UNICEF 

2014-05-14 12:26:34,269 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-82) START, ConnectStoragePoolVDSCommand(HostName =
unc-srv-hyp2, HostId = 2d8722cc-5041-427d-964d-8980f40c5aa6, storagePoolId =
4eeccf64-715d-4ebe-a44c-eeca94a09a05, vds_spm_id = 2, masterDomainId =
9bf9ed01-43ab-4372-acdd-3500645f3bd0, masterVersion = 1), log id: aaf1e60 

2014-05-14 12:26:35,998 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-82) FINISH, ConnectStoragePoolVDSCommand, log id: aaf1e60 

2014-05-14 12:26:35,998 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-82) IrsBroker::Failed::GetStoragePoolInfoVDS due to:
IRSNonOperationalException: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Could not connect host to Data Center(Storage issue) 

2014-05-14 12:26:36,082 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-82) Irs placed on server
2d8722cc-5041-427d-964d-8980f40c5aa6 failed. Proceed Failover 

2014-05-14 12:26:36,126 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-82) hostFromVds::selectedVds - unc-srv-hyp1, spmStatus
Unknown_Pool, storage pool UNICEF 

2014-05-14 12:26:36,147 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-82) START, ConnectStoragePoolVDSCommand(HostName =
unc-srv-hyp1, HostId = 4987bc7d-82b2-444f-ad6c-8289da5e4fb9, storagePoolId =
4eeccf64-715d-4ebe-a44c-eeca94a09a05, vds_spm_id = 1, masterDomainId =
9bf9ed01-43ab-4372-acdd-3500645f3bd0, masterVersion = 1), log id: 3b569aaf 

2014-05-14 12:26:37,177 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-82) FINISH, ConnectStoragePoolVDSCommand, log id: 3b569aaf

2014-05-14 12:26:37,182 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-82) IrsBroker::Failed::GetStoragePoolInfoVDS due to:
IRSNonOperationalException: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Could not connect host to Data Center(Storage issue) 

2014-05-14 12:26:47,305 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-13) Command
org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value 

TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]]

2014-05-14 12:26:47,321 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-13) HostName = unc-srv-hyp1 

2014-05-14 12:26:47,325 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-13) Command HSMGetAllTasksStatusesVDSCommand(HostName =
unc-srv-hyp1, HostId = 4987bc7d-82b2-444f-ad6c-8289da5e4fb9) execution failed. Exception:
IRSNonOperationalException: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Not SPM: () 

2014-05-14 12:26:47,375 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-13) hostFromVds::selectedVds - unc-srv-hyp2, spmStatus
Unknown_Pool, storage pool UNICEF 

2014-05-14 12:26:47,396 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-13) START, ConnectStoragePoolVDSCommand(HostName =
unc-srv-hyp2, HostId = 2d8722cc-5041-427d-964d-8980f40c5aa6, storagePoolId =
4eeccf64-715d-4ebe-a44c-eeca94a09a05, vds_spm_id = 2, masterDomainId =
9bf9ed01-43ab-4372-acdd-3500645f3bd0, masterVersion = 1), log id: 2fce851e 

2014-05-14 12:26:49,042 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-13) FINISH, ConnectStoragePoolVDSCommand, log id: 2fce851e

2014-05-14 12:26:49,042 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-13) IrsBroker::Failed::GetStoragePoolInfoVDS due to:
IRSNonOperationalException: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Could not connect host to Data Center(Storage issue) 

Vdsm.log : 

Traceback (most recent call last): 

File "/usr/share/vdsm/storage/task.py", line 873, in _run 

return fn(*args, **kargs) 

File "/usr/share/vdsm/logUtils.py", line 45, in wrapper 

res = f(*args, **kwargs) 

File "/usr/share/vdsm/storage/hsm.py", line 1009, in connectStoragePool 

spUUID, hostID, msdUUID, masterVersion, domainsMap) 

File "/usr/share/vdsm/storage/hsm.py", line 1080, in _connectStoragePool 

res = pool.connect(hostID, msdUUID, masterVersion) 

File "/usr/share/vdsm/storage/sp.py", line 638, in connect 

self.__createMailboxMonitor() 

File "/usr/share/vdsm/storage/sp.py", line 459, in __createMailboxMonitor 

self.masterDomain.supportsMailbox): 

File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ 

return getattr(self.getRealDomain(), attrName) 

AttributeError: BlockStorageDomain instance has no attribute 'supportsMailbox' 

Thread-4535::DEBUG::2014-05-14 12:27:52,895::task::885::TaskManager.Task::(_run)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::Task._run:
2096f0da-3c8a-4837-9d0d-2e4b8abed77a ('4eeccf64-715d-4ebe-a44c-eeca94a09a05', 1,
'9bf9ed01-43ab-4372-acdd-3500645f3bd0', 1, None) {} failed - stopping task 

Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::1211::TaskManager.Task::(stop)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::stopping in state preparing (force False) 

Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::990::TaskManager.Task::(_decref)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::ref 1 aborting True 

Thread-4535::INFO::2014-05-14 12:27:52,896::task::1168::TaskManager.Task::(prepare)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::aborting: Task is aborted:
u"BlockStorageDomain instance has no attribute 'supportsMailbox'" - code
100 

Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::1173::TaskManager.Task::(prepare)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::Prepare: aborted: BlockStorageDomain instance
has no attribute 'supportsMailbox' 

Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::990::TaskManager.Task::(_decref)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::ref 0 aborting True 

Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::925::TaskManager.Task::(_doAbort)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::Task._doAbort: force False 

Thread-4535::DEBUG::2014-05-14
12:27:52,896::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll
requests {} 

Thread-4535::DEBUG::2014-05-14 12:27:52,897::task::595::TaskManager.Task::(_updateState)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::moving from state preparing -> state
aborting 

Thread-4535::DEBUG::2014-05-14
12:27:52,897::task::550::TaskManager.Task::(__state_aborting)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::_aborting: recover policy none 

Thread-4535::DEBUG::2014-05-14 12:27:52,897::task::595::TaskManager.Task::(_updateState)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::moving from state aborting -> state failed

Thread-4535::DEBUG::2014-05-14
12:27:52,897::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll
requests {} resources {} 

Thread-4535::DEBUG::2014-05-14
12:27:52,897::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll
requests {} 

Thread-4535::ERROR::2014-05-14
12:27:52,897::dispatcher::68::Storage.Dispatcher.Protect::(run) BlockStorageDomain
instance has no attribute 'supportsMailbox' 

Traceback (most recent call last): 

File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run 

result = ctask.prepare(self.func, *args, **kwargs) 

File "/usr/share/vdsm/storage/task.py", line 103, in wrapper 

return m(self, *a, **kw) 

File "/usr/share/vdsm/storage/task.py", line 1176, in prepare 

raise self.error 

AttributeError: BlockStorageDomain instance has no attribute 'supportsMailbox' 

Thread-16::DEBUG::2014-05-14 12:27:55,639::lvm::295::Storage.Misc.excCmd::(cmd)
'/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names =
[\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0
disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [
\'a|/dev/mapper/3690b11c0005592a9000004855344bf5c|\', \'r|.*|\' ] } global
{ locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50
retain_days = 0 } " 9bf9ed01-43ab-4372-acdd-3500645f3bd0' (cwd None) 

Thread-16::DEBUG::2014-05-14 12:27:55,765::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS:
<err> = ''; <rc> = 0 

Thread-16::DEBUG::2014-05-14
12:27:55,768::blockSD::605::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct
if=/dev/9bf9ed01-43ab-4372-acdd-3500645f3bd0/metadata bs=4096 count=1' (cwd None) 

Thread-16::DEBUG::2014-05-14
12:27:55,773::blockSD::605::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> =
'1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0003455 s, 11.9
MB/s\n'; <rc> = 0 

Thread-4535::DEBUG::2014-05-14 12:27:57,995::task::595::TaskManager.Task::(_updateState)
Task=`89ef3bd0-f222-470b-b678-6228d5254831`::moving from state init -> state preparing

Thread-4535::INFO::2014-05-14 12:27:57,995::logUtils::44::dispatcher::(wrapper) Run and
protect: repoStats(options=None) 

Thread-4535::INFO::2014-05-14 12:27:57,996::logUtils::47::dispatcher::(wrapper) Run and
protect: repoStats, Return response: {'9bf9ed01-43ab-4372-acdd-3500645f3bd0':
{'code': 0, 'version': 3, 'acquired': True, 'delay':
'0.0003455', 'lastCheck': '2.2', 'valid': True}} 

Thread-4535::DEBUG::2014-05-14 12:27:57,996::task::1185::TaskManager.Task::(prepare)
Task=`89ef3bd0-f222-470b-b678-6228d5254831`::finished:
{'9bf9ed01-43ab-4372-acdd-3500645f3bd0': {'code': 0, 'version': 3,
'acquired': True, 'delay': '0.0003455', 'lastCheck':
'2.2', 'valid': True}} 

Thread-4535::DEBUG::2014-05-14 12:27:57,996::task::595::TaskManager.Task::(_updateState)
Task=`89ef3bd0-f222-470b-b678-6228d5254831`::moving from state preparing -> state
finished 

Thread-4535::DEBUG::2014-05-14
12:27:57,996::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll
requests {} resources {} 

Thread-4535::DEBUG::2014-05-14
12:27:57,996::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll
requests {} 

Thread-4535::DEBUG::2014-05-14 12:27:57,996::task::990::TaskManager.Task::(_decref)
Task=`89ef3bd0-f222-470b-b678-6228d5254831`::ref 0 aborting False 

Alain VONDRA 
Chargé d'exploitation des Systèmes d'Information 
Direction Administrative et Financière 
+33 1 44 39 77 76 
UNICEF France 
3 rue Duguay Trouin 75006 PARIS 
www.unicef.fr 	

_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Re: [ovirt-users] Datacenter in no more responsive