[ovirt-users] Datacenter in no more responsive

Wed May 14 13:33:48 UTC 2014

Hi again,
Just to complete my first mail, I've tried 4 or 5 times a new install from scratch using engine-cleanup and engine-setup, but nothing works...
Tank you

Alain VONDRA
Chargé d'exploitation des Systèmes d'Information
Direction Administrative et Financière
+33 1 44 39 77 76
UNICEF France
3 rue Duguay Trouin  75006 PARIS
www.unicef.fr

-----Message d'origine-----
De : Elad Ben Aharon [mailto:ebenahar at redhat.com]
Envoyé : mercredi 14 mai 2014 15:09
À : VONDRA Alain; Allon Mureinik
Cc : users at ovirt.org
Objet : Re: [ovirt-users] Datacenter in no more responsive

Allon, could it be related to https://bugzilla.redhat.com/show_bug.cgi?id=1083476 ?

----- Original Message -----
From: "VONDRA Alain" <AVONDRA at unicef.fr>
To: users at ovirt.org
Sent: Wednesday, May 14, 2014 1:29:13 PM
Subject: [ovirt-users] Datacenter in no more responsive

Hi,

I have a very weird issue with the last version of oVirt 3.4.1, I’ve upgraded because I already had this issue.

It came when I wanted to re-create a new export domain on my first hypervisor larger than the first one.

After this point, the datacenter came in a non responsive state, so I’ve upgraded oVirt to 3.4.1, but the issue is still there.

I join the tail of the engine.log and the vdsm.log of one of the two hypervisors.

Tell me if you want more logs.

Thank you in advance for your help J

Engine.log :

TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]]

2014-05-14 12:26:34,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-82) HostName = unc-srv-hyp2

2014-05-14 12:26:34,199 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-82) Command HSMGetAllTasksStatusesVDSCommand(HostName = unc-srv-hyp2, HostId = 2d8722cc-5041-427d-964d-8980f40c5aa6) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()

2014-05-14 12:26:34,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) hostFromVds::selectedVds - unc-srv-hyp2, spmStatus Unknown_Pool, storage pool UNICEF

2014-05-14 12:26:34,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-82) START, ConnectStoragePoolVDSCommand(HostName = unc-srv-hyp2, HostId = 2d8722cc-5041-427d-964d-8980f40c5aa6, storagePoolId = 4eeccf64-715d-4ebe-a44c-eeca94a09a05, vds_spm_id = 2, masterDomainId = 9bf9ed01-43ab-4372-acdd-3500645f3bd0, masterVersion = 1), log id: aaf1e60

2014-05-14 12:26:35,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-82) FINISH, ConnectStoragePoolVDSCommand, log id: aaf1e60

2014-05-14 12:26:35,998 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)

2014-05-14 12:26:36,082 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) Irs placed on server 2d8722cc-5041-427d-964d-8980f40c5aa6 failed. Proceed Failover

2014-05-14 12:26:36,126 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) hostFromVds::selectedVds - unc-srv-hyp1, spmStatus Unknown_Pool, storage pool UNICEF

2014-05-14 12:26:36,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-82) START, ConnectStoragePoolVDSCommand(HostName = unc-srv-hyp1, HostId = 4987bc7d-82b2-444f-ad6c-8289da5e4fb9, storagePoolId = 4eeccf64-715d-4ebe-a44c-eeca94a09a05, vds_spm_id = 1, masterDomainId = 9bf9ed01-43ab-4372-acdd-3500645f3bd0, masterVersion = 1), log id: 3b569aaf

2014-05-14 12:26:37,177 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-82) FINISH, ConnectStoragePoolVDSCommand, log id: 3b569aaf

2014-05-14 12:26:37,182 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)

2014-05-14 12:26:47,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-13) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value

TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]]

2014-05-14 12:26:47,321 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-13) HostName = unc-srv-hyp1

2014-05-14 12:26:47,325 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-13) Command HSMGetAllTasksStatusesVDSCommand(HostName = unc-srv-hyp1, HostId = 4987bc7d-82b2-444f-ad6c-8289da5e4fb9) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()

2014-05-14 12:26:47,375 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-13) hostFromVds::selectedVds - unc-srv-hyp2, spmStatus Unknown_Pool, storage pool UNICEF

2014-05-14 12:26:47,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-13) START, ConnectStoragePoolVDSCommand(HostName = unc-srv-hyp2, HostId = 2d8722cc-5041-427d-964d-8980f40c5aa6, storagePoolId = 4eeccf64-715d-4ebe-a44c-eeca94a09a05, vds_spm_id = 2, masterDomainId = 9bf9ed01-43ab-4372-acdd-3500645f3bd0, masterVersion = 1), log id: 2fce851e

2014-05-14 12:26:49,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-13) FINISH, ConnectStoragePoolVDSCommand, log id: 2fce851e

2014-05-14 12:26:49,042 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-13) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue)

Vdsm.log :

Traceback (most recent call last):

File "/usr/share/vdsm/storage/task.py", line 873, in _run

return fn(*args, **kargs)

File "/usr/share/vdsm/logUtils.py", line 45, in wrapper

res = f(*args, **kwargs)

File "/usr/share/vdsm/storage/hsm.py", line 1009, in connectStoragePool

spUUID, hostID, msdUUID, masterVersion, domainsMap)

File "/usr/share/vdsm/storage/hsm.py", line 1080, in _connectStoragePool

res = pool.connect(hostID, msdUUID, masterVersion)

File "/usr/share/vdsm/storage/sp.py", line 638, in connect

self.__createMailboxMonitor()

File "/usr/share/vdsm/storage/sp.py", line 459, in __createMailboxMonitor

self.masterDomain.supportsMailbox):

File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__

return getattr(self.getRealDomain(), attrName)

AttributeError: BlockStorageDomain instance has no attribute 'supportsMailbox'

Thread-4535::DEBUG::2014-05-14 12:27:52,895::task::885::TaskManager.Task::(_run) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::Task._run: 2096f0da-3c8a-4837-9d0d-2e4b8abed77a ('4eeccf64-715d-4ebe-a44c-eeca94a09a05', 1, '9bf9ed01-43ab-4372-acdd-3500645f3bd0', 1, None) {} failed - stopping task

Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::1211::TaskManager.Task::(stop) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::stopping in state preparing (force False)

Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::990::TaskManager.Task::(_decref) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::ref 1 aborting True

Thread-4535::INFO::2014-05-14 12:27:52,896::task::1168::TaskManager.Task::(prepare) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::aborting: Task is aborted: u"BlockStorageDomain instance has no attribute 'supportsMailbox'" - code 100

Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::1173::TaskManager.Task::(prepare) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::Prepare: aborted: BlockStorageDomain instance has no attribute 'supportsMailbox'

Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::990::TaskManager.Task::(_decref) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::ref 0 aborting True

Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::925::TaskManager.Task::(_doAbort) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::Task._doAbort: force False

Thread-4535::DEBUG::2014-05-14 12:27:52,896::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}

Thread-4535::DEBUG::2014-05-14 12:27:52,897::task::595::TaskManager.Task::(_updateState) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::moving from state preparing -> state aborting

Thread-4535::DEBUG::2014-05-14 12:27:52,897::task::550::TaskManager.Task::(__state_aborting) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::_aborting: recover policy none

Thread-4535::DEBUG::2014-05-14 12:27:52,897::task::595::TaskManager.Task::(_updateState) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::moving from state aborting -> state failed

Thread-4535::DEBUG::2014-05-14 12:27:52,897::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}

Thread-4535::DEBUG::2014-05-14 12:27:52,897::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}

Thread-4535::ERROR::2014-05-14 12:27:52,897::dispatcher::68::Storage.Dispatcher.Protect::(run) BlockStorageDomain instance has no attribute 'supportsMailbox'

Traceback (most recent call last):

File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run

result = ctask.prepare(self.func, *args, **kwargs)

File "/usr/share/vdsm/storage/task.py", line 103, in wrapper

return m(self, *a, **kw)

File "/usr/share/vdsm/storage/task.py", line 1176, in prepare

raise self.error

AttributeError: BlockStorageDomain instance has no attribute 'supportsMailbox'

Thread-16::DEBUG::2014-05-14 12:27:55,639::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'a|/dev/mapper/3690b11c0005592a9000004855344bf5c|\', \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 9bf9ed01-43ab-4372-acdd-3500645f3bd0' (cwd None)

Thread-16::DEBUG::2014-05-14 12:27:55,765::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0

Thread-16::DEBUG::2014-05-14 12:27:55,768::blockSD::605::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/dev/9bf9ed01-43ab-4372-acdd-3500645f3bd0/metadata bs=4096 count=1' (cwd None)

Thread-16::DEBUG::2014-05-14 12:27:55,773::blockSD::605::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0003455 s, 11.9 MB/s\n'; <rc> = 0

Thread-4535::DEBUG::2014-05-14 12:27:57,995::task::595::TaskManager.Task::(_updateState) Task=`89ef3bd0-f222-470b-b678-6228d5254831`::moving from state init -> state preparing

Thread-4535::INFO::2014-05-14 12:27:57,995::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)

Thread-4535::INFO::2014-05-14 12:27:57,996::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'9bf9ed01-43ab-4372-acdd-3500645f3bd0': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0003455', 'lastCheck': '2.2', 'valid': True}}

Thread-4535::DEBUG::2014-05-14 12:27:57,996::task::1185::TaskManager.Task::(prepare) Task=`89ef3bd0-f222-470b-b678-6228d5254831`::finished: {'9bf9ed01-43ab-4372-acdd-3500645f3bd0': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0003455', 'lastCheck': '2.2', 'valid': True}}

Thread-4535::DEBUG::2014-05-14 12:27:57,996::task::595::TaskManager.Task::(_updateState) Task=`89ef3bd0-f222-470b-b678-6228d5254831`::moving from state preparing -> state finished

Thread-4535::DEBUG::2014-05-14 12:27:57,996::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}

Thread-4535::DEBUG::2014-05-14 12:27:57,996::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}

Thread-4535::DEBUG::2014-05-14 12:27:57,996::task::990::TaskManager.Task::(_decref) Task=`89ef3bd0-f222-470b-b678-6228d5254831`::ref 0 aborting False

Alain VONDRA
Chargé d'exploitation des Systèmes d'Information Direction Administrative et Financière
+33 1 44 39 77 76
UNICEF France
3 rue Duguay Trouin 75006 PARIS
www.unicef.fr

_______________________________________________
Users mailing list
Users at ovirt.org
http://lists.ovirt.org/mailman/listinfo/users