9:33 a.m.
Hi again,
Just to complete my first mail, I've tried 4 or 5 times a new install from scratch
using engine-cleanup and engine-setup, but nothing works...
Tank you
Alain VONDRA
Chargé d'exploitation des Systèmes d'Information
Direction Administrative et Financière
+33 1 44 39 77 76
UNICEF France
3 rue Duguay Trouin 75006 PARIS
www.unicef.fr
-----Message d'origine-----
De : Elad Ben Aharon [mailto:ebenahar@redhat.com]
Envoyé : mercredi 14 mai 2014 15:09
À : VONDRA Alain; Allon Mureinik
Cc : users(a)ovirt.org
Objet : Re: [ovirt-users] Datacenter in no more responsive
Allon, could it be related to https://bugzilla.redhat.com/show_bug.cgi?id=1083476 ?
----- Original Message -----
From: "VONDRA Alain" <AVONDRA(a)unicef.fr>
To: users(a)ovirt.org
Sent: Wednesday, May 14, 2014 1:29:13 PM
Subject: [ovirt-users] Datacenter in no more responsive
Hi,
I have a very weird issue with the last version of oVirt 3.4.1, I’ve upgraded because I
already had this issue.
It came when I wanted to re-create a new export domain on my first hypervisor larger than
the first one.
After this point, the datacenter came in a non responsive state, so I’ve upgraded oVirt to
3.4.1, but the issue is still there.
I join the tail of the engine.log and the vdsm.log of one of the two hypervisors.
Tell me if you want more logs.
Thank you in advance for your help J
Engine.log :
TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM:
()]]
2014-05-14 12:26:34,195 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-82) HostName = unc-srv-hyp2
2014-05-14 12:26:34,199 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-82) Command HSMGetAllTasksStatusesVDSCommand(HostName =
unc-srv-hyp2, HostId = 2d8722cc-5041-427d-964d-8980f40c5aa6) execution failed. Exception:
IRSNonOperationalException: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Not SPM: ()
2014-05-14 12:26:34,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-82) hostFromVds::selectedVds - unc-srv-hyp2, spmStatus
Unknown_Pool, storage pool UNICEF
2014-05-14 12:26:34,269 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-82) START, ConnectStoragePoolVDSCommand(HostName =
unc-srv-hyp2, HostId = 2d8722cc-5041-427d-964d-8980f40c5aa6, storagePoolId =
4eeccf64-715d-4ebe-a44c-eeca94a09a05, vds_spm_id = 2, masterDomainId =
9bf9ed01-43ab-4372-acdd-3500645f3bd0, masterVersion = 1), log id: aaf1e60
2014-05-14 12:26:35,998 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-82) FINISH, ConnectStoragePoolVDSCommand, log id: aaf1e60
2014-05-14 12:26:35,998 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-82) IrsBroker::Failed::GetStoragePoolInfoVDS due to:
IRSNonOperationalException: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-05-14 12:26:36,082 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-82) Irs placed on server
2d8722cc-5041-427d-964d-8980f40c5aa6 failed. Proceed Failover
2014-05-14 12:26:36,126 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-82) hostFromVds::selectedVds - unc-srv-hyp1, spmStatus
Unknown_Pool, storage pool UNICEF
2014-05-14 12:26:36,147 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-82) START, ConnectStoragePoolVDSCommand(HostName =
unc-srv-hyp1, HostId = 4987bc7d-82b2-444f-ad6c-8289da5e4fb9, storagePoolId =
4eeccf64-715d-4ebe-a44c-eeca94a09a05, vds_spm_id = 1, masterDomainId =
9bf9ed01-43ab-4372-acdd-3500645f3bd0, masterVersion = 1), log id: 3b569aaf
2014-05-14 12:26:37,177 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-82) FINISH, ConnectStoragePoolVDSCommand, log id:
3b569aaf
2014-05-14 12:26:37,182 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-82) IrsBroker::Failed::GetStoragePoolInfoVDS due to:
IRSNonOperationalException: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-05-14 12:26:47,305 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-13) Command
org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM:
()]]
2014-05-14 12:26:47,321 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-13) HostName = unc-srv-hyp1
2014-05-14 12:26:47,325 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-13) Command HSMGetAllTasksStatusesVDSCommand(HostName =
unc-srv-hyp1, HostId = 4987bc7d-82b2-444f-ad6c-8289da5e4fb9) execution failed. Exception:
IRSNonOperationalException: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Not SPM: ()
2014-05-14 12:26:47,375 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-13) hostFromVds::selectedVds - unc-srv-hyp2, spmStatus
Unknown_Pool, storage pool UNICEF
2014-05-14 12:26:47,396 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-13) START, ConnectStoragePoolVDSCommand(HostName =
unc-srv-hyp2, HostId = 2d8722cc-5041-427d-964d-8980f40c5aa6, storagePoolId =
4eeccf64-715d-4ebe-a44c-eeca94a09a05, vds_spm_id = 2, masterDomainId =
9bf9ed01-43ab-4372-acdd-3500645f3bd0, masterVersion = 1), log id: 2fce851e
2014-05-14 12:26:49,042 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-13) FINISH, ConnectStoragePoolVDSCommand, log id:
2fce851e
2014-05-14 12:26:49,042 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-13) IrsBroker::Failed::GetStoragePoolInfoVDS due to:
IRSNonOperationalException: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
Vdsm.log :
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 873, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 1009, in connectStoragePool
spUUID, hostID, msdUUID, masterVersion, domainsMap)
File "/usr/share/vdsm/storage/hsm.py", line 1080, in _connectStoragePool
res = pool.connect(hostID, msdUUID, masterVersion)
File "/usr/share/vdsm/storage/sp.py", line 638, in connect
self.__createMailboxMonitor()
File "/usr/share/vdsm/storage/sp.py", line 459, in __createMailboxMonitor
self.masterDomain.supportsMailbox):
File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
return getattr(self.getRealDomain(), attrName)
AttributeError: BlockStorageDomain instance has no attribute 'supportsMailbox'
Thread-4535::DEBUG::2014-05-14 12:27:52,895::task::885::TaskManager.Task::(_run)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::Task._run:
2096f0da-3c8a-4837-9d0d-2e4b8abed77a ('4eeccf64-715d-4ebe-a44c-eeca94a09a05', 1,
'9bf9ed01-43ab-4372-acdd-3500645f3bd0', 1, None) {} failed - stopping task
Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::1211::TaskManager.Task::(stop)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::stopping in state preparing (force False)
Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::990::TaskManager.Task::(_decref)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::ref 1 aborting True
Thread-4535::INFO::2014-05-14 12:27:52,896::task::1168::TaskManager.Task::(prepare)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::aborting: Task is aborted:
u"BlockStorageDomain instance has no attribute 'supportsMailbox'" - code
100
Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::1173::TaskManager.Task::(prepare)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::Prepare: aborted: BlockStorageDomain instance
has no attribute 'supportsMailbox'
Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::990::TaskManager.Task::(_decref)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::ref 0 aborting True
Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::925::TaskManager.Task::(_doAbort)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::Task._doAbort: force False
Thread-4535::DEBUG::2014-05-14
12:27:52,896::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll
requests {}
Thread-4535::DEBUG::2014-05-14 12:27:52,897::task::595::TaskManager.Task::(_updateState)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::moving from state preparing -> state
aborting
Thread-4535::DEBUG::2014-05-14
12:27:52,897::task::550::TaskManager.Task::(__state_aborting)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::_aborting: recover policy none
Thread-4535::DEBUG::2014-05-14 12:27:52,897::task::595::TaskManager.Task::(_updateState)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::moving from state aborting -> state
failed
Thread-4535::DEBUG::2014-05-14
12:27:52,897::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll
requests {} resources {}
Thread-4535::DEBUG::2014-05-14
12:27:52,897::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll
requests {}
Thread-4535::ERROR::2014-05-14
12:27:52,897::dispatcher::68::Storage.Dispatcher.Protect::(run) BlockStorageDomain
instance has no attribute 'supportsMailbox'
Traceback (most recent call last):
File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
result = ctask.prepare(self.func, *args, **kwargs)
File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
return m(self, *a, **kw)
File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
raise self.error
AttributeError: BlockStorageDomain instance has no attribute 'supportsMailbox'
Thread-16::DEBUG::2014-05-14 12:27:55,639::lvm::295::Storage.Misc.excCmd::(cmd)
'/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names =
[\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0
disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [
\'a|/dev/mapper/3690b11c0005592a9000004855344bf5c|\', \'r|.*|\' ] } global
{ locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50
retain_days = 0 } " 9bf9ed01-43ab-4372-acdd-3500645f3bd0' (cwd None)
Thread-16::DEBUG::2014-05-14 12:27:55,765::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS:
<err> = ''; <rc> = 0
Thread-16::DEBUG::2014-05-14
12:27:55,768::blockSD::605::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct
if=/dev/9bf9ed01-43ab-4372-acdd-3500645f3bd0/metadata bs=4096 count=1' (cwd None)
Thread-16::DEBUG::2014-05-14
12:27:55,773::blockSD::605::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> =
'1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0003455 s, 11.9
MB/s\n'; <rc> = 0
Thread-4535::DEBUG::2014-05-14 12:27:57,995::task::595::TaskManager.Task::(_updateState)
Task=`89ef3bd0-f222-470b-b678-6228d5254831`::moving from state init -> state
preparing
Thread-4535::INFO::2014-05-14 12:27:57,995::logUtils::44::dispatcher::(wrapper) Run and
protect: repoStats(options=None)
Thread-4535::INFO::2014-05-14 12:27:57,996::logUtils::47::dispatcher::(wrapper) Run and
protect: repoStats, Return response: {'9bf9ed01-43ab-4372-acdd-3500645f3bd0':
{'code': 0, 'version': 3, 'acquired': True, 'delay':
'0.0003455', 'lastCheck': '2.2', 'valid': True}}
Thread-4535::DEBUG::2014-05-14 12:27:57,996::task::1185::TaskManager.Task::(prepare)
Task=`89ef3bd0-f222-470b-b678-6228d5254831`::finished:
{'9bf9ed01-43ab-4372-acdd-3500645f3bd0': {'code': 0, 'version': 3,
'acquired': True, 'delay': '0.0003455', 'lastCheck':
'2.2', 'valid': True}}
Thread-4535::DEBUG::2014-05-14 12:27:57,996::task::595::TaskManager.Task::(_updateState)
Task=`89ef3bd0-f222-470b-b678-6228d5254831`::moving from state preparing -> state
finished
Thread-4535::DEBUG::2014-05-14
12:27:57,996::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll
requests {} resources {}
Thread-4535::DEBUG::2014-05-14
12:27:57,996::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll
requests {}
Thread-4535::DEBUG::2014-05-14 12:27:57,996::task::990::TaskManager.Task::(_decref)
Task=`89ef3bd0-f222-470b-b678-6228d5254831`::ref 0 aborting False
Alain VONDRA
Chargé d'exploitation des Systèmes d'Information Direction Administrative et
Financière
+33 1 44 39 77 76
UNICEF France
3 rue Duguay Trouin 75006 PARIS
www.unicef.fr
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users