Datacenter in no more responsive
--_004_E73CCB6CDF087B47A36DEC60AE765BF8A4706DUNCSRVEXC1cfuloca_ Content-Type: multipart/alternative; boundary="_000_E73CCB6CDF087B47A36DEC60AE765BF8A4706DUNCSRVEXC1cfuloca_" --_000_E73CCB6CDF087B47A36DEC60AE765BF8A4706DUNCSRVEXC1cfuloca_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi, I have a very weird issue with the last version of oVirt 3.4.1, I've upgrad= ed because I already had this issue. It came when I wanted to re-create a new export domain on my first hypervis= or larger than the first one. After this point, the datacenter came in a non responsive state, so I've up= graded oVirt to 3.4.1, but the issue is still there. I join the tail of the engine.log and the vdsm.log of one of the two hyperv= isors. Tell me if you want more logs. Thank you in advance for your help :) Engine.log : TaskStatusListReturnForXmlRpc [mStatus=3DStatusForXmlRpc [mCode=3D654, mMes= sage=3DNot SPM: ()]] 2014-05-14 12:26:34,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HS= MGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-82) HostName= =3D unc-srv-hyp2 2014-05-14 12:26:34,199 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HS= MGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-82) Command = HSMGetAllTasksStatusesVDSCommand(HostName =3D unc-srv-hyp2, HostId =3D 2d87= 22cc-5041-427d-964d-8980f40c5aa6) execution failed. Exception: IRSNonOperat= ionalException: IRSGenericException: IRSErrorException: IRSNonOperationalEx= ception: Not SPM: () 2014-05-14 12:26:34,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.Ir= sBrokerCommand] (DefaultQuartzScheduler_Worker-82) hostFromVds::selectedVds= - unc-srv-hyp2, spmStatus Unknown_Pool, storage pool UNICEF 2014-05-14 12:26:34,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.Co= nnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-82) START, Conne= ctStoragePoolVDSCommand(HostName =3D unc-srv-hyp2, HostId =3D 2d8722cc-5041= -427d-964d-8980f40c5aa6, storagePoolId =3D 4eeccf64-715d-4ebe-a44c-eeca94a0= 9a05, vds_spm_id =3D 2, masterDomainId =3D 9bf9ed01-43ab-4372-acdd-3500645f= 3bd0, masterVersion =3D 1), log id: aaf1e60 2014-05-14 12:26:35,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.Co= nnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-82) FINISH, Conn= ectStoragePoolVDSCommand, log id: aaf1e60 2014-05-14 12:26:35,998 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.Ir= sBrokerCommand] (DefaultQuartzScheduler_Worker-82) IrsBroker::Failed::GetSt= oragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: I= RSErrorException: IRSNonOperationalException: Could not connect host to Dat= a Center(Storage issue) 2014-05-14 12:26:36,082 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.Ir= sBrokerCommand] (DefaultQuartzScheduler_Worker-82) Irs placed on server 2d8= 722cc-5041-427d-964d-8980f40c5aa6 failed. Proceed Failover 2014-05-14 12:26:36,126 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.Ir= sBrokerCommand] (DefaultQuartzScheduler_Worker-82) hostFromVds::selectedVds= - unc-srv-hyp1, spmStatus Unknown_Pool, storage pool UNICEF 2014-05-14 12:26:36,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.Co= nnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-82) START, Conne= ctStoragePoolVDSCommand(HostName =3D unc-srv-hyp1, HostId =3D 4987bc7d-82b2= -444f-ad6c-8289da5e4fb9, storagePoolId =3D 4eeccf64-715d-4ebe-a44c-eeca94a0= 9a05, vds_spm_id =3D 1, masterDomainId =3D 9bf9ed01-43ab-4372-acdd-3500645f= 3bd0, masterVersion =3D 1), log id: 3b569aaf 2014-05-14 12:26:37,177 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.Co= nnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-82) FINISH, Conn= ectStoragePoolVDSCommand, log id: 3b569aaf 2014-05-14 12:26:37,182 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.Ir= sBrokerCommand] (DefaultQuartzScheduler_Worker-82) IrsBroker::Failed::GetSt= oragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: I= RSErrorException: IRSNonOperationalException: Could not connect host to Dat= a Center(Storage issue) 2014-05-14 12:26:47,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HS= MGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-13) Command = org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand = return value TaskStatusListReturnForXmlRpc [mStatus=3DStatusForXmlRpc [mCode=3D654, mMes= sage=3DNot SPM: ()]] 2014-05-14 12:26:47,321 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HS= MGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-13) HostName= =3D unc-srv-hyp1 2014-05-14 12:26:47,325 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HS= MGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-13) Command = HSMGetAllTasksStatusesVDSCommand(HostName =3D unc-srv-hyp1, HostId =3D 4987= bc7d-82b2-444f-ad6c-8289da5e4fb9) execution failed. Exception: IRSNonOperat= ionalException: IRSGenericException: IRSErrorException: IRSNonOperationalEx= ception: Not SPM: () 2014-05-14 12:26:47,375 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.Ir= sBrokerCommand] (DefaultQuartzScheduler_Worker-13) hostFromVds::selectedVds= - unc-srv-hyp2, spmStatus Unknown_Pool, storage pool UNICEF 2014-05-14 12:26:47,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.Co= nnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-13) START, Conne= ctStoragePoolVDSCommand(HostName =3D unc-srv-hyp2, HostId =3D 2d8722cc-5041= -427d-964d-8980f40c5aa6, storagePoolId =3D 4eeccf64-715d-4ebe-a44c-eeca94a0= 9a05, vds_spm_id =3D 2, masterDomainId =3D 9bf9ed01-43ab-4372-acdd-3500645f= 3bd0, masterVersion =3D 1), log id: 2fce851e 2014-05-14 12:26:49,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.Co= nnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-13) FINISH, Conn= ectStoragePoolVDSCommand, log id: 2fce851e 2014-05-14 12:26:49,042 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.Ir= sBrokerCommand] (DefaultQuartzScheduler_Worker-13) IrsBroker::Failed::GetSt= oragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: I= RSErrorException: IRSNonOperationalException: Could not connect host to Dat= a Center(Storage issue) Vdsm.log : Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 873, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 45, in wrapper res =3D f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 1009, in connectStoragePool spUUID, hostID, msdUUID, masterVersion, domainsMap) File "/usr/share/vdsm/storage/hsm.py", line 1080, in _connectStoragePool res =3D pool.connect(hostID, msdUUID, masterVersion) File "/usr/share/vdsm/storage/sp.py", line 638, in connect self.__createMailboxMonitor() File "/usr/share/vdsm/storage/sp.py", line 459, in __createMailboxMonitor self.masterDomain.supportsMailbox): File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ return getattr(self.getRealDomain(), attrName) AttributeError: BlockStorageDomain instance has no attribute 'supportsMailb= ox' Thread-4535::DEBUG::2014-05-14 12:27:52,895::task::885::TaskManager.Task::(= _run) Task=3D`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::Task._run: 2096f0da-3c= 8a-4837-9d0d-2e4b8abed77a ('4eeccf64-715d-4ebe-a44c-eeca94a09a05', 1, '9bf9= ed01-43ab-4372-acdd-3500645f3bd0', 1, None) {} failed - stopping task Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::1211::TaskManager.Task::= (stop) Task=3D`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::stopping in state pre= paring (force False) Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::990::TaskManager.Task::(= _decref) Task=3D`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::ref 1 aborting True Thread-4535::INFO::2014-05-14 12:27:52,896::task::1168::TaskManager.Task::(= prepare) Task=3D`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::aborting: Task is a= borted: u"BlockStorageDomain instance has no attribute 'supportsMailbox'" -= code 100 Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::1173::TaskManager.Task::= (prepare) Task=3D`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::Prepare: aborted: = BlockStorageDomain instance has no attribute 'supportsMailbox' Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::990::TaskManager.Task::(= _decref) Task=3D`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::ref 0 aborting True Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::925::TaskManager.Task::(= _doAbort) Task=3D`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::Task._doAbort: for= ce False Thread-4535::DEBUG::2014-05-14 12:27:52,896::resourceManager::977::Resource= Manager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-4535::DEBUG::2014-05-14 12:27:52,897::task::595::TaskManager.Task::(= _updateState) Task=3D`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::moving from st= ate preparing -> state aborting Thread-4535::DEBUG::2014-05-14 12:27:52,897::task::550::TaskManager.Task::(= __state_aborting) Task=3D`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::_aborting:= recover policy none Thread-4535::DEBUG::2014-05-14 12:27:52,897::task::595::TaskManager.Task::(= _updateState) Task=3D`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::moving from st= ate aborting -> state failed Thread-4535::DEBUG::2014-05-14 12:27:52,897::resourceManager::940::Resource= Manager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-4535::DEBUG::2014-05-14 12:27:52,897::resourceManager::977::Resource= Manager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-4535::ERROR::2014-05-14 12:27:52,897::dispatcher::68::Storage.Dispat= cher.Protect::(run) BlockStorageDomain instance has no attribute 'supportsM= ailbox' Traceback (most recent call last): File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run result =3D ctask.prepare(self.func, *args, **kwargs) File "/usr/share/vdsm/storage/task.py", line 103, in wrapper return m(self, *a, **kw) File "/usr/share/vdsm/storage/task.py", line 1176, in prepare raise self.error AttributeError: BlockStorageDomain instance has no attribute 'supportsMailb= ox' Thread-16::DEBUG::2014-05-14 12:27:55,639::lvm::295::Storage.Misc.excCmd::(= cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = =3D [\\"^/dev/mapper/\\"] ignore_suspended_devices=3D1 write_cache_state=3D= 0 disable_after_error_count=3D3 obtain_device_list_from_udev=3D0 filter =3D= [ \'a|/dev/mapper/3690b11c0005592a9000004855344bf5c|\', \'r|.*|\' ] } glo= bal { locking_type=3D1 prioritise_write_locks=3D1 wait_for_locks=3D1 } = backup { retain_min =3D 50 retain_days =3D 0 } " 9bf9ed01-43ab-4372-acdd-= 3500645f3bd0' (cwd None) Thread-16::DEBUG::2014-05-14 12:27:55,765::lvm::295::Storage.Misc.excCmd::(= cmd) SUCCESS: <err> =3D ''; <rc> =3D 0 Thread-16::DEBUG::2014-05-14 12:27:55,768::blockSD::605::Storage.Misc.excCm= d::(getReadDelay) '/bin/dd iflag=3Ddirect if=3D/dev/9bf9ed01-43ab-4372-acdd= -3500645f3bd0/metadata bs=3D4096 count=3D1' (cwd None) Thread-16::DEBUG::2014-05-14 12:27:55,773::blockSD::605::Storage.Misc.excCm= d::(getReadDelay) SUCCESS: <err> =3D '1+0 records in\n1+0 records out\n4096= bytes (4.1 kB) copied, 0.0003455 s, 11.9 MB/s\n'; <rc> =3D 0 Thread-4535::DEBUG::2014-05-14 12:27:57,995::task::595::TaskManager.Task::(= _updateState) Task=3D`89ef3bd0-f222-470b-b678-6228d5254831`::moving from st= ate init -> state preparing Thread-4535::INFO::2014-05-14 12:27:57,995::logUtils::44::dispatcher::(wrap= per) Run and protect: repoStats(options=3DNone) Thread-4535::INFO::2014-05-14 12:27:57,996::logUtils::47::dispatcher::(wrap= per) Run and protect: repoStats, Return response: {'9bf9ed01-43ab-4372-acdd= -3500645f3bd0': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.000= 3455', 'lastCheck': '2.2', 'valid': True}} Thread-4535::DEBUG::2014-05-14 12:27:57,996::task::1185::TaskManager.Task::= (prepare) Task=3D`89ef3bd0-f222-470b-b678-6228d5254831`::finished: {'9bf9ed= 01-43ab-4372-acdd-3500645f3bd0': {'code': 0, 'version': 3, 'acquired': True= , 'delay': '0.0003455', 'lastCheck': '2.2', 'valid': True}} Thread-4535::DEBUG::2014-05-14 12:27:57,996::task::595::TaskManager.Task::(= _updateState) Task=3D`89ef3bd0-f222-470b-b678-6228d5254831`::moving from st= ate preparing -> state finished Thread-4535::DEBUG::2014-05-14 12:27:57,996::resourceManager::940::Resource= Manager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-4535::DEBUG::2014-05-14 12:27:57,996::resourceManager::977::Resource= Manager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-4535::DEBUG::2014-05-14 12:27:57,996::task::990::TaskManager.Task::(= _decref) Task=3D`89ef3bd0-f222-470b-b678-6228d5254831`::ref 0 aborting Fals= e ________________________________ Alain VONDRA Charg? d'exploitation des Syst?mes d'Information Direction Administrative et Financi?re +33 1 44 39 77 76 UNICEF France 3 rue Duguay Trouin 75006 PARIS www.unicef.fr<http://www.unicef.fr/> [cid:signature_email_consultation223f6c]<http://www.jeparledemesdroits.fr> ________________________________ --_000_E73CCB6CDF087B47A36DEC60AE765BF8A4706DUNCSRVEXC1cfuloca_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr= osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:= //www.w3.org/TR/REC-html40"> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
<meta name=3D"Generator" content=3D"Microsoft Word 12 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:Wingdings; panose-1:5 0 0 0 0 0 0 0 0 0;} @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0cm; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri","sans-serif";} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:purple; text-decoration:underline;} span.EmailStyle17 {mso-style-type:personal-compose; font-family:"Calibri","sans-serif"; color:windowtext;} .MsoChpDefault {mso-style-type:export-only;} @page WordSection1 {size:612.0pt 792.0pt; margin:70.85pt 70.85pt 70.85pt 70.85pt;} div.WordSection1 {page:WordSection1;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--> </head> <body lang=3D"FR" link=3D"blue" vlink=3D"purple"> <div class=3D"WordSection1"> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Hi,<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">I have a very weird issue with = the last version of oVirt 3.4.1, I’ve upgraded because I already had = this issue.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">It came when I wanted to re-cre= ate a new export domain on my first hypervisor larger than the first one.<o= :p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">After this point, the datacente= r came in a non responsive state, so I’ve upgraded oVirt to 3.4.1, bu= t the issue is still there.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">I join the tail of the engine.l= og and the vdsm.log of one of the two hypervisors.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Tell me if you want more logs.<= o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Thank you in advance for your h= elp </span> <span lang=3D"EN-GB" style=3D"font-family:Wingdings">J</span><span lang=3D"= EN-GB"><o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Engine.log :<o:p></o:p></span><= /p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">TaskStatusListReturnForXmlRpc [= mStatus=3DStatusForXmlRpc [mCode=3D654, mMessage=3DNot SPM: ()]]<o:p></o:p>= </span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">2014-05-14 12:26:34,195 INFO&nb= sp; [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCom= mand] (DefaultQuartzScheduler_Worker-82) HostName =3D unc-srv-hyp2<o:p></o:= p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">2014-05-14 12:26:34,199 ERROR [= org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]= (DefaultQuartzScheduler_Worker-82) Command HSMGetAllTasksStatusesVDSComman= d(HostName =3D unc-srv-hyp2, HostId =3D 2d8722cc-5041-427d-964d-8980f40c5aa6) execution failed. Exception: IRSNonO= perationalException: IRSGenericException: IRSErrorException: IRSNonOperatio= nalException: Not SPM: ()<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">2014-05-14 12:26:34,248 INFO&nb= sp; [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQu= artzScheduler_Worker-82) hostFromVds::selectedVds - unc-srv-hyp2, spmStatus= Unknown_Pool, storage pool UNICEF<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">2014-05-14 12:26:34,269 INFO&nb= sp; [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand= ] (DefaultQuartzScheduler_Worker-82) START, ConnectStoragePoolVDSCommand(Ho= stName =3D unc-srv-hyp2, HostId =3D 2d8722cc-5041-427d-964d-8980f40c5aa6, storagePoolId =3D 4eeccf64-715d-4ebe-a44c-eeca94a09a05, vds_spm_id =3D 2, = masterDomainId =3D 9bf9ed01-43ab-4372-acdd-3500645f3bd0, masterVersion =3D = 1), log id: aaf1e60<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">2014-05-14 12:26:35,998 INFO&nb= sp; [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand= ] (DefaultQuartzScheduler_Worker-82) FINISH, ConnectStoragePoolVDSCommand, = log id: aaf1e60<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">2014-05-14 12:26:35,998 ERROR [= org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzS= cheduler_Worker-82) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNon= OperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to D= ata Center(Storage issue)<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">2014-05-14 12:26:36,082 INFO&nb= sp; [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQu= artzScheduler_Worker-82) Irs placed on server 2d8722cc-5041-427d-964d-8980f= 40c5aa6 failed. Proceed Failover<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">2014-05-14 12:26:36,126 INFO&nb= sp; [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQu= artzScheduler_Worker-82) hostFromVds::selectedVds - unc-srv-hyp1, spmStatus= Unknown_Pool, storage pool UNICEF<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">2014-05-14 12:26:36,147 INFO&nb= sp; [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand= ] (DefaultQuartzScheduler_Worker-82) START, ConnectStoragePoolVDSCommand(Ho= stName =3D unc-srv-hyp1, HostId =3D 4987bc7d-82b2-444f-ad6c-8289da5e4fb9, storagePoolId =3D 4eeccf64-715d-4ebe-a44c-eeca94a09a05, vds_spm_id =3D 1, = masterDomainId =3D 9bf9ed01-43ab-4372-acdd-3500645f3bd0, masterVersion =3D = 1), log id: 3b569aaf<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">2014-05-14 12:26:37,177 INFO&nb= sp; [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand= ] (DefaultQuartzScheduler_Worker-82) FINISH, ConnectStoragePoolVDSCommand, = log id: 3b569aaf<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">2014-05-14 12:26:37,182 ERROR [= org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzS= cheduler_Worker-82) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNon= OperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to D= ata Center(Storage issue)<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">2014-05-14 12:26:47,305 INFO&nb= sp; [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCom= mand] (DefaultQuartzScheduler_Worker-13) Command org.ovirt.engine.core.vdsb= roker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">TaskStatusListReturnForXmlRpc [= mStatus=3DStatusForXmlRpc [mCode=3D654, mMessage=3DNot SPM: ()]]<o:p></o:p>= </span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">2014-05-14 12:26:47,321 INFO&nb= sp; [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCom= mand] (DefaultQuartzScheduler_Worker-13) HostName =3D unc-srv-hyp1<o:p></o:= p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">2014-05-14 12:26:47,325 ERROR [= org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]= (DefaultQuartzScheduler_Worker-13) Command HSMGetAllTasksStatusesVDSComman= d(HostName =3D unc-srv-hyp1, HostId =3D 4987bc7d-82b2-444f-ad6c-8289da5e4fb9) execution failed. Exception: IRSNonO= perationalException: IRSGenericException: IRSErrorException: IRSNonOperatio= nalException: Not SPM: ()<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">2014-05-14 12:26:47,375 INFO&nb= sp; [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQu= artzScheduler_Worker-13) hostFromVds::selectedVds - unc-srv-hyp2, spmStatus= Unknown_Pool, storage pool UNICEF<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">2014-05-14 12:26:47,396 INFO&nb= sp; [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand= ] (DefaultQuartzScheduler_Worker-13) START, ConnectStoragePoolVDSCommand(Ho= stName =3D unc-srv-hyp2, HostId =3D 2d8722cc-5041-427d-964d-8980f40c5aa6, storagePoolId =3D 4eeccf64-715d-4ebe-a44c-eeca94a09a05, vds_spm_id =3D 2, = masterDomainId =3D 9bf9ed01-43ab-4372-acdd-3500645f3bd0, masterVersion =3D = 1), log id: 2fce851e<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">2014-05-14 12:26:49,042 INFO&nb= sp; [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand= ] (DefaultQuartzScheduler_Worker-13) FINISH, ConnectStoragePoolVDSCommand, = log id: 2fce851e<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">2014-05-14 12:26:49,042 ERROR [= org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzS= cheduler_Worker-13) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNon= OperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to D= ata Center(Storage issue)<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Vdsm.log :<o:p></o:p></span></p=
</p> <p class=3D"MsoNormal"><span lang=3D"EN-GB"> res =3D pool= .connect(hostID, msdUUID, masterVersion)<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB"> File "/usr/share/vd= sm/storage/sp.py", line 638, in connect<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB"> self.__creat= eMailboxMonitor()<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB"> File "/usr/share/vd= sm/storage/sp.py", line 459, in __createMailboxMonitor<o:p></o:p></spa= n></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB"> self.masterD= omain.supportsMailbox):<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB"> File "/usr/share/vd= sm/storage/sdc.py", line 49, in __getattr__<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB"> return getat=
<p class=3D"MsoNormal"><span lang=3D"EN-GB"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Traceback (most recent call las= t):<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB"> File "/usr/share/vd= sm/storage/task.py", line 873, in _run<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB"> return fn(*a= rgs, **kargs)<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB"> File "/usr/share/vd= sm/logUtils.py", line 45, in wrapper<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB"> res =3D f(*a= rgs, **kwargs)<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB"> File "/usr/share/vd= sm/storage/hsm.py", line 1009, in connectStoragePool<o:p></o:p></span>= </p> <p class=3D"MsoNormal"><span lang=3D"EN-GB"> spUUID, host= ID, msdUUID, masterVersion, domainsMap)<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB"> File "/usr/share/vd= sm/storage/hsm.py", line 1080, in _connectStoragePool<o:p></o:p></span= tr(self.getRealDomain(), attrName)<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">AttributeError: BlockStorageDom= ain instance has no attribute 'supportsMailbox'<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Thread-4535::DEBUG::2014-05-14 = 12:27:52,895::task::885::TaskManager.Task::(_run) Task=3D`2096f0da-3c8a-483= 7-9d0d-2e4b8abed77a`::Task._run: 2096f0da-3c8a-4837-9d0d-2e4b8abed77a ('4ee= ccf64-715d-4ebe-a44c-eeca94a09a05', 1, '9bf9ed01-43ab-4372-acdd-3500645f3bd0', 1, None) {} failed - stopping task= <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Thread-4535::DEBUG::2014-05-14 = 12:27:52,896::task::1211::TaskManager.Task::(stop) Task=3D`2096f0da-3c8a-48= 37-9d0d-2e4b8abed77a`::stopping in state preparing (force False)<o:p></o:p>= </span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Thread-4535::DEBUG::2014-05-14 = 12:27:52,896::task::990::TaskManager.Task::(_decref) Task=3D`2096f0da-3c8a-= 4837-9d0d-2e4b8abed77a`::ref 1 aborting True<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Thread-4535::INFO::2014-05-14 1= 2:27:52,896::task::1168::TaskManager.Task::(prepare) Task=3D`2096f0da-3c8a-= 4837-9d0d-2e4b8abed77a`::aborting: Task is aborted: u"BlockStorageDoma= in instance has no attribute 'supportsMailbox'" - code 100<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Thread-4535::DEBUG::2014-05-14 = 12:27:52,896::task::1173::TaskManager.Task::(prepare) Task=3D`2096f0da-3c8a= -4837-9d0d-2e4b8abed77a`::Prepare: aborted: BlockStorageDomain instance has= no attribute 'supportsMailbox'<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Thread-4535::DEBUG::2014-05-14 = 12:27:52,896::task::990::TaskManager.Task::(_decref) Task=3D`2096f0da-3c8a-= 4837-9d0d-2e4b8abed77a`::ref 0 aborting True<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Thread-4535::DEBUG::2014-05-14 = 12:27:52,896::task::925::TaskManager.Task::(_doAbort) Task=3D`2096f0da-3c8a= -4837-9d0d-2e4b8abed77a`::Task._doAbort: force False<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Thread-4535::DEBUG::2014-05-14 = 12:27:52,896::resourceManager::977::ResourceManager.Owner::(cancelAll) Owne= r.cancelAll requests {}<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Thread-4535::DEBUG::2014-05-14 = 12:27:52,897::task::595::TaskManager.Task::(_updateState) Task=3D`2096f0da-= 3c8a-4837-9d0d-2e4b8abed77a`::moving from state preparing -> state abort= ing<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Thread-4535::DEBUG::2014-05-14 = 12:27:52,897::task::550::TaskManager.Task::(__state_aborting) Task=3D`2096f= 0da-3c8a-4837-9d0d-2e4b8abed77a`::_aborting: recover policy none<o:p></o:p>= </span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Thread-4535::DEBUG::2014-05-14 = 12:27:52,897::task::595::TaskManager.Task::(_updateState) Task=3D`2096f0da-= 3c8a-4837-9d0d-2e4b8abed77a`::moving from state aborting -> state failed= <o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Thread-4535::DEBUG::2014-05-14 = 12:27:52,897::resourceManager::940::ResourceManager.Owner::(releaseAll) Own= er.releaseAll requests {} resources {}<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Thread-4535::DEBUG::2014-05-14 = 12:27:52,897::resourceManager::977::ResourceManager.Owner::(cancelAll) Owne= r.cancelAll requests {}<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Thread-4535::ERROR::2014-05-14 = 12:27:52,897::dispatcher::68::Storage.Dispatcher.Protect::(run) BlockStorag= eDomain instance has no attribute 'supportsMailbox'<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Traceback (most recent call las= t):<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB"> File "/usr/share/vd= sm/storage/dispatcher.py", line 60, in run<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB"> result =3D c= task.prepare(self.func, *args, **kwargs)<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB"> File "/usr/share/vd= sm/storage/task.py", line 103, in wrapper<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB"> return m(sel= f, *a, **kw)<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB"> File "/usr/share/vd= sm/storage/task.py", line 1176, in prepare<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB"> raise self.e= rror<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">AttributeError: BlockStorageDom= ain instance has no attribute 'supportsMailbox'<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Thread-16::DEBUG::2014-05-14 12= :27:55,639::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lv= m vgck --config " devices { preferred_names =3D [\\"^/dev/mapper/= \\"] ignore_suspended_devices=3D1 write_cache_state=3D0 disable_after_error_count=3D3 obtain_device_list_from_udev=3D0 filter =3D = [ \'a|/dev/mapper/3690b11c0005592a9000004855344bf5c|\', \'r|.*|\' ] } = global { locking_type=3D1 prioritise_write_locks=3D1 wai= t_for_locks=3D1 } backup { retain_min =3D 50 retain_days = =3D 0 } " 9bf9ed01-43ab-4372-acdd-3500645f3bd0' (cwd None)<o:p></o:p></span><= /p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Thread-16::DEBUG::2014-05-14 12= :27:55,765::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> =3D '= '; <rc> =3D 0<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Thread-16::DEBUG::2014-05-14 12= :27:55,768::blockSD::605::Storage.Misc.excCmd::(getReadDelay) '/bin/dd ifla= g=3Ddirect if=3D/dev/9bf9ed01-43ab-4372-acdd-3500645f3bd0/metadata bs=3D409= 6 count=3D1' (cwd None)<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Thread-16::DEBUG::2014-05-14 12= :27:55,773::blockSD::605::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <= err> =3D '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) c= opied, 0.0003455 s, 11.9 MB/s\n'; <rc> =3D 0<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Thread-4535::DEBUG::2014-05-14 = 12:27:57,995::task::595::TaskManager.Task::(_updateState) Task=3D`89ef3bd0-= f222-470b-b678-6228d5254831`::moving from state init -> state preparing<= o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Thread-4535::INFO::2014-05-14 1= 2:27:57,995::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats= (options=3DNone)<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Thread-4535::INFO::2014-05-14 1= 2:27:57,996::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats= , Return response: {'9bf9ed01-43ab-4372-acdd-3500645f3bd0': {'code': 0, 've= rsion': 3, 'acquired': True, 'delay': '0.0003455', 'lastCheck': '2.2', 'valid': True}}<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Thread-4535::DEBUG::2014-05-14 = 12:27:57,996::task::1185::TaskManager.Task::(prepare) Task=3D`89ef3bd0-f222= -470b-b678-6228d5254831`::finished: {'9bf9ed01-43ab-4372-acdd-3500645f3bd0'= : {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0003455', 'lastCheck': '2.2', 'valid': True}}<o:p></o:p>= </span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Thread-4535::DEBUG::2014-05-14 = 12:27:57,996::task::595::TaskManager.Task::(_updateState) Task=3D`89ef3bd0-= f222-470b-b678-6228d5254831`::moving from state preparing -> state finis= hed<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Thread-4535::DEBUG::2014-05-14 = 12:27:57,996::resourceManager::940::ResourceManager.Owner::(releaseAll) Own= er.releaseAll requests {} resources {}<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Thread-4535::DEBUG::2014-05-14 = 12:27:57,996::resourceManager::977::ResourceManager.Owner::(cancelAll) Owne= r.cancelAll requests {}<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-GB">Thread-4535::DEBUG::2014-05-14 = 12:27:57,996::task::990::TaskManager.Task::(_decref) Task=3D`89ef3bd0-f222-= 470b-b678-6228d5254831`::ref 0 aborting False<o:p></o:p></span></p> </div> <br> <table style=3D"BORDER-COLLAPSE: collapse" class=3D"3a8ded91-40ea-4011-8a93= -273b83b3e250Table" border=3D"0" cellspacing=3D"0" cellpadding=3D"0"> <tbody> <tr> <td> <p class=3D"3a8ded91-40ea-4011-8a93-273b83b3e250"></p> <hr color=3D"#0099ff" size=3D"1"> <p></p> <p></p> </td> </tr> <tr style=3D"TEXT-DECORATION: "> <td> <p class=3D"d25f83d3-f6d0-434b-863d-6b70d6fe505d"> <table style=3D"BORDER-BOTTOM: medium none; BORDER-COLLAPSE: collapse; BORD= ER-TOP: medium none" class=3D"d25f83d3-f6d0-434b-863d-6b70d6fe505dTable" bo= rder=3D"0" cellspacing=3D"0" cellpadding=3D"0"> <tbody> <tr style=3D"TEXT-DECORATION: "> <td class=3D"WIDTH"> <p class=3D"d25f83d3-f6d0-434b-863d-6b70d6fe505d" align=3D"left"><font colo= r=3D"#0099ff" size=3D"2" face=3D"Arial"><strong>Alain VONDRA </s= trong></font><br> <font color=3D"#808080" size=3D"1" face=3D"Arial"><strong>Chargé d'e= xploitation des Systèmes d'Information </stro= ng></font><br> <font color=3D"#808080" size=3D"1" face=3D"Arial"><strong>Direction Adminis= trative et Financière</strong></font><strong><br> <font color=3D"#808080" size=3D"1">+33 1 44 39 77 76 &= nbsp; </font></strong><strong><font color=3D"#808080" size=3D"1"></font></s= trong><strong><br> </strong><strong><font color=3D"#808080" size=3D"1">UNICEF France</font><br=
<font color=3D"#808080" size=3D"1">3 rue Duguay Trouin 75006 PAR= IS</font></strong><strong><br> <font color=3D"#0099ff" size=3D"1" face=3D"Arial"><strong><a href=3D"http:/= /www.unicef.fr/">www.unicef.fr</a></strong></font> </p> </strong></td> <td class=3D"WIDTH"> <p class=3D"d25f83d3-f6d0-434b-863d-6b70d6fe505d" align=3D"right"><a href= =3D"http://www.jeparledemesdroits.fr"><img src=3D"cid:signature_email_consu= ltation223f6c" width=3D"87" height=3D"80"></a> </p> </td> <td> <p class=3D"d25f83d3-f6d0-434b-863d-6b70d6fe505d" align=3D"right"> </p=
</td> </tr> </tbody> </table> </p> </td> </tr> <tr> <td> <p class=3D"d25f83d3-f6d0-434b-863d-6b70d6fe505d"></p> <hr color=3D"#0099ff" size=3D"1"> </td> </tr> </tbody> </table> <br> </body> </html> --_000_E73CCB6CDF087B47A36DEC60AE765BF8A4706DUNCSRVEXC1cfuloca_-- --_004_E73CCB6CDF087B47A36DEC60AE765BF8A4706DUNCSRVEXC1cfuloca_ Content-Type: image/gif; name="signature_email_consultation223f6c" Content-Description: signature_email_consultation223f6c Content-Disposition: inline; filename="signature_email_consultation223f6c"; size=7700; creation-date="Wed, 14 May 2014 10:29:15 GMT"; modification-date="Wed, 14 May 2014 10:29:15 GMT" Content-ID: <signature_email_consultation223f6c> Content-Transfer-Encoding: base64 R0lGODlhZABkAOYAAJF1aFi39/39/NO5qQJYkrCVhbGJdvvm1+vHswCR8suOcYxrW0ZNWSYuONnv /v3z6FyVvHBVTMfGxbDc+HvK/pjU/LBxVHltZzuv/ZiBdLWkltXCtuDh44ZdT9fW1sSYgvDLt723 tfbr48qnlGZIPzxET6J6avvcykxWZOC7pzA5Rv768efUx6SioNKtl67L3+TCr7OppK27xAKC18ra 5e/5/9i0n/XRvEY2M/Hw8ICAgPLTwZuIfgVwuHtiWch8XvnVwuGvl8ahjODb2R4lL9iih/rZxp+X khESGb/l/t3Ry8/Kx/nf0fDbzhUbJZairPzh0Lawrhmi/aGQhpCapI5PPf327vTXxdDR0uvs7qm0 u+3PvkGUy+7i2+bKvG12f1RBPFtnciYXF/T19vPVyYaUnvT5/cDR3BuR4IKJkDgoJcawoqSrr/3t 3+3CqvrXyJxhaZG/3mdeXLqBZKNiSCIgJPTNvJKQjfr6+r/AwKqclca+u/7+/gCZ/////wAAACH/ C1hNUCBEYXRhWE1QPD94cGFja2V0IGJlZ2luPSLvu78iIGlkPSJXNU0wTXBDZWhpSHpyZVN6TlRj emtjOWQiPz4gPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iQWRv YmUgWE1QIENvcmUgNS4wLWMwNjEgNjQuMTQwOTQ5LCAyMDEwLzEyLzA3LTEwOjU3OjAxICAgICAg ICAiPiA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRm LXN5bnRheC1ucyMiPiA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIiB4bWxuczp4bXBNTT0i aHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL21tLyIgeG1sbnM6c3RSZWY9Imh0dHA6Ly9ucy5h ZG9iZS5jb20veGFwLzEuMC9zVHlwZS9SZXNvdXJjZVJlZiMiIHhtbG5zOnhtcD0iaHR0cDovL25z LmFkb2JlLmNvbS94YXAvMS4wLyIgeG1wTU06T3JpZ2luYWxEb2N1bWVudElEPSJ4bXAuZGlkOjdB NzA0QjUzMzFCM0UzMTE4QTY4Qzg0MzBEODk5Nzk1IiB4bXBNTTpEb2N1bWVudElEPSJ4bXAuZGlk OkUzMEI5MEZEQjMzMjExRTNBODY5REU5MEREQUM5RjZDIiB4bXBNTTpJbnN0YW5jZUlEPSJ4bXAu aWlkOkUzMEI5MEZDQjMzMjExRTNBODY5REU5MEREQUM5RjZDIiB4bXA6Q3JlYXRvclRvb2w9IkFk b2JlIFBob3Rvc2hvcCBDUzUuMSBXaW5kb3dzIj4gPHhtcE1NOkRlcml2ZWRGcm9tIHN0UmVmOmlu c3RhbmNlSUQ9InhtcC5paWQ6N0U3MDRCNTMzMUIzRTMxMThBNjhDODQzMEQ4OTk3OTUiIHN0UmVm OmRvY3VtZW50SUQ9InhtcC5kaWQ6N0E3MDRCNTMzMUIzRTMxMThBNjhDODQzMEQ4OTk3OTUiLz4g PC9yZGY6RGVzY3JpcHRpb24+IDwvcmRmOlJERj4gPC94OnhtcG1ldGE+IDw/eHBhY2tldCBlbmQ9 InIiPz4B//79/Pv6+fj39vX08/Lx8O/u7ezr6uno5+bl5OPi4eDf3t3c29rZ2NfW1dTT0tHQz87N zMvKycjHxsXEw8LBwL++vby7urm4t7a1tLOysbCvrq2sq6qpqKempaSjoqGgn56dnJuamZiXlpWU k5KRkI+OjYyLiomIh4aFhIOCgYB/fn18e3p5eHd2dXRzcnFwb25tbGtqaWhnZmVkY2JhYF9eXVxb WllYV1ZVVFNSUVBPTk1MS0pJSEdGRURDQkFAPz49PDs6OTg3NjU0MzIxMC8uLSwrKikoJyYlJCMi ISAfHh0cGxoZGBcWFRQTEhEQDw4NDAsKCQgHBgUEAwIBAAAh+QQAAAAAACwAAAAAZABkAAAH/4B4 gngChYaHfImKfnwPImtyEXJTMCB2ICBuKTY2QSluIEIdcy4fBgamBapTPj5gYHI4dWq0OLZgERck crpgFzwXCxe/GiFRURJ5MRJDHlFdfIPSh9SK1ol+Vm0DEZIFXiA3mOM35UYnWxkmQkEfpnOnPAU+ JAyvYGqz+Wo4sHKwki5MATAMwBENevTECJFMAhYse6LkECBNEDVE164JeKAEQLcjXsqJC2cEyBYE G7ZcsdHBwAgXQkyZMAGARywwt9SIEVNnVj97/XgdmXLBR4YpAzMozCNhSZ5je/ZQrEjoYqGM2Pxo tdJFgxw5BSqVA4EAhg0XIzRoGLFmQ4GWZ/9jmljQYYEPHLT45ePJzxaJvxcA6Bl4IQMAOT4u3JkS hSmWFjqOLIFG1epVrFoZ+RHQhUUGH5TC3dgBI8UADUI0GJhytICJljA/zKHbIQI+Wjjz1mLwtfCU IzwAECSBa5gOZB6W6JDDxoOhipax8smMrVGbNQsygLoBZMeWLV68wBiwZkQBwxkWRFgnxABdj7b4 vZL/ikQ3HzV5ZCjKwFYJXHIckccQWUigwx17ZPHcIFZJN11mEG7UhDopiHbFDhh+5913MKzhnnoL uGMAHSZYQAIOxL3STzcAzJTBi4bZhtMuFxwYQx4eZJHFEne0gMUKhUB3kYMQFrnCAwUsYEP/OEBc 4eR34MFQ2gbieVjbAqfMNdtfJLTSygImGKDKKR69whsAd+ihQRQhLMFBFm8u0UIIYyw4VVUYXVPk nn6sMAAAI5Dk5BVQmjWAF1vsMYIeY7YIgAFzmUDHXz4sYOmjHwhRAA/01JNYjTUV4yYHNHjAgQch DMEHNQzmqSefRXJWwAcknWDEhVuUBsMGMQCDAm4+iHlKAacYAABtYM6qwXk+2FYCDkTg0AAYJPyy jAfJjOoBHqsuaJGr1sDKSFZWsCCEGzecw8QJV4Bg1nktlKHFBWrYF0wBMQkxwghC9PsBvvqaMtxX YTTgBA4lfKEDGDosFEIMN3KQwxibWYWn/wBEilskHxvYwN0JUEDB7g4pCLHBGA6wUdQFR0ThWhED DODCzPzqW/NAd+HwBRVfqIDwzmnAwoMeQ+mxx0TdWgbuIhrv2QUMdgABcsghkxFeFzVwoAUbeZwR AqQ2vNGEeJugFdOmADRLywVayFBGGlQ8kUYEPvBgt916LEGxIkpfxndWsAKu2QPgSB3yAYgzscUG GyihI6l5ePhBEFscwMQVNyCQAkzEph0BcV3qQAUbbJTxBWIZ3KFfBkMrMZ01SmfctFaKrMDCFoYj rjsTZGwQcwhrxOBOpi5s0IXlJ2Rug6b7dWMffnCk7SV+i63OQwxdjAt7g5jNTt10AjQBgv/hUOh+ ABS8e7GB2cN/4AIZXTBxgK0g2GDeXHJwmQuM6B2FFIxTCIEIsBKdjHiPT3xowhbSVT7zrYt3diBb EPZlgy0cjwkYPAFpXAApu0TiPiaQxyqQwhoY6eFk2tve0h50QKchgIENTNwJyLADFoinNClIgReO ZzkmhOwNCHCBa+zSCrotAA6sG+HqXsQDDXjgdRkZkgFb6EI7nCOGIWPCGzAHgvCEhwzyO98JfHgA MgTxAzSxVKVaAQB18MAAb+QfDwIognFhTIWG6B4VISQCL0TtBCBDXBa36J0uboEMYMRgBhF3BbMI 7FjCuJRwXmQCdQDwhBSjnUZWuEcITUf/BFBL19TKt643YGgHoyHDG3yYxcOV0SztMQwkF3As4dBk JpUM4RRiMAQ8aIaAeXxVJx+0AiWA42OjzOAb3gAEIDBzmYqkGuLMuDweVFI4lsrmsXBpySkYrQsC +F4U/ca0YfZpA2sAR9TOMUb0resEy2wmd94AyHY2kAyOhNSLZgkHS9GkRW4sQAyUkANPAvOO4dIY Cz15zg8UAQYIsAN32JlFeC7zBnawww7ewM52Wg6fyzOWo9KWTTC1iCas+2YmNZkIjN3Rb3fcIzas MIAPKMAFKUAAk4xwxXrG8xIZbSY7r8gEO+STlm0EQAc60E9/AnSOGthAFhaqSZe2FKEx/21aQv2Q Aw4qwKE2qIREpVZPW130EmSJWjPL+gZ3tWMuSFXPUp2K0qGtwQNWQOCq9totPQlOXNgQQXsMoAAF FMEFMNgOEDrKU2bu4BIIiKxEb8BRWwFBc6WAqxprM1ekVnKXxvMlVYH5KqpqzAOnmINqvwoziAaV sSV5LFkQAIrJcvQNdnBDEN5Km9p0o7N11cMAPCCCcO6JtEybzhgU5EmtUSENOkjDF/4TATCt1n2f sIQoz8FTIGCUthC1RFCZaYcUBMEFCpCUXWpThft4do4xmMwYTMtS2G3VD1oIQxme8IQWUEEHKCiB ClTQgAYQoQ5OmAUJOmABEyhACAPIqf8lFsvT7oojsrQNR9RWiVvzFkE2dLFUB+yz1EnarQAhyBG3 AhfFcLW0DGFAAYGJ0AAVlIABBC6wE3aMBCSIoV4k+sAIcqjYZjZTtpGFKAKaSU/L7UC3pZhNdniw APvQzcRT0MAeJCZaFts3puHkwBdkbOAd77gOBS4wgsXQ4x6LAQxVmIP7bPAJnYYDlWgti5LHusoy YlYmSsGOlTvQohPHtwtjwEMm/9ricp6BzGVuMxISTAQa0xjBbf5xS4TgAjorGRNbmG1iIztbTJBB gwjYrWxQvAce/LYubZyjHqKQIz/k4Qs00Ipx68s3hE5HzALWsRMkPekE9+TADehJHXr/rAYwCbnT SsawnnWK4UsoWUr2+8BC9OCRCCy1UrEW6B6GYIYsoKABKNCCaLPSaGxwIAwMYICwedzmMyf4zDTO Rx1+bBQNrMEFEZYSht1AbQyL9ySlscEa1hAFpUagCt++1FFm7SYBlKHGDShBGQrKwkaHkwYBZkAJ yjzsTDuBJ2suubKVDYYM+HsNainAFAowggFI6eYpCG+oo0SeRZmgUt+uFFLhuwcPmEEGaS5wCdKQ Bc2ksKUPwoKMRT5yIph55ZU+8IFXjuBZXGAPG9jDGv5lLEsVRgMR5oSnr51wIXaQLkKX+NDaBKcS OCHNNFYB03+pkc08Acc2LoHgc5z0/wJnHe9qOLAYcKAHJSiBPDFfzYtWxoOFz4wTpimNzPqlqTSW dOgtkwAHavAFJ1TawKjX+6I1gocyDPjGgqf6yA1sdUpbmsZprgMYQsACJaCzPDFfRXD2c73ydJoT AziLKdaCxmNVqjCsg2oIsFADNuz49LQvMAOeAEXYXTzj8Rb84El++KxXWtlI2P0QfA88hKhC5ta7 g/yPALzLm2ZzpxjB8mZiqaMAoyYKIQFZQAMqMGymZ3inpwJhkGt81VIGJmCBF2y0Z2kTuHXL1mN1 oAFNwAKMMwAxkBbBhxT6wQNH0AImuAeosS8zExPPZgov8hsZ8DkRkAFz8iYMgAS1d/93OpZ3DMAG T7cqRDBgrweBuJd0ltYTJedmF7CBSzAAHbgGIaAWapEQGmCCanIMefA1WQIHM4EKIkVLrRAMrcAD UYAFWZAGPcZjB1hm2kcF3eeAQ4hjtDdghbd1J2dsdcAALNAZvpMWjCKFgAgxMdACGgAxbRIC8FAA cLCIi9gBVcAloPF/15MHDqAFdtdjVjdphmd6NlYG68Y3gSdydCiEhJdsdaB1p9gAiVcAB8ACf+YO MVEs3jSFU3A3dnMHwsOIc8CIi1gFcHA9esA6R7AMHHAGQpiGBoh7nKh6TycAsQd4NSaEElh4SVcH ctAETYA5bsAJ5WADRWAB7VUbleL/f6xTPZERBbyYjt40BTowNI2BBe+mdEQwaZmogztmY2mwN7AD e8EmjaRIitGoBqYYAm0gMiCQU0YgMm7wjVXwiH9BLXQTGL4BHPSXAX/RXt1QEG00BXOSBxyQBjZG YJeohsKGjxwHOyIXb/+4kqOIAznmBHLABAiwPEKQAidwAAVpB0HwAw3JJX/hbbRUSadgGAHEA0sF cd7mAx0QGCS4DGwQhIKXcfNoegZYZjamA/p4VSkZkqQofnSYcRDoMwzwCQpgAaSwBW3wAA/QBkaw k3TQkxe5AHLWLyOARhbQCnmjiL3FWfjBGFigAn+gAoAneMmoiVaXcV9QA5tEdaNo/2PhR4cuKYrS cgHN5AZlOQc2wARtYAWcCQVBoABvSQKPWAVYon8QVTKQQhA60AIbkAF0sJcdgB+SkQZ/gATopn3y honDVmmmlzBZ2S2PSWADJnvSKHgu2QAAgDgnAAKXCQMH8AArwJkHYJng2JANuQ4poBJXMJMuaDcC EgUitlRzRYYS0ABpGGBOUAIowACFeXe9+QU5sJgBRorhJ3uMKWBysJltcAAg8I0K4JxWEJ1ruZAW 8JYNKZdFsAUhkzyo+VlRIAMS8BZMNVcZEANYcAdtRgToqaEyRo87ZjDp+QVNtz0MgALrGXviB3tU J3K2oAcrEJ1tcANF8AMKkAIHEP+gnPkAMvoDdEAHHdCjpNAE+wkFN5ACaHRNdzB9e6BUFOojIUAC O4GD8TZsKrCeyWh6VpcwI3pVzmiiJjp4w1mfYBCVPiACAboCbDmjNXqjAvoAbVmWPfqjFmADz7mf 21kKaCQMd+AQGqCU2aEHWCABrsBmSCCYI4eDJqoCmvihTqACX5CVhqCeXgqBklqiJ6oCDNME0Bmd VnACO4mZUICjagkFKaAAP2ABBUoHFjACbXAk57MD6+MawgEAbHAGEnAENUFrIfAPOMBm6RlsVGqi U2kwBuao8+VSkTqp8WailsqsJXABawAFacmpVuCZCjAHRbADovoA01kEBjAHBYr/qh/wAFawlgew AyFFE+14B0yxB3MiAXeAE7LQYxlHYFTJoQXonpWmAjrALVzapevppQLrpSSgB3YAMtNarlYwnaZa o9BZrtzan9daoOCqADnaBkwAA6UAKSTYAqGnBfDKALQQpUhQBwJWaba5Y+qZmzR2d3qnPYdQpQHL rCgQYyYqBxsgMoH0sOXaBgupWh/ABG0ao+0Ap2apANC5lrmSAkUArgswazHABksQA3ugA/tGso1q r/Q4j1VKmHdHYxoHs4bQrOtJtiggB0vwACJTElBArhAbozN6qinwotrQBuhaBBNrAd8qAtHJO29Q quBIaFGwB20SBVwjB5LmspFm/5sgmpIGY2klQAV2dAgDW6mWegRoej5AEA6b+bZA8JmnagEHIABo 2gZA4AYxAa4WcK1k0LcH8Lc82pAAsAwSIAGDCK9piGYzloRBOGw3ZmMHSAQloAWTawg1S7PIywBL cCQFaQRu4AZGMLRGUKqnSgdFQLrcKpNRBq7gagPlKgJtQAafCY50wANt4hQxoAd5IAP5enqZWG8q MI/oJpgFWGAqIAMVQw02K7BkOwQwCgXOWwRFcABHspYn4AYf8AM8SgdtsBHzAwIu0LTcq1p1KgI6 CadwgBwSEAVEwxBhUGy8mYSaaJ5OEGA3lncNkAf5S7nJu6y8MQQCoA0HEMCr+v+iawkFC6nAPJkC G4Gx9WNTc/ADqkUHMAC+Mgm6JhAFSoAFeeCxU3AHyJAGZlZ7xIaDJPy7IheE97sZSUMIDLC/Iuel XwxOmRvAp0rANyyxCkwHBsAHMrwF9nOt4MqTBdAE6ZMCaTEASiABGqADMHIcIfCUxUZviWswtolj 6ql0KoysFCEABPDIkAzJPTDJzuGqOAy6QRDD3IqupnqqP0C6dorHssG9b6l//LIvGhAczTI9BkE6 kfzKjzzJPTADtDwDCXDLuJzLugzLkkzJ2Hs+IBAEahqgazm9cSvExNwGjbS9qDopHTAX4okP/WAf xCEJBqEFsUwAPcDLslzLtmz/y7ncBwnQB+Q8zpCsApHczXmAvQUZzHHbqjeMAHirw8XFrU2gsaOM qgtAByjyiDqxE/XyFyhCLXKgA9gsy7L8ygjtzeB8y+JczuXMy9o8yzMAAaSbuUU6zzuwERE7zwo8 umspkzZgCuG6ACYysm5GH7cwzWwzyTOA0Nus0N1cy+E8zg/dBxLt0jPABTlgwzO8kDPqBhx9AHBc WArMAm7cBuIz0nB6KREgBrXpYzvRD31BHxHwBAz90pMMy9s80w1NzmBNzjEt07TcA2fAzgcszAoQ BAJ6AFeg1qbqBTGMsV4QUhawz3SQNmwG1QCtG7WwIhBAywnA0GMt0zr91TYt/9ZbTQDoPNG1XAZ8 8L/uvNZJy5YwgLemysNoqjgl8wEWIJ60BNVSvRM/9s99QS1coMu1vNhcPdPhHNZa3dWOjQZoMANp QLfcegO6JcBJ+wAijdmZfCRQwAL4XCJBmQG3kRe2gBenTdBlgAZSQNvRTdta/cgoEMsLTdvSLQXc DdZowAUQEN5lEN7hHQAYwAVp0MD/uwXPmwLqbQUiMDabsFuEsLAKtD7LogpqogMX8BW94QNf4Ty8 UBMtEADmbeAGzgUKTt4Mngbh/QQQEAARjuAYgAHdzQUBEAcvcAZnsOEV8AIVEAdx8ASjS7oy3ARk oKlBsgIiIAJdgI1kYAXR4P8I2MgCved4LDAES5AMWRgCPh4FHssDaTK4EqAENFABE5DkSR7iIi4D L/DkTr7hZzABNEADSaDkE1ABFBAA3B0AE1BQfGIGDnAGfFsx0bliWuEAChXDOZADvuQ9NeAHWeAB Zjg7fBDnflADeN40ce4AFcDlcZAEYU3OFOAHUtAHFeAHvnToau4HFdAHFEABh94HGOAAFADWif7o UkC6AjDont4HExAA5CwFhe4Hgg7WAWDpYY0BiT4Bky4FFXDqg17h5IwBFGDpGBAASRDphx7pSXDq UpAZl17qGEDp5MzlhA7pfSAFDnDoSeDGeBDpxR4Akg7pkR7plF7tai7otl7/7KSe7aIO6rVe7MpO 6sXe7dnO6KHu5VpR7Jkh6s6e5svuBw6w7OTc6PDeB7tu74WOBznwAFpx6UngB+6eGY+e6rs+8IJe 6qJe7MTO71rx6LXuB8PuB6J+6wmfBOZNAU1X8DWw7BPQBwGQGcXu5+Vu8GCdBDVw6FIQ5ywuAhQj 8AQ/6B9f64lu6pAe8GAtBbTO833Q6PU+8RUf9JQ+AfROAWig6zMf8SJ/6nh+8MX+6BiA8uScBM1O znEuAC1eRzJ/7hU+9TVAAeSe6AvP9KNe6zUP9GA99RV/9JNeAZYuBexe8OSO6Vrx8YeOAWmvFeGu 794u8lshAgXV9X0AIeat/+ZSj/MMz+9BPwFRb/aUTvE5b95xfvBhz/OJ7u71jgG/runvTs6ZT+jk LurwfvViXwGcKfkD7/DX3vBbnvPcju3KXuiaLvEGfuyRX/GXTu1Rv+s8z/HuLuo3P/MD7+hYn+d1 LwUTcOlVj/OErvXRru8zH9avj/bOP+r7vvjUTv3cLvmFXgN9r/ehngC5PgG/Tu8qnwMOUAMOYPUO sP5Wz8XoX/ztH/9aceWwLgJD0AUcAAhJNX4OSYaGDn5+SYl+AjWHioSMipBJfpaDkImFDgKVjGYT FBh9UhgBFRM0Ey9xr3EVshQUFWcOIk1NQ105vlYiXUNYIVoSWRMBeDQbKf8DG0tLSh5DHBxDHkpY Skt7ISFLQzljNKxnra8vL+XsNGfv7+qusRNJExWkUglSaKgBARAgfOnBpWAALgH8BYgTIorDKGyO 6PhCMYzFL09kvJCRZwuCFCMKGODBY4pJkiRPkozigcOTgAAhcAlIs6bNmgb/JcQgpc+fn0CDCh1K tKjRo0iTKl3KtKnTp1CjSp1KtarVq0vVNMDKFaiiPJIUlfijI2weJH/kKKpDNqyfFm6/NuCgiIOO En7ASuJwIe5PsFLz7GHwswRftEO/+tHBWAfbsi100L3wR6+OtpHvKGK8Ge4eunl05HibN68OsKPt Nv7rR6pmP3xVjM7Rgm3/UMVDy47FG6UObNhtx/7B4oetoj946xa+UxbsZd9fhwKWWucC3Rxp9W71 WjrsT8gtRl8oKyeKHwZlseQZHeXncRySWgTFCzbPnetu8/yEO7WOZD85ICEHcX6ocVtpjV3WVnx/ 0MWAZlGkxx53yDH4BxJIwKfXVwzA1pgc33HgmiJDhIHDbI4hlweIfqjXmlC6AdWhW3CVUMdohP1x XHLAtVXfYkDRl5t+UeURhXCGyYHYhQCaF4VzjY1VFg5AmXdBCTjANRqVHQaoY2saEsdBeH6o5RxQ 8KnG2E93ENkVUBeMhkUdG24W3E9IAAhUA5IIZ558O+aFhF5YMEAfkIX5EVXZi2826uijkEYq6aSU BhUIADs= --_004_E73CCB6CDF087B47A36DEC60AE765BF8A4706DUNCSRVEXC1cfuloca_--
Allon, could it be related to https://bugzilla.redhat.com/show_bug.cgi?id=1083476 ? ----- Original Message ----- From: "VONDRA Alain" <AVONDRA@unicef.fr> To: users@ovirt.org Sent: Wednesday, May 14, 2014 1:29:13 PM Subject: [ovirt-users] Datacenter in no more responsive Hi, I have a very weird issue with the last version of oVirt 3.4.1, I’ve upgraded because I already had this issue. It came when I wanted to re-create a new export domain on my first hypervisor larger than the first one. After this point, the datacenter came in a non responsive state, so I’ve upgraded oVirt to 3.4.1, but the issue is still there. I join the tail of the engine.log and the vdsm.log of one of the two hypervisors. Tell me if you want more logs. Thank you in advance for your help J Engine.log : TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2014-05-14 12:26:34,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-82) HostName = unc-srv-hyp2 2014-05-14 12:26:34,199 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-82) Command HSMGetAllTasksStatusesVDSCommand(HostName = unc-srv-hyp2, HostId = 2d8722cc-5041-427d-964d-8980f40c5aa6) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2014-05-14 12:26:34,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) hostFromVds::selectedVds - unc-srv-hyp2, spmStatus Unknown_Pool, storage pool UNICEF 2014-05-14 12:26:34,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-82) START, ConnectStoragePoolVDSCommand(HostName = unc-srv-hyp2, HostId = 2d8722cc-5041-427d-964d-8980f40c5aa6, storagePoolId = 4eeccf64-715d-4ebe-a44c-eeca94a09a05, vds_spm_id = 2, masterDomainId = 9bf9ed01-43ab-4372-acdd-3500645f3bd0, masterVersion = 1), log id: aaf1e60 2014-05-14 12:26:35,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-82) FINISH, ConnectStoragePoolVDSCommand, log id: aaf1e60 2014-05-14 12:26:35,998 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) 2014-05-14 12:26:36,082 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) Irs placed on server 2d8722cc-5041-427d-964d-8980f40c5aa6 failed. Proceed Failover 2014-05-14 12:26:36,126 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) hostFromVds::selectedVds - unc-srv-hyp1, spmStatus Unknown_Pool, storage pool UNICEF 2014-05-14 12:26:36,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-82) START, ConnectStoragePoolVDSCommand(HostName = unc-srv-hyp1, HostId = 4987bc7d-82b2-444f-ad6c-8289da5e4fb9, storagePoolId = 4eeccf64-715d-4ebe-a44c-eeca94a09a05, vds_spm_id = 1, masterDomainId = 9bf9ed01-43ab-4372-acdd-3500645f3bd0, masterVersion = 1), log id: 3b569aaf 2014-05-14 12:26:37,177 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-82) FINISH, ConnectStoragePoolVDSCommand, log id: 3b569aaf 2014-05-14 12:26:37,182 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) 2014-05-14 12:26:47,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-13) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2014-05-14 12:26:47,321 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-13) HostName = unc-srv-hyp1 2014-05-14 12:26:47,325 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-13) Command HSMGetAllTasksStatusesVDSCommand(HostName = unc-srv-hyp1, HostId = 4987bc7d-82b2-444f-ad6c-8289da5e4fb9) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2014-05-14 12:26:47,375 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-13) hostFromVds::selectedVds - unc-srv-hyp2, spmStatus Unknown_Pool, storage pool UNICEF 2014-05-14 12:26:47,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-13) START, ConnectStoragePoolVDSCommand(HostName = unc-srv-hyp2, HostId = 2d8722cc-5041-427d-964d-8980f40c5aa6, storagePoolId = 4eeccf64-715d-4ebe-a44c-eeca94a09a05, vds_spm_id = 2, masterDomainId = 9bf9ed01-43ab-4372-acdd-3500645f3bd0, masterVersion = 1), log id: 2fce851e 2014-05-14 12:26:49,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-13) FINISH, ConnectStoragePoolVDSCommand, log id: 2fce851e 2014-05-14 12:26:49,042 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-13) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) Vdsm.log : Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 873, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 45, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 1009, in connectStoragePool spUUID, hostID, msdUUID, masterVersion, domainsMap) File "/usr/share/vdsm/storage/hsm.py", line 1080, in _connectStoragePool res = pool.connect(hostID, msdUUID, masterVersion) File "/usr/share/vdsm/storage/sp.py", line 638, in connect self.__createMailboxMonitor() File "/usr/share/vdsm/storage/sp.py", line 459, in __createMailboxMonitor self.masterDomain.supportsMailbox): File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ return getattr(self.getRealDomain(), attrName) AttributeError: BlockStorageDomain instance has no attribute 'supportsMailbox' Thread-4535::DEBUG::2014-05-14 12:27:52,895::task::885::TaskManager.Task::(_run) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::Task._run: 2096f0da-3c8a-4837-9d0d-2e4b8abed77a ('4eeccf64-715d-4ebe-a44c-eeca94a09a05', 1, '9bf9ed01-43ab-4372-acdd-3500645f3bd0', 1, None) {} failed - stopping task Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::1211::TaskManager.Task::(stop) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::stopping in state preparing (force False) Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::990::TaskManager.Task::(_decref) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::ref 1 aborting True Thread-4535::INFO::2014-05-14 12:27:52,896::task::1168::TaskManager.Task::(prepare) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::aborting: Task is aborted: u"BlockStorageDomain instance has no attribute 'supportsMailbox'" - code 100 Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::1173::TaskManager.Task::(prepare) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::Prepare: aborted: BlockStorageDomain instance has no attribute 'supportsMailbox' Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::990::TaskManager.Task::(_decref) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::ref 0 aborting True Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::925::TaskManager.Task::(_doAbort) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::Task._doAbort: force False Thread-4535::DEBUG::2014-05-14 12:27:52,896::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-4535::DEBUG::2014-05-14 12:27:52,897::task::595::TaskManager.Task::(_updateState) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::moving from state preparing -> state aborting Thread-4535::DEBUG::2014-05-14 12:27:52,897::task::550::TaskManager.Task::(__state_aborting) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::_aborting: recover policy none Thread-4535::DEBUG::2014-05-14 12:27:52,897::task::595::TaskManager.Task::(_updateState) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::moving from state aborting -> state failed Thread-4535::DEBUG::2014-05-14 12:27:52,897::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-4535::DEBUG::2014-05-14 12:27:52,897::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-4535::ERROR::2014-05-14 12:27:52,897::dispatcher::68::Storage.Dispatcher.Protect::(run) BlockStorageDomain instance has no attribute 'supportsMailbox' Traceback (most recent call last): File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run result = ctask.prepare(self.func, *args, **kwargs) File "/usr/share/vdsm/storage/task.py", line 103, in wrapper return m(self, *a, **kw) File "/usr/share/vdsm/storage/task.py", line 1176, in prepare raise self.error AttributeError: BlockStorageDomain instance has no attribute 'supportsMailbox' Thread-16::DEBUG::2014-05-14 12:27:55,639::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'a|/dev/mapper/3690b11c0005592a9000004855344bf5c|\', \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 9bf9ed01-43ab-4372-acdd-3500645f3bd0' (cwd None) Thread-16::DEBUG::2014-05-14 12:27:55,765::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 Thread-16::DEBUG::2014-05-14 12:27:55,768::blockSD::605::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/dev/9bf9ed01-43ab-4372-acdd-3500645f3bd0/metadata bs=4096 count=1' (cwd None) Thread-16::DEBUG::2014-05-14 12:27:55,773::blockSD::605::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0003455 s, 11.9 MB/s\n'; <rc> = 0 Thread-4535::DEBUG::2014-05-14 12:27:57,995::task::595::TaskManager.Task::(_updateState) Task=`89ef3bd0-f222-470b-b678-6228d5254831`::moving from state init -> state preparing Thread-4535::INFO::2014-05-14 12:27:57,995::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) Thread-4535::INFO::2014-05-14 12:27:57,996::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'9bf9ed01-43ab-4372-acdd-3500645f3bd0': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0003455', 'lastCheck': '2.2', 'valid': True}} Thread-4535::DEBUG::2014-05-14 12:27:57,996::task::1185::TaskManager.Task::(prepare) Task=`89ef3bd0-f222-470b-b678-6228d5254831`::finished: {'9bf9ed01-43ab-4372-acdd-3500645f3bd0': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0003455', 'lastCheck': '2.2', 'valid': True}} Thread-4535::DEBUG::2014-05-14 12:27:57,996::task::595::TaskManager.Task::(_updateState) Task=`89ef3bd0-f222-470b-b678-6228d5254831`::moving from state preparing -> state finished Thread-4535::DEBUG::2014-05-14 12:27:57,996::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-4535::DEBUG::2014-05-14 12:27:57,996::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-4535::DEBUG::2014-05-14 12:27:57,996::task::990::TaskManager.Task::(_decref) Task=`89ef3bd0-f222-470b-b678-6228d5254831`::ref 0 aborting False Alain VONDRA Chargé d'exploitation des Systèmes d'Information Direction Administrative et Financière +33 1 44 39 77 76 UNICEF France 3 rue Duguay Trouin 75006 PARIS www.unicef.fr _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi again, Just to complete my first mail, I've tried 4 or 5 times a new install from scratch using engine-cleanup and engine-setup, but nothing works... Tank you Alain VONDRA Chargé d'exploitation des Systèmes d'Information Direction Administrative et Financière +33 1 44 39 77 76 UNICEF France 3 rue Duguay Trouin 75006 PARIS www.unicef.fr -----Message d'origine----- De : Elad Ben Aharon [mailto:ebenahar@redhat.com] Envoyé : mercredi 14 mai 2014 15:09 À : VONDRA Alain; Allon Mureinik Cc : users@ovirt.org Objet : Re: [ovirt-users] Datacenter in no more responsive Allon, could it be related to https://bugzilla.redhat.com/show_bug.cgi?id=1083476 ? ----- Original Message ----- From: "VONDRA Alain" <AVONDRA@unicef.fr> To: users@ovirt.org Sent: Wednesday, May 14, 2014 1:29:13 PM Subject: [ovirt-users] Datacenter in no more responsive Hi, I have a very weird issue with the last version of oVirt 3.4.1, I’ve upgraded because I already had this issue. It came when I wanted to re-create a new export domain on my first hypervisor larger than the first one. After this point, the datacenter came in a non responsive state, so I’ve upgraded oVirt to 3.4.1, but the issue is still there. I join the tail of the engine.log and the vdsm.log of one of the two hypervisors. Tell me if you want more logs. Thank you in advance for your help J Engine.log : TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2014-05-14 12:26:34,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-82) HostName = unc-srv-hyp2 2014-05-14 12:26:34,199 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-82) Command HSMGetAllTasksStatusesVDSCommand(HostName = unc-srv-hyp2, HostId = 2d8722cc-5041-427d-964d-8980f40c5aa6) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2014-05-14 12:26:34,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) hostFromVds::selectedVds - unc-srv-hyp2, spmStatus Unknown_Pool, storage pool UNICEF 2014-05-14 12:26:34,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-82) START, ConnectStoragePoolVDSCommand(HostName = unc-srv-hyp2, HostId = 2d8722cc-5041-427d-964d-8980f40c5aa6, storagePoolId = 4eeccf64-715d-4ebe-a44c-eeca94a09a05, vds_spm_id = 2, masterDomainId = 9bf9ed01-43ab-4372-acdd-3500645f3bd0, masterVersion = 1), log id: aaf1e60 2014-05-14 12:26:35,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-82) FINISH, ConnectStoragePoolVDSCommand, log id: aaf1e60 2014-05-14 12:26:35,998 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) 2014-05-14 12:26:36,082 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) Irs placed on server 2d8722cc-5041-427d-964d-8980f40c5aa6 failed. Proceed Failover 2014-05-14 12:26:36,126 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) hostFromVds::selectedVds - unc-srv-hyp1, spmStatus Unknown_Pool, storage pool UNICEF 2014-05-14 12:26:36,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-82) START, ConnectStoragePoolVDSCommand(HostName = unc-srv-hyp1, HostId = 4987bc7d-82b2-444f-ad6c-8289da5e4fb9, storagePoolId = 4eeccf64-715d-4ebe-a44c-eeca94a09a05, vds_spm_id = 1, masterDomainId = 9bf9ed01-43ab-4372-acdd-3500645f3bd0, masterVersion = 1), log id: 3b569aaf 2014-05-14 12:26:37,177 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-82) FINISH, ConnectStoragePoolVDSCommand, log id: 3b569aaf 2014-05-14 12:26:37,182 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) 2014-05-14 12:26:47,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-13) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2014-05-14 12:26:47,321 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-13) HostName = unc-srv-hyp1 2014-05-14 12:26:47,325 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-13) Command HSMGetAllTasksStatusesVDSCommand(HostName = unc-srv-hyp1, HostId = 4987bc7d-82b2-444f-ad6c-8289da5e4fb9) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2014-05-14 12:26:47,375 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-13) hostFromVds::selectedVds - unc-srv-hyp2, spmStatus Unknown_Pool, storage pool UNICEF 2014-05-14 12:26:47,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-13) START, ConnectStoragePoolVDSCommand(HostName = unc-srv-hyp2, HostId = 2d8722cc-5041-427d-964d-8980f40c5aa6, storagePoolId = 4eeccf64-715d-4ebe-a44c-eeca94a09a05, vds_spm_id = 2, masterDomainId = 9bf9ed01-43ab-4372-acdd-3500645f3bd0, masterVersion = 1), log id: 2fce851e 2014-05-14 12:26:49,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-13) FINISH, ConnectStoragePoolVDSCommand, log id: 2fce851e 2014-05-14 12:26:49,042 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-13) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) Vdsm.log : Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 873, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 45, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 1009, in connectStoragePool spUUID, hostID, msdUUID, masterVersion, domainsMap) File "/usr/share/vdsm/storage/hsm.py", line 1080, in _connectStoragePool res = pool.connect(hostID, msdUUID, masterVersion) File "/usr/share/vdsm/storage/sp.py", line 638, in connect self.__createMailboxMonitor() File "/usr/share/vdsm/storage/sp.py", line 459, in __createMailboxMonitor self.masterDomain.supportsMailbox): File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ return getattr(self.getRealDomain(), attrName) AttributeError: BlockStorageDomain instance has no attribute 'supportsMailbox' Thread-4535::DEBUG::2014-05-14 12:27:52,895::task::885::TaskManager.Task::(_run) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::Task._run: 2096f0da-3c8a-4837-9d0d-2e4b8abed77a ('4eeccf64-715d-4ebe-a44c-eeca94a09a05', 1, '9bf9ed01-43ab-4372-acdd-3500645f3bd0', 1, None) {} failed - stopping task Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::1211::TaskManager.Task::(stop) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::stopping in state preparing (force False) Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::990::TaskManager.Task::(_decref) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::ref 1 aborting True Thread-4535::INFO::2014-05-14 12:27:52,896::task::1168::TaskManager.Task::(prepare) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::aborting: Task is aborted: u"BlockStorageDomain instance has no attribute 'supportsMailbox'" - code 100 Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::1173::TaskManager.Task::(prepare) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::Prepare: aborted: BlockStorageDomain instance has no attribute 'supportsMailbox' Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::990::TaskManager.Task::(_decref) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::ref 0 aborting True Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::925::TaskManager.Task::(_doAbort) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::Task._doAbort: force False Thread-4535::DEBUG::2014-05-14 12:27:52,896::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-4535::DEBUG::2014-05-14 12:27:52,897::task::595::TaskManager.Task::(_updateState) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::moving from state preparing -> state aborting Thread-4535::DEBUG::2014-05-14 12:27:52,897::task::550::TaskManager.Task::(__state_aborting) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::_aborting: recover policy none Thread-4535::DEBUG::2014-05-14 12:27:52,897::task::595::TaskManager.Task::(_updateState) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::moving from state aborting -> state failed Thread-4535::DEBUG::2014-05-14 12:27:52,897::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-4535::DEBUG::2014-05-14 12:27:52,897::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-4535::ERROR::2014-05-14 12:27:52,897::dispatcher::68::Storage.Dispatcher.Protect::(run) BlockStorageDomain instance has no attribute 'supportsMailbox' Traceback (most recent call last): File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run result = ctask.prepare(self.func, *args, **kwargs) File "/usr/share/vdsm/storage/task.py", line 103, in wrapper return m(self, *a, **kw) File "/usr/share/vdsm/storage/task.py", line 1176, in prepare raise self.error AttributeError: BlockStorageDomain instance has no attribute 'supportsMailbox' Thread-16::DEBUG::2014-05-14 12:27:55,639::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'a|/dev/mapper/3690b11c0005592a9000004855344bf5c|\', \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 9bf9ed01-43ab-4372-acdd-3500645f3bd0' (cwd None) Thread-16::DEBUG::2014-05-14 12:27:55,765::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 Thread-16::DEBUG::2014-05-14 12:27:55,768::blockSD::605::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/dev/9bf9ed01-43ab-4372-acdd-3500645f3bd0/metadata bs=4096 count=1' (cwd None) Thread-16::DEBUG::2014-05-14 12:27:55,773::blockSD::605::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0003455 s, 11.9 MB/s\n'; <rc> = 0 Thread-4535::DEBUG::2014-05-14 12:27:57,995::task::595::TaskManager.Task::(_updateState) Task=`89ef3bd0-f222-470b-b678-6228d5254831`::moving from state init -> state preparing Thread-4535::INFO::2014-05-14 12:27:57,995::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) Thread-4535::INFO::2014-05-14 12:27:57,996::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'9bf9ed01-43ab-4372-acdd-3500645f3bd0': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0003455', 'lastCheck': '2.2', 'valid': True}} Thread-4535::DEBUG::2014-05-14 12:27:57,996::task::1185::TaskManager.Task::(prepare) Task=`89ef3bd0-f222-470b-b678-6228d5254831`::finished: {'9bf9ed01-43ab-4372-acdd-3500645f3bd0': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0003455', 'lastCheck': '2.2', 'valid': True}} Thread-4535::DEBUG::2014-05-14 12:27:57,996::task::595::TaskManager.Task::(_updateState) Task=`89ef3bd0-f222-470b-b678-6228d5254831`::moving from state preparing -> state finished Thread-4535::DEBUG::2014-05-14 12:27:57,996::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-4535::DEBUG::2014-05-14 12:27:57,996::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-4535::DEBUG::2014-05-14 12:27:57,996::task::990::TaskManager.Task::(_decref) Task=`89ef3bd0-f222-470b-b678-6228d5254831`::ref 0 aborting False Alain VONDRA Chargé d'exploitation des Systèmes d'Information Direction Administrative et Financière +33 1 44 39 77 76 UNICEF France 3 rue Duguay Trouin 75006 PARIS www.unicef.fr _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi, Is there any news about this bug ? Thank you Alain VONDRA Chargé d'exploitation des Systèmes d'Information Direction Administrative et Financière +33 1 44 39 77 76 UNICEF France 3 rue Duguay Trouin 75006 PARIS www.unicef.fr -----Message d'origine----- De : Elad Ben Aharon [mailto:ebenahar@redhat.com] Envoyé : mercredi 14 mai 2014 15:09 À : VONDRA Alain; Allon Mureinik Cc : users@ovirt.org Objet : Re: [ovirt-users] Datacenter in no more responsive Allon, could it be related to https://bugzilla.redhat.com/show_bug.cgi?id=1083476 ? ----- Original Message ----- From: "VONDRA Alain" <AVONDRA@unicef.fr> To: users@ovirt.org Sent: Wednesday, May 14, 2014 1:29:13 PM Subject: [ovirt-users] Datacenter in no more responsive Hi, I have a very weird issue with the last version of oVirt 3.4.1, I’ve upgraded because I already had this issue. It came when I wanted to re-create a new export domain on my first hypervisor larger than the first one. After this point, the datacenter came in a non responsive state, so I’ve upgraded oVirt to 3.4.1, but the issue is still there. I join the tail of the engine.log and the vdsm.log of one of the two hypervisors. Tell me if you want more logs. Thank you in advance for your help J Engine.log : TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2014-05-14 12:26:34,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-82) HostName = unc-srv-hyp2 2014-05-14 12:26:34,199 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-82) Command HSMGetAllTasksStatusesVDSCommand(HostName = unc-srv-hyp2, HostId = 2d8722cc-5041-427d-964d-8980f40c5aa6) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2014-05-14 12:26:34,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) hostFromVds::selectedVds - unc-srv-hyp2, spmStatus Unknown_Pool, storage pool UNICEF 2014-05-14 12:26:34,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-82) START, ConnectStoragePoolVDSCommand(HostName = unc-srv-hyp2, HostId = 2d8722cc-5041-427d-964d-8980f40c5aa6, storagePoolId = 4eeccf64-715d-4ebe-a44c-eeca94a09a05, vds_spm_id = 2, masterDomainId = 9bf9ed01-43ab-4372-acdd-3500645f3bd0, masterVersion = 1), log id: aaf1e60 2014-05-14 12:26:35,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-82) FINISH, ConnectStoragePoolVDSCommand, log id: aaf1e60 2014-05-14 12:26:35,998 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) 2014-05-14 12:26:36,082 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) Irs placed on server 2d8722cc-5041-427d-964d-8980f40c5aa6 failed. Proceed Failover 2014-05-14 12:26:36,126 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) hostFromVds::selectedVds - unc-srv-hyp1, spmStatus Unknown_Pool, storage pool UNICEF 2014-05-14 12:26:36,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-82) START, ConnectStoragePoolVDSCommand(HostName = unc-srv-hyp1, HostId = 4987bc7d-82b2-444f-ad6c-8289da5e4fb9, storagePoolId = 4eeccf64-715d-4ebe-a44c-eeca94a09a05, vds_spm_id = 1, masterDomainId = 9bf9ed01-43ab-4372-acdd-3500645f3bd0, masterVersion = 1), log id: 3b569aaf 2014-05-14 12:26:37,177 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-82) FINISH, ConnectStoragePoolVDSCommand, log id: 3b569aaf 2014-05-14 12:26:37,182 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-82) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) 2014-05-14 12:26:47,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-13) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2014-05-14 12:26:47,321 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-13) HostName = unc-srv-hyp1 2014-05-14 12:26:47,325 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-13) Command HSMGetAllTasksStatusesVDSCommand(HostName = unc-srv-hyp1, HostId = 4987bc7d-82b2-444f-ad6c-8289da5e4fb9) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2014-05-14 12:26:47,375 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-13) hostFromVds::selectedVds - unc-srv-hyp2, spmStatus Unknown_Pool, storage pool UNICEF 2014-05-14 12:26:47,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-13) START, ConnectStoragePoolVDSCommand(HostName = unc-srv-hyp2, HostId = 2d8722cc-5041-427d-964d-8980f40c5aa6, storagePoolId = 4eeccf64-715d-4ebe-a44c-eeca94a09a05, vds_spm_id = 2, masterDomainId = 9bf9ed01-43ab-4372-acdd-3500645f3bd0, masterVersion = 1), log id: 2fce851e 2014-05-14 12:26:49,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-13) FINISH, ConnectStoragePoolVDSCommand, log id: 2fce851e 2014-05-14 12:26:49,042 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-13) IrsBroker::Failed::GetStoragePoolInfoVDS due to: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) Vdsm.log : Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 873, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 45, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 1009, in connectStoragePool spUUID, hostID, msdUUID, masterVersion, domainsMap) File "/usr/share/vdsm/storage/hsm.py", line 1080, in _connectStoragePool res = pool.connect(hostID, msdUUID, masterVersion) File "/usr/share/vdsm/storage/sp.py", line 638, in connect self.__createMailboxMonitor() File "/usr/share/vdsm/storage/sp.py", line 459, in __createMailboxMonitor self.masterDomain.supportsMailbox): File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ return getattr(self.getRealDomain(), attrName) AttributeError: BlockStorageDomain instance has no attribute 'supportsMailbox' Thread-4535::DEBUG::2014-05-14 12:27:52,895::task::885::TaskManager.Task::(_run) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::Task._run: 2096f0da-3c8a-4837-9d0d-2e4b8abed77a ('4eeccf64-715d-4ebe-a44c-eeca94a09a05', 1, '9bf9ed01-43ab-4372-acdd-3500645f3bd0', 1, None) {} failed - stopping task Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::1211::TaskManager.Task::(stop) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::stopping in state preparing (force False) Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::990::TaskManager.Task::(_decref) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::ref 1 aborting True Thread-4535::INFO::2014-05-14 12:27:52,896::task::1168::TaskManager.Task::(prepare) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::aborting: Task is aborted: u"BlockStorageDomain instance has no attribute 'supportsMailbox'" - code 100 Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::1173::TaskManager.Task::(prepare) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::Prepare: aborted: BlockStorageDomain instance has no attribute 'supportsMailbox' Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::990::TaskManager.Task::(_decref) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::ref 0 aborting True Thread-4535::DEBUG::2014-05-14 12:27:52,896::task::925::TaskManager.Task::(_doAbort) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::Task._doAbort: force False Thread-4535::DEBUG::2014-05-14 12:27:52,896::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-4535::DEBUG::2014-05-14 12:27:52,897::task::595::TaskManager.Task::(_updateState) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::moving from state preparing -> state aborting Thread-4535::DEBUG::2014-05-14 12:27:52,897::task::550::TaskManager.Task::(__state_aborting) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::_aborting: recover policy none Thread-4535::DEBUG::2014-05-14 12:27:52,897::task::595::TaskManager.Task::(_updateState) Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::moving from state aborting -> state failed Thread-4535::DEBUG::2014-05-14 12:27:52,897::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-4535::DEBUG::2014-05-14 12:27:52,897::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-4535::ERROR::2014-05-14 12:27:52,897::dispatcher::68::Storage.Dispatcher.Protect::(run) BlockStorageDomain instance has no attribute 'supportsMailbox' Traceback (most recent call last): File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run result = ctask.prepare(self.func, *args, **kwargs) File "/usr/share/vdsm/storage/task.py", line 103, in wrapper return m(self, *a, **kw) File "/usr/share/vdsm/storage/task.py", line 1176, in prepare raise self.error AttributeError: BlockStorageDomain instance has no attribute 'supportsMailbox' Thread-16::DEBUG::2014-05-14 12:27:55,639::lvm::295::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'a|/dev/mapper/3690b11c0005592a9000004855344bf5c|\', \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 9bf9ed01-43ab-4372-acdd-3500645f3bd0' (cwd None) Thread-16::DEBUG::2014-05-14 12:27:55,765::lvm::295::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 Thread-16::DEBUG::2014-05-14 12:27:55,768::blockSD::605::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/dev/9bf9ed01-43ab-4372-acdd-3500645f3bd0/metadata bs=4096 count=1' (cwd None) Thread-16::DEBUG::2014-05-14 12:27:55,773::blockSD::605::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0003455 s, 11.9 MB/s\n'; <rc> = 0 Thread-4535::DEBUG::2014-05-14 12:27:57,995::task::595::TaskManager.Task::(_updateState) Task=`89ef3bd0-f222-470b-b678-6228d5254831`::moving from state init -> state preparing Thread-4535::INFO::2014-05-14 12:27:57,995::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) Thread-4535::INFO::2014-05-14 12:27:57,996::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'9bf9ed01-43ab-4372-acdd-3500645f3bd0': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0003455', 'lastCheck': '2.2', 'valid': True}} Thread-4535::DEBUG::2014-05-14 12:27:57,996::task::1185::TaskManager.Task::(prepare) Task=`89ef3bd0-f222-470b-b678-6228d5254831`::finished: {'9bf9ed01-43ab-4372-acdd-3500645f3bd0': {'code': 0, 'version': 3, 'acquired': True, 'delay': '0.0003455', 'lastCheck': '2.2', 'valid': True}} Thread-4535::DEBUG::2014-05-14 12:27:57,996::task::595::TaskManager.Task::(_updateState) Task=`89ef3bd0-f222-470b-b678-6228d5254831`::moving from state preparing -> state finished Thread-4535::DEBUG::2014-05-14 12:27:57,996::resourceManager::940::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-4535::DEBUG::2014-05-14 12:27:57,996::resourceManager::977::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-4535::DEBUG::2014-05-14 12:27:57,996::task::990::TaskManager.Task::(_decref) Task=`89ef3bd0-f222-470b-b678-6228d5254831`::ref 0 aborting False Alain VONDRA Chargé d'exploitation des Systèmes d'Information Direction Administrative et Financière +33 1 44 39 77 76 UNICEF France 3 rue Duguay Trouin 75006 PARIS www.unicef.fr _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (2)
-
Elad Ben Aharon -
VONDRA Alain