
This is a multi-part message in MIME format. --------------000705000900010300040900 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit On 29-04-2015 11:56, Liron Aravot wrote:
From: "Christopher Pereira" <kripper@imatronix.cl> To: "Nir Soffer" <nsoffer@redhat.com> Cc: devel@ovirt.org, "Liron Aravot" <laravot@redhat.com> Sent: Wednesday, April 29, 2015 6:14:32 AM Subject: Re: [ovirt-devel] "Please activate the master Storage Domain first"
On 28-04-2015 18:14, Nir Soffer wrote:
The DC storage master domain is on a (unrecoverable) storage on a remote dead host. Engine is automatically setting another storage as the "Data (Master)". Seconds later, the unrecoverable storage is marked as "Data (Master)" again. There is no way to start the Datacenter.
Both storages are gluster. The old (unrecoverable) one worked fine as a master. This may be related to this bug: https://bugzilla.redhat.com/1183977. Ok. I added a comment and explained more in detail the issue on BZ.
[...] Hi Chris, Can you please attach the engine/vdsm logs from the time the issue occurred?
----- Original Message ----- thanks.
Hi, relevant VDSM logs were published here: https://bugzilla.redhat.com/show_bug.cgi?id=1183977 And engine says "Failed in 'ReconstructMasterVDS' method" because of "Cannot acquire host id: (u'ba7be27f-aee5-4436-ae9a-0764f551f9a7', SanlockException(5, 'Sanlock lockspace add failure', 'Input/output error'))": 2015-04-28 16:34:01,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-56) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] ' 2015-04-28 16:34:01,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-56) [] HostName = H4 2015-04-28 16:34:01,915 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-56) [] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = H4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='0146bec9-9e8a-4611-a2ca-e4f3c8ccdb26'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2015-04-28 16:34:01,951 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-56) [] hostFromVds::selectedVds - 'H6', spmStatus 'Unknown_Pool', storage pool 'Default', storage pool version '3.6' 2015-04-28 16:34:01,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-56) [] START, ConnectStoragePoolVDSCommand(HostName = H6, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsId = 75e32383-90dd-42c5-9bc9-73b6e5333fe1, storagePoolId = 00000001-0001-0001-0001-00000000007e, masterVersion = 2625), log id: 15842e4f 2015-04-28 16:34:02,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-56) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=00000001-0001-0001-0001-00000000007e, msdUUID=3233144b-7be1-445f-9ea6-6aebbacbb93f']]' 2015-04-28 16:34:02,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-56) [] HostName = H6 2015-04-28 16:34:02,036 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-56) [] Command 'ConnectStoragePoolVDSCommand(HostName = H6, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsId = 75e32383-90dd-42c5-9bc9-73b6e5333fe1, storagePoolId = 00000001-0001-0001-0001-00000000007e, masterVersion = 2625)' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=00000001-0001-0001-0001-00000000007e, msdUUID=3233144b-7be1-445f-9ea6-6aebbacbb93f' 2015-04-28 16:34:02,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-56) [] FINISH, ConnectStoragePoolVDSCommand, log id: 15842e4f 2015-04-28 16:34:02,036 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-56) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=00000001-0001-0001-0001-00000000007e, msdUUID=3233144b-7be1-445f-9ea6-6aebbacbb93f' 2015-04-28 16:34:02,137 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-25) [1339ee5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 3233144b-7be1-445f-9ea6-6aebbacbb93f Type: Storage 2015-04-28 16:34:02,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1339ee5] START, DisconnectStoragePoolVDSCommand(HostName = H6, DisconnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, storagePoolId = 00000001-0001-0001-0001-00000000007e, vds_spm_id = 3), log id: 782490bb 2015-04-28 16:34:02,239 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1339ee5] FINISH, DisconnectStoragePoolVDSCommand, log id: 782490bb 2015-04-28 16:34:02,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1339ee5] START, ReconstructMasterVDSCommand(HostName = H6, ReconstructMasterVDSCommandParameters:{runAsync='true', hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsSpmId = 3, storagePoolId = 00000001-0001-0001-0001-00000000007e, storagePoolName = Default, masterDomainId = ba7be27f-aee5-4436-ae9a-0764f551f9a7, masterVersion = 2626, domainsList = [{ domainId: ad08adbc-b3a7-4198-a1cb-4ad70c724454, status: Unknown };{ domainId: 15674257-e14a-4693-9b95-cc4bb92aa765, status: Unknown };{ domainId: ba7be27f-aee5-4436-ae9a-0764f551f9a7, status: Unknown };{ domainId: 3233144b-7be1-445f-9ea6-6aebbacbb93f, status: Inactive };]), log id: 626fc611 2015-04-28 16:34:03,318 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1339ee5] Failed in 'ReconstructMasterVDS' method 2015-04-28 16:34:03,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand' return value 'StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=661, mMessage=Cannot acquire host id: (u'ba7be27f-aee5-4436-ae9a-0764f551f9a7', SanlockException(5, 'Sanlock lockspace add failure', 'Input/output error'))]]' 2015-04-28 16:34:03,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1339ee5] HostName = H6 2015-04-28 16:34:03,318 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command 'ReconstructMasterVDSCommand(HostName = H6, ReconstructMasterVDSCommandParameters:{runAsync='true', hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsSpmId = 3, storagePoolId = 00000001-0001-0001-0001-00000000007e, storagePoolName = Default, masterDomainId = ba7be27f-aee5-4436-ae9a-0764f551f9a7, masterVersion = 2626, domainsList = [{ domainId: ad08adbc-b3a7-4198-a1cb-4ad70c724454, status: Unknown };{ domainId: 15674257-e14a-4693-9b95-cc4bb92aa765, status: Unknown };{ domainId: ba7be27f-aee5-4436-ae9a-0764f551f9a7, status: Unknown };{ domainId: 3233144b-7be1-445f-9ea6-6aebbacbb93f, status: Inactive };])' execution failed: VDSGenericException: VDSErrorException: Failed to ReconstructMasterVDS, error = Cannot acquire host id: (u'ba7be27f-aee5-4436-ae9a-0764f551f9a7', SanlockException(5, 'Sanlock lockspace add failure', 'Input/output error')), code = 661 2015-04-28 16:34:03,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1339ee5] FINISH, ReconstructMasterVDSCommand, log id: 626fc611 2015-04-28 16:34:03,319 ERROR [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command 'org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand' failed: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to ReconstructMasterVDS, error = Cannot acquire host id: (u'ba7be27f-aee5-4436-ae9a-0764f551f9a7', SanlockException(5, 'Sanlock lockspace add failure', 'Input/output error')), code = 661 (Failed with error AcquireHostIdFailure and code 661) 2015-04-28 16:34:03,320 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command [id=449f2035-96bb-4b70-867c-05a7179dc75b]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: EntityStatusSnapshot:{id='storagePoolId = 00000001-0001-0001-0001-00000000007e, storageId = 3233144b-7be1-445f-9ea6-6aebbacbb93f', status='Unknown'}. 2015-04-28 16:34:03,321 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command [id=449f2035-96bb-4b70-867c-05a7179dc75b]: Compensating UPDATED_ONLY_ENTITY of org.ovirt.engine.core.common.businessentities.StorageDomainStatic; snapshot: StorageDomainStatic[vdisks-ssd, 3233144b-7be1-445f-9ea6-6aebbacbb93f]. 2015-04-28 16:34:03,321 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command [id=449f2035-96bb-4b70-867c-05a7179dc75b]: Compensating UPDATED_ONLY_ENTITY of org.ovirt.engine.core.common.businessentities.StorageDomainStatic; snapshot: StorageDomainStatic[vdisks, ba7be27f-aee5-4436-ae9a-0764f551f9a7]. 2015-04-28 16:34:03,370 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-6) [] Finished reconstruct for pool '00000001-0001-0001-0001-00000000007e'. Clearing event queue 2015-04-28 16:34:04,154 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerObjectsBuilder] (DefaultQuartzScheduler_Worker-65) [c1df557] Can't set graphics data from XML. 2015-04-28 16:34:04,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-65) [c1df557] START, DestroyVDSCommand(HostName = H4, DestroyVmVDSCommandParameters:{runAsync='true', hostId='0146bec9-9e8a-4611-a2ca-e4f3c8ccdb26', vmId='6ec9c0a0-2879-4bfe-9a79-92471881ebfe', force='false', secondsToWait='0', gracefully='false', reason=''}), log id: 238de0d4 2015-04-28 16:34:04,158 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-65) [c1df557] Unexpected return value: StatusForXmlRpc [mCode=-32603, mMessage=u'6ec9c0a0-2879-4bfe-9a79-92471881ebfe'] 2015-04-28 16:34:04,158 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-65) [c1df557] Failed in 'DestroyVDS' method 2015-04-28 16:34:04,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-65) [c1df557] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand' return value 'StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=-32603, mMessage=u'6ec9c0a0-2879-4bfe-9a79-92471881ebfe']]' 2015-04-28 16:34:04,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-65) [c1df557] HostName = H4 2015-04-28 16:34:04,158 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-65) [c1df557] Command 'DestroyVDSCommand(HostName = H4, DestroyVmVDSCommandParameters:{runAsync='true', hostId='0146bec9-9e8a-4611-a2ca-e4f3c8ccdb26', vmId='6ec9c0a0-2879-4bfe-9a79-92471881ebfe', force='false', secondsToWait='0', gracefully='false', reason=''})' execution failed: VDSGenericException: VDSErrorException: Failed to DestroyVDS, error = u'6ec9c0a0-2879-4bfe-9a79-92471881ebfe', code = -32603 2015-04-28 16:34:04,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-65) [c1df557] FINISH, DestroyVDSCommand, log id: 238de0d4 DC is now up, because I attached the old MSD, but I could try reproducing the failing command if you want. How do I specify the dataDict in the vdsClient command? I guess it should be something like: vdsClient -s 0 reconstructMaster '00000001-0001-0001-0001-00000000007e' 'Default' 'ba7be27f-aee5-4436-ae9a-0764f551f9a7' '[{ domainId: ad08adbc-b3a7-4198-a1cb-4ad70c724454, status: Unknown };{ domainId: 15674257-e14a-4693-9b95-cc4bb92aa765, status: Unknown };{ domainId: ba7be27f-aee5-4436-ae9a-0764f551f9a7, status: Unknown };{ domainId: 3233144b-7be1-445f-9ea6-6aebbacbb93f, status: Inactive };]' Maybe related with sanlock giving me "add_lockspace fail result -5" all the time. Best regards, Chris --------------000705000900010300040900 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> <br> <div class="moz-cite-prefix">On 29-04-2015 11:56, Liron Aravot wrote:<br> </div> <blockquote cite="mid:1347581894.5682332.1430319376995.JavaMail.zimbra@redhat.com" type="cite"> <pre wrap=""> ----- Original Message ----- </pre> <blockquote type="cite"> <pre wrap="">From: "Christopher Pereira" <a class="moz-txt-link-rfc2396E" href="mailto:kripper@imatronix.cl"><kripper@imatronix.cl></a> To: "Nir Soffer" <a class="moz-txt-link-rfc2396E" href="mailto:nsoffer@redhat.com"><nsoffer@redhat.com></a> Cc: <a class="moz-txt-link-abbreviated" href="mailto:devel@ovirt.org">devel@ovirt.org</a>, "Liron Aravot" <a class="moz-txt-link-rfc2396E" href="mailto:laravot@redhat.com"><laravot@redhat.com></a> Sent: Wednesday, April 29, 2015 6:14:32 AM Subject: Re: [ovirt-devel] "Please activate the master Storage Domain first" On 28-04-2015 18:14, Nir Soffer wrote: </pre> <blockquote type="cite"> <blockquote type="cite"> <pre wrap="">The DC storage master domain is on a (unrecoverable) storage on a remote dead host. Engine is automatically setting another storage as the "Data (Master)". Seconds later, the unrecoverable storage is marked as "Data (Master)" again. There is no way to start the Datacenter. Both storages are gluster. The old (unrecoverable) one worked fine as a master. </pre> </blockquote> <pre wrap="">This may be related to this bug: <a class="moz-txt-link-freetext" href="https://bugzilla.redhat.com/1183977">https://bugzilla.redhat.com/1183977</a>. </pre> </blockquote> <pre wrap=""> Ok. I added a comment and explained more in detail the issue on BZ. [...] </pre> </blockquote> <pre wrap=""> Hi Chris, Can you please attach the engine/vdsm logs from the time the issue occurred? thanks.</pre> </blockquote> <br> Hi, relevant VDSM logs were published here:<br> <a class="moz-txt-link-freetext" href="https://bugzilla.redhat.com/show_bug.cgi?id=1183977">https://bugzilla.redhat.com/show_bug.cgi?id=1183977</a><br> <br> And engine says "Failed in 'ReconstructMasterVDS' method" because of "Cannot acquire host id: (u'ba7be27f-aee5-4436-ae9a-0764f551f9a7', SanlockException(5, 'Sanlock lockspace add failure', 'Input/output error'))":<br> <blockquote>2015-04-28 16:34:01,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-56) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value '<br> TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]]<br> '<br> 2015-04-28 16:34:01,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-56) [] HostName = H4<br> 2015-04-28 16:34:01,915 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-56) [] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = H4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='0146bec9-9e8a-4611-a2ca-e4f3c8ccdb26'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()<br> 2015-04-28 16:34:01,951 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-56) [] hostFromVds::selectedVds - 'H6', spmStatus 'Unknown_Pool', storage pool 'Default', storage pool version '3.6'<br> 2015-04-28 16:34:01,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-56) [] START, ConnectStoragePoolVDSCommand(HostName = H6, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsId = 75e32383-90dd-42c5-9bc9-73b6e5333fe1, storagePoolId = 00000001-0001-0001-0001-00000000007e, masterVersion = 2625), log id: 15842e4f<br> 2015-04-28 16:34:02,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-56) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=00000001-0001-0001-0001-00000000007e, msdUUID=3233144b-7be1-445f-9ea6-6aebbacbb93f']]'<br> 2015-04-28 16:34:02,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-56) [] HostName = H6<br> 2015-04-28 16:34:02,036 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-56) [] Command 'ConnectStoragePoolVDSCommand(HostName = H6, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsId = 75e32383-90dd-42c5-9bc9-73b6e5333fe1, storagePoolId = 00000001-0001-0001-0001-00000000007e, masterVersion = 2625)' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=00000001-0001-0001-0001-00000000007e, msdUUID=3233144b-7be1-445f-9ea6-6aebbacbb93f'<br> 2015-04-28 16:34:02,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-56) [] FINISH, ConnectStoragePoolVDSCommand, log id: 15842e4f<br> 2015-04-28 16:34:02,036 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-56) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=00000001-0001-0001-0001-00000000007e, msdUUID=3233144b-7be1-445f-9ea6-6aebbacbb93f'<br> 2015-04-28 16:34:02,137 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-25) [1339ee5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 3233144b-7be1-445f-9ea6-6aebbacbb93f Type: Storage<br> 2015-04-28 16:34:02,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1339ee5] START, DisconnectStoragePoolVDSCommand(HostName = H6, DisconnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, storagePoolId = 00000001-0001-0001-0001-00000000007e, vds_spm_id = 3), log id: 782490bb<br> 2015-04-28 16:34:02,239 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1339ee5] FINISH, DisconnectStoragePoolVDSCommand, log id: 782490bb<br> 2015-04-28 16:34:02,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1339ee5] START, ReconstructMasterVDSCommand(HostName = H6, ReconstructMasterVDSCommandParameters:{runAsync='true', hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsSpmId = 3, storagePoolId = 00000001-0001-0001-0001-00000000007e, storagePoolName = Default, masterDomainId = ba7be27f-aee5-4436-ae9a-0764f551f9a7, masterVersion = 2626, domainsList = [{ domainId: ad08adbc-b3a7-4198-a1cb-4ad70c724454, status: Unknown };{ domainId: 15674257-e14a-4693-9b95-cc4bb92aa765, status: Unknown };{ domainId: ba7be27f-aee5-4436-ae9a-0764f551f9a7, status: Unknown };{ domainId: 3233144b-7be1-445f-9ea6-6aebbacbb93f, status: Inactive };]), log id: 626fc611<br> 2015-04-28 16:34:03,318 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1339ee5] Failed in 'ReconstructMasterVDS' method<br> 2015-04-28 16:34:03,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand' return value 'StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=661, mMessage=Cannot acquire host id: (u'ba7be27f-aee5-4436-ae9a-0764f551f9a7', SanlockException(5, 'Sanlock lockspace add failure', 'Input/output error'))]]'<br> 2015-04-28 16:34:03,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1339ee5] HostName = H6<br> 2015-04-28 16:34:03,318 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command 'ReconstructMasterVDSCommand(HostName = H6, ReconstructMasterVDSCommandParameters:{runAsync='true', hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsSpmId = 3, storagePoolId = 00000001-0001-0001-0001-00000000007e, storagePoolName = Default, masterDomainId = ba7be27f-aee5-4436-ae9a-0764f551f9a7, masterVersion = 2626, domainsList = [{ domainId: ad08adbc-b3a7-4198-a1cb-4ad70c724454, status: Unknown };{ domainId: 15674257-e14a-4693-9b95-cc4bb92aa765, status: Unknown };{ domainId: ba7be27f-aee5-4436-ae9a-0764f551f9a7, status: Unknown };{ domainId: 3233144b-7be1-445f-9ea6-6aebbacbb93f, status: Inactive };])' execution failed: VDSGenericException: VDSErrorException: Failed to ReconstructMasterVDS, error = Cannot acquire host id: (u'ba7be27f-aee5-4436-ae9a-0764f551f9a7', SanlockException(5, 'Sanlock lockspace add failure', 'Input/output error')), code = 661<br> 2015-04-28 16:34:03,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand] (org.ovirt.thread.pool-8-thread-25) [1339ee5] FINISH, ReconstructMasterVDSCommand, log id: 626fc611<br> 2015-04-28 16:34:03,319 ERROR [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command 'org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand' failed: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to ReconstructMasterVDS, error = Cannot acquire host id: (u'ba7be27f-aee5-4436-ae9a-0764f551f9a7', SanlockException(5, 'Sanlock lockspace add failure', 'Input/output error')), code = 661 (Failed with error AcquireHostIdFailure and code 661)<br> 2015-04-28 16:34:03,320 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command [id=449f2035-96bb-4b70-867c-05a7179dc75b]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: EntityStatusSnapshot:{id='storagePoolId = 00000001-0001-0001-0001-00000000007e, storageId = 3233144b-7be1-445f-9ea6-6aebbacbb93f', status='Unknown'}.<br> 2015-04-28 16:34:03,321 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command [id=449f2035-96bb-4b70-867c-05a7179dc75b]: Compensating UPDATED_ONLY_ENTITY of org.ovirt.engine.core.common.businessentities.StorageDomainStatic; snapshot: StorageDomainStatic[vdisks-ssd, 3233144b-7be1-445f-9ea6-6aebbacbb93f].<br> 2015-04-28 16:34:03,321 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command [id=449f2035-96bb-4b70-867c-05a7179dc75b]: Compensating UPDATED_ONLY_ENTITY of org.ovirt.engine.core.common.businessentities.StorageDomainStatic; snapshot: StorageDomainStatic[vdisks, ba7be27f-aee5-4436-ae9a-0764f551f9a7].<br> 2015-04-28 16:34:03,370 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-8-thread-6) [] Finished reconstruct for pool '00000001-0001-0001-0001-00000000007e'. Clearing event queue<br> 2015-04-28 16:34:04,154 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerObjectsBuilder] (DefaultQuartzScheduler_Worker-65) [c1df557] Can't set graphics data from XML.<br> 2015-04-28 16:34:04,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-65) [c1df557] START, DestroyVDSCommand(HostName = H4, DestroyVmVDSCommandParameters:{runAsync='true', hostId='0146bec9-9e8a-4611-a2ca-e4f3c8ccdb26', vmId='6ec9c0a0-2879-4bfe-9a79-92471881ebfe', force='false', secondsToWait='0', gracefully='false', reason=''}), log id: 238de0d4<br> 2015-04-28 16:34:04,158 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-65) [c1df557] Unexpected return value: StatusForXmlRpc [mCode=-32603, mMessage=u'6ec9c0a0-2879-4bfe-9a79-92471881ebfe']<br> 2015-04-28 16:34:04,158 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-65) [c1df557] Failed in 'DestroyVDS' method<br> 2015-04-28 16:34:04,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-65) [c1df557] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand' return value 'StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=-32603, mMessage=u'6ec9c0a0-2879-4bfe-9a79-92471881ebfe']]'<br> 2015-04-28 16:34:04,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-65) [c1df557] HostName = H4<br> 2015-04-28 16:34:04,158 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-65) [c1df557] Command 'DestroyVDSCommand(HostName = H4, DestroyVmVDSCommandParameters:{runAsync='true', hostId='0146bec9-9e8a-4611-a2ca-e4f3c8ccdb26', vmId='6ec9c0a0-2879-4bfe-9a79-92471881ebfe', force='false', secondsToWait='0', gracefully='false', reason=''})' execution failed: VDSGenericException: VDSErrorException: Failed to DestroyVDS, error = u'6ec9c0a0-2879-4bfe-9a79-92471881ebfe', code = -32603<br> 2015-04-28 16:34:04,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-65) [c1df557] FINISH, DestroyVDSCommand, log id: 238de0d4<br> </blockquote> DC is now up, because I attached the old MSD, but I could try reproducing the failing command if you want.<br> <br> How do I specify the dataDict in the vdsClient command?<br> <br> I guess it should be something like:<br> <blockquote>vdsClient -s 0 reconstructMaster '00000001-0001-0001-0001-00000000007e' 'Default' 'ba7be27f-aee5-4436-ae9a-0764f551f9a7' '[{ domainId: ad08adbc-b3a7-4198-a1cb-4ad70c724454, status: Unknown };{ domainId: 15674257-e14a-4693-9b95-cc4bb92aa765, status: Unknown };{ domainId: ba7be27f-aee5-4436-ae9a-0764f551f9a7, status: Unknown };{ domainId: 3233144b-7be1-445f-9ea6-6aebbacbb93f, status: Inactive };]'<br> </blockquote> Maybe related with sanlock giving me "add_lockspace fail result -5" all the time.<br> <br> Best regards,<br> Chris<br> </body> </html> --------------000705000900010300040900--