<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
<div class="moz-cite-prefix">On 29-04-2015 11:56, Liron Aravot
wrote:<br>
</div>
<blockquote
cite="mid:1347581894.5682332.1430319376995.JavaMail.zimbra@redhat.com"
type="cite">
<pre wrap="">
----- Original Message -----
</pre>
<blockquote type="cite">
<pre wrap="">From: "Christopher Pereira" <a class="moz-txt-link-rfc2396E" href="mailto:kripper@imatronix.cl"><kripper@imatronix.cl></a>
To: "Nir Soffer" <a class="moz-txt-link-rfc2396E" href="mailto:nsoffer@redhat.com"><nsoffer@redhat.com></a>
Cc: <a class="moz-txt-link-abbreviated" href="mailto:devel@ovirt.org">devel@ovirt.org</a>, "Liron Aravot" <a class="moz-txt-link-rfc2396E" href="mailto:laravot@redhat.com"><laravot@redhat.com></a>
Sent: Wednesday, April 29, 2015 6:14:32 AM
Subject: Re: [ovirt-devel] "Please activate the master Storage Domain first"
On 28-04-2015 18:14, Nir Soffer wrote:
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">The DC storage master domain is on a (unrecoverable) storage on a remote
dead
host.
Engine is automatically setting another storage as the "Data (Master)".
Seconds later, the unrecoverable storage is marked as "Data (Master)"
again.
There is no way to start the Datacenter.
Both storages are gluster. The old (unrecoverable) one worked fine as a
master.
</pre>
</blockquote>
<pre wrap="">This may be related to this bug:
<a class="moz-txt-link-freetext" href="https://bugzilla.redhat.com/1183977">https://bugzilla.redhat.com/1183977</a>.
</pre>
</blockquote>
<pre wrap="">
Ok. I added a comment and explained more in detail the issue on BZ.
[...]
</pre>
</blockquote>
<pre wrap="">
Hi Chris,
Can you please attach the engine/vdsm logs from the time the issue occurred?
thanks.</pre>
</blockquote>
<br>
Hi, relevant VDSM logs were published here:<br>
<a class="moz-txt-link-freetext" href="https://bugzilla.redhat.com/show_bug.cgi?id=1183977">https://bugzilla.redhat.com/show_bug.cgi?id=1183977</a><br>
<br>
And engine says "Failed in 'ReconstructMasterVDS' method" because of
"Cannot acquire host id: (u'ba7be27f-aee5-4436-ae9a-0764f551f9a7',
SanlockException(5, 'Sanlock lockspace add failure', 'Input/output
error'))":<br>
<blockquote>2015-04-28 16:34:01,915 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-56) [] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand'
return value '<br>
TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654,
mMessage=Not SPM: ()]]<br>
'<br>
2015-04-28 16:34:01,915 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-56) [] HostName = H4<br>
2015-04-28 16:34:01,915 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-56) [] Command
'HSMGetAllTasksStatusesVDSCommand(HostName = H4,
VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='0146bec9-9e8a-4611-a2ca-e4f3c8ccdb26'})' execution failed:
IRSGenericException: IRSErrorException:
IRSNonOperationalException: Not SPM: ()<br>
2015-04-28 16:34:01,951 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(DefaultQuartzScheduler_Worker-56) [] hostFromVds::selectedVds -
'H6', spmStatus 'Unknown_Pool', storage pool 'Default', storage
pool version '3.6'<br>
2015-04-28 16:34:01,953 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-56) [] START,
ConnectStoragePoolVDSCommand(HostName = H6,
ConnectStoragePoolVDSCommandParameters:{runAsync='true',
hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsId =
75e32383-90dd-42c5-9bc9-73b6e5333fe1, storagePoolId =
00000001-0001-0001-0001-00000000007e, masterVersion = 2625), log
id: 15842e4f<br>
2015-04-28 16:34:02,035 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-56) [] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand'
return value 'StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc
[mCode=304, mMessage=Cannot find master domain:
u'spUUID=00000001-0001-0001-0001-00000000007e,
msdUUID=3233144b-7be1-445f-9ea6-6aebbacbb93f']]'<br>
2015-04-28 16:34:02,036 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-56) [] HostName = H6<br>
2015-04-28 16:34:02,036 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-56) [] Command
'ConnectStoragePoolVDSCommand(HostName = H6,
ConnectStoragePoolVDSCommandParameters:{runAsync='true',
hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsId =
75e32383-90dd-42c5-9bc9-73b6e5333fe1, storagePoolId =
00000001-0001-0001-0001-00000000007e, masterVersion = 2625)'
execution failed: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
u'spUUID=00000001-0001-0001-0001-00000000007e,
msdUUID=3233144b-7be1-445f-9ea6-6aebbacbb93f'<br>
2015-04-28 16:34:02,036 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-56) [] FINISH,
ConnectStoragePoolVDSCommand, log id: 15842e4f<br>
2015-04-28 16:34:02,036 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-56) []
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master
domain: u'spUUID=00000001-0001-0001-0001-00000000007e,
msdUUID=3233144b-7be1-445f-9ea6-6aebbacbb93f'<br>
2015-04-28 16:34:02,137 INFO
[org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] Running command:
ReconstructMasterDomainCommand internal: true. Entities affected
: ID: 3233144b-7be1-445f-9ea6-6aebbacbb93f Type: Storage<br>
2015-04-28 16:34:02,235 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] START,
DisconnectStoragePoolVDSCommand(HostName = H6,
DisconnectStoragePoolVDSCommandParameters:{runAsync='true',
hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, storagePoolId =
00000001-0001-0001-0001-00000000007e, vds_spm_id = 3), log id:
782490bb<br>
2015-04-28 16:34:02,239 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] FINISH,
DisconnectStoragePoolVDSCommand, log id: 782490bb<br>
2015-04-28 16:34:02,240 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] START,
ReconstructMasterVDSCommand(HostName = H6,
ReconstructMasterVDSCommandParameters:{runAsync='true',
hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsSpmId = 3,
storagePoolId = 00000001-0001-0001-0001-00000000007e,
storagePoolName = Default, masterDomainId =
ba7be27f-aee5-4436-ae9a-0764f551f9a7, masterVersion = 2626,
domainsList = [{ domainId: ad08adbc-b3a7-4198-a1cb-4ad70c724454,
status: Unknown };{ domainId:
15674257-e14a-4693-9b95-cc4bb92aa765, status: Unknown };{
domainId: ba7be27f-aee5-4436-ae9a-0764f551f9a7, status: Unknown
};{ domainId: 3233144b-7be1-445f-9ea6-6aebbacbb93f, status:
Inactive };]), log id: 626fc611<br>
2015-04-28 16:34:03,318 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] Failed in
'ReconstructMasterVDS' method<br>
2015-04-28 16:34:03,318 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand'
return value 'StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc
[mCode=661, mMessage=Cannot acquire host id:
(u'ba7be27f-aee5-4436-ae9a-0764f551f9a7', SanlockException(5,
'Sanlock lockspace add failure', 'Input/output error'))]]'<br>
2015-04-28 16:34:03,318 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] HostName = H6<br>
2015-04-28 16:34:03,318 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
'ReconstructMasterVDSCommand(HostName = H6,
ReconstructMasterVDSCommandParameters:{runAsync='true',
hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsSpmId = 3,
storagePoolId = 00000001-0001-0001-0001-00000000007e,
storagePoolName = Default, masterDomainId =
ba7be27f-aee5-4436-ae9a-0764f551f9a7, masterVersion = 2626,
domainsList = [{ domainId: ad08adbc-b3a7-4198-a1cb-4ad70c724454,
status: Unknown };{ domainId:
15674257-e14a-4693-9b95-cc4bb92aa765, status: Unknown };{
domainId: ba7be27f-aee5-4436-ae9a-0764f551f9a7, status: Unknown
};{ domainId: 3233144b-7be1-445f-9ea6-6aebbacbb93f, status:
Inactive };])' execution failed: VDSGenericException:
VDSErrorException: Failed to ReconstructMasterVDS, error = Cannot
acquire host id: (u'ba7be27f-aee5-4436-ae9a-0764f551f9a7',
SanlockException(5, 'Sanlock lockspace add failure', 'Input/output
error')), code = 661<br>
2015-04-28 16:34:03,319 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] FINISH,
ReconstructMasterVDSCommand, log id: 626fc611<br>
2015-04-28 16:34:03,319 ERROR
[org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
'org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand'
failed: VdcBLLException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
VDSGenericException: VDSErrorException: Failed to
ReconstructMasterVDS, error = Cannot acquire host id:
(u'ba7be27f-aee5-4436-ae9a-0764f551f9a7', SanlockException(5,
'Sanlock lockspace add failure', 'Input/output error')), code =
661 (Failed with error AcquireHostIdFailure and code 661)<br>
2015-04-28 16:34:03,320 INFO
[org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
[id=449f2035-96bb-4b70-867c-05a7179dc75b]: Compensating
CHANGED_STATUS_ONLY of
org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap;
snapshot: EntityStatusSnapshot:{id='storagePoolId =
00000001-0001-0001-0001-00000000007e, storageId =
3233144b-7be1-445f-9ea6-6aebbacbb93f', status='Unknown'}.<br>
2015-04-28 16:34:03,321 INFO
[org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
[id=449f2035-96bb-4b70-867c-05a7179dc75b]: Compensating
UPDATED_ONLY_ENTITY of
org.ovirt.engine.core.common.businessentities.StorageDomainStatic;
snapshot: StorageDomainStatic[vdisks-ssd,
3233144b-7be1-445f-9ea6-6aebbacbb93f].<br>
2015-04-28 16:34:03,321 INFO
[org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
[id=449f2035-96bb-4b70-867c-05a7179dc75b]: Compensating
UPDATED_ONLY_ENTITY of
org.ovirt.engine.core.common.businessentities.StorageDomainStatic;
snapshot: StorageDomainStatic[vdisks,
ba7be27f-aee5-4436-ae9a-0764f551f9a7].<br>
2015-04-28 16:34:03,370 INFO
[org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor]
(org.ovirt.thread.pool-8-thread-6) [] Finished reconstruct for
pool '00000001-0001-0001-0001-00000000007e'. Clearing event queue<br>
2015-04-28 16:34:04,154 WARN
[org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerObjectsBuilder]
(DefaultQuartzScheduler_Worker-65) [c1df557] Can't set graphics
data from XML.<br>
2015-04-28 16:34:04,155 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
(DefaultQuartzScheduler_Worker-65) [c1df557] START,
DestroyVDSCommand(HostName = H4,
DestroyVmVDSCommandParameters:{runAsync='true',
hostId='0146bec9-9e8a-4611-a2ca-e4f3c8ccdb26',
vmId='6ec9c0a0-2879-4bfe-9a79-92471881ebfe', force='false',
secondsToWait='0', gracefully='false', reason=''}), log id:
238de0d4<br>
2015-04-28 16:34:04,158 WARN
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
(DefaultQuartzScheduler_Worker-65) [c1df557] Unexpected return
value: StatusForXmlRpc [mCode=-32603,
mMessage=u'6ec9c0a0-2879-4bfe-9a79-92471881ebfe']<br>
2015-04-28 16:34:04,158 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
(DefaultQuartzScheduler_Worker-65) [c1df557] Failed in
'DestroyVDS' method<br>
2015-04-28 16:34:04,158 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
(DefaultQuartzScheduler_Worker-65) [c1df557] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand'
return value 'StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc
[mCode=-32603, mMessage=u'6ec9c0a0-2879-4bfe-9a79-92471881ebfe']]'<br>
2015-04-28 16:34:04,158 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
(DefaultQuartzScheduler_Worker-65) [c1df557] HostName = H4<br>
2015-04-28 16:34:04,158 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
(DefaultQuartzScheduler_Worker-65) [c1df557] Command
'DestroyVDSCommand(HostName = H4,
DestroyVmVDSCommandParameters:{runAsync='true',
hostId='0146bec9-9e8a-4611-a2ca-e4f3c8ccdb26',
vmId='6ec9c0a0-2879-4bfe-9a79-92471881ebfe', force='false',
secondsToWait='0', gracefully='false', reason=''})' execution
failed: VDSGenericException: VDSErrorException: Failed to
DestroyVDS, error = u'6ec9c0a0-2879-4bfe-9a79-92471881ebfe', code
= -32603<br>
2015-04-28 16:34:04,158 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
(DefaultQuartzScheduler_Worker-65) [c1df557] FINISH,
DestroyVDSCommand, log id: 238de0d4<br>
</blockquote>
DC is now up, because I attached the old MSD, but I could try
reproducing the failing command if you want.<br>
<br>
How do I specify the dataDict in the vdsClient command?<br>
<br>
I guess it should be something like:<br>
<blockquote>vdsClient -s 0 reconstructMaster
'00000001-0001-0001-0001-00000000007e' 'Default'
'ba7be27f-aee5-4436-ae9a-0764f551f9a7' '[{ domainId:
ad08adbc-b3a7-4198-a1cb-4ad70c724454, status: Unknown };{
domainId: 15674257-e14a-4693-9b95-cc4bb92aa765, status: Unknown
};{ domainId: ba7be27f-aee5-4436-ae9a-0764f551f9a7, status:
Unknown };{ domainId: 3233144b-7be1-445f-9ea6-6aebbacbb93f,
status: Inactive };]'<br>
</blockquote>
Maybe related with sanlock giving me "add_lockspace fail result -5"
all the time.<br>
<br>
Best regards,<br>
Chris<br>
</body>
</html>