[ovirt-devel] "Please activate the master Storage Domain first"
Christopher Pereira
kripper at imatronix.cl
Wed Apr 29 18:58:58 UTC 2015
On 29-04-2015 11:56, Liron Aravot wrote:
>
> ----- Original Message -----
>> From: "Christopher Pereira" <kripper at imatronix.cl>
>> To: "Nir Soffer" <nsoffer at redhat.com>
>> Cc: devel at ovirt.org, "Liron Aravot" <laravot at redhat.com>
>> Sent: Wednesday, April 29, 2015 6:14:32 AM
>> Subject: Re: [ovirt-devel] "Please activate the master Storage Domain first"
>>
>>
>> On 28-04-2015 18:14, Nir Soffer wrote:
>>>> The DC storage master domain is on a (unrecoverable) storage on a remote
>>>> dead
>>>> host.
>>>> Engine is automatically setting another storage as the "Data (Master)".
>>>> Seconds later, the unrecoverable storage is marked as "Data (Master)"
>>>> again.
>>>> There is no way to start the Datacenter.
>>>>
>>>> Both storages are gluster. The old (unrecoverable) one worked fine as a
>>>> master.
>>> This may be related to this bug:
>>> https://bugzilla.redhat.com/1183977.
>> Ok. I added a comment and explained more in detail the issue on BZ.
>>
>> [...]
> Hi Chris,
> Can you please attach the engine/vdsm logs from the time the issue occurred?
> thanks.
Hi, relevant VDSM logs were published here:
https://bugzilla.redhat.com/show_bug.cgi?id=1183977
And engine says "Failed in 'ReconstructMasterVDS' method" because of
"Cannot acquire host id: (u'ba7be27f-aee5-4436-ae9a-0764f551f9a7',
SanlockException(5, 'Sanlock lockspace add failure', 'Input/output
error'))":
2015-04-28 16:34:01,915 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-56) [] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand'
return value '
TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654,
mMessage=Not SPM: ()]]
'
2015-04-28 16:34:01,915 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-56) [] HostName = H4
2015-04-28 16:34:01,915 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-56) [] Command
'HSMGetAllTasksStatusesVDSCommand(HostName = H4,
VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='0146bec9-9e8a-4611-a2ca-e4f3c8ccdb26'})' execution failed:
IRSGenericException: IRSErrorException: IRSNonOperationalException:
Not SPM: ()
2015-04-28 16:34:01,951 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(DefaultQuartzScheduler_Worker-56) [] hostFromVds::selectedVds -
'H6', spmStatus 'Unknown_Pool', storage pool 'Default', storage pool
version '3.6'
2015-04-28 16:34:01,953 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-56) [] START,
ConnectStoragePoolVDSCommand(HostName = H6,
ConnectStoragePoolVDSCommandParameters:{runAsync='true',
hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsId =
75e32383-90dd-42c5-9bc9-73b6e5333fe1, storagePoolId =
00000001-0001-0001-0001-00000000007e, masterVersion = 2625), log id:
15842e4f
2015-04-28 16:34:02,035 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-56) [] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand'
return value 'StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc
[mCode=304, mMessage=Cannot find master domain:
u'spUUID=00000001-0001-0001-0001-00000000007e,
msdUUID=3233144b-7be1-445f-9ea6-6aebbacbb93f']]'
2015-04-28 16:34:02,036 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-56) [] HostName = H6
2015-04-28 16:34:02,036 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-56) [] Command
'ConnectStoragePoolVDSCommand(HostName = H6,
ConnectStoragePoolVDSCommandParameters:{runAsync='true',
hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsId =
75e32383-90dd-42c5-9bc9-73b6e5333fe1, storagePoolId =
00000001-0001-0001-0001-00000000007e, masterVersion = 2625)'
execution failed: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
u'spUUID=00000001-0001-0001-0001-00000000007e,
msdUUID=3233144b-7be1-445f-9ea6-6aebbacbb93f'
2015-04-28 16:34:02,036 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-56) [] FINISH,
ConnectStoragePoolVDSCommand, log id: 15842e4f
2015-04-28 16:34:02,036 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-56) []
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master
domain: u'spUUID=00000001-0001-0001-0001-00000000007e,
msdUUID=3233144b-7be1-445f-9ea6-6aebbacbb93f'
2015-04-28 16:34:02,137 INFO
[org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] Running command:
ReconstructMasterDomainCommand internal: true. Entities affected :
ID: 3233144b-7be1-445f-9ea6-6aebbacbb93f Type: Storage
2015-04-28 16:34:02,235 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] START,
DisconnectStoragePoolVDSCommand(HostName = H6,
DisconnectStoragePoolVDSCommandParameters:{runAsync='true',
hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, storagePoolId =
00000001-0001-0001-0001-00000000007e, vds_spm_id = 3), log id: 782490bb
2015-04-28 16:34:02,239 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] FINISH,
DisconnectStoragePoolVDSCommand, log id: 782490bb
2015-04-28 16:34:02,240 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] START,
ReconstructMasterVDSCommand(HostName = H6,
ReconstructMasterVDSCommandParameters:{runAsync='true',
hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsSpmId = 3,
storagePoolId = 00000001-0001-0001-0001-00000000007e,
storagePoolName = Default, masterDomainId =
ba7be27f-aee5-4436-ae9a-0764f551f9a7, masterVersion = 2626,
domainsList = [{ domainId: ad08adbc-b3a7-4198-a1cb-4ad70c724454,
status: Unknown };{ domainId: 15674257-e14a-4693-9b95-cc4bb92aa765,
status: Unknown };{ domainId: ba7be27f-aee5-4436-ae9a-0764f551f9a7,
status: Unknown };{ domainId: 3233144b-7be1-445f-9ea6-6aebbacbb93f,
status: Inactive };]), log id: 626fc611
2015-04-28 16:34:03,318 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] Failed in
'ReconstructMasterVDS' method
2015-04-28 16:34:03,318 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand'
return value 'StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc
[mCode=661, mMessage=Cannot acquire host id:
(u'ba7be27f-aee5-4436-ae9a-0764f551f9a7', SanlockException(5,
'Sanlock lockspace add failure', 'Input/output error'))]]'
2015-04-28 16:34:03,318 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] HostName = H6
2015-04-28 16:34:03,318 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
'ReconstructMasterVDSCommand(HostName = H6,
ReconstructMasterVDSCommandParameters:{runAsync='true',
hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsSpmId = 3,
storagePoolId = 00000001-0001-0001-0001-00000000007e,
storagePoolName = Default, masterDomainId =
ba7be27f-aee5-4436-ae9a-0764f551f9a7, masterVersion = 2626,
domainsList = [{ domainId: ad08adbc-b3a7-4198-a1cb-4ad70c724454,
status: Unknown };{ domainId: 15674257-e14a-4693-9b95-cc4bb92aa765,
status: Unknown };{ domainId: ba7be27f-aee5-4436-ae9a-0764f551f9a7,
status: Unknown };{ domainId: 3233144b-7be1-445f-9ea6-6aebbacbb93f,
status: Inactive };])' execution failed: VDSGenericException:
VDSErrorException: Failed to ReconstructMasterVDS, error = Cannot
acquire host id: (u'ba7be27f-aee5-4436-ae9a-0764f551f9a7',
SanlockException(5, 'Sanlock lockspace add failure', 'Input/output
error')), code = 661
2015-04-28 16:34:03,319 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] FINISH,
ReconstructMasterVDSCommand, log id: 626fc611
2015-04-28 16:34:03,319 ERROR
[org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
'org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand'
failed: VdcBLLException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
VDSGenericException: VDSErrorException: Failed to
ReconstructMasterVDS, error = Cannot acquire host id:
(u'ba7be27f-aee5-4436-ae9a-0764f551f9a7', SanlockException(5,
'Sanlock lockspace add failure', 'Input/output error')), code = 661
(Failed with error AcquireHostIdFailure and code 661)
2015-04-28 16:34:03,320 INFO
[org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
[id=449f2035-96bb-4b70-867c-05a7179dc75b]: Compensating
CHANGED_STATUS_ONLY of
org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap;
snapshot: EntityStatusSnapshot:{id='storagePoolId =
00000001-0001-0001-0001-00000000007e, storageId =
3233144b-7be1-445f-9ea6-6aebbacbb93f', status='Unknown'}.
2015-04-28 16:34:03,321 INFO
[org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
[id=449f2035-96bb-4b70-867c-05a7179dc75b]: Compensating
UPDATED_ONLY_ENTITY of
org.ovirt.engine.core.common.businessentities.StorageDomainStatic;
snapshot: StorageDomainStatic[vdisks-ssd,
3233144b-7be1-445f-9ea6-6aebbacbb93f].
2015-04-28 16:34:03,321 INFO
[org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
[id=449f2035-96bb-4b70-867c-05a7179dc75b]: Compensating
UPDATED_ONLY_ENTITY of
org.ovirt.engine.core.common.businessentities.StorageDomainStatic;
snapshot: StorageDomainStatic[vdisks,
ba7be27f-aee5-4436-ae9a-0764f551f9a7].
2015-04-28 16:34:03,370 INFO
[org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor]
(org.ovirt.thread.pool-8-thread-6) [] Finished reconstruct for pool
'00000001-0001-0001-0001-00000000007e'. Clearing event queue
2015-04-28 16:34:04,154 WARN
[org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerObjectsBuilder]
(DefaultQuartzScheduler_Worker-65) [c1df557] Can't set graphics data
from XML.
2015-04-28 16:34:04,155 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
(DefaultQuartzScheduler_Worker-65) [c1df557] START,
DestroyVDSCommand(HostName = H4,
DestroyVmVDSCommandParameters:{runAsync='true',
hostId='0146bec9-9e8a-4611-a2ca-e4f3c8ccdb26',
vmId='6ec9c0a0-2879-4bfe-9a79-92471881ebfe', force='false',
secondsToWait='0', gracefully='false', reason=''}), log id: 238de0d4
2015-04-28 16:34:04,158 WARN
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
(DefaultQuartzScheduler_Worker-65) [c1df557] Unexpected return
value: StatusForXmlRpc [mCode=-32603,
mMessage=u'6ec9c0a0-2879-4bfe-9a79-92471881ebfe']
2015-04-28 16:34:04,158 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
(DefaultQuartzScheduler_Worker-65) [c1df557] Failed in 'DestroyVDS'
method
2015-04-28 16:34:04,158 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
(DefaultQuartzScheduler_Worker-65) [c1df557] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand' return
value 'StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc
[mCode=-32603, mMessage=u'6ec9c0a0-2879-4bfe-9a79-92471881ebfe']]'
2015-04-28 16:34:04,158 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
(DefaultQuartzScheduler_Worker-65) [c1df557] HostName = H4
2015-04-28 16:34:04,158 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
(DefaultQuartzScheduler_Worker-65) [c1df557] Command
'DestroyVDSCommand(HostName = H4,
DestroyVmVDSCommandParameters:{runAsync='true',
hostId='0146bec9-9e8a-4611-a2ca-e4f3c8ccdb26',
vmId='6ec9c0a0-2879-4bfe-9a79-92471881ebfe', force='false',
secondsToWait='0', gracefully='false', reason=''})' execution
failed: VDSGenericException: VDSErrorException: Failed to
DestroyVDS, error = u'6ec9c0a0-2879-4bfe-9a79-92471881ebfe', code =
-32603
2015-04-28 16:34:04,158 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
(DefaultQuartzScheduler_Worker-65) [c1df557] FINISH,
DestroyVDSCommand, log id: 238de0d4
DC is now up, because I attached the old MSD, but I could try
reproducing the failing command if you want.
How do I specify the dataDict in the vdsClient command?
I guess it should be something like:
vdsClient -s 0 reconstructMaster
'00000001-0001-0001-0001-00000000007e' 'Default'
'ba7be27f-aee5-4436-ae9a-0764f551f9a7' '[{ domainId:
ad08adbc-b3a7-4198-a1cb-4ad70c724454, status: Unknown };{ domainId:
15674257-e14a-4693-9b95-cc4bb92aa765, status: Unknown };{ domainId:
ba7be27f-aee5-4436-ae9a-0764f551f9a7, status: Unknown };{ domainId:
3233144b-7be1-445f-9ea6-6aebbacbb93f, status: Inactive };]'
Maybe related with sanlock giving me "add_lockspace fail result -5" all
the time.
Best regards,
Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20150429/fd04c844/attachment.html>
More information about the Devel
mailing list