[ovirt-devel] "Please activate the master Storage Domain first"

Christopher Pereira kripper at imatronix.cl
Wed Apr 29 18:58:58 UTC 2015


On 29-04-2015 11:56, Liron Aravot wrote:
>
> ----- Original Message -----
>> From: "Christopher Pereira" <kripper at imatronix.cl>
>> To: "Nir Soffer" <nsoffer at redhat.com>
>> Cc: devel at ovirt.org, "Liron Aravot" <laravot at redhat.com>
>> Sent: Wednesday, April 29, 2015 6:14:32 AM
>> Subject: Re: [ovirt-devel] "Please activate the master Storage Domain first"
>>
>>
>> On 28-04-2015 18:14, Nir Soffer wrote:
>>>> The DC storage master domain is on a (unrecoverable) storage on a remote
>>>> dead
>>>> host.
>>>> Engine is automatically setting another storage as the "Data (Master)".
>>>> Seconds later, the unrecoverable storage is marked as "Data (Master)"
>>>> again.
>>>> There is no way to start the Datacenter.
>>>>
>>>> Both storages are gluster. The old (unrecoverable) one worked fine as a
>>>> master.
>>> This may be related to this bug:
>>> https://bugzilla.redhat.com/1183977.
>> Ok. I added a comment and explained more in detail the issue on BZ.
>>
>> [...]
> Hi Chris,
> Can you please attach the engine/vdsm logs from the time the issue occurred?
> thanks.

Hi, relevant VDSM logs were published here:
https://bugzilla.redhat.com/show_bug.cgi?id=1183977

And engine says "Failed in 'ReconstructMasterVDS' method" because of 
"Cannot acquire host id: (u'ba7be27f-aee5-4436-ae9a-0764f551f9a7', 
SanlockException(5, 'Sanlock lockspace add failure', 'Input/output 
error'))":

    2015-04-28 16:34:01,915 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
    (DefaultQuartzScheduler_Worker-56) [] Command
    'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand'
    return value '
    TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654,
    mMessage=Not SPM: ()]]
    '
    2015-04-28 16:34:01,915 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
    (DefaultQuartzScheduler_Worker-56) [] HostName = H4
    2015-04-28 16:34:01,915 ERROR
    [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
    (DefaultQuartzScheduler_Worker-56) [] Command
    'HSMGetAllTasksStatusesVDSCommand(HostName = H4,
    VdsIdVDSCommandParametersBase:{runAsync='true',
    hostId='0146bec9-9e8a-4611-a2ca-e4f3c8ccdb26'})' execution failed:
    IRSGenericException: IRSErrorException: IRSNonOperationalException:
    Not SPM: ()
    2015-04-28 16:34:01,951 INFO
    [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
    (DefaultQuartzScheduler_Worker-56) [] hostFromVds::selectedVds -
    'H6', spmStatus 'Unknown_Pool', storage pool 'Default', storage pool
    version '3.6'
    2015-04-28 16:34:01,953 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
    (DefaultQuartzScheduler_Worker-56) [] START,
    ConnectStoragePoolVDSCommand(HostName = H6,
    ConnectStoragePoolVDSCommandParameters:{runAsync='true',
    hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsId =
    75e32383-90dd-42c5-9bc9-73b6e5333fe1, storagePoolId =
    00000001-0001-0001-0001-00000000007e, masterVersion = 2625), log id:
    15842e4f
    2015-04-28 16:34:02,035 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
    (DefaultQuartzScheduler_Worker-56) [] Command
    'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand'
    return value 'StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc
    [mCode=304, mMessage=Cannot find master domain:
    u'spUUID=00000001-0001-0001-0001-00000000007e,
    msdUUID=3233144b-7be1-445f-9ea6-6aebbacbb93f']]'
    2015-04-28 16:34:02,036 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
    (DefaultQuartzScheduler_Worker-56) [] HostName = H6
    2015-04-28 16:34:02,036 ERROR
    [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
    (DefaultQuartzScheduler_Worker-56) [] Command
    'ConnectStoragePoolVDSCommand(HostName = H6,
    ConnectStoragePoolVDSCommandParameters:{runAsync='true',
    hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsId =
    75e32383-90dd-42c5-9bc9-73b6e5333fe1, storagePoolId =
    00000001-0001-0001-0001-00000000007e, masterVersion = 2625)'
    execution failed: IRSGenericException: IRSErrorException:
    IRSNoMasterDomainException: Cannot find master domain:
    u'spUUID=00000001-0001-0001-0001-00000000007e,
    msdUUID=3233144b-7be1-445f-9ea6-6aebbacbb93f'
    2015-04-28 16:34:02,036 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
    (DefaultQuartzScheduler_Worker-56) [] FINISH,
    ConnectStoragePoolVDSCommand, log id: 15842e4f
    2015-04-28 16:34:02,036 ERROR
    [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
    (DefaultQuartzScheduler_Worker-56) []
    IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException:
    IRSErrorException: IRSNoMasterDomainException: Cannot find master
    domain: u'spUUID=00000001-0001-0001-0001-00000000007e,
    msdUUID=3233144b-7be1-445f-9ea6-6aebbacbb93f'
    2015-04-28 16:34:02,137 INFO
    [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
    (org.ovirt.thread.pool-8-thread-25) [1339ee5] Running command:
    ReconstructMasterDomainCommand internal: true. Entities affected : 
    ID: 3233144b-7be1-445f-9ea6-6aebbacbb93f Type: Storage
    2015-04-28 16:34:02,235 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand]
    (org.ovirt.thread.pool-8-thread-25) [1339ee5] START,
    DisconnectStoragePoolVDSCommand(HostName = H6,
    DisconnectStoragePoolVDSCommandParameters:{runAsync='true',
    hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, storagePoolId =
    00000001-0001-0001-0001-00000000007e, vds_spm_id = 3), log id: 782490bb
    2015-04-28 16:34:02,239 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand]
    (org.ovirt.thread.pool-8-thread-25) [1339ee5] FINISH,
    DisconnectStoragePoolVDSCommand, log id: 782490bb
    2015-04-28 16:34:02,240 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
    (org.ovirt.thread.pool-8-thread-25) [1339ee5] START,
    ReconstructMasterVDSCommand(HostName = H6,
    ReconstructMasterVDSCommandParameters:{runAsync='true',
    hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsSpmId = 3,
    storagePoolId = 00000001-0001-0001-0001-00000000007e,
    storagePoolName = Default, masterDomainId =
    ba7be27f-aee5-4436-ae9a-0764f551f9a7, masterVersion = 2626,
    domainsList = [{ domainId: ad08adbc-b3a7-4198-a1cb-4ad70c724454,
    status: Unknown };{ domainId: 15674257-e14a-4693-9b95-cc4bb92aa765,
    status: Unknown };{ domainId: ba7be27f-aee5-4436-ae9a-0764f551f9a7,
    status: Unknown };{ domainId: 3233144b-7be1-445f-9ea6-6aebbacbb93f,
    status: Inactive };]), log id: 626fc611
    2015-04-28 16:34:03,318 ERROR
    [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
    (org.ovirt.thread.pool-8-thread-25) [1339ee5] Failed in
    'ReconstructMasterVDS' method
    2015-04-28 16:34:03,318 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
    (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
    'org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand'
    return value 'StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc
    [mCode=661, mMessage=Cannot acquire host id:
    (u'ba7be27f-aee5-4436-ae9a-0764f551f9a7', SanlockException(5,
    'Sanlock lockspace add failure', 'Input/output error'))]]'
    2015-04-28 16:34:03,318 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
    (org.ovirt.thread.pool-8-thread-25) [1339ee5] HostName = H6
    2015-04-28 16:34:03,318 ERROR
    [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
    (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
    'ReconstructMasterVDSCommand(HostName = H6,
    ReconstructMasterVDSCommandParameters:{runAsync='true',
    hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsSpmId = 3,
    storagePoolId = 00000001-0001-0001-0001-00000000007e,
    storagePoolName = Default, masterDomainId =
    ba7be27f-aee5-4436-ae9a-0764f551f9a7, masterVersion = 2626,
    domainsList = [{ domainId: ad08adbc-b3a7-4198-a1cb-4ad70c724454,
    status: Unknown };{ domainId: 15674257-e14a-4693-9b95-cc4bb92aa765,
    status: Unknown };{ domainId: ba7be27f-aee5-4436-ae9a-0764f551f9a7,
    status: Unknown };{ domainId: 3233144b-7be1-445f-9ea6-6aebbacbb93f,
    status: Inactive };])' execution failed: VDSGenericException:
    VDSErrorException: Failed to ReconstructMasterVDS, error = Cannot
    acquire host id: (u'ba7be27f-aee5-4436-ae9a-0764f551f9a7',
    SanlockException(5, 'Sanlock lockspace add failure', 'Input/output
    error')), code = 661
    2015-04-28 16:34:03,319 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
    (org.ovirt.thread.pool-8-thread-25) [1339ee5] FINISH,
    ReconstructMasterVDSCommand, log id: 626fc611
    2015-04-28 16:34:03,319 ERROR
    [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
    (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
    'org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand'
    failed: VdcBLLException:
    org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
    VDSGenericException: VDSErrorException: Failed to
    ReconstructMasterVDS, error = Cannot acquire host id:
    (u'ba7be27f-aee5-4436-ae9a-0764f551f9a7', SanlockException(5,
    'Sanlock lockspace add failure', 'Input/output error')), code = 661
    (Failed with error AcquireHostIdFailure and code 661)
    2015-04-28 16:34:03,320 INFO
    [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
    (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
    [id=449f2035-96bb-4b70-867c-05a7179dc75b]: Compensating
    CHANGED_STATUS_ONLY of
    org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap;
    snapshot: EntityStatusSnapshot:{id='storagePoolId =
    00000001-0001-0001-0001-00000000007e, storageId =
    3233144b-7be1-445f-9ea6-6aebbacbb93f', status='Unknown'}.
    2015-04-28 16:34:03,321 INFO
    [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
    (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
    [id=449f2035-96bb-4b70-867c-05a7179dc75b]: Compensating
    UPDATED_ONLY_ENTITY of
    org.ovirt.engine.core.common.businessentities.StorageDomainStatic;
    snapshot: StorageDomainStatic[vdisks-ssd,
    3233144b-7be1-445f-9ea6-6aebbacbb93f].
    2015-04-28 16:34:03,321 INFO
    [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
    (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
    [id=449f2035-96bb-4b70-867c-05a7179dc75b]: Compensating
    UPDATED_ONLY_ENTITY of
    org.ovirt.engine.core.common.businessentities.StorageDomainStatic;
    snapshot: StorageDomainStatic[vdisks,
    ba7be27f-aee5-4436-ae9a-0764f551f9a7].
    2015-04-28 16:34:03,370 INFO
    [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor]
    (org.ovirt.thread.pool-8-thread-6) [] Finished reconstruct for pool
    '00000001-0001-0001-0001-00000000007e'. Clearing event queue
    2015-04-28 16:34:04,154 WARN
    [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerObjectsBuilder]
    (DefaultQuartzScheduler_Worker-65) [c1df557] Can't set graphics data
    from XML.
    2015-04-28 16:34:04,155 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
    (DefaultQuartzScheduler_Worker-65) [c1df557] START,
    DestroyVDSCommand(HostName = H4,
    DestroyVmVDSCommandParameters:{runAsync='true',
    hostId='0146bec9-9e8a-4611-a2ca-e4f3c8ccdb26',
    vmId='6ec9c0a0-2879-4bfe-9a79-92471881ebfe', force='false',
    secondsToWait='0', gracefully='false', reason=''}), log id: 238de0d4
    2015-04-28 16:34:04,158 WARN
    [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
    (DefaultQuartzScheduler_Worker-65) [c1df557] Unexpected return
    value: StatusForXmlRpc [mCode=-32603,
    mMessage=u'6ec9c0a0-2879-4bfe-9a79-92471881ebfe']
    2015-04-28 16:34:04,158 ERROR
    [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
    (DefaultQuartzScheduler_Worker-65) [c1df557] Failed in 'DestroyVDS'
    method
    2015-04-28 16:34:04,158 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
    (DefaultQuartzScheduler_Worker-65) [c1df557] Command
    'org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand' return
    value 'StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc
    [mCode=-32603, mMessage=u'6ec9c0a0-2879-4bfe-9a79-92471881ebfe']]'
    2015-04-28 16:34:04,158 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
    (DefaultQuartzScheduler_Worker-65) [c1df557] HostName = H4
    2015-04-28 16:34:04,158 ERROR
    [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
    (DefaultQuartzScheduler_Worker-65) [c1df557] Command
    'DestroyVDSCommand(HostName = H4,
    DestroyVmVDSCommandParameters:{runAsync='true',
    hostId='0146bec9-9e8a-4611-a2ca-e4f3c8ccdb26',
    vmId='6ec9c0a0-2879-4bfe-9a79-92471881ebfe', force='false',
    secondsToWait='0', gracefully='false', reason=''})' execution
    failed: VDSGenericException: VDSErrorException: Failed to
    DestroyVDS, error = u'6ec9c0a0-2879-4bfe-9a79-92471881ebfe', code =
    -32603
    2015-04-28 16:34:04,158 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
    (DefaultQuartzScheduler_Worker-65) [c1df557] FINISH,
    DestroyVDSCommand, log id: 238de0d4

DC is now up, because I attached the old MSD, but I could try 
reproducing the failing command if you want.

How do I specify the dataDict in the vdsClient command?

I guess it should be something like:

    vdsClient -s 0 reconstructMaster
    '00000001-0001-0001-0001-00000000007e' 'Default'
    'ba7be27f-aee5-4436-ae9a-0764f551f9a7' '[{ domainId:
    ad08adbc-b3a7-4198-a1cb-4ad70c724454, status: Unknown };{ domainId:
    15674257-e14a-4693-9b95-cc4bb92aa765, status: Unknown };{ domainId:
    ba7be27f-aee5-4436-ae9a-0764f551f9a7, status: Unknown };{ domainId:
    3233144b-7be1-445f-9ea6-6aebbacbb93f, status: Inactive };]'

Maybe related with sanlock giving me "add_lockspace fail result -5" all 
the time.

Best regards,
Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20150429/fd04c844/attachment.html>


More information about the Devel mailing list