Re: [ovirt-devel] "Please activate the master Storage Domain first"

Wednesday, 29 April 2015

This is a multi-part message in MIME format.
--------------000705000900010300040900
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

On 29-04-2015 11:56, Liron Aravot wrote:
...

 ----- Original Message -----
> From: "Christopher Pereira" <kripper(a)imatronix.cl&gt;
> To: "Nir Soffer" <nsoffer(a)redhat.com&gt;
> Cc: devel(a)ovirt.org, "Liron Aravot" <laravot(a)redhat.com&gt;
> Sent: Wednesday, April 29, 2015 6:14:32 AM
> Subject: Re: [ovirt-devel] "Please activate the master Storage Domain
first"
>
>
> On 28-04-2015 18:14, Nir Soffer wrote:
>>> The DC storage master domain is on a (unrecoverable) storage on a remote
>>> dead
>>> host.
>>> Engine is automatically setting another storage as the "Data
(Master)".
>>> Seconds later, the unrecoverable storage is marked as "Data
(Master)"
>>> again.
>>> There is no way to start the Datacenter.
>>>
>>> Both storages are gluster. The old (unrecoverable) one worked fine as a
>>> master.
>> This may be related to this bug:
>> https://bugzilla.redhat.com/1183977.
> Ok. I added a comment and explained more in detail the issue on BZ.
>
> [...]
 Hi Chris,
 Can you please attach the engine/vdsm logs from the time the issue occurred?
 thanks. 
Hi, relevant VDSM logs were published here:
https://bugzilla.redhat.com/show_bug.cgi?id=1183977

And engine says "Failed in 'ReconstructMasterVDS' method" because of 
"Cannot acquire host id: (u'ba7be27f-aee5-4436-ae9a-0764f551f9a7', 
SanlockException(5, 'Sanlock lockspace add failure', 'Input/output 
error'))":

    2015-04-28 16:34:01,915 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
    (DefaultQuartzScheduler_Worker-56) [] Command
    'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand'
    return value '
    TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654,
    mMessage=Not SPM: ()]]
    '
    2015-04-28 16:34:01,915 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
    (DefaultQuartzScheduler_Worker-56) [] HostName = H4
    2015-04-28 16:34:01,915 ERROR
    [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
    (DefaultQuartzScheduler_Worker-56) [] Command
    'HSMGetAllTasksStatusesVDSCommand(HostName = H4,
    VdsIdVDSCommandParametersBase:{runAsync='true',
    hostId='0146bec9-9e8a-4611-a2ca-e4f3c8ccdb26'})' execution failed:
    IRSGenericException: IRSErrorException: IRSNonOperationalException:
    Not SPM: ()
    2015-04-28 16:34:01,951 INFO
    [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
    (DefaultQuartzScheduler_Worker-56) [] hostFromVds::selectedVds -
    'H6', spmStatus 'Unknown_Pool', storage pool 'Default',
storage pool
    version '3.6'
    2015-04-28 16:34:01,953 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
    (DefaultQuartzScheduler_Worker-56) [] START,
    ConnectStoragePoolVDSCommand(HostName = H6,
    ConnectStoragePoolVDSCommandParameters:{runAsync='true',
    hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsId =
    75e32383-90dd-42c5-9bc9-73b6e5333fe1, storagePoolId =
    00000001-0001-0001-0001-00000000007e, masterVersion = 2625), log id:
    15842e4f
    2015-04-28 16:34:02,035 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
    (DefaultQuartzScheduler_Worker-56) [] Command
    'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand'
    return value 'StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc
    [mCode=304, mMessage=Cannot find master domain:
    u'spUUID=00000001-0001-0001-0001-00000000007e,
    msdUUID=3233144b-7be1-445f-9ea6-6aebbacbb93f']]'
    2015-04-28 16:34:02,036 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
    (DefaultQuartzScheduler_Worker-56) [] HostName = H6
    2015-04-28 16:34:02,036 ERROR
    [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
    (DefaultQuartzScheduler_Worker-56) [] Command
    'ConnectStoragePoolVDSCommand(HostName = H6,
    ConnectStoragePoolVDSCommandParameters:{runAsync='true',
    hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsId =
    75e32383-90dd-42c5-9bc9-73b6e5333fe1, storagePoolId =
    00000001-0001-0001-0001-00000000007e, masterVersion = 2625)'
    execution failed: IRSGenericException: IRSErrorException:
    IRSNoMasterDomainException: Cannot find master domain:
    u'spUUID=00000001-0001-0001-0001-00000000007e,
    msdUUID=3233144b-7be1-445f-9ea6-6aebbacbb93f'
    2015-04-28 16:34:02,036 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
    (DefaultQuartzScheduler_Worker-56) [] FINISH,
    ConnectStoragePoolVDSCommand, log id: 15842e4f
    2015-04-28 16:34:02,036 ERROR
    [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
    (DefaultQuartzScheduler_Worker-56) []
    IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException:
    IRSErrorException: IRSNoMasterDomainException: Cannot find master
    domain: u'spUUID=00000001-0001-0001-0001-00000000007e,
    msdUUID=3233144b-7be1-445f-9ea6-6aebbacbb93f'
    2015-04-28 16:34:02,137 INFO
    [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
    (org.ovirt.thread.pool-8-thread-25) [1339ee5] Running command:
    ReconstructMasterDomainCommand internal: true. Entities affected : 
    ID: 3233144b-7be1-445f-9ea6-6aebbacbb93f Type: Storage
    2015-04-28 16:34:02,235 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand]
    (org.ovirt.thread.pool-8-thread-25) [1339ee5] START,
    DisconnectStoragePoolVDSCommand(HostName = H6,
    DisconnectStoragePoolVDSCommandParameters:{runAsync='true',
    hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, storagePoolId =
    00000001-0001-0001-0001-00000000007e, vds_spm_id = 3), log id: 782490bb
    2015-04-28 16:34:02,239 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand]
    (org.ovirt.thread.pool-8-thread-25) [1339ee5] FINISH,
    DisconnectStoragePoolVDSCommand, log id: 782490bb
    2015-04-28 16:34:02,240 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
    (org.ovirt.thread.pool-8-thread-25) [1339ee5] START,
    ReconstructMasterVDSCommand(HostName = H6,
    ReconstructMasterVDSCommandParameters:{runAsync='true',
    hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsSpmId = 3,
    storagePoolId = 00000001-0001-0001-0001-00000000007e,
    storagePoolName = Default, masterDomainId =
    ba7be27f-aee5-4436-ae9a-0764f551f9a7, masterVersion = 2626,
    domainsList = [{ domainId: ad08adbc-b3a7-4198-a1cb-4ad70c724454,
    status: Unknown };{ domainId: 15674257-e14a-4693-9b95-cc4bb92aa765,
    status: Unknown };{ domainId: ba7be27f-aee5-4436-ae9a-0764f551f9a7,
    status: Unknown };{ domainId: 3233144b-7be1-445f-9ea6-6aebbacbb93f,
    status: Inactive };]), log id: 626fc611
    2015-04-28 16:34:03,318 ERROR
    [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
    (org.ovirt.thread.pool-8-thread-25) [1339ee5] Failed in
    'ReconstructMasterVDS' method
    2015-04-28 16:34:03,318 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
    (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
    'org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand'
    return value 'StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc
    [mCode=661, mMessage=Cannot acquire host id:
    (u'ba7be27f-aee5-4436-ae9a-0764f551f9a7', SanlockException(5,
    'Sanlock lockspace add failure', 'Input/output error'))]]'
    2015-04-28 16:34:03,318 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
    (org.ovirt.thread.pool-8-thread-25) [1339ee5] HostName = H6
    2015-04-28 16:34:03,318 ERROR
    [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
    (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
    'ReconstructMasterVDSCommand(HostName = H6,
    ReconstructMasterVDSCommandParameters:{runAsync='true',
    hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsSpmId = 3,
    storagePoolId = 00000001-0001-0001-0001-00000000007e,
    storagePoolName = Default, masterDomainId =
    ba7be27f-aee5-4436-ae9a-0764f551f9a7, masterVersion = 2626,
    domainsList = [{ domainId: ad08adbc-b3a7-4198-a1cb-4ad70c724454,
    status: Unknown };{ domainId: 15674257-e14a-4693-9b95-cc4bb92aa765,
    status: Unknown };{ domainId: ba7be27f-aee5-4436-ae9a-0764f551f9a7,
    status: Unknown };{ domainId: 3233144b-7be1-445f-9ea6-6aebbacbb93f,
    status: Inactive };])' execution failed: VDSGenericException:
    VDSErrorException: Failed to ReconstructMasterVDS, error = Cannot
    acquire host id: (u'ba7be27f-aee5-4436-ae9a-0764f551f9a7',
    SanlockException(5, 'Sanlock lockspace add failure', 'Input/output
    error')), code = 661
    2015-04-28 16:34:03,319 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
    (org.ovirt.thread.pool-8-thread-25) [1339ee5] FINISH,
    ReconstructMasterVDSCommand, log id: 626fc611
    2015-04-28 16:34:03,319 ERROR
    [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
    (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
    'org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand'
    failed: VdcBLLException:
    org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
    VDSGenericException: VDSErrorException: Failed to
    ReconstructMasterVDS, error = Cannot acquire host id:
    (u'ba7be27f-aee5-4436-ae9a-0764f551f9a7', SanlockException(5,
    'Sanlock lockspace add failure', 'Input/output error')), code = 661
    (Failed with error AcquireHostIdFailure and code 661)
    2015-04-28 16:34:03,320 INFO
    [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
    (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
    [id=449f2035-96bb-4b70-867c-05a7179dc75b]: Compensating
    CHANGED_STATUS_ONLY of
    org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap;
    snapshot: EntityStatusSnapshot:{id='storagePoolId =
    00000001-0001-0001-0001-00000000007e, storageId =
    3233144b-7be1-445f-9ea6-6aebbacbb93f', status='Unknown'}.
    2015-04-28 16:34:03,321 INFO
    [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
    (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
    [id=449f2035-96bb-4b70-867c-05a7179dc75b]: Compensating
    UPDATED_ONLY_ENTITY of
    org.ovirt.engine.core.common.businessentities.StorageDomainStatic;
    snapshot: StorageDomainStatic[vdisks-ssd,
    3233144b-7be1-445f-9ea6-6aebbacbb93f].
    2015-04-28 16:34:03,321 INFO
    [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
    (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
    [id=449f2035-96bb-4b70-867c-05a7179dc75b]: Compensating
    UPDATED_ONLY_ENTITY of
    org.ovirt.engine.core.common.businessentities.StorageDomainStatic;
    snapshot: StorageDomainStatic[vdisks,
    ba7be27f-aee5-4436-ae9a-0764f551f9a7].
    2015-04-28 16:34:03,370 INFO
    [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor]
    (org.ovirt.thread.pool-8-thread-6) [] Finished reconstruct for pool
    '00000001-0001-0001-0001-00000000007e'. Clearing event queue
    2015-04-28 16:34:04,154 WARN
    [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerObjectsBuilder]
    (DefaultQuartzScheduler_Worker-65) [c1df557] Can't set graphics data
    from XML.
    2015-04-28 16:34:04,155 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
    (DefaultQuartzScheduler_Worker-65) [c1df557] START,
    DestroyVDSCommand(HostName = H4,
    DestroyVmVDSCommandParameters:{runAsync='true',
    hostId='0146bec9-9e8a-4611-a2ca-e4f3c8ccdb26',
    vmId='6ec9c0a0-2879-4bfe-9a79-92471881ebfe', force='false',
    secondsToWait='0', gracefully='false', reason=''}), log id:
238de0d4
    2015-04-28 16:34:04,158 WARN
    [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
    (DefaultQuartzScheduler_Worker-65) [c1df557] Unexpected return
    value: StatusForXmlRpc [mCode=-32603,
    mMessage=u'6ec9c0a0-2879-4bfe-9a79-92471881ebfe']
    2015-04-28 16:34:04,158 ERROR
    [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
    (DefaultQuartzScheduler_Worker-65) [c1df557] Failed in 'DestroyVDS'
    method
    2015-04-28 16:34:04,158 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
    (DefaultQuartzScheduler_Worker-65) [c1df557] Command
    'org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand' return
    value 'StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc
    [mCode=-32603, mMessage=u'6ec9c0a0-2879-4bfe-9a79-92471881ebfe']]'
    2015-04-28 16:34:04,158 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
    (DefaultQuartzScheduler_Worker-65) [c1df557] HostName = H4
    2015-04-28 16:34:04,158 ERROR
    [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
    (DefaultQuartzScheduler_Worker-65) [c1df557] Command
    'DestroyVDSCommand(HostName = H4,
    DestroyVmVDSCommandParameters:{runAsync='true',
    hostId='0146bec9-9e8a-4611-a2ca-e4f3c8ccdb26',
    vmId='6ec9c0a0-2879-4bfe-9a79-92471881ebfe', force='false',
    secondsToWait='0', gracefully='false', reason=''})'
execution
    failed: VDSGenericException: VDSErrorException: Failed to
    DestroyVDS, error = u'6ec9c0a0-2879-4bfe-9a79-92471881ebfe', code =
    -32603
    2015-04-28 16:34:04,158 INFO
    [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
    (DefaultQuartzScheduler_Worker-65) [c1df557] FINISH,
    DestroyVDSCommand, log id: 238de0d4

DC is now up, because I attached the old MSD, but I could try 
reproducing the failing command if you want.

How do I specify the dataDict in the vdsClient command?

I guess it should be something like:

    vdsClient -s 0 reconstructMaster
    '00000001-0001-0001-0001-00000000007e' 'Default'
    'ba7be27f-aee5-4436-ae9a-0764f551f9a7' '[{ domainId:
    ad08adbc-b3a7-4198-a1cb-4ad70c724454, status: Unknown };{ domainId:
    15674257-e14a-4693-9b95-cc4bb92aa765, status: Unknown };{ domainId:
    ba7be27f-aee5-4436-ae9a-0764f551f9a7, status: Unknown };{ domainId:
    3233144b-7be1-445f-9ea6-6aebbacbb93f, status: Inactive };]'

Maybe related with sanlock giving me "add_lockspace fail result -5" all 
the time.

Best regards,
Chris

--------------000705000900010300040900
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta content="text/html; charset=utf-8"
http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <br>
    <div class="moz-cite-prefix">On 29-04-2015 11:56, Liron Aravot
      wrote:<br>
    </div>
    <blockquote
      cite="mid:1347581894.5682332.1430319376995.JavaMail.zimbra@redhat.com"
      type="cite">
      <pre wrap="">

----- Original Message -----
</pre>
      <blockquote type="cite">
        <pre wrap="">From: "Christopher Pereira" <a
class="moz-txt-link-rfc2396E"
href="mailto:kripper@imatronix.cl">&lt;kripper@imatronix.cl&gt;</a>
To: "Nir Soffer" <a class="moz-txt-link-rfc2396E"
href="mailto:nsoffer@redhat.com">&lt;nsoffer@redhat.com&gt;</a>
Cc: <a class="moz-txt-link-abbreviated"
href="mailto:devel@ovirt.org">devel@ovirt.org</a>, "Liron
Aravot" <a class="moz-txt-link-rfc2396E"
href="mailto:laravot@redhat.com">&lt;laravot@redhat.com&gt;</a>
Sent: Wednesday, April 29, 2015 6:14:32 AM
Subject: Re: [ovirt-devel] "Please activate the master Storage Domain first"

On 28-04-2015 18:14, Nir Soffer wrote:
</pre>
        <blockquote type="cite">
          <blockquote type="cite">
            <pre wrap="">The DC storage master domain is on a
(unrecoverable) storage on a remote
dead
host.
Engine is automatically setting another storage as the "Data (Master)".
Seconds later, the unrecoverable storage is marked as "Data (Master)"
again.
There is no way to start the Datacenter.

Both storages are gluster. The old (unrecoverable) one worked fine as a
master.
</pre>
          </blockquote>
          <pre wrap="">This may be related to this bug:
<a class="moz-txt-link-freetext"
href="https://bugzilla.redhat.com/1183977">https://bugzilla....;.
</pre>
        </blockquote>
        <pre wrap="">
Ok. I added a comment and explained more in detail the issue on BZ.

[...]
</pre>
      </blockquote>
      <pre wrap="">
Hi Chris,
Can you please attach the engine/vdsm logs from the time the issue occurred?
thanks.</pre>
    </blockquote>
    <br>
    Hi, relevant VDSM logs were published here:<br>
    <a class="moz-txt-link-freetext"
href="https://bugzilla.redhat.com/show_bug.cgi?id=1183977">h...
    <br>
    And engine says "Failed in 'ReconstructMasterVDS' method" because
of
    "Cannot acquire host id: (u'ba7be27f-aee5-4436-ae9a-0764f551f9a7',
    SanlockException(5, 'Sanlock lockspace add failure', 'Input/output
    error'))":<br>
    <blockquote>2015-04-28 16:34:01,915 INFO 
      [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
      (DefaultQuartzScheduler_Worker-56) [] Command

'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand'
      return value '<br>
      TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654,
      mMessage=Not SPM: ()]]<br>
      '<br>
      2015-04-28 16:34:01,915 INFO 
      [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
      (DefaultQuartzScheduler_Worker-56) [] HostName = H4<br>
      2015-04-28 16:34:01,915 ERROR
      [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
      (DefaultQuartzScheduler_Worker-56) [] Command
      'HSMGetAllTasksStatusesVDSCommand(HostName = H4,
      VdsIdVDSCommandParametersBase:{runAsync='true',
      hostId='0146bec9-9e8a-4611-a2ca-e4f3c8ccdb26'})' execution failed:
      IRSGenericException: IRSErrorException:
      IRSNonOperationalException: Not SPM: ()<br>
      2015-04-28 16:34:01,951 INFO 
      [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
      (DefaultQuartzScheduler_Worker-56) [] hostFromVds::selectedVds -
      'H6', spmStatus 'Unknown_Pool', storage pool 'Default',
storage
      pool version '3.6'<br>
      2015-04-28 16:34:01,953 INFO 
      [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
      (DefaultQuartzScheduler_Worker-56) [] START,
      ConnectStoragePoolVDSCommand(HostName = H6,
      ConnectStoragePoolVDSCommandParameters:{runAsync='true',
      hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsId =
      75e32383-90dd-42c5-9bc9-73b6e5333fe1, storagePoolId =
      00000001-0001-0001-0001-00000000007e, masterVersion = 2625), log
      id: 15842e4f<br>
      2015-04-28 16:34:02,035 INFO 
      [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
      (DefaultQuartzScheduler_Worker-56) [] Command
      'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand'
      return value 'StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc
      [mCode=304, mMessage=Cannot find master domain:
      u'spUUID=00000001-0001-0001-0001-00000000007e,
      msdUUID=3233144b-7be1-445f-9ea6-6aebbacbb93f']]'<br>
      2015-04-28 16:34:02,036 INFO 
      [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
      (DefaultQuartzScheduler_Worker-56) [] HostName = H6<br>
      2015-04-28 16:34:02,036 ERROR
      [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
      (DefaultQuartzScheduler_Worker-56) [] Command
      'ConnectStoragePoolVDSCommand(HostName = H6,
      ConnectStoragePoolVDSCommandParameters:{runAsync='true',
      hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsId =
      75e32383-90dd-42c5-9bc9-73b6e5333fe1, storagePoolId =
      00000001-0001-0001-0001-00000000007e, masterVersion = 2625)'
      execution failed: IRSGenericException: IRSErrorException:
      IRSNoMasterDomainException: Cannot find master domain:
      u'spUUID=00000001-0001-0001-0001-00000000007e,
      msdUUID=3233144b-7be1-445f-9ea6-6aebbacbb93f'<br>
      2015-04-28 16:34:02,036 INFO 
      [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
      (DefaultQuartzScheduler_Worker-56) [] FINISH,
      ConnectStoragePoolVDSCommand, log id: 15842e4f<br>
      2015-04-28 16:34:02,036 ERROR
      [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
      (DefaultQuartzScheduler_Worker-56) []
      IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException:
      IRSErrorException: IRSNoMasterDomainException: Cannot find master
      domain: u'spUUID=00000001-0001-0001-0001-00000000007e,
      msdUUID=3233144b-7be1-445f-9ea6-6aebbacbb93f'<br>
      2015-04-28 16:34:02,137 INFO 
      [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
      (org.ovirt.thread.pool-8-thread-25) [1339ee5] Running command:
      ReconstructMasterDomainCommand internal: true. Entities affected
      :  ID: 3233144b-7be1-445f-9ea6-6aebbacbb93f Type: Storage<br>
      2015-04-28 16:34:02,235 INFO 
      [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand]
      (org.ovirt.thread.pool-8-thread-25) [1339ee5] START,
      DisconnectStoragePoolVDSCommand(HostName = H6,
      DisconnectStoragePoolVDSCommandParameters:{runAsync='true',
      hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, storagePoolId =
      00000001-0001-0001-0001-00000000007e, vds_spm_id = 3), log id:
      782490bb<br>
      2015-04-28 16:34:02,239 INFO 
      [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand]
      (org.ovirt.thread.pool-8-thread-25) [1339ee5] FINISH,
      DisconnectStoragePoolVDSCommand, log id: 782490bb<br>
      2015-04-28 16:34:02,240 INFO 
      [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
      (org.ovirt.thread.pool-8-thread-25) [1339ee5] START,
      ReconstructMasterVDSCommand(HostName = H6,
      ReconstructMasterVDSCommandParameters:{runAsync='true',
      hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsSpmId = 3,
      storagePoolId = 00000001-0001-0001-0001-00000000007e,
      storagePoolName = Default, masterDomainId =
      ba7be27f-aee5-4436-ae9a-0764f551f9a7, masterVersion = 2626,
      domainsList = [{ domainId: ad08adbc-b3a7-4198-a1cb-4ad70c724454,
      status: Unknown };{ domainId:
      15674257-e14a-4693-9b95-cc4bb92aa765, status: Unknown };{
      domainId: ba7be27f-aee5-4436-ae9a-0764f551f9a7, status: Unknown
      };{ domainId: 3233144b-7be1-445f-9ea6-6aebbacbb93f, status:
      Inactive };]), log id: 626fc611<br>
      2015-04-28 16:34:03,318 ERROR
      [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
      (org.ovirt.thread.pool-8-thread-25) [1339ee5] Failed in
      'ReconstructMasterVDS' method<br>
      2015-04-28 16:34:03,318 INFO 
      [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
      (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
      'org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand'
      return value 'StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc
      [mCode=661, mMessage=Cannot acquire host id:
      (u'ba7be27f-aee5-4436-ae9a-0764f551f9a7', SanlockException(5,
      'Sanlock lockspace add failure', 'Input/output
error'))]]'<br>
      2015-04-28 16:34:03,318 INFO 
      [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
      (org.ovirt.thread.pool-8-thread-25) [1339ee5] HostName = H6<br>
      2015-04-28 16:34:03,318 ERROR
      [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
      (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
      'ReconstructMasterVDSCommand(HostName = H6,
      ReconstructMasterVDSCommandParameters:{runAsync='true',
      hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsSpmId = 3,
      storagePoolId = 00000001-0001-0001-0001-00000000007e,
      storagePoolName = Default, masterDomainId =
      ba7be27f-aee5-4436-ae9a-0764f551f9a7, masterVersion = 2626,
      domainsList = [{ domainId: ad08adbc-b3a7-4198-a1cb-4ad70c724454,
      status: Unknown };{ domainId:
      15674257-e14a-4693-9b95-cc4bb92aa765, status: Unknown };{
      domainId: ba7be27f-aee5-4436-ae9a-0764f551f9a7, status: Unknown
      };{ domainId: 3233144b-7be1-445f-9ea6-6aebbacbb93f, status:
      Inactive };])' execution failed: VDSGenericException:
      VDSErrorException: Failed to ReconstructMasterVDS, error = Cannot
      acquire host id: (u'ba7be27f-aee5-4436-ae9a-0764f551f9a7',
      SanlockException(5, 'Sanlock lockspace add failure', 'Input/output
      error')), code = 661<br>
      2015-04-28 16:34:03,319 INFO 
      [org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
      (org.ovirt.thread.pool-8-thread-25) [1339ee5] FINISH,
      ReconstructMasterVDSCommand, log id: 626fc611<br>
      2015-04-28 16:34:03,319 ERROR
      [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
      (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
      'org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand'
      failed: VdcBLLException:
      org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
      VDSGenericException: VDSErrorException: Failed to
      ReconstructMasterVDS, error = Cannot acquire host id:
      (u'ba7be27f-aee5-4436-ae9a-0764f551f9a7', SanlockException(5,
      'Sanlock lockspace add failure', 'Input/output error')), code =
      661 (Failed with error AcquireHostIdFailure and code 661)<br>
      2015-04-28 16:34:03,320 INFO 
      [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
      (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
      [id=449f2035-96bb-4b70-867c-05a7179dc75b]: Compensating
      CHANGED_STATUS_ONLY of
      org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap;
      snapshot: EntityStatusSnapshot:{id='storagePoolId =
      00000001-0001-0001-0001-00000000007e, storageId =
      3233144b-7be1-445f-9ea6-6aebbacbb93f', status='Unknown'}.<br>
      2015-04-28 16:34:03,321 INFO 
      [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
      (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
      [id=449f2035-96bb-4b70-867c-05a7179dc75b]: Compensating
      UPDATED_ONLY_ENTITY of
      org.ovirt.engine.core.common.businessentities.StorageDomainStatic;
      snapshot: StorageDomainStatic[vdisks-ssd,
      3233144b-7be1-445f-9ea6-6aebbacbb93f].<br>
      2015-04-28 16:34:03,321 INFO 
      [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
      (org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
      [id=449f2035-96bb-4b70-867c-05a7179dc75b]: Compensating
      UPDATED_ONLY_ENTITY of
      org.ovirt.engine.core.common.businessentities.StorageDomainStatic;
      snapshot: StorageDomainStatic[vdisks,
      ba7be27f-aee5-4436-ae9a-0764f551f9a7].<br>
      2015-04-28 16:34:03,370 INFO 
      [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor]
      (org.ovirt.thread.pool-8-thread-6) [] Finished reconstruct for
      pool '00000001-0001-0001-0001-00000000007e'. Clearing event queue<br>
      2015-04-28 16:34:04,154 WARN 
      [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerObjectsBuilder]
      (DefaultQuartzScheduler_Worker-65) [c1df557] Can't set graphics
      data from XML.<br>
      2015-04-28 16:34:04,155 INFO 
      [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
      (DefaultQuartzScheduler_Worker-65) [c1df557] START,
      DestroyVDSCommand(HostName = H4,
      DestroyVmVDSCommandParameters:{runAsync='true',
      hostId='0146bec9-9e8a-4611-a2ca-e4f3c8ccdb26',
      vmId='6ec9c0a0-2879-4bfe-9a79-92471881ebfe', force='false',
      secondsToWait='0', gracefully='false', reason=''}), log id:
      238de0d4<br>
      2015-04-28 16:34:04,158 WARN 
      [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
      (DefaultQuartzScheduler_Worker-65) [c1df557] Unexpected return
      value: StatusForXmlRpc [mCode=-32603,
      mMessage=u'6ec9c0a0-2879-4bfe-9a79-92471881ebfe']<br>
      2015-04-28 16:34:04,158 ERROR
      [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
      (DefaultQuartzScheduler_Worker-65) [c1df557] Failed in
      'DestroyVDS' method<br>
      2015-04-28 16:34:04,158 INFO 
      [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
      (DefaultQuartzScheduler_Worker-65) [c1df557] Command
      'org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand'
      return value 'StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc
      [mCode=-32603,
mMessage=u'6ec9c0a0-2879-4bfe-9a79-92471881ebfe']]'<br>
      2015-04-28 16:34:04,158 INFO 
      [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
      (DefaultQuartzScheduler_Worker-65) [c1df557] HostName = H4<br>
      2015-04-28 16:34:04,158 ERROR
      [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
      (DefaultQuartzScheduler_Worker-65) [c1df557] Command
      'DestroyVDSCommand(HostName = H4,
      DestroyVmVDSCommandParameters:{runAsync='true',
      hostId='0146bec9-9e8a-4611-a2ca-e4f3c8ccdb26',
      vmId='6ec9c0a0-2879-4bfe-9a79-92471881ebfe', force='false',
      secondsToWait='0', gracefully='false', reason=''})'
execution
      failed: VDSGenericException: VDSErrorException: Failed to
      DestroyVDS, error = u'6ec9c0a0-2879-4bfe-9a79-92471881ebfe', code
      = -32603<br>
      2015-04-28 16:34:04,158 INFO 
      [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
      (DefaultQuartzScheduler_Worker-65) [c1df557] FINISH,
      DestroyVDSCommand, log id: 238de0d4<br>
    </blockquote>
    DC is now up, because I attached the old MSD, but I could try
    reproducing the failing command if you want.<br>
    <br>
    How do I specify the dataDict in the vdsClient command?<br>
    <br>
    I guess it should be something like:<br>
    <blockquote>vdsClient -s 0 reconstructMaster
      '00000001-0001-0001-0001-00000000007e' 'Default'
      'ba7be27f-aee5-4436-ae9a-0764f551f9a7' '[{ domainId:
      ad08adbc-b3a7-4198-a1cb-4ad70c724454, status: Unknown };{
      domainId: 15674257-e14a-4693-9b95-cc4bb92aa765, status: Unknown
      };{ domainId: ba7be27f-aee5-4436-ae9a-0764f551f9a7, status:
      Unknown };{ domainId: 3233144b-7be1-445f-9ea6-6aebbacbb93f,
      status: Inactive };]'<br>
    </blockquote>
    Maybe related with sanlock giving me "add_lockspace fail result -5"
    all the time.<br>
    <br>
    Best regards,<br>
    Chris<br>
  </body>
</html>

--------------000705000900010300040900--

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Re: [ovirt-devel] "Please activate the master Storage Domain first"