<div dir="ltr"><div>Can you share the REST API data of the Storage domain and Data Center?<br></div>Here an example of the URLs, you will need to replace with correct ids.<br><div><br><a href="http://MY-SERVER/ovirt-engine/api/v4/storagedomains/13461356-f6f7-4a58-9897-2fac61ff40af">http://MY-SERVER/ovirt-engine/api/v4/storagedomains/13461356-f6f7-4a58-9897-2fac61ff40af</a><br><br><a href="http://MY-SERVER/ovirt-engine/api/v4/datacenters/5a5df553-022d-036d-01e8-000000000071/storagedomains">http://MY-SERVER/ovirt-engine/api/v4/datacenters/5a5df553-022d-036d-01e8-000000000071/storagedomains</a><br><br><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 2, 2018 at 12:53 PM, Callum Smith <span dir="ltr"><<a href="mailto:callum@well.ox.ac.uk" target="_blank">callum@well.ox.ac.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word;line-break:after-white-space">
This is on 4.2.0.2-1, I've linked the main logs to dropbox simply because they're big, full of noise right now.
<div><a href="https://www.dropbox.com/s/f8q3m5amro2a1b2/engine.log?dl=0" target="_blank">https://www.dropbox.com/s/<wbr>f8q3m5amro2a1b2/engine.log?dl=<wbr>0</a></div>
<div><a href="https://www.dropbox.com/s/uods85jk65halo3/vdsm.log?dl=0" target="_blank">https://www.dropbox.com/s/<wbr>uods85jk65halo3/vdsm.log?dl=0</a></div>
<div><br>
<div>
<div><span class="">
<div>
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">
<div style="word-wrap:break-word">
<div style="color:rgb(0,0,0);font-family:Arial;font-size:13px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
Regards,</div>
<div style="color:rgb(0,0,0);font-family:Arial;font-size:13px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
Callum</div>
<div style="color:rgb(0,0,0);font-family:Arial;font-size:13px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<br>
</div>
<div style="color:rgb(0,0,0);font-family:Arial;font-size:13px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
--<br>
<br>
Callum Smith</div>
<div style="color:rgb(0,0,0);font-family:Arial;font-size:13px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
Research Computing Core</div>
<div style="color:rgb(0,0,0);font-family:Arial;font-size:13px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
Wellcome Trust Centre for Human Genetics<br>
University of Oxford<br>
e. <a href="mailto:callum@well.ox.ac.uk" target="_blank">callum@well.ox.ac.uk</a></div>
</div>
</div>
</div>
</span><div><div class="h5"><div><br>
<blockquote type="cite">
<div>On 2 May 2018, at 10:43, Fred Rolland <<a href="mailto:frolland@redhat.com" target="_blank">frolland@redhat.com</a>> wrote:</div>
<br class="m_2571234484608545856Apple-interchange-newline">
<div>
<div dir="ltr">
<div>Which version are you using?<br>
Can you provide the whole log?<br>
<br>
</div>
For some reason, it looks like the Vdsm thinks that the Storage Domain is not part of the pool.<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, May 2, 2018 at 11:20 AM, Callum Smith <span dir="ltr">
<<a href="mailto:callum@well.ox.ac.uk" target="_blank">callum@well.ox.ac.uk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word;line-break:after-white-space">State is maintenance for the ISOs storage. I've extracted what is hopefully the relevant bits of the log.
<div>
<div><br>
</div>
<div>VDSM.log (SPM)</div>
<div><br>
</div>
<div>
<div>2018-05-02 09:16:03,455+0100 INFO (ioprocess communication (179084)) [IOProcess] Starting ioprocess (__init__:447)</div>
<div>2018-05-02 09:16:03,456+0100 INFO (ioprocess communication (179091)) [IOProcess] Starting ioprocess (__init__:447)</div>
<div>2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [vdsm.api] FINISH activateStorageDomain error=Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b6<wbr>66-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-0<wbr>00000000304' from=::ffff:192.168.64.254,589<wbr>68,
flow_id=93433989-8e26-48a9-bd3<wbr>a-2ab95f296c08, task_id=7f21f911-348f-45a3-b79<wbr>c-e3cb11642035 (api:50)</div>
<div>2018-05-02 09:16:03,461+0100 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='7f21f911-348f-45a3-b79c<wbr>-e3cb11642035') Unexpected error (task:875)</div>
<div>Traceback (most recent call last):</div>
<div> File "/usr/lib/python2.7/site-packa<wbr>ges/vdsm/storage/task.py", line 882, in _run</div>
<div> return fn(*args, **kargs)</div>
<div> File "<string>", line 2, in activateStorageDomain</div>
<div> File "/usr/lib/python2.7/site-packa<wbr>ges/vdsm/common/api.py", line 48, in method</div>
<div> ret = func(*args, **kwargs)</div>
<div> File "/usr/lib/python2.7/site-packa<wbr>ges/vdsm/storage/hsm.py", line 1256, in activateStorageDomain</div>
<div> pool.activateSD(sdUUID)</div>
<div> File "/usr/lib/python2.7/site-packa<wbr>ges/vdsm/storage/securable.py"<wbr>, line 79, in wrapper</div>
<div> return method(self, *args, **kwargs)</div>
<div> File "/usr/lib/python2.7/site-packa<wbr>ges/vdsm/storage/sp.py", line 1130, in activateSD</div>
<div> self.validateAttachedDomain(do<wbr>m)</div>
<div> File "/usr/lib/python2.7/site-packa<wbr>ges/vdsm/storage/securable.py"<wbr>, line 79, in wrapper</div>
<div> return method(self, *args, **kwargs)</div>
<div> File "/usr/lib/python2.7/site-packa<wbr>ges/vdsm/storage/sp.py", line 557, in validateAttachedDomain</div>
<div> raise se.StorageDomainNotInPool(self<wbr>.spUUID, dom.sdUUID)</div>
<div>StorageDomainNotInPool: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b6<wbr>66-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-0<wbr>00000000304'</div>
<div>2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [storage.TaskManager.Task] (Task='7f21f911-348f-45a3-b79c<wbr>-e3cb11642035') aborting: Task is aborted: "Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b6<wbr>66-c929aa0225ae,
pool=5a54bf81-0228-02bc-0358-0<wbr>00000000304'" - code 353 (task:1181)</div>
<div>2018-05-02 09:16:03,462+0100 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH activateStorageDomain error=Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b6<wbr>66-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-0<wbr>00000000304'
(dispatcher:82)</div>
</div>
<div><br>
</div>
<div>engine.log</div>
<div><br>
</div>
<div>
<div>2018-05-02 09:16:02,326+01 INFO [<a href="http://org.ovirt.engine.core.bll.st">org.ovirt.engine.core.bll.st</a><wbr>orage.domain.ActivateStorageDo<wbr>mainCommand] (default task-20) [93433989-8e26-48a9-bd3a-2ab95<wbr>f296c08] Lock Acquired to object 'EngineLock:{exclusiveLocks='[<wbr>f5914df0-f46c-4cc0-b666-c929aa<wbr>0225ae=STORAGE]',
sharedLocks=''}'</div>
<div>2018-05-02 09:16:02,376+01 INFO [<a href="http://org.ovirt.engine.core.bll.st">org.ovirt.engine.core.bll.st</a><wbr>orage.domain.ActivateStorageDo<wbr>mainCommand] (EE-ManagedThreadFactory-engin<wbr>e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95<wbr>f296c08]
Running command: ActivateStorageDomainCommand internal: false. Entities affected : ID: f5914df0-f46c-4cc0-b666-c929aa<wbr>0225ae Type: StorageAction group MANIPULATE_STORAGE_DOMA</div>
<div>IN with role type ADMIN</div>
<div>2018-05-02 09:16:02,385+01 INFO [<a href="http://org.ovirt.engine.core.bll.st">org.ovirt.engine.core.bll.st</a><wbr>orage.domain.ActivateStorageDo<wbr>mainCommand] (EE-ManagedThreadFactory-engin<wbr>e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95<wbr>f296c08]
Lock freed to object 'EngineLock:{exclusiveLocks='[<wbr>f5914df0-f46c-4cc0-b666-c929aa<wbr>0225ae=STORAGE]', sharedLocks=''}'</div>
<div>2018-05-02 09:16:02,385+01 INFO [<a href="http://org.ovirt.engine.core.bll.st">org.ovirt.engine.core.bll.st</a><wbr>orage.domain.ActivateStorageDo<wbr>mainCommand] (EE-ManagedThreadFactory-engin<wbr>e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95<wbr>f296c08]
ActivateStorage Domain. Before Connect all hosts to pool. Time: Wed May 02 09:16:02 BST 2018</div>
<div>2018-05-02 09:16:02,407+01 INFO [<a href="http://org.ovirt.engine.core.bll.st">org.ovirt.engine.core.bll.st</a><wbr>orage.connection.ConnectStorag<wbr>eToVdsCommand] (EE-ManagedThreadFactory-engin<wbr>e-Thread-33456) [40a82b47] Running command: ConnectStorageToVdsCommand
internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456<wbr>789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN</div>
<div>2018-05-02 09:16:02,421+01 INFO [org.ovirt.engine.core.vdsbro<wbr>ker.vdsbroker.ConnectStorageSe<wbr>rverVDSCommand] (EE-ManagedThreadFactory-engin<wbr>e-Thread-33456) [40a82b47] START, ConnectStorageServerVDSCommand<wbr>(HostName
= virtA003, StorageServerConnectionManagem<wbr>entVDSParameters:{hostId='fe28<wbr>61fc-2b47-4807-b054-470198eda4<wbr>73', storagePoolId='00000000-0000-0<wbr>000-0000-000000</div>
<div>000000', storageType='NFS', connectionList='[StorageServer<wbr>Connections:{id='da392861-<wbr>aedc-4f1e-97f4-6919fb01f1e9', connection='backoffice01.clust<wbr>er:/vm-iso', iqn='null', vfsType='null', mountOptions='null',
nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 23ce648f</div>
<div>2018-05-02 09:16:02,446+01 INFO [org.ovirt.engine.core.vdsbro<wbr>ker.vdsbroker.ConnectStorageSe<wbr>rverVDSCommand] (EE-ManagedThreadFactory-engin<wbr>e-Thread-33456) [40a82b47] FINISH, ConnectStorageServerVDSCommand<wbr>,
return: {da392861-aedc-4f1e-97f4-6919f<wbr>b01f1e9=0}, log id: 23ce648f</div>
<div>2018-05-02 09:16:02,450+01 INFO [org.ovirt.engine.core.vdsbro<wbr>ker.irsbroker.ActivateStorageD<wbr>omainVDSCommand] (EE-ManagedThreadFactory-engin<wbr>e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95<wbr>f296c08]
START, ActivateStorageDomainVDSComman<wbr>d( ActivateStorageDomainVDSComman<wbr>dParameters:{storagePoolId='5a<wbr>54bf81-0228-02bc-0358-00000000<wbr>0304', ignoreFailoverLimit='false', stor</div>
<div>ageDomainId='f5914df0-f46c-4cc<wbr>0-b666-c929aa0225ae'}), log id: 5c864594</div>
<div>2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.dal.dbb<wbr>roker.auditloghandling.AuditLo<wbr>gDirector] (EE-ManagedThreadFactory-engin<wbr>e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95<wbr>f296c08]
EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,<wbr>803), VDSM command ActivateStorageDomainVDS failed: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b6<wbr>66-c929aa0225ae, pool=5a5</div>
<div>4bf81-0228-02bc-0358-000000000<wbr>304'</div>
<div>2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.irsbroker.ActivateStorageDo<wbr>mainVDSCommand] (EE-ManagedThreadFactory-engin<wbr>e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95<wbr>f296c08]
Command 'ActivateStorageDomainVDSComma<wbr>nd( ActivateStorageDomainVDSComman<wbr>dParameters:{storagePoolId='5a<wbr>54bf81-0228-02bc-0358-00000000<wbr>0304', ignoreFailoverLimit='false', st</div>
<div>orageDomainId='f5914df0-f46c-4<wbr>cc0-b666-c929aa0225ae'})' execution failed: IRSGenericException: IRSErrorException: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b6<wbr>66-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-0<wbr>00000000304'</div>
<div>2018-05-02 09:16:02,635+01 INFO [org.ovirt.engine.core.vdsbro<wbr>ker.irsbroker.ActivateStorageD<wbr>omainVDSCommand] (EE-ManagedThreadFactory-engin<wbr>e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95<wbr>f296c08]
FINISH, ActivateStorageDomainVDSComman<wbr>d, log id: 5c864594</div>
<div>2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.bll.sto<wbr>rage.domain.ActivateStorageDom<wbr>ainCommand] (EE-ManagedThreadFactory-engin<wbr>e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95<wbr>f296c08]
Command 'org.ovirt.engine.core.bll.sto<wbr>rage.domain.ActivateStorageDom<wbr>ainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroke<wbr>r.irsbroker.IrsOperationFailed<wbr>NoFailove</div>
<div>rException: IRSGenericException: IRSErrorException: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b6<wbr>66-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-0<wbr>00000000304' (Failed with error StorageDomainNotInPool and
code 353)</div>
<div>2018-05-02 09:16:02,636+01 INFO [<a href="http://org.ovirt.engine.core.bll.st">org.ovirt.engine.core.bll.st</a><wbr>orage.domain.ActivateStorageDo<wbr>mainCommand] (EE-ManagedThreadFactory-engin<wbr>e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95<wbr>f296c08]
Command [id=22b0f3c1-9a09-4e26-8096-d8<wbr>3465c8f4ee]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.b<wbr>usinessentities.StoragePoolIso<wbr>Map; snapshot: EntityStatus</div>
<div>Snapshot:{id='StoragePoolIsoMa<wbr>pId:{storagePoolId='5a54bf81-<wbr>0228-02bc-0358-000000000304', storageId='f5914df0-f46c-4cc0-<wbr>b666-c929aa0225ae'}', status='Maintenance'}.</div>
<div>2018-05-02 09:16:02,660+01 ERROR [org.ovirt.engine.core.dal.dbb<wbr>roker.auditloghandling.AuditLo<wbr>gDirector] (EE-ManagedThreadFactory-engin<wbr>e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95<wbr>f296c08]
EVENT_ID: USER_ACTIVATE_STORAGE_DOMAIN_F<wbr>AILED(967), Failed to activate Storage Domain VMISOs (Data Center Default) by admin@internal-authz</div>
</div>
<div><span><br>
<div>
<div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">
<div style="word-wrap:break-word">
<div style="font-family:Arial;font-size:13px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
Regards,</div>
<div style="font-family:Arial;font-size:13px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
Callum</div>
<div style="font-family:Arial;font-size:13px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<br>
</div>
<div style="font-family:Arial;font-size:13px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
--<br>
<br>
Callum Smith</div>
<div style="font-family:Arial;font-size:13px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
Research Computing Core</div>
<div style="font-family:Arial;font-size:13px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
Wellcome Trust Centre for Human Genetics<br>
University of Oxford<br>
e. <a href="mailto:callum@well.ox.ac.uk" target="_blank">callum@well.ox.ac.uk</a></div>
</div>
</div>
</div>
</span>
<div>
<div class="m_2571234484608545856h5">
<div><br>
<blockquote type="cite">
<div>On 2 May 2018, at 08:44, Fred Rolland <<a href="mailto:frolland@redhat.com" target="_blank">frolland@redhat.com</a>> wrote:</div>
<br class="m_2571234484608545856m_-2093273250354586386Apple-interchange-newline">
<div>
<div dir="ltr">
<div>
<div>
<div>
<div>Hi,<br>
<br>
</div>
Can you provide logs from engine and Vdsm(SPM)?<br>
</div>
What is the state now?<br>
<br>
</div>
Thanks,<br>
</div>
Fred<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, May 1, 2018 at 4:11 PM, Callum Smith <span dir="ltr">
<<a href="mailto:callum@well.ox.ac.uk" target="_blank">callum@well.ox.ac.uk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word;line-break:after-white-space">Dear All,
<div><br>
</div>
<div>It appears that clicking "detach" on the ISO storage domain is a really bad idea. This has gotten half way through the procedure and now can't be recovered from. Is there any advice for re-attaching the ISO storage domain manually? An NFS mount
didn't add it back to the "pool" unfortunately.</div>
<div><br>
</div>
<div>On a separate note, is it possible to migrate this storage to a new location? And if so how.</div>
<div><br>
<div>
<div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">
<div style="word-wrap:break-word">
<div style="font-family:Arial;font-size:13px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
Regards,</div>
<div style="font-family:Arial;font-size:13px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
Callum</div>
<div style="font-family:Arial;font-size:13px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<br>
</div>
<div style="font-family:Arial;font-size:13px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
--<br>
<br>
Callum Smith</div>
<div style="font-family:Arial;font-size:13px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
Research Computing Core</div>
<div style="font-family:Arial;font-size:13px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
Wellcome Trust Centre for Human Genetics<br>
University of Oxford<br>
e. <a href="mailto:callum@well.ox.ac.uk" target="_blank">callum@well.ox.ac.uk</a></div>
</div>
</div>
</div>
<br>
</div>
</div>
<br>
______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div></div></div>
</div>
</div>
</div>
</blockquote></div><br></div>