Re-attaching ISOs and moving ISOs storage
--_000_DDD9531266F74CA59A0180B14810823Cwelloxacuk_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Dear All, It appears that clicking "detach" on the ISO storage domain is a really bad= idea. This has gotten half way through the procedure and now can't be reco= vered from. Is there any advice for re-attaching the ISO storage domain man= ually? An NFS mount didn't add it back to the "pool" unfortunately. On a separate note, is it possible to migrate this storage to a new locatio= n? And if so how. Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> --_000_DDD9531266F74CA59A0180B14810823Cwelloxacuk_ Content-Type: text/html; charset="us-ascii" Content-ID: <1847FC58E1CB3740851381BF7C769B9A@well.ox.ac.uk> Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
</head> <body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; line-break:= after-white-space;" class=3D""> Dear All, <div class=3D""><br class=3D""> </div> <div class=3D"">It appears that clicking "detach" on the ISO stor= age domain is a really bad idea. This has gotten half way through the proce= dure and now can't be recovered from. Is there any advice for re-attaching = the ISO storage domain manually? An NFS mount didn't add it back to the "pool" unfortunately.</div> <div class=3D""><br class=3D""> </div> <div class=3D"">On a separate note, is it possible to migrate this storage = to a new location? And if so how.</div> <div class=3D""><br class=3D""> <div class=3D""> <div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; text-align: star= t; text-indent: 0px; text-transform: none; white-space: normal; word-spacin= g: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp= -mode: space; -webkit-line-break: after-white-space;" class=3D""> <div style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line= -break: after-white-space;" class=3D""> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Regards,</div> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Callum</div> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> <br class=3D""> </div> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> --<br class=3D""> <br class=3D""> Callum Smith</div> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Research Computing Core</div> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Wellcome Trust Centre for Human Genetics<br class=3D""> University of Oxford<br class=3D""> e. <a href=3D"mailto:callum@well.ox.ac.uk" class=3D"">callum@well.ox.ac.uk<= /a></div> </div> </div> </div> <br class=3D""> </div> </body> </html> --_000_DDD9531266F74CA59A0180B14810823Cwelloxacuk_--
Hi, Can you provide logs from engine and Vdsm(SPM)? What is the state now? Thanks, Fred On Tue, May 1, 2018 at 4:11 PM, Callum Smith <callum@well.ox.ac.uk> wrote:
Dear All,
It appears that clicking "detach" on the ISO storage domain is a really bad idea. This has gotten half way through the procedure and now can't be recovered from. Is there any advice for re-attaching the ISO storage domain manually? An NFS mount didn't add it back to the "pool" unfortunately.
On a separate note, is it possible to migrate this storage to a new location? And if so how.
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--_000_F4071341D5B9476A9A215EAFC436F6C1welloxacuk_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable State is maintenance for the ISOs storage. I've extracted what is hopefully= the relevant bits of the log. VDSM.log (SPM) 2018-05-02 09:16:03,455+0100 INFO (ioprocess communication (179084)) [IOPr= ocess] Starting ioprocess (__init__:447) 2018-05-02 09:16:03,456+0100 INFO (ioprocess communication (179091)) [IOPr= ocess] Starting ioprocess (__init__:447) 2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [vdsm.api] FINISH activateSt= orageDomain error=3DStorage domain not in pool: u'domain=3Df5914df0-f46c-4c= c0-b666-c929aa0225ae, pool=3D5a54bf81-0228-02bc-0358-000000000304' from=3D:= :ffff:192.168.64.254,58968, flow_id=3D93433989-8e26-48a9-bd3a-2ab95f296c08,= task_id=3D7f21f911-348f-45a3-b79c-e3cb11642035 (api:50) 2018-05-02 09:16:03,461+0100 ERROR (jsonrpc/0) [storage.TaskManager.Task] (= Task=3D'7f21f911-348f-45a3-b79c-e3cb11642035') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, i= n _run return fn(*args, **kargs) File "<string>", line 2, in activateStorageDomain File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in m= ethod ret =3D func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1256, i= n activateStorageDomain pool.activateSD(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 7= 9, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1130, in= activateSD self.validateAttachedDomain(dom) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 7= 9, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 557, in = validateAttachedDomain raise se.StorageDomainNotInPool(self.spUUID, dom.sdUUID) StorageDomainNotInPool: Storage domain not in pool: u'domain=3Df5914df0-f46= c-4cc0-b666-c929aa0225ae, pool=3D5a54bf81-0228-02bc-0358-000000000304' 2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [storage.TaskManager.Task] (= Task=3D'7f21f911-348f-45a3-b79c-e3cb11642035') aborting: Task is aborted: "= Storage domain not in pool: u'domain=3Df5914df0-f46c-4cc0-b666-c929aa0225ae= , pool=3D5a54bf81-0228-02bc-0358-000000000304'" - code 353 (task:1181) 2018-05-02 09:16:03,462+0100 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH = activateStorageDomain error=3DStorage domain not in pool: u'domain=3Df5914d= f0-f46c-4cc0-b666-c929aa0225ae, pool=3D5a54bf81-0228-02bc-0358-000000000304= ' (dispatcher:82) engine.log 2018-05-02 09:16:02,326+01 INFO [org.ovirt.engine.core.bll.storage.domain.= ActivateStorageDomainCommand] (default task-20) [93433989-8e26-48a9-bd3a-2a= b95f296c08] Lock Acquired to object 'EngineLock:{exclusiveLocks=3D'[f5914df= 0-f46c-4cc0-b666-c929aa0225ae=3DSTORAGE]', sharedLocks=3D''}' 2018-05-02 09:16:02,376+01 INFO [org.ovirt.engine.core.bll.storage.domain.= ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455)= [93433989-8e26-48a9-bd3a-2ab95f296c08] Running command: ActivateStorageDom= ainCommand internal: false. Entities affected : ID: f5914df0-f46c-4cc0-b66= 6-c929aa0225ae Type: StorageAction group MANIPULATE_STORAGE_DOMA IN with role type ADMIN 2018-05-02 09:16:02,385+01 INFO [org.ovirt.engine.core.bll.storage.domain.= ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455)= [93433989-8e26-48a9-bd3a-2ab95f296c08] Lock freed to object 'EngineLock:{e= xclusiveLocks=3D'[f5914df0-f46c-4cc0-b666-c929aa0225ae=3DSTORAGE]', sharedL= ocks=3D''}' 2018-05-02 09:16:02,385+01 INFO [org.ovirt.engine.core.bll.storage.domain.= ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455)= [93433989-8e26-48a9-bd3a-2ab95f296c08] ActivateStorage Domain. Before Conn= ect all hosts to pool. Time: Wed May 02 09:16:02 BST 2018 2018-05-02 09:16:02,407+01 INFO [org.ovirt.engine.core.bll.storage.connect= ion.ConnectStorageToVdsCommand] (EE-ManagedThreadFactory-engine-Thread-3345= 6) [40a82b47] Running command: ConnectStorageToVdsCommand internal: true. E= ntities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAc= tion group CREATE_STORAGE_DOMAIN with role type ADMIN 2018-05-02 09:16:02,421+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker= .ConnectStorageServerVDSCommand] (EE-ManagedThreadFactory-engine-Thread-334= 56) [40a82b47] START, ConnectStorageServerVDSCommand(HostName =3D virtA003,= StorageServerConnectionManagementVDSParameters:{hostId=3D'fe2861fc-2b47-48= 07-b054-470198eda473', storagePoolId=3D'00000000-0000-0000-0000-000000 000000', storageType=3D'NFS', connectionList=3D'[StorageServerConnections:{= id=3D'da392861-aedc-4f1e-97f4-6919fb01f1e9', connection=3D'backoffice01.clu= ster:/vm-iso', iqn=3D'null', vfsType=3D'null', mountOptions=3D'null', nfsVe= rsion=3D'null', nfsRetrans=3D'null', nfsTimeo=3D'null', iface=3D'null', net= IfaceName=3D'null'}]'}), log id: 23ce648f 2018-05-02 09:16:02,446+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker= .ConnectStorageServerVDSCommand] (EE-ManagedThreadFactory-engine-Thread-334= 56) [40a82b47] FINISH, ConnectStorageServerVDSCommand, return: {da392861-ae= dc-4f1e-97f4-6919fb01f1e9=3D0}, log id: 23ce648f 2018-05-02 09:16:02,450+01 INFO [org.ovirt.engine.core.vdsbroker.irsbroker= .ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33= 455) [93433989-8e26-48a9-bd3a-2ab95f296c08] START, ActivateStorageDomainVDS= Command( ActivateStorageDomainVDSCommandParameters:{storagePoolId=3D'5a54bf= 81-0228-02bc-0358-000000000304', ignoreFailoverLimit=3D'false', stor ageDomainId=3D'f5914df0-f46c-4cc0-b666-c929aa0225ae'}), log id: 5c864594 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditl= oghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-33455) = [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID: IRS_BROKER_COMMAND_FAILURE= (10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain not = in pool: u'domain=3Df5914df0-f46c-4cc0-b666-c929aa0225ae, pool=3D5a5 4bf81-0228-02bc-0358-000000000304' 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker= .ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33= 455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command 'ActivateStorageDomainV= DSCommand( ActivateStorageDomainVDSCommandParameters:{storagePoolId=3D'5a54= bf81-0228-02bc-0358-000000000304', ignoreFailoverLimit=3D'false', st orageDomainId=3D'f5914df0-f46c-4cc0-b666-c929aa0225ae'})' execution failed:= IRSGenericException: IRSErrorException: Storage domain not in pool: u'doma= in=3Df5914df0-f46c-4cc0-b666-c929aa0225ae, pool=3D5a54bf81-0228-02bc-0358-0= 00000000304' 2018-05-02 09:16:02,635+01 INFO [org.ovirt.engine.core.vdsbroker.irsbroker= .ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33= 455) [93433989-8e26-48a9-bd3a-2ab95f296c08] FINISH, ActivateStorageDomainVD= SCommand, log id: 5c864594 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.bll.storage.domain.= ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455)= [93433989-8e26-48a9-bd3a-2ab95f296c08] Command 'org.ovirt.engine.core.bll.= storage.domain.ActivateStorageDomainCommand' failed: EngineException: org.o= virt.engine.core.vdsbroker.irsbroker.IrsOperationFailedNoFailove rException: IRSGenericException: IRSErrorException: Storage domain not in p= ool: u'domain=3Df5914df0-f46c-4cc0-b666-c929aa0225ae, pool=3D5a54bf81-0228-= 02bc-0358-000000000304' (Failed with error StorageDomainNotInPool and code = 353) 2018-05-02 09:16:02,636+01 INFO [org.ovirt.engine.core.bll.storage.domain.= ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455)= [93433989-8e26-48a9-bd3a-2ab95f296c08] Command [id=3D22b0f3c1-9a09-4e26-80= 96-d83465c8f4ee]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core= .common.businessentities.StoragePoolIsoMap; snapshot: EntityStatus Snapshot:{id=3D'StoragePoolIsoMapId:{storagePoolId=3D'5a54bf81-0228-02bc-03= 58-000000000304', storageId=3D'f5914df0-f46c-4cc0-b666-c929aa0225ae'}', sta= tus=3D'Maintenance'}. 2018-05-02 09:16:02,660+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditl= oghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-33455) = [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID: USER_ACTIVATE_STORAGE_DOMA= IN_FAILED(967), Failed to activate Storage Domain VMISOs (Data Center Defau= lt) by admin@internal-authz Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> On 2 May 2018, at 08:44, Fred Rolland <frolland@redhat.com<mailto:frolland@= redhat.com>> wrote: Hi, Can you provide logs from engine and Vdsm(SPM)? What is the state now? Thanks, Fred On Tue, May 1, 2018 at 4:11 PM, Callum Smith <callum@well.ox.ac.uk<mailto:c= allum@well.ox.ac.uk>> wrote: Dear All, It appears that clicking "detach" on the ISO storage domain is a really bad= idea. This has gotten half way through the procedure and now can't be reco= vered from. Is there any advice for re-attaching the ISO storage domain man= ually? An NFS mount didn't add it back to the "pool" unfortunately. On a separate note, is it possible to migrate this storage to a new locatio= n? And if so how. Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users --_000_F4071341D5B9476A9A215EAFC436F6C1welloxacuk_ Content-Type: text/html; charset="us-ascii" Content-ID: <96203274F7E1B84B84B53161A74B1D23@well.ox.ac.uk> Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
Dear All, <div class=3D""><br class=3D""> </div> <div class=3D"">It appears that clicking "detach" on the ISO stor= age domain is a really bad idea. This has gotten half way through the proce= dure and now can't be recovered from. Is there any advice for re-attaching =
</head> <body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; line-break:= after-white-space;" class=3D""> State is maintenance for the ISOs storage. I've extracted what is hopefully= the relevant bits of the log. <div class=3D""> <div class=3D""><br class=3D""> </div> <div class=3D"">VDSM.log (SPM)</div> <div class=3D""><br class=3D""> </div> <div class=3D""> <div class=3D"">2018-05-02 09:16:03,455+0100 INFO (ioprocess comm= unication (179084)) [IOProcess] Starting ioprocess (__init__:447)</div> <div class=3D"">2018-05-02 09:16:03,456+0100 INFO (ioprocess comm= unication (179091)) [IOProcess] Starting ioprocess (__init__:447)</div> <div class=3D"">2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [vd= sm.api] FINISH activateStorageDomain error=3DStorage domain not in pool: u'= domain=3Df5914df0-f46c-4cc0-b666-c929aa0225ae, pool=3D5a54bf81-0228-02bc-03= 58-000000000304' from=3D::ffff:192.168.64.254,58968, flow_id=3D93433989-8e26-48a9-bd3a-2ab95f296c08, task_id=3D7f21f911-348f-45= a3-b79c-e3cb11642035 (api:50)</div> <div class=3D"">2018-05-02 09:16:03,461+0100 ERROR (jsonrpc/0) [storage= .TaskManager.Task] (Task=3D'7f21f911-348f-45a3-b79c-e3cb11642035') Unexpect= ed error (task:875)</div> <div class=3D"">Traceback (most recent call last):</div> <div class=3D""> File "/usr/lib/python2.7/site-packages/vdsm/sto= rage/task.py", line 882, in _run</div> <div class=3D""> return fn(*args, **kargs)</div> <div class=3D""> File "<string>", line 2, in activate= StorageDomain</div> <div class=3D""> File "/usr/lib/python2.7/site-packages/vdsm/com= mon/api.py", line 48, in method</div> <div class=3D""> ret =3D func(*args, **kwargs)</div> <div class=3D""> File "/usr/lib/python2.7/site-packages/vdsm/sto= rage/hsm.py", line 1256, in activateStorageDomain</div> <div class=3D""> pool.activateSD(sdUUID)</div> <div class=3D""> File "/usr/lib/python2.7/site-packages/vdsm/sto= rage/securable.py", line 79, in wrapper</div> <div class=3D""> return method(self, *args, **kwargs)</div> <div class=3D""> File "/usr/lib/python2.7/site-packages/vdsm/sto= rage/sp.py", line 1130, in activateSD</div> <div class=3D""> self.validateAttachedDomain(dom)</div> <div class=3D""> File "/usr/lib/python2.7/site-packages/vdsm/sto= rage/securable.py", line 79, in wrapper</div> <div class=3D""> return method(self, *args, **kwargs)</div> <div class=3D""> File "/usr/lib/python2.7/site-packages/vdsm/sto= rage/sp.py", line 557, in validateAttachedDomain</div> <div class=3D""> raise se.StorageDomainNotInPool(self.spUUID, = dom.sdUUID)</div> <div class=3D"">StorageDomainNotInPool: Storage domain not in pool: u'domai= n=3Df5914df0-f46c-4cc0-b666-c929aa0225ae, pool=3D5a54bf81-0228-02bc-0358-00= 0000000304'</div> <div class=3D"">2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [st= orage.TaskManager.Task] (Task=3D'7f21f911-348f-45a3-b79c-e3cb11642035') abo= rting: Task is aborted: "Storage domain not in pool: u'domain=3Df5914d= f0-f46c-4cc0-b666-c929aa0225ae, pool=3D5a54bf81-0228-02bc-0358-000000000304= '" - code 353 (task:1181)</div> <div class=3D"">2018-05-02 09:16:03,462+0100 ERROR (jsonrpc/0) [storage= .Dispatcher] FINISH activateStorageDomain error=3DStorage domain not in poo= l: u'domain=3Df5914df0-f46c-4cc0-b666-c929aa0225ae, pool=3D5a54bf81-0228-02= bc-0358-000000000304' (dispatcher:82)</div> </div> <div class=3D""><br class=3D""> </div> <div class=3D"">engine.log</div> <div class=3D""><br class=3D""> </div> <div class=3D""> <div class=3D"">2018-05-02 09:16:02,326+01 INFO [org.ovirt.engine= .core.bll.storage.domain.ActivateStorageDomainCommand] (default task-20) [9= 3433989-8e26-48a9-bd3a-2ab95f296c08] Lock Acquired to object 'EngineLock:{e= xclusiveLocks=3D'[f5914df0-f46c-4cc0-b666-c929aa0225ae=3DSTORAGE]', sharedLocks=3D''}'</div> <div class=3D"">2018-05-02 09:16:02,376+01 INFO [org.ovirt.engine= .core.bll.storage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFac= tory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Running co= mmand: ActivateStorageDomainCommand internal: false. Entities affected : ID: f5914df0-f46c-4cc0-b666-c929aa0225ae = Type: StorageAction group MANIPULATE_STORAGE_DOMA</div> <div class=3D"">IN with role type ADMIN</div> <div class=3D"">2018-05-02 09:16:02,385+01 INFO [org.ovirt.engine= .core.bll.storage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFac= tory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Lock freed= to object 'EngineLock:{exclusiveLocks=3D'[f5914df0-f46c-4cc0-b666-c929aa02= 25ae=3DSTORAGE]', sharedLocks=3D''}'</div> <div class=3D"">2018-05-02 09:16:02,385+01 INFO [org.ovirt.engine= .core.bll.storage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFac= tory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] ActivateSt= orage Domain. Before Connect all hosts to pool. Time: Wed May 02 09:16:02 BST 2018</div> <div class=3D"">2018-05-02 09:16:02,407+01 INFO [org.ovirt.engine= .core.bll.storage.connection.ConnectStorageToVdsCommand] (EE-ManagedThreadF= actory-engine-Thread-33456) [40a82b47] Running command: ConnectStorageToVds= Command internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CR= EATE_STORAGE_DOMAIN with role type ADMIN</div> <div class=3D"">2018-05-02 09:16:02,421+01 INFO [org.ovirt.engine= .core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedThread= Factory-engine-Thread-33456) [40a82b47] START, ConnectStorageServerVDSComma= nd(HostName =3D virtA003, StorageServerConnectionManagementVDSParameters:{h= ostId=3D'fe2861fc-2b47-4807-b054-470198eda473', storagePoolId=3D'00000000-0000-0000-0000-000000</div> <div class=3D"">000000', storageType=3D'NFS', connectionList=3D'[StorageSer= verConnections:{id=3D'da392861-aedc-4f1e-97f4-6919fb01f1e9', connection=3D'= backoffice01.cluster:/vm-iso', iqn=3D'null', vfsType=3D'null', mountOptions= =3D'null', nfsVersion=3D'null', nfsRetrans=3D'null', nfsTimeo=3D'null', iface=3D'null', netIfaceName=3D'null'}]'}), log id: 23c= e648f</div> <div class=3D"">2018-05-02 09:16:02,446+01 INFO [org.ovirt.engine= .core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedThread= Factory-engine-Thread-33456) [40a82b47] FINISH, ConnectStorageServerVDSComm= and, return: {da392861-aedc-4f1e-97f4-6919fb01f1e9=3D0}, log id: 23ce648f</div> <div class=3D"">2018-05-02 09:16:02,450+01 INFO [org.ovirt.engine= .core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThrea= dFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] START,= ActivateStorageDomainVDSCommand( ActivateStorageDomainVDSCommandParameters= :{storagePoolId=3D'5a54bf81-0228-02bc-0358-000000000304', ignoreFailoverLimit=3D'false', stor</div> <div class=3D"">ageDomainId=3D'f5914df0-f46c-4cc0-b666-c929aa0225ae'}), log= id: 5c864594</div> <div class=3D"">2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core= .dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-e= ngine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID: IRS_BR= OKER_COMMAND_FAILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain not in pool: u'domain=3Df5= 914df0-f46c-4cc0-b666-c929aa0225ae, pool=3D5a5</div> <div class=3D"">4bf81-0228-02bc-0358-000000000304'</div> <div class=3D"">2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core= .vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFact= ory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command 'Ac= tivateStorageDomainVDSCommand( ActivateStorageDomainVDSCommandParameters:{s= toragePoolId=3D'5a54bf81-0228-02bc-0358-000000000304', ignoreFailoverLimit=3D'false', st</div> <div class=3D"">orageDomainId=3D'f5914df0-f46c-4cc0-b666-c929aa0225ae'})' e= xecution failed: IRSGenericException: IRSErrorException: Storage domain not= in pool: u'domain=3Df5914df0-f46c-4cc0-b666-c929aa0225ae, pool=3D5a54bf81-= 0228-02bc-0358-000000000304'</div> <div class=3D"">2018-05-02 09:16:02,635+01 INFO [org.ovirt.engine= .core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThrea= dFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] FINISH= , ActivateStorageDomainVDSCommand, log id: 5c864594</div> <div class=3D"">2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core= .bll.storage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-= engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command 'org.ov= irt.engine.core.bll.storage.domain.ActivateStorageDomainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.irsbroker.IrsOper= ationFailedNoFailove</div> <div class=3D"">rException: IRSGenericException: IRSErrorException: Storage= domain not in pool: u'domain=3Df5914df0-f46c-4cc0-b666-c929aa0225ae, pool= =3D5a54bf81-0228-02bc-0358-000000000304' (Failed with error StorageDomainNo= tInPool and code 353)</div> <div class=3D"">2018-05-02 09:16:02,636+01 INFO [org.ovirt.engine= .core.bll.storage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFac= tory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command [i= d=3D22b0f3c1-9a09-4e26-8096-d83465c8f4ee]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.Stora= gePoolIsoMap; snapshot: EntityStatus</div> <div class=3D"">Snapshot:{id=3D'StoragePoolIsoMapId:{storagePoolId=3D'5a54b= f81-0228-02bc-0358-000000000304', storageId=3D'f5914df0-f46c-4cc0-b666-c929= aa0225ae'}', status=3D'Maintenance'}.</div> <div class=3D"">2018-05-02 09:16:02,660+01 ERROR [org.ovirt.engine.core= .dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-e= ngine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID: USER_A= CTIVATE_STORAGE_DOMAIN_FAILED(967), Failed to activate Storage Domain VMISOs (Data Center Default) by admin@internal-= authz</div> </div> <div class=3D""><br class=3D""> <div class=3D""> <div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; text-align: star= t; text-indent: 0px; text-transform: none; white-space: normal; word-spacin= g: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp= -mode: space; -webkit-line-break: after-white-space;" class=3D""> <div style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line= -break: after-white-space;" class=3D""> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Regards,</div> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Callum</div> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> <br class=3D""> </div> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> --<br class=3D""> <br class=3D""> Callum Smith</div> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Research Computing Core</div> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Wellcome Trust Centre for Human Genetics<br class=3D""> University of Oxford<br class=3D""> e. <a href=3D"mailto:callum@well.ox.ac.uk" class=3D"">callum@well.ox.ac.uk<= /a></div> </div> </div> </div> <div><br class=3D""> <blockquote type=3D"cite" class=3D""> <div class=3D"">On 2 May 2018, at 08:44, Fred Rolland <<a href=3D"mailto= :frolland@redhat.com" class=3D"">frolland@redhat.com</a>> wrote:</div> <br class=3D"Apple-interchange-newline"> <div class=3D""> <div dir=3D"ltr" class=3D""> <div class=3D""> <div class=3D""> <div class=3D""> <div class=3D"">Hi,<br class=3D""> <br class=3D""> </div> Can you provide logs from engine and Vdsm(SPM)?<br class=3D""> </div> What is the state now?<br class=3D""> <br class=3D""> </div> Thanks,<br class=3D""> </div> Fred<br class=3D""> </div> <div class=3D"gmail_extra"><br class=3D""> <div class=3D"gmail_quote">On Tue, May 1, 2018 at 4:11 PM, Callum Smith <sp= an dir=3D"ltr" class=3D""> <<a href=3D"mailto:callum@well.ox.ac.uk" target=3D"_blank" class=3D"">ca= llum@well.ox.ac.uk</a>></span> wrote:<br class=3D""> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <div style=3D"word-wrap:break-word;line-break:after-white-space" class=3D""= the ISO storage domain manually? An NFS mount didn't add it back to the "pool" unfortunately.</div> <div class=3D""><br class=3D""> </div> <div class=3D"">On a separate note, is it possible to migrate this storage = to a new location? And if so how.</div> <div class=3D""><br class=3D""> <div class=3D""> <div style=3D"letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: br= eak-word;" class=3D""> <div style=3D"word-wrap:break-word" class=3D""> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px;" class=3D""> Regards,</div> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px;" class=3D""> Callum</div> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px;" class=3D""> <br class=3D""> </div> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px;" class=3D""> --<br class=3D""> <br class=3D""> Callum Smith</div> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px;" class=3D""> Research Computing Core</div> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px;" class=3D""> Wellcome Trust Centre for Human Genetics<br class=3D""> University of Oxford<br class=3D""> e. <a href=3D"mailto:callum@well.ox.ac.uk" target=3D"_blank" class=3D"">cal= lum@well.ox.ac.uk</a></div> </div> </div> </div> <br class=3D""> </div> </div> <br class=3D""> ______________________________<wbr class=3D"">_________________<br class=3D= ""> Users mailing list<br class=3D""> <a href=3D"mailto:Users@ovirt.org" class=3D"">Users@ovirt.org</a><br class= =3D""> <a href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer= " target=3D"_blank" class=3D"">http://lists.ovirt.org/<wbr class=3D"">mailm= an/listinfo/users</a><br class=3D""> <br class=3D""> </blockquote> </div> <br class=3D""> </div> </div> </blockquote> </div> <br class=3D""> </div> </div> </body> </html> --_000_F4071341D5B9476A9A215EAFC436F6C1welloxacuk_--
Which version are you using? Can you provide the whole log? For some reason, it looks like the Vdsm thinks that the Storage Domain is not part of the pool. On Wed, May 2, 2018 at 11:20 AM, Callum Smith <callum@well.ox.ac.uk> wrote:
State is maintenance for the ISOs storage. I've extracted what is hopefully the relevant bits of the log.
VDSM.log (SPM)
2018-05-02 09:16:03,455+0100 INFO (ioprocess communication (179084)) [IOProcess] Starting ioprocess (__init__:447) 2018-05-02 09:16:03,456+0100 INFO (ioprocess communication (179091)) [IOProcess] Starting ioprocess (__init__:447) 2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [vdsm.api] FINISH activateStorageDomain error=Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' from=::ffff:192.168.64.254,58968, flow_id=93433989-8e26-48a9-bd3a-2ab95f296c08, task_id=7f21f911-348f-45a3-b79c-e3cb11642035 (api:50) 2018-05-02 09:16:03,461+0100 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='7f21f911-348f-45a3-b79c-e3cb11642035') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in activateStorageDomain File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1256, in activateStorageDomain pool.activateSD(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1130, in activateSD self.validateAttachedDomain(dom) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 557, in validateAttachedDomain raise se.StorageDomainNotInPool(self.spUUID, dom.sdUUID) StorageDomainNotInPool: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' 2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [storage.TaskManager.Task] (Task='7f21f911-348f-45a3-b79c-e3cb11642035') aborting: Task is aborted: "Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304'" - code 353 (task:1181) 2018-05-02 09:16:03,462+0100 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH activateStorageDomain error=Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' (dispatcher:82)
engine.log
2018-05-02 09:16:02,326+01 INFO [org.ovirt.engine.core.bll. storage.domain.ActivateStorageDomainCommand] (default task-20) [93433989-8e26-48a9-bd3a-2ab95f296c08] Lock Acquired to object 'EngineLock:{exclusiveLocks='[f5914df0-f46c-4cc0-b666-c929aa0225ae=STORAGE]', sharedLocks=''}' 2018-05-02 09:16:02,376+01 INFO [org.ovirt.engine.core.bll. storage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Running command: ActivateStorageDomainCommand internal: false. Entities affected : ID: f5914df0-f46c-4cc0-b666-c929aa0225ae Type: StorageAction group MANIPULATE_STORAGE_DOMA IN with role type ADMIN 2018-05-02 09:16:02,385+01 INFO [org.ovirt.engine.core.bll. storage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Lock freed to object 'EngineLock:{exclusiveLocks='[f5914df0-f46c-4cc0-b666-c929aa0225ae=STORAGE]', sharedLocks=''}' 2018-05-02 09:16:02,385+01 INFO [org.ovirt.engine.core.bll. storage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] ActivateStorage Domain. Before Connect all hosts to pool. Time: Wed May 02 09:16:02 BST 2018 2018-05-02 09:16:02,407+01 INFO [org.ovirt.engine.core.bll. storage.connection.ConnectStorageToVdsCommand] (EE-ManagedThreadFactory-engine-Thread-33456) [40a82b47] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2018-05-02 09:16:02,421+01 INFO [org.ovirt.engine.core. vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33456) [40a82b47] START, ConnectStorageServerVDSCommand(HostName = virtA003, StorageServerConnectionManagementVDSParameters:{hostId=' fe2861fc-2b47-4807-b054-470198eda473', storagePoolId='00000000-0000- 0000-0000-000000 000000', storageType='NFS', connectionList='[ StorageServerConnections:{id='da392861-aedc-4f1e-97f4-6919fb01f1e9', connection='backoffice01.cluster:/vm-iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 23ce648f 2018-05-02 09:16:02,446+01 INFO [org.ovirt.engine.core. vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33456) [40a82b47] FINISH, ConnectStorageServerVDSCommand, return: {da392861-aedc-4f1e-97f4-6919fb01f1e9=0}, log id: 23ce648f 2018-05-02 09:16:02,450+01 INFO [org.ovirt.engine.core. vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] START, ActivateStorageDomainVDSCommand( ActivateStorageDomainVDSComman dParameters:{storagePoolId='5a54bf81-0228-02bc-0358-000000000304', ignoreFailoverLimit='false', stor ageDomainId='f5914df0-f46c-4cc0-b666-c929aa0225ae'}), log id: 5c864594 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a5 4bf81-0228-02bc-0358-000000000304' 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core. vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command 'ActivateStorageDomainVDSCommand( ActivateStorageDomainVDSComman dParameters:{storagePoolId='5a54bf81-0228-02bc-0358-000000000304', ignoreFailoverLimit='false', st orageDomainId='f5914df0-f46c-4cc0-b666-c929aa0225ae'})' execution failed: IRSGenericException: IRSErrorException: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' 2018-05-02 09:16:02,635+01 INFO [org.ovirt.engine.core. vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] FINISH, ActivateStorageDomainVDSCommand, log id: 5c864594 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.bll. storage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command 'org.ovirt.engine.core.bll. storage.domain.ActivateStorageDomainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.irsbroker.IrsOperationFailedNoFailove rException: IRSGenericException: IRSErrorException: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' (Failed with error StorageDomainNotInPool and code 353) 2018-05-02 09:16:02,636+01 INFO [org.ovirt.engine.core.bll. storage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command [id=22b0f3c1-9a09-4e26-8096-d83465c8f4ee]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: EntityStatus Snapshot:{id='StoragePoolIsoMapId:{storagePoolId='5a54bf81-0228-02bc-0358-000000000304', storageId='f5914df0-f46c-4cc0-b666-c929aa0225ae'}', status='Maintenance'}. 2018-05-02 09:16:02,660+01 ERROR [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID: USER_ACTIVATE_STORAGE_DOMAIN_FAILED(967), Failed to activate Storage Domain VMISOs (Data Center Default) by admin@internal-authz
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk
On 2 May 2018, at 08:44, Fred Rolland <frolland@redhat.com> wrote:
Hi,
Can you provide logs from engine and Vdsm(SPM)? What is the state now?
Thanks, Fred
On Tue, May 1, 2018 at 4:11 PM, Callum Smith <callum@well.ox.ac.uk> wrote:
Dear All,
It appears that clicking "detach" on the ISO storage domain is a really bad idea. This has gotten half way through the procedure and now can't be recovered from. Is there any advice for re-attaching the ISO storage domain manually? An NFS mount didn't add it back to the "pool" unfortunately.
On a separate note, is it possible to migrate this storage to a new location? And if so how.
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--_000_288C5B8ED9BC48E5A130EA2F9FA9E942welloxacuk_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable This is on 4.2.0.2-1, I've linked the main logs to dropbox simply because t= hey're big, full of noise right now. https://www.dropbox.com/s/f8q3m5amro2a1b2/engine.log?dl=3D0 https://www.dropbox.com/s/uods85jk65halo3/vdsm.log?dl=3D0 Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> On 2 May 2018, at 10:43, Fred Rolland <frolland@redhat.com<mailto:frolland@= redhat.com>> wrote: Which version are you using? Can you provide the whole log? For some reason, it looks like the Vdsm thinks that the Storage Domain is n= ot part of the pool. On Wed, May 2, 2018 at 11:20 AM, Callum Smith <callum@well.ox.ac.uk<mailto:= callum@well.ox.ac.uk>> wrote: State is maintenance for the ISOs storage. I've extracted what is hopefully= the relevant bits of the log. VDSM.log (SPM) 2018-05-02 09:16:03,455+0100 INFO (ioprocess communication (179084)) [IOPr= ocess] Starting ioprocess (__init__:447) 2018-05-02 09:16:03,456+0100 INFO (ioprocess communication (179091)) [IOPr= ocess] Starting ioprocess (__init__:447) 2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [vdsm.api] FINISH activateSt= orageDomain error=3DStorage domain not in pool: u'domain=3Df5914df0-f46c-4c= c0-b666-c929aa0225ae, pool=3D5a54bf81-0228-02bc-0358-000000000304' from=3D:= :ffff:192.168.64.254,58968, flow_id=3D93433989-8e26-48a9-bd3a-2ab95f296c08,= task_id=3D7f21f911-348f-45a3-b79c-e3cb11642035 (api:50) 2018-05-02 09:16:03,461+0100 ERROR (jsonrpc/0) [storage.TaskManager.Task] (= Task=3D'7f21f911-348f-45a3-b79c-e3cb11642035') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, i= n _run return fn(*args, **kargs) File "<string>", line 2, in activateStorageDomain File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in m= ethod ret =3D func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1256, i= n activateStorageDomain pool.activateSD(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 7= 9, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1130, in= activateSD self.validateAttachedDomain(dom) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 7= 9, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 557, in = validateAttachedDomain raise se.StorageDomainNotInPool(self.spUUID, dom.sdUUID) StorageDomainNotInPool: Storage domain not in pool: u'domain=3Df5914df0-f46= c-4cc0-b666-c929aa0225ae, pool=3D5a54bf81-0228-02bc-0358-000000000304' 2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [storage.TaskManager.Task] (= Task=3D'7f21f911-348f-45a3-b79c-e3cb11642035') aborting: Task is aborted: "= Storage domain not in pool: u'domain=3Df5914df0-f46c-4cc0-b666-c929aa0225ae= , pool=3D5a54bf81-0228-02bc-0358-000000000304'" - code 353 (task:1181) 2018-05-02 09:16:03,462+0100 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH = activateStorageDomain error=3DStorage domain not in pool: u'domain=3Df5914d= f0-f46c-4cc0-b666-c929aa0225ae, pool=3D5a54bf81-0228-02bc-0358-000000000304= ' (dispatcher:82) engine.log 2018-05-02 09:16:02,326+01 INFO [org.ovirt.engine.core.bll.storage.domain.= ActivateStorageDomainCommand] (default task-20) [93433989-8e26-48a9-bd3a-2a= b95f296c08] Lock Acquired to object 'EngineLock:{exclusiveLocks=3D'[f5914df= 0-f46c-4cc0-b666-c929aa0225ae=3DSTORAGE]', sharedLocks=3D''}' 2018-05-02 09:16:02,376+01 INFO [org.ovirt.engine.core.bll.storage.domain.= ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455)= [93433989-8e26-48a9-bd3a-2ab95f296c08] Running command: ActivateStorageDom= ainCommand internal: false. Entities affected : ID: f5914df0-f46c-4cc0-b66= 6-c929aa0225ae Type: StorageAction group MANIPULATE_STORAGE_DOMA IN with role type ADMIN 2018-05-02 09:16:02,385+01 INFO [org.ovirt.engine.core.bll.storage.domain.= ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455)= [93433989-8e26-48a9-bd3a-2ab95f296c08] Lock freed to object 'EngineLock:{e= xclusiveLocks=3D'[f5914df0-f46c-4cc0-b666-c929aa0225ae=3DSTORAGE]', sharedL= ocks=3D''}' 2018-05-02 09:16:02,385+01 INFO [org.ovirt.engine.core.bll.storage.domain.= ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455)= [93433989-8e26-48a9-bd3a-2ab95f296c08] ActivateStorage Domain. Before Conn= ect all hosts to pool. Time: Wed May 02 09:16:02 BST 2018 2018-05-02 09:16:02,407+01 INFO [org.ovirt.engine.core.bll.storage.connect= ion.ConnectStorageToVdsCommand] (EE-ManagedThreadFactory-engine-Thread-3345= 6) [40a82b47] Running command: ConnectStorageToVdsCommand internal: true. E= ntities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAc= tion group CREATE_STORAGE_DOMAIN with role type ADMIN 2018-05-02 09:16:02,421+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker= .ConnectStorageServerVDSCommand] (EE-ManagedThreadFactory-engine-Thread-334= 56) [40a82b47] START, ConnectStorageServerVDSCommand(HostName =3D virtA003,= StorageServerConnectionManagementVDSParameters:{hostId=3D'fe2861fc-2b47-48= 07-b054-470198eda473', storagePoolId=3D'00000000-0000-0000-0000-000000 000000', storageType=3D'NFS', connectionList=3D'[StorageServerConnections:{= id=3D'da392861-aedc-4f1e-97f4-6919fb01f1e9', connection=3D'backoffice01.clu= ster:/vm-iso', iqn=3D'null', vfsType=3D'null', mountOptions=3D'null', nfsVe= rsion=3D'null', nfsRetrans=3D'null', nfsTimeo=3D'null', iface=3D'null', net= IfaceName=3D'null'}]'}), log id: 23ce648f 2018-05-02 09:16:02,446+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker= .ConnectStorageServerVDSCommand] (EE-ManagedThreadFactory-engine-Thread-334= 56) [40a82b47] FINISH, ConnectStorageServerVDSCommand, return: {da392861-ae= dc-4f1e-97f4-6919fb01f1e9=3D0}, log id: 23ce648f 2018-05-02 09:16:02,450+01 INFO [org.ovirt.engine.core.vdsbroker.irsbroker= .ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33= 455) [93433989-8e26-48a9-bd3a-2ab95f296c08] START, ActivateStorageDomainVDS= Command( ActivateStorageDomainVDSCommandParameters:{storagePoolId=3D'5a54bf= 81-0228-02bc-0358-000000000304', ignoreFailoverLimit=3D'false', stor ageDomainId=3D'f5914df0-f46c-4cc0-b666-c929aa0225ae'}), log id: 5c864594 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditl= oghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-33455) = [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID: IRS_BROKER_COMMAND_FAILURE= (10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain not = in pool: u'domain=3Df5914df0-f46c-4cc0-b666-c929aa0225ae, pool=3D5a5 4bf81-0228-02bc-0358-000000000304' 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker= .ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33= 455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command 'ActivateStorageDomainV= DSCommand( ActivateStorageDomainVDSCommandParameters:{storagePoolId=3D'5a54= bf81-0228-02bc-0358-000000000304', ignoreFailoverLimit=3D'false', st orageDomainId=3D'f5914df0-f46c-4cc0-b666-c929aa0225ae'})' execution failed:= IRSGenericException: IRSErrorException: Storage domain not in pool: u'doma= in=3Df5914df0-f46c-4cc0-b666-c929aa0225ae, pool=3D5a54bf81-0228-02bc-0358-0= 00000000304' 2018-05-02 09:16:02,635+01 INFO [org.ovirt.engine.core.vdsbroker.irsbroker= .ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33= 455) [93433989-8e26-48a9-bd3a-2ab95f296c08] FINISH, ActivateStorageDomainVD= SCommand, log id: 5c864594 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.bll.storage.domain.= ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455)= [93433989-8e26-48a9-bd3a-2ab95f296c08] Command 'org.ovirt.engine.core.bll.= storage.domain.ActivateStorageDomainCommand' failed: EngineException: org.o= virt.engine.core.vdsbroker.irsbroker.IrsOperationFailedNoFailove rException: IRSGenericException: IRSErrorException: Storage domain not in p= ool: u'domain=3Df5914df0-f46c-4cc0-b666-c929aa0225ae, pool=3D5a54bf81-0228-= 02bc-0358-000000000304' (Failed with error StorageDomainNotInPool and code = 353) 2018-05-02 09:16:02,636+01 INFO [org.ovirt.engine.core.bll.storage.domain.= ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455)= [93433989-8e26-48a9-bd3a-2ab95f296c08] Command [id=3D22b0f3c1-9a09-4e26-80= 96-d83465c8f4ee]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core= .common.businessentities.StoragePoolIsoMap; snapshot: EntityStatus Snapshot:{id=3D'StoragePoolIsoMapId:{storagePoolId=3D'5a54bf81-0228-02bc-03= 58-000000000304', storageId=3D'f5914df0-f46c-4cc0-b666-c929aa0225ae'}', sta= tus=3D'Maintenance'}. 2018-05-02 09:16:02,660+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditl= oghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-33455) = [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID: USER_ACTIVATE_STORAGE_DOMA= IN_FAILED(967), Failed to activate Storage Domain VMISOs (Data Center Defau= lt) by admin@internal-authz Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> On 2 May 2018, at 08:44, Fred Rolland <frolland@redhat.com<mailto:frolland@= redhat.com>> wrote: Hi, Can you provide logs from engine and Vdsm(SPM)? What is the state now? Thanks, Fred On Tue, May 1, 2018 at 4:11 PM, Callum Smith <callum@well.ox.ac.uk<mailto:c= allum@well.ox.ac.uk>> wrote: Dear All, It appears that clicking "detach" on the ISO storage domain is a really bad= idea. This has gotten half way through the procedure and now can't be reco= vered from. Is there any advice for re-attaching the ISO storage domain man= ually? An NFS mount didn't add it back to the "pool" unfortunately. On a separate note, is it possible to migrate this storage to a new locatio= n? And if so how. Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users --_000_288C5B8ED9BC48E5A130EA2F9FA9E942welloxacuk_ Content-Type: text/html; charset="us-ascii" Content-ID: <995C5595881C87499E993EB0AD84590E@well.ox.ac.uk> Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
</head> <body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; line-break:= after-white-space;" class=3D""> This is on 4.2.0.2-1, I've linked the main logs to dropbox simply because t= hey're big, full of noise right now. <div class=3D""><a href=3D"https://www.dropbox.com/s/f8q3m5amro2a1b2/engine= .log?dl=3D0" class=3D"">https://www.dropbox.com/s/f8q3m5amro2a1b2/engine.lo= g?dl=3D0</a></div> <div class=3D""><a href=3D"https://www.dropbox.com/s/uods85jk65halo3/vdsm.l= og?dl=3D0" class=3D"">https://www.dropbox.com/s/uods85jk65halo3/vdsm.log?dl= =3D0</a></div> <div class=3D""><br class=3D""> <div class=3D""> <div class=3D""> <div class=3D""> <div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; text-align: star= t; text-indent: 0px; text-transform: none; white-space: normal; word-spacin= g: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp= -mode: space; -webkit-line-break: after-white-space;" class=3D""> <div style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line= -break: after-white-space;" class=3D""> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Regards,</div> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Callum</div> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> <br class=3D""> </div> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> --<br class=3D""> <br class=3D""> Callum Smith</div> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Research Computing Core</div> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Wellcome Trust Centre for Human Genetics<br class=3D""> University of Oxford<br class=3D""> e. <a href=3D"mailto:callum@well.ox.ac.uk" class=3D"">callum@well.ox.ac.uk<= /a></div> </div> </div> </div> <div><br class=3D""> <blockquote type=3D"cite" class=3D""> <div class=3D"">On 2 May 2018, at 10:43, Fred Rolland <<a href=3D"mailto= :frolland@redhat.com" class=3D"">frolland@redhat.com</a>> wrote:</div> <br class=3D"Apple-interchange-newline"> <div class=3D""> <div dir=3D"ltr" class=3D""> <div class=3D"">Which version are you using?<br class=3D""> Can you provide the whole log?<br class=3D""> <br class=3D""> </div> For some reason, it looks like the Vdsm thinks that the Storage Domain is n= ot part of the pool.<br class=3D""> </div> <div class=3D"gmail_extra"><br class=3D""> <div class=3D"gmail_quote">On Wed, May 2, 2018 at 11:20 AM, Callum Smith <s= pan dir=3D"ltr" class=3D""> <<a href=3D"mailto:callum@well.ox.ac.uk" target=3D"_blank" class=3D"">ca= llum@well.ox.ac.uk</a>></span> wrote:<br class=3D""> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <div style=3D"word-wrap:break-word;line-break:after-white-space" class=3D""=
State is maintenance for the ISOs storage. I've extracted what is hopefull= y the relevant bits of the log. <div class=3D""> <div class=3D""><br class=3D""> </div> <div class=3D"">VDSM.log (SPM)</div> <div class=3D""><br class=3D""> </div> <div class=3D""> <div class=3D"">2018-05-02 09:16:03,455+0100 INFO (ioprocess comm= unication (179084)) [IOProcess] Starting ioprocess (__init__:447)</div> <div class=3D"">2018-05-02 09:16:03,456+0100 INFO (ioprocess comm= unication (179091)) [IOProcess] Starting ioprocess (__init__:447)</div> <div class=3D"">2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [vd= sm.api] FINISH activateStorageDomain error=3DStorage domain not in pool: u'= domain=3Df5914df0-f46c-4cc0-<wbr class=3D"">b666-c929aa0225ae, pool=3D5a54b= f81-0228-02bc-0358-<wbr class=3D"">000000000304' from=3D::ffff:192.168.64.2= 54,<wbr class=3D"">58968, flow_id=3D93433989-8e26-48a9-<wbr class=3D"">bd3a-2ab95f296c08, task_id=3D= 7f21f911-348f-45a3-<wbr class=3D"">b79c-e3cb11642035 (api:50)</div> <div class=3D"">2018-05-02 09:16:03,461+0100 ERROR (jsonrpc/0) [storage= .TaskManager.Task] (Task=3D'7f21f911-348f-45a3-<wbr class=3D"">b79c-e3cb116= 42035') Unexpected error (task:875)</div> <div class=3D"">Traceback (most recent call last):</div> <div class=3D""> File "/usr/lib/python2.7/site-<wbr class=3D"">p= ackages/vdsm/storage/task.py"<wbr class=3D"">, line 882, in _run</div> <div class=3D""> return fn(*args, **kargs)</div> <div class=3D""> File "<string>", line 2, in activate= StorageDomain</div> <div class=3D""> File "/usr/lib/python2.7/site-<wbr class=3D"">p= ackages/vdsm/common/api.py", line 48, in method</div> <div class=3D""> ret =3D func(*args, **kwargs)</div> <div class=3D""> File "/usr/lib/python2.7/site-<wbr class=3D"">p= ackages/vdsm/storage/hsm.py", line 1256, in activateStorageDomain</div=
businessentities.<wbr class=3D"">StoragePoolIsoMap; snapshot: EntityStatus= </div> <div class=3D"">Snapshot:{id=3D'<wbr class=3D"">StoragePoolIsoMapId:{<wbr c= lass=3D"">storagePoolId=3D'5a54bf81-0228-<wbr class=3D"">02bc-0358-00000000= 0304', storageId=3D'f5914df0-f46c-4cc0-<wbr class=3D"">b666-c929aa0225ae'}'= , status=3D'Maintenance'}.</div> <div class=3D"">2018-05-02 09:16:02,660+01 ERROR [org.ovirt.engine.core= .dal.<wbr class=3D"">dbbroker.auditloghandling.<wbr class=3D"">AuditLogDire= ctor] (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-33455) [934339= 89-8e26-48a9-bd3a-<wbr class=3D"">2ab95f296c08] EVENT_ID: USER_ACTIVATE_STORAGE_DOMAIN_<wbr class=3D"">FAILED(967), Failed= to activate Storage Domain VMISOs (Data Center Default) by admin@internal-= authz</div> </div> <div class=3D""><span class=3D""><br class=3D""> <div class=3D""> <div style=3D"letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: br= eak-word;" class=3D""> <div style=3D"word-wrap:break-word" class=3D""> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px;" class=3D""> Regards,</div> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px;" class=3D""> Callum</div> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px;" class=3D""> <br class=3D""> </div> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px;" class=3D""> --<br class=3D""> <br class=3D""> Callum Smith</div> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px;" class=3D""> Research Computing Core</div> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px;" class=3D""> Wellcome Trust Centre for Human Genetics<br class=3D""> University of Oxford<br class=3D""> e. <a href=3D"mailto:callum@well.ox.ac.uk" target=3D"_blank" class=3D"">cal= lum@well.ox.ac.uk</a></div> </div> </div> </div> </span> <div class=3D""> <div class=3D"h5"> <div class=3D""><br class=3D""> <blockquote type=3D"cite" class=3D""> <div class=3D"">On 2 May 2018, at 08:44, Fred Rolland <<a href=3D"mailto= :frolland@redhat.com" target=3D"_blank" class=3D"">frolland@redhat.com</a>&= gt; wrote:</div> <br class=3D"m_-2093273250354586386Apple-interchange-newline"> <div class=3D""> <div dir=3D"ltr" class=3D""> <div class=3D""> <div class=3D""> <div class=3D""> <div class=3D"">Hi,<br class=3D""> <br class=3D""> </div> Can you provide logs from engine and Vdsm(SPM)?<br class=3D""> </div> What is the state now?<br class=3D""> <br class=3D""> </div> Thanks,<br class=3D""> </div> Fred<br class=3D""> </div> <div class=3D"gmail_extra"><br class=3D""> <div class=3D"gmail_quote">On Tue, May 1, 2018 at 4:11 PM, Callum Smith <sp= an dir=3D"ltr" class=3D""> <<a href=3D"mailto:callum@well.ox.ac.uk" target=3D"_blank" class=3D"">ca= llum@well.ox.ac.uk</a>></span> wrote:<br class=3D""> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <div style=3D"word-wrap:break-word;line-break:after-white-space" class=3D""= Dear All, <div class=3D""><br class=3D""> </div> <div class=3D"">It appears that clicking "detach" on the ISO stor= age domain is a really bad idea. This has gotten half way through the proce= dure and now can't be recovered from. Is there any advice for re-attaching =
<div class=3D""> pool.activateSD(sdUUID)</div> <div class=3D""> File "/usr/lib/python2.7/site-<wbr class=3D"">p= ackages/vdsm/storage/<wbr class=3D"">securable.py", line 79, in wrappe= r</div> <div class=3D""> return method(self, *args, **kwargs)</div> <div class=3D""> File "/usr/lib/python2.7/site-<wbr class=3D"">p= ackages/vdsm/storage/sp.py", line 1130, in activateSD</div> <div class=3D""> self.validateAttachedDomain(<wbr class=3D"">d= om)</div> <div class=3D""> File "/usr/lib/python2.7/site-<wbr class=3D"">p= ackages/vdsm/storage/<wbr class=3D"">securable.py", line 79, in wrappe= r</div> <div class=3D""> return method(self, *args, **kwargs)</div> <div class=3D""> File "/usr/lib/python2.7/site-<wbr class=3D"">p= ackages/vdsm/storage/sp.py", line 557, in validateAttachedDomain</div> <div class=3D""> raise se.StorageDomainNotInPool(<wbr class=3D= "">self.spUUID, dom.sdUUID)</div> <div class=3D"">StorageDomainNotInPool: Storage domain not in pool: u'domai= n=3Df5914df0-f46c-4cc0-<wbr class=3D"">b666-c929aa0225ae, pool=3D5a54bf81-0= 228-02bc-0358-<wbr class=3D"">000000000304'</div> <div class=3D"">2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [st= orage.TaskManager.Task] (Task=3D'7f21f911-348f-45a3-<wbr class=3D"">b79c-e3= cb11642035') aborting: Task is aborted: "Storage domain not in pool: u= 'domain=3Df5914df0-f46c-4cc0-<wbr class=3D"">b666-c929aa0225ae, pool=3D5a54bf81-0228-02bc-0358-<wbr class=3D"">000000000304'" - code = 353 (task:1181)</div> <div class=3D"">2018-05-02 09:16:03,462+0100 ERROR (jsonrpc/0) [storage= .Dispatcher] FINISH activateStorageDomain error=3DStorage domain not in poo= l: u'domain=3Df5914df0-f46c-4cc0-<wbr class=3D"">b666-c929aa0225ae, pool=3D= 5a54bf81-0228-02bc-0358-<wbr class=3D"">000000000304' (dispatcher:82)</div> </div> <div class=3D""><br class=3D""> </div> <div class=3D"">engine.log</div> <div class=3D""><br class=3D""> </div> <div class=3D""> <div class=3D"">2018-05-02 09:16:02,326+01 INFO [org.ovirt.engine= .core.bll.<wbr class=3D"">storage.domain.<wbr class=3D"">ActivateStorageDom= ainCommand] (default task-20) [93433989-8e26-48a9-bd3a-<wbr class=3D"">2ab9= 5f296c08] Lock Acquired to object 'EngineLock:{exclusiveLocks=3D'[<wbr clas= s=3D"">f5914df0-f46c-4cc0-b666-<wbr class=3D"">c929aa0225ae=3DSTORAGE]', sharedLocks=3D''}'</div> <div class=3D"">2018-05-02 09:16:02,376+01 INFO [org.ovirt.engine= .core.bll.<wbr class=3D"">storage.domain.<wbr class=3D"">ActivateStorageDom= ainCommand] (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-33455) [= 93433989-8e26-48a9-bd3a-<wbr class=3D"">2ab95f296c08] Running command: ActivateStorageDomainCommand internal: false. Entities af= fected : ID: f5914df0-f46c-4cc0-b666-<wbr class=3D"">c929aa0225ae Typ= e: StorageAction group MANIPULATE_STORAGE_DOMA</div> <div class=3D"">IN with role type ADMIN</div> <div class=3D"">2018-05-02 09:16:02,385+01 INFO [org.ovirt.engine= .core.bll.<wbr class=3D"">storage.domain.<wbr class=3D"">ActivateStorageDom= ainCommand] (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-33455) [= 93433989-8e26-48a9-bd3a-<wbr class=3D"">2ab95f296c08] Lock freed to object 'EngineLock:{exclusiveLocks=3D'[<wbr class=3D"">f5914= df0-f46c-4cc0-b666-<wbr class=3D"">c929aa0225ae=3DSTORAGE]', sharedLocks=3D= ''}'</div> <div class=3D"">2018-05-02 09:16:02,385+01 INFO [org.ovirt.engine= .core.bll.<wbr class=3D"">storage.domain.<wbr class=3D"">ActivateStorageDom= ainCommand] (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-33455) [= 93433989-8e26-48a9-bd3a-<wbr class=3D"">2ab95f296c08] ActivateStorage Domain. Before Connect all hosts to pool. Time: Wed May 02= 09:16:02 BST 2018</div> <div class=3D"">2018-05-02 09:16:02,407+01 INFO [org.ovirt.engine= .core.bll.<wbr class=3D"">storage.connection.<wbr class=3D"">ConnectStorage= ToVdsCommand] (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-33456)= [40a82b47] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-<wbr= class=3D"">123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN wit= h role type ADMIN</div> <div class=3D"">2018-05-02 09:16:02,421+01 INFO [org.ovirt.engine= .core.<wbr class=3D"">vdsbroker.vdsbroker.<wbr class=3D"">ConnectStorageSer= verVDSCommand<wbr class=3D"">] (EE-ManagedThreadFactory-<wbr class=3D"">eng= ine-Thread-33456) [40a82b47] START, ConnectStorageServerVDSCommand<wbr clas= s=3D"">(HostName =3D virtA003, StorageServerConnectionManagem<wbr class=3D"">entVDSParamete= rs:{hostId=3D'<wbr class=3D"">fe2861fc-2b47-4807-b054-<wbr class=3D"">47019= 8eda473', storagePoolId=3D'00000000-0000-<wbr class=3D"">0000-0000-000000</= div> <div class=3D"">000000', storageType=3D'NFS', connectionList=3D'[<wbr class= =3D"">StorageServerConnections:{id=3D'<wbr class=3D"">da392861-aedc-4f1e-97= f4-<wbr class=3D"">6919fb01f1e9', connection=3D'backoffice01.<wbr class=3D"= ">cluster:/vm-iso', iqn=3D'null', vfsType=3D'null', mountOptions=3D'null', nfsVersion=3D'null', nfsRetrans=3D'null', nfsTimeo=3D'null', iface=3D'null= ', netIfaceName=3D'null'}]'}), log id: 23ce648f</div> <div class=3D"">2018-05-02 09:16:02,446+01 INFO [org.ovirt.engine= .core.<wbr class=3D"">vdsbroker.vdsbroker.<wbr class=3D"">ConnectStorageSer= verVDSCommand<wbr class=3D"">] (EE-ManagedThreadFactory-<wbr class=3D"">eng= ine-Thread-33456) [40a82b47] FINISH, ConnectStorageServerVDSCommand<wbr cla= ss=3D"">, return: {da392861-aedc-4f1e-97f4-<wbr class=3D"">6919fb01f1e9=3D0}, log id= : 23ce648f</div> <div class=3D"">2018-05-02 09:16:02,450+01 INFO [org.ovirt.engine= .core.<wbr class=3D"">vdsbroker.irsbroker.<wbr class=3D"">ActivateStorageDo= mainVDSComman<wbr class=3D"">d] (EE-ManagedThreadFactory-<wbr class=3D"">en= gine-Thread-33455) [93433989-8e26-48a9-bd3a-<wbr class=3D"">2ab95f296c08] START, ActivateStorageDomainVDSComman<wbr class=3D"">d( ActivateStorageDom= ainVDSComman<wbr class=3D"">dParameters:{storagePoolId=3D'<wbr class=3D"">5= a54bf81-0228-02bc-0358-<wbr class=3D"">000000000304', ignoreFailoverLimit= =3D'false', stor</div> <div class=3D"">ageDomainId=3D'f5914df0-f46c-<wbr class=3D"">4cc0-b666-c929= aa0225ae'}), log id: 5c864594</div> <div class=3D"">2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core= .dal.<wbr class=3D"">dbbroker.auditloghandling.<wbr class=3D"">AuditLogDire= ctor] (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-33455) [934339= 89-8e26-48a9-bd3a-<wbr class=3D"">2ab95f296c08] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,<wbr class=3D"">803), VDSM command= ActivateStorageDomainVDS failed: Storage domain not in pool: u'domain=3Df5= 914df0-f46c-4cc0-<wbr class=3D"">b666-c929aa0225ae, pool=3D5a5</div> <div class=3D"">4bf81-0228-02bc-0358-<wbr class=3D"">000000000304'</div> <div class=3D"">2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core= .<wbr class=3D"">vdsbroker.irsbroker.<wbr class=3D"">ActivateStorageDomainV= DSComman<wbr class=3D"">d] (EE-ManagedThreadFactory-<wbr class=3D"">engine-= Thread-33455) [93433989-8e26-48a9-bd3a-<wbr class=3D"">2ab95f296c08] Command '<wbr class=3D"">ActivateStorageDomainVDSComman<wbr class=3D"">d( = ActivateStorageDomainVDSComman<wbr class=3D"">dParameters:{storagePoolId=3D= '<wbr class=3D"">5a54bf81-0228-02bc-0358-<wbr class=3D"">000000000304', ign= oreFailoverLimit=3D'false', st</div> <div class=3D"">orageDomainId=3D'f5914df0-f46c-<wbr class=3D"">4cc0-b666-c9= 29aa0225ae'})' execution failed: IRSGenericException: IRSErrorException: St= orage domain not in pool: u'domain=3Df5914df0-f46c-4cc0-<wbr class=3D"">b66= 6-c929aa0225ae, pool=3D5a54bf81-0228-02bc-0358-<wbr class=3D"">000000000304= '</div> <div class=3D"">2018-05-02 09:16:02,635+01 INFO [org.ovirt.engine= .core.<wbr class=3D"">vdsbroker.irsbroker.<wbr class=3D"">ActivateStorageDo= mainVDSComman<wbr class=3D"">d] (EE-ManagedThreadFactory-<wbr class=3D"">en= gine-Thread-33455) [93433989-8e26-48a9-bd3a-<wbr class=3D"">2ab95f296c08] FINISH, ActivateStorageDomainVDSComman<wbr class=3D"">d, log id: 5c864594<= /div> <div class=3D"">2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core= .bll.<wbr class=3D"">storage.domain.<wbr class=3D"">ActivateStorageDomainCo= mmand] (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-33455) [93433= 989-8e26-48a9-bd3a-<wbr class=3D"">2ab95f296c08] Command 'org.ovirt.engine.core.bll.<wbr class=3D"">storage.domain.<wbr cla= ss=3D"">ActivateStorageDomainCommand' failed: EngineException: org.ovirt.en= gine.core.<wbr class=3D"">vdsbroker.irsbroker.<wbr class=3D"">IrsOperationF= ailedNoFailove</div> <div class=3D"">rException: IRSGenericException: IRSErrorException: Storage= domain not in pool: u'domain=3Df5914df0-f46c-4cc0-<wbr class=3D"">b666-c92= 9aa0225ae, pool=3D5a54bf81-0228-02bc-0358-<wbr class=3D"">000000000304' (Fa= iled with error StorageDomainNotInPool and code 353)</div> <div class=3D"">2018-05-02 09:16:02,636+01 INFO [org.ovirt.engine= .core.bll.<wbr class=3D"">storage.domain.<wbr class=3D"">ActivateStorageDom= ainCommand] (EE-ManagedThreadFactory-<wbr class=3D"">engine-Thread-33455) [= 93433989-8e26-48a9-bd3a-<wbr class=3D"">2ab95f296c08] Command [id=3D22b0f3c1-9a09-4e26-8096-<wbr class=3D"">d83465c8f4ee]: Compe= nsating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.<wbr class=3D""= the ISO storage domain manually? An NFS mount didn't add it back to the "pool" unfortunately.</div> <div class=3D""><br class=3D""> </div> <div class=3D"">On a separate note, is it possible to migrate this storage = to a new location? And if so how.</div> <div class=3D""><br class=3D""> <div class=3D""> <div style=3D"letter-spacing:normal;text-align:start;text-indent:0px;text-t= ransform:none;white-space:normal;word-spacing:0px;word-wrap:break-word" cla= ss=3D""> <div style=3D"word-wrap:break-word" class=3D""> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> Regards,</div> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> Callum</div> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> <br class=3D""> </div> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> --<br class=3D""> <br class=3D""> Callum Smith</div> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> Research Computing Core</div> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> Wellcome Trust Centre for Human Genetics<br class=3D""> University of Oxford<br class=3D""> e. <a href=3D"mailto:callum@well.ox.ac.uk" target=3D"_blank" class=3D"">cal= lum@well.ox.ac.uk</a></div> </div> </div> </div> <br class=3D""> </div> </div> <br class=3D""> ______________________________<wbr class=3D"">_________________<br class=3D= ""> Users mailing list<br class=3D""> <a href=3D"mailto:Users@ovirt.org" target=3D"_blank" class=3D"">Users@ovirt= .org</a><br class=3D""> <a href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer= " target=3D"_blank" class=3D"">http://lists.ovirt.org/mailman<wbr class=3D"= ">/listinfo/users</a><br class=3D""> <br class=3D""> </blockquote> </div> <br class=3D""> </div> </div> </blockquote> </div> <br class=3D""> </div> </div> </div> </div> </div> </blockquote> </div> <br class=3D""> </div> </div> </blockquote> </div> <br class=3D""> </div> </div> </div> </body> </html> --_000_288C5B8ED9BC48E5A130EA2F9FA9E942welloxacuk_--
Can you share the REST API data of the Storage domain and Data Center? Here an example of the URLs, you will need to replace with correct ids. http://MY-SERVER/ovirt-engine/api/v4/storagedomains/13461356-f6f7-4a58-9897-... http://MY-SERVER/ovirt-engine/api/v4/datacenters/5a5df553-022d-036d-01e8-000... On Wed, May 2, 2018 at 12:53 PM, Callum Smith <callum@well.ox.ac.uk> wrote:
This is on 4.2.0.2-1, I've linked the main logs to dropbox simply because they're big, full of noise right now. https://www.dropbox.com/s/f8q3m5amro2a1b2/engine.log?dl=0 https://www.dropbox.com/s/uods85jk65halo3/vdsm.log?dl=0
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk
On 2 May 2018, at 10:43, Fred Rolland <frolland@redhat.com> wrote:
Which version are you using? Can you provide the whole log?
For some reason, it looks like the Vdsm thinks that the Storage Domain is not part of the pool.
On Wed, May 2, 2018 at 11:20 AM, Callum Smith <callum@well.ox.ac.uk> wrote:
State is maintenance for the ISOs storage. I've extracted what is hopefully the relevant bits of the log.
VDSM.log (SPM)
2018-05-02 09:16:03,455+0100 INFO (ioprocess communication (179084)) [IOProcess] Starting ioprocess (__init__:447) 2018-05-02 09:16:03,456+0100 INFO (ioprocess communication (179091)) [IOProcess] Starting ioprocess (__init__:447) 2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [vdsm.api] FINISH activateStorageDomain error=Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' from=::ffff:192.168.64.254,58968, flow_id=93433989-8e26-48a9-bd3a-2ab95f296c08, task_id=7f21f911-348f-45a3-b79c-e3cb11642035 (api:50) 2018-05-02 09:16:03,461+0100 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='7f21f911-348f-45a3-b79c-e3cb11642035') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in activateStorageDomain File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1256, in activateStorageDomain pool.activateSD(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1130, in activateSD self.validateAttachedDomain(dom) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 557, in validateAttachedDomain raise se.StorageDomainNotInPool(self.spUUID, dom.sdUUID) StorageDomainNotInPool: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' 2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [storage.TaskManager.Task] (Task='7f21f911-348f-45a3-b79c-e3cb11642035') aborting: Task is aborted: "Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304'" - code 353 (task:1181) 2018-05-02 09:16:03,462+0100 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH activateStorageDomain error=Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' (dispatcher:82)
engine.log
2018-05-02 09:16:02,326+01 INFO [org.ovirt.engine.core.bll.st orage.domain.ActivateStorageDomainCommand] (default task-20) [93433989-8e26-48a9-bd3a-2ab95f296c08] Lock Acquired to object 'EngineLock:{exclusiveLocks='[f5914df0-f46c-4cc0-b666-c929aa0225ae=STORAGE]', sharedLocks=''}' 2018-05-02 09:16:02,376+01 INFO [org.ovirt.engine.core.bll.st orage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Running command: ActivateStorageDomainCommand internal: false. Entities affected : ID: f5914df0-f46c-4cc0-b666-c929aa0225ae Type: StorageAction group MANIPULATE_STORAGE_DOMA IN with role type ADMIN 2018-05-02 09:16:02,385+01 INFO [org.ovirt.engine.core.bll.st orage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Lock freed to object 'EngineLock:{exclusiveLocks='[f5914df0-f46c-4cc0-b666-c929aa0225ae=STORAGE]', sharedLocks=''}' 2018-05-02 09:16:02,385+01 INFO [org.ovirt.engine.core.bll.st orage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] ActivateStorage Domain. Before Connect all hosts to pool. Time: Wed May 02 09:16:02 BST 2018 2018-05-02 09:16:02,407+01 INFO [org.ovirt.engine.core.bll.st orage.connection.ConnectStorageToVdsCommand] (EE-ManagedThreadFactory-engine-Thread-33456) [40a82b47] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2018-05-02 09:16:02,421+01 INFO [org.ovirt.engine.core.vdsbro ker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33456) [40a82b47] START, ConnectStorageServerVDSCommand(HostName = virtA003, StorageServerConnectionManagementVDSParameters:{hostId='fe28 61fc-2b47-4807-b054-470198eda473', storagePoolId='00000000-0000-0 000-0000-000000 000000', storageType='NFS', connectionList='[StorageServer Connections:{id='da392861-aedc-4f1e-97f4-6919fb01f1e9', connection='backoffice01.cluster:/vm-iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 23ce648f 2018-05-02 09:16:02,446+01 INFO [org.ovirt.engine.core.vdsbro ker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33456) [40a82b47] FINISH, ConnectStorageServerVDSCommand, return: {da392861-aedc-4f1e-97f4-6919fb01f1e9=0}, log id: 23ce648f 2018-05-02 09:16:02,450+01 INFO [org.ovirt.engine.core.vdsbro ker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] START, ActivateStorageDomainVDSCommand( ActivateStorageDomainVDSComman dParameters:{storagePoolId='5a54bf81-0228-02bc-0358-000000000304', ignoreFailoverLimit='false', stor ageDomainId='f5914df0-f46c-4cc0-b666-c929aa0225ae'}), log id: 5c864594 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a5 4bf81-0228-02bc-0358-000000000304' 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.vdsbrok er.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command 'ActivateStorageDomainVDSCommand( ActivateStorageDomainVDSComman dParameters:{storagePoolId='5a54bf81-0228-02bc-0358-000000000304', ignoreFailoverLimit='false', st orageDomainId='f5914df0-f46c-4cc0-b666-c929aa0225ae'})' execution failed: IRSGenericException: IRSErrorException: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' 2018-05-02 09:16:02,635+01 INFO [org.ovirt.engine.core.vdsbro ker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] FINISH, ActivateStorageDomainVDSCommand, log id: 5c864594 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.bll.sto rage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command 'org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroke r.irsbroker.IrsOperationFailedNoFailove rException: IRSGenericException: IRSErrorException: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' (Failed with error StorageDomainNotInPool and code 353) 2018-05-02 09:16:02,636+01 INFO [org.ovirt.engine.core.bll.st orage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command [id=22b0f3c1-9a09-4e26-8096-d83465c8f4ee]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.b usinessentities.StoragePoolIsoMap; snapshot: EntityStatus Snapshot:{id='StoragePoolIsoMapId:{storagePoolId='5a54bf81-0228-02bc-0358-000000000304', storageId='f5914df0-f46c-4cc0-b666-c929aa0225ae'}', status='Maintenance'}. 2018-05-02 09:16:02,660+01 ERROR [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID: USER_ACTIVATE_STORAGE_DOMAIN_FAILED(967), Failed to activate Storage Domain VMISOs (Data Center Default) by admin@internal-authz
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk
On 2 May 2018, at 08:44, Fred Rolland <frolland@redhat.com> wrote:
Hi,
Can you provide logs from engine and Vdsm(SPM)? What is the state now?
Thanks, Fred
On Tue, May 1, 2018 at 4:11 PM, Callum Smith <callum@well.ox.ac.uk> wrote:
Dear All,
It appears that clicking "detach" on the ISO storage domain is a really bad idea. This has gotten half way through the procedure and now can't be recovered from. Is there any advice for re-attaching the ISO storage domain manually? An NFS mount didn't add it back to the "pool" unfortunately.
On a separate note, is it possible to migrate this storage to a new location? And if so how.
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--_004_FB71A3FB6A03472B8BC9B2EB92C1C9C2welloxacuk_ Content-Type: multipart/alternative; boundary="_000_FB71A3FB6A03472B8BC9B2EB92C1C9C2welloxacuk_" --_000_FB71A3FB6A03472B8BC9B2EB92C1C9C2welloxacuk_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Attached, thank you for looking into this https://HOSTNAME/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666= -c929aa0225ae <?xml version=3D"1.0" encoding=3D"UTF-8" standalone=3D"yes"?> <storage_domain href=3D"/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4= cc0-b666-c929aa0225ae<view-source:https://virtman.cluster/ovirt-engine/api/= v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae>" id=3D"f5914df0-f46= c-4cc0-b666-c929aa0225ae"> <actions> <link href=3D"/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc= 0-b666-c929aa0225ae/isattached<view-source:https://virtman.cluster/ovirt-en= gine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/isattached>= " rel=3D"isattached"/> <link href=3D"/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc= 0-b666-c929aa0225ae/reduceluns<view-source:https://virtman.cluster/ovirt-en= gine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/reduceluns>= " rel=3D"reduceluns"/> <link href=3D"/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc= 0-b666-c929aa0225ae/refreshluns<view-source:https://virtman.cluster/ovirt-e= ngine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/refreshlun= s>" rel=3D"refreshluns"/> <link href=3D"/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc= 0-b666-c929aa0225ae/updateovfstore<view-source:https://virtman.cluster/ovir= t-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/updateo= vfstore>" rel=3D"updateovfstore"/> </actions> <name>VMISOs</name> <link href=3D"/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b6= 66-c929aa0225ae/disksnapshots<view-source:https://virtman.cluster/ovirt-eng= ine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/disksnapshot= s>" rel=3D"disksnapshots"/> <link href=3D"/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b6= 66-c929aa0225ae/files<view-source:https://virtman.cluster/ovirt-engine/api/= v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/files>" rel=3D"files= "/> <link href=3D"/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b6= 66-c929aa0225ae/diskprofiles<view-source:https://virtman.cluster/ovirt-engi= ne/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/diskprofiles>= " rel=3D"diskprofiles"/> <link href=3D"/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b6= 66-c929aa0225ae/storageconnections<view-source:https://virtman.cluster/ovir= t-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/storage= connections>" rel=3D"storageconnections"/> <link href=3D"/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b6= 66-c929aa0225ae/permissions<view-source:https://virtman.cluster/ovirt-engin= e/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/permissions>" = rel=3D"permissions"/> <available>11770357874688</available> <backup>false</backup> <committed>0</committed> <critical_space_action_blocker>5</critical_space_action_blocker> <discard_after_delete>false</discard_after_delete> <external_status>ok</external_status> <master>false</master> <storage> <address>backoffice01.cluster</address> <path>/vm-iso</path> <type>nfs</type> </storage> <storage_format>v1</storage_format> <supports_discard>false</supports_discard> <supports_discard_zeroes_data>false</supports_discard_zeroes_data> <type>iso</type> <used>38654705664</used> <warning_low_space_indicator>10</warning_low_space_indicator> <wipe_after_delete>false</wipe_after_delete> <data_centers> <data_center href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228= -02bc-0358-000000000304<view-source:https://virtman.cluster/ovirt-engine/ap= i/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304>" id=3D"5a54bf81-0228= -02bc-0358-000000000304"/> </data_centers> </storage_domain> https://HOSTNAME/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-00= 0000000304/storagedomains <?xml version=3D"1.0" encoding=3D"UTF-8" standalone=3D"yes"?> <storage_domains> <storage_domain href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-= 02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91a613ef98ba<= view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenters/5a54bf8= 1-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91a613= ef98ba>" id=3D"d9f8093b-71cd-423d-9b7d-91a613ef98ba"> <actions> <link href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02= bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91a613ef98ba/ac= tivate<view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenters/= 5a54bf81-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d= -91a613ef98ba/activate>" rel=3D"activate"/> <link href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02= bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91a613ef98ba/de= activate<view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenter= s/5a54bf81-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b= 7d-91a613ef98ba/deactivate>" rel=3D"deactivate"/> </actions> <name>tegile-virtman-backup</name> <link href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0= 358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91a613ef98ba/disks<= view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenters/5a54bf8= 1-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91a613= ef98ba/disks>" rel=3D"disks"/> <available>17519171600384</available> <backup>false</backup> <committed>0</committed> <critical_space_action_blocker>5</critical_space_action_blocker> <discard_after_delete>false</discard_after_delete> <external_status>ok</external_status> <master>false</master> <status>maintenance</status> <storage> <address>192.168.64.248</address> <nfs_version>auto</nfs_version> <path>/export/virtman/backup</path> <type>nfs</type> </storage> <storage_format>v1</storage_format> <supports_discard>false</supports_discard> <supports_discard_zeroes_data>false</supports_discard_zeroes_data> <type>export</type> <used>8589934592</used> <warning_low_space_indicator>10</warning_low_space_indicator> <wipe_after_delete>false</wipe_after_delete> <data_center href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228= -02bc-0358-000000000304<view-source:https://virtman.cluster/ovirt-engine/ap= i/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304>" id=3D"5a54bf81-0228= -02bc-0358-000000000304"/> <data_centers> <data_center href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-= 0228-02bc-0358-000000000304<view-source:https://virtman.cluster/ovirt-engin= e/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304>" id=3D"5a54bf81-= 0228-02bc-0358-000000000304"/> </data_centers> </storage_domain> <storage_domain href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-= 02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-67737dedd7a1<= view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenters/5a54bf8= 1-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-67737d= edd7a1>" id=3D"017dd620-65ec-4fa4-8d46-67737dedd7a1"> <actions> <link href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02= bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-67737dedd7a1/ac= tivate<view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenters/= 5a54bf81-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46= -67737dedd7a1/activate>" rel=3D"activate"/> <link href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02= bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-67737dedd7a1/de= activate<view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenter= s/5a54bf81-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d= 46-67737dedd7a1/deactivate>" rel=3D"deactivate"/> </actions> <name>VMStorage</name> <link href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0= 358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-67737dedd7a1/disks<= view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenters/5a54bf8= 1-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-67737d= edd7a1/disks>" rel=3D"disks"/> <available>11770357874688</available> <backup>false</backup> <committed>118111600640</committed> <critical_space_action_blocker>5</critical_space_action_blocker> <discard_after_delete>false</discard_after_delete> <external_status>ok</external_status> <master>true</master> <status>active</status> <storage> <address>backoffice01.cluster</address> <path>/vm-storage2</path> <type>nfs</type> </storage> <storage_format>v4</storage_format> <supports_discard>false</supports_discard> <supports_discard_zeroes_data>false</supports_discard_zeroes_data> <type>data</type> <used>38654705664</used> <warning_low_space_indicator>10</warning_low_space_indicator> <wipe_after_delete>false</wipe_after_delete> <data_center href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228= -02bc-0358-000000000304<view-source:https://virtman.cluster/ovirt-engine/ap= i/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304>" id=3D"5a54bf81-0228= -02bc-0358-000000000304"/> <data_centers> <data_center href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-= 0228-02bc-0358-000000000304<view-source:https://virtman.cluster/ovirt-engin= e/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304>" id=3D"5a54bf81-= 0228-02bc-0358-000000000304"/> </data_centers> </storage_domain> <storage_domain href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-= 02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b6f563a9fc17<= view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenters/5a54bf8= 1-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b6f563= a9fc17>" id=3D"e572c874-17f8-4ad8-9bac-b6f563a9fc17"> <actions> <link href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02= bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b6f563a9fc17/ac= tivate<view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenters/= 5a54bf81-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac= -b6f563a9fc17/activate>" rel=3D"activate"/> <link href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02= bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b6f563a9fc17/de= activate<view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenter= s/5a54bf81-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9b= ac-b6f563a9fc17/deactivate>" rel=3D"deactivate"/> </actions> <name>tegile-virtman</name> <link href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0= 358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b6f563a9fc17/disks<= view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenters/5a54bf8= 1-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b6f563= a9fc17/disks>" rel=3D"disks"/> <available>2190433320960</available> <backup>false</backup> <committed>226559524864</committed> <critical_space_action_blocker>5</critical_space_action_blocker> <discard_after_delete>false</discard_after_delete> <external_status>ok</external_status> <master>false</master> <status>active</status> <storage> <address>192.168.64.248</address> <nfs_version>auto</nfs_version> <path>/export/virtman/VirtualServerShare_1</path> <type>nfs</type> </storage> <storage_format>v4</storage_format> <supports_discard>false</supports_discard> <supports_discard_zeroes_data>false</supports_discard_zeroes_data> <type>data</type> <used>8589934592</used> <warning_low_space_indicator>10</warning_low_space_indicator> <wipe_after_delete>false</wipe_after_delete> <data_center href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228= -02bc-0358-000000000304<view-source:https://virtman.cluster/ovirt-engine/ap= i/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304>" id=3D"5a54bf81-0228= -02bc-0358-000000000304"/> <data_centers> <data_center href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-= 0228-02bc-0358-000000000304<view-source:https://virtman.cluster/ovirt-engin= e/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304>" id=3D"5a54bf81-= 0228-02bc-0358-000000000304"/> </data_centers> </storage_domain> <storage_domain href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-= 02bc-0358-000000000304/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae<= view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenters/5a54bf8= 1-0228-02bc-0358-000000000304/storagedomains/f5914df0-f46c-4cc0-b666-c929aa= 0225ae>" id=3D"f5914df0-f46c-4cc0-b666-c929aa0225ae"> <actions> <link href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02= bc-0358-000000000304/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/ac= tivate<view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenters/= 5a54bf81-0228-02bc-0358-000000000304/storagedomains/f5914df0-f46c-4cc0-b666= -c929aa0225ae/activate>" rel=3D"activate"/> <link href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02= bc-0358-000000000304/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/de= activate<view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenter= s/5a54bf81-0228-02bc-0358-000000000304/storagedomains/f5914df0-f46c-4cc0-b6= 66-c929aa0225ae/deactivate>" rel=3D"deactivate"/> </actions> <name>VMISOs</name> <available>11770357874688</available> <backup>false</backup> <committed>0</committed> <critical_space_action_blocker>5</critical_space_action_blocker> <discard_after_delete>false</discard_after_delete> <external_status>ok</external_status> <master>false</master> <status>maintenance</status> <storage> <address>backoffice01.cluster</address> <path>/vm-iso</path> <type>nfs</type> </storage> <storage_format>v1</storage_format> <supports_discard>false</supports_discard> <supports_discard_zeroes_data>false</supports_discard_zeroes_data> <type>iso</type> <used>38654705664</used> <warning_low_space_indicator>10</warning_low_space_indicator> <wipe_after_delete>false</wipe_after_delete> <data_center href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228= -02bc-0358-000000000304<view-source:https://virtman.cluster/ovirt-engine/ap= i/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304>" id=3D"5a54bf81-0228= -02bc-0358-000000000304"/> <data_centers> <data_center href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-= 0228-02bc-0358-000000000304<view-source:https://virtman.cluster/ovirt-engin= e/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304>" id=3D"5a54bf81-= 0228-02bc-0358-000000000304"/> </data_centers> </storage_domain> </storage_domains> [cid:921E5257-26F4-4E0D-83B0-68F996842D08@well.ox.ac.uk] Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> On 2 May 2018, at 14:46, Fred Rolland <frolland@redhat.com<mailto:frolland@= redhat.com>> wrote: Can you share the REST API data of the Storage domain and Data Center? Here an example of the URLs, you will need to replace with correct ids. http://MY-SERVER/ovirt-engine/api/v4/storagedomains/13461356-f6f7-4a58-9897= -2fac61ff40af<http://my-server/ovirt-engine/api/v4/storagedomains/13461356-= f6f7-4a58-9897-2fac61ff40af> http://MY-SERVER/ovirt-engine/api/v4/datacenters/5a5df553-022d-036d-01e8-00= 0000000071/storagedomains<http://my-server/ovirt-engine/api/v4/datacenters/= 5a5df553-022d-036d-01e8-000000000071/storagedomains> On Wed, May 2, 2018 at 12:53 PM, Callum Smith <callum@well.ox.ac.uk<mailto:= callum@well.ox.ac.uk>> wrote: This is on 4.2.0.2-1, I've linked the main logs to dropbox simply because t= hey're big, full of noise right now. https://www.dropbox.com/s/f8q3m5amro2a1b2/engine.log?dl=3D0 https://www.dropbox.com/s/uods85jk65halo3/vdsm.log?dl=3D0 Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> On 2 May 2018, at 10:43, Fred Rolland <frolland@redhat.com<mailto:frolland@= redhat.com>> wrote: Which version are you using? Can you provide the whole log? For some reason, it looks like the Vdsm thinks that the Storage Domain is n= ot part of the pool. On Wed, May 2, 2018 at 11:20 AM, Callum Smith <callum@well.ox.ac.uk<mailto:= callum@well.ox.ac.uk>> wrote: State is maintenance for the ISOs storage. I've extracted what is hopefully= the relevant bits of the log. VDSM.log (SPM) 2018-05-02 09:16:03,455+0100 INFO (ioprocess communication (179084)) [IOPr= ocess] Starting ioprocess (__init__:447) 2018-05-02 09:16:03,456+0100 INFO (ioprocess communication (179091)) [IOPr= ocess] Starting ioprocess (__init__:447) 2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [vdsm.api] FINISH activateSt= orageDomain error=3DStorage domain not in pool: u'domain=3Df5914df0-f46c-4c= c0-b666-c929aa0225ae, pool=3D5a54bf81-0228-02bc-0358-000000000304' from=3D:= :ffff:192.168.64.254,58968, flow_id=3D93433989-8e26-48a9-bd3a-2ab95f296c08,= task_id=3D7f21f911-348f-45a3-b79c-e3cb11642035 (api:50) 2018-05-02 09:16:03,461+0100 ERROR (jsonrpc/0) [storage.TaskManager.Task] (= Task=3D'7f21f911-348f-45a3-b79c-e3cb11642035') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, i= n _run return fn(*args, **kargs) File "<string>", line 2, in activateStorageDomain File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in m= ethod ret =3D func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1256, i= n activateStorageDomain pool.activateSD(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 7= 9, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1130, in= activateSD self.validateAttachedDomain(dom) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 7= 9, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 557, in = validateAttachedDomain raise se.StorageDomainNotInPool(self.spUUID, dom.sdUUID) StorageDomainNotInPool: Storage domain not in pool: u'domain=3Df5914df0-f46= c-4cc0-b666-c929aa0225ae, pool=3D5a54bf81-0228-02bc-0358-000000000304' 2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [storage.TaskManager.Task] (= Task=3D'7f21f911-348f-45a3-b79c-e3cb11642035') aborting: Task is aborted: "= Storage domain not in pool: u'domain=3Df5914df0-f46c-4cc0-b666-c929aa0225ae= , pool=3D5a54bf81-0228-02bc-0358-000000000304'" - code 353 (task:1181) 2018-05-02 09:16:03,462+0100 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH = activateStorageDomain error=3DStorage domain not in pool: u'domain=3Df5914d= f0-f46c-4cc0-b666-c929aa0225ae, pool=3D5a54bf81-0228-02bc-0358-000000000304= ' (dispatcher:82) engine.log 2018-05-02 09:16:02,326+01 INFO [org.ovirt.engine.core.bll.st<http://org.o= virt.engine.core.bll.st/>orage.domain.ActivateStorageDomainCommand] (defaul= t task-20) [93433989-8e26-48a9-bd3a-2ab95f296c08] Lock Acquired to object '= EngineLock:{exclusiveLocks=3D'[f5914df0-f46c-4cc0-b666-c929aa0225ae=3DSTORA= GE]', sharedLocks=3D''}' 2018-05-02 09:16:02,376+01 INFO [org.ovirt.engine.core.bll.st<http://org.o= virt.engine.core.bll.st/>orage.domain.ActivateStorageDomainCommand] (EE-Man= agedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c0= 8] Running command: ActivateStorageDomainCommand internal: false. Entities = affected : ID: f5914df0-f46c-4cc0-b666-c929aa0225ae Type: StorageAction gr= oup MANIPULATE_STORAGE_DOMA IN with role type ADMIN 2018-05-02 09:16:02,385+01 INFO [org.ovirt.engine.core.bll.st<http://org.o= virt.engine.core.bll.st/>orage.domain.ActivateStorageDomainCommand] (EE-Man= agedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c0= 8] Lock freed to object 'EngineLock:{exclusiveLocks=3D'[f5914df0-f46c-4cc0-= b666-c929aa0225ae=3DSTORAGE]', sharedLocks=3D''}' 2018-05-02 09:16:02,385+01 INFO [org.ovirt.engine.core.bll.st<http://org.o= virt.engine.core.bll.st/>orage.domain.ActivateStorageDomainCommand] (EE-Man= agedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c0= 8] ActivateStorage Domain. Before Connect all hosts to pool. Time: Wed May = 02 09:16:02 BST 2018 2018-05-02 09:16:02,407+01 INFO [org.ovirt.engine.core.bll.st<http://org.o= virt.engine.core.bll.st/>orage.connection.ConnectStorageToVdsCommand] (EE-M= anagedThreadFactory-engine-Thread-33456) [40a82b47] Running command: Connec= tStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000= -0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with= role type ADMIN 2018-05-02 09:16:02,421+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker= .ConnectStorageServerVDSCommand] (EE-ManagedThreadFactory-engine-Thread-334= 56) [40a82b47] START, ConnectStorageServerVDSCommand(HostName =3D virtA003,= StorageServerConnectionManagementVDSParameters:{hostId=3D'fe2861fc-2b47-48= 07-b054-470198eda473', storagePoolId=3D'00000000-0000-0000-0000-000000 000000', storageType=3D'NFS', connectionList=3D'[StorageServerConnections:{= id=3D'da392861-aedc-4f1e-97f4-6919fb01f1e9', connection=3D'backoffice01.clu= ster:/vm-iso', iqn=3D'null', vfsType=3D'null', mountOptions=3D'null', nfsVe= rsion=3D'null', nfsRetrans=3D'null', nfsTimeo=3D'null', iface=3D'null', net= IfaceName=3D'null'}]'}), log id: 23ce648f 2018-05-02 09:16:02,446+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker= .ConnectStorageServerVDSCommand] (EE-ManagedThreadFactory-engine-Thread-334= 56) [40a82b47] FINISH, ConnectStorageServerVDSCommand, return: {da392861-ae= dc-4f1e-97f4-6919fb01f1e9=3D0}, log id: 23ce648f 2018-05-02 09:16:02,450+01 INFO [org.ovirt.engine.core.vdsbroker.irsbroker= .ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33= 455) [93433989-8e26-48a9-bd3a-2ab95f296c08] START, ActivateStorageDomainVDS= Command( ActivateStorageDomainVDSCommandParameters:{storagePoolId=3D'5a54bf= 81-0228-02bc-0358-000000000304', ignoreFailoverLimit=3D'false', stor ageDomainId=3D'f5914df0-f46c-4cc0-b666-c929aa0225ae'}), log id: 5c864594 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditl= oghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-33455) = [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID: IRS_BROKER_COMMAND_FAILURE= (10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain not = in pool: u'domain=3Df5914df0-f46c-4cc0-b666-c929aa0225ae, pool=3D5a5 4bf81-0228-02bc-0358-000000000304' 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker= .ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33= 455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command 'ActivateStorageDomainV= DSCommand( ActivateStorageDomainVDSCommandParameters:{storagePoolId=3D'5a54= bf81-0228-02bc-0358-000000000304', ignoreFailoverLimit=3D'false', st orageDomainId=3D'f5914df0-f46c-4cc0-b666-c929aa0225ae'})' execution failed:= IRSGenericException: IRSErrorException: Storage domain not in pool: u'doma= in=3Df5914df0-f46c-4cc0-b666-c929aa0225ae, pool=3D5a54bf81-0228-02bc-0358-0= 00000000304' 2018-05-02 09:16:02,635+01 INFO [org.ovirt.engine.core.vdsbroker.irsbroker= .ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33= 455) [93433989-8e26-48a9-bd3a-2ab95f296c08] FINISH, ActivateStorageDomainVD= SCommand, log id: 5c864594 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.bll.storage.domain.= ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455)= [93433989-8e26-48a9-bd3a-2ab95f296c08] Command 'org.ovirt.engine.core.bll.= storage.domain.ActivateStorageDomainCommand' failed: EngineException: org.o= virt.engine.core.vdsbroker.irsbroker.IrsOperationFailedNoFailove rException: IRSGenericException: IRSErrorException: Storage domain not in p= ool: u'domain=3Df5914df0-f46c-4cc0-b666-c929aa0225ae, pool=3D5a54bf81-0228-= 02bc-0358-000000000304' (Failed with error StorageDomainNotInPool and code = 353) 2018-05-02 09:16:02,636+01 INFO [org.ovirt.engine.core.bll.st<http://org.o= virt.engine.core.bll.st/>orage.domain.ActivateStorageDomainCommand] (EE-Man= agedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c0= 8] Command [id=3D22b0f3c1-9a09-4e26-8096-d83465c8f4ee]: Compensating CHANGE= D_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.StoragePoolI= soMap; snapshot: EntityStatus Snapshot:{id=3D'StoragePoolIsoMapId:{storagePoolId=3D'5a54bf81-0228-02bc-03= 58-000000000304', storageId=3D'f5914df0-f46c-4cc0-b666-c929aa0225ae'}', sta= tus=3D'Maintenance'}. 2018-05-02 09:16:02,660+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditl= oghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-33455) = [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID: USER_ACTIVATE_STORAGE_DOMA= IN_FAILED(967), Failed to activate Storage Domain VMISOs (Data Center Defau= lt) by admin@internal-authz Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> On 2 May 2018, at 08:44, Fred Rolland <frolland@redhat.com<mailto:frolland@= redhat.com>> wrote: Hi, Can you provide logs from engine and Vdsm(SPM)? What is the state now? Thanks, Fred On Tue, May 1, 2018 at 4:11 PM, Callum Smith <callum@well.ox.ac.uk<mailto:c= allum@well.ox.ac.uk>> wrote: Dear All, It appears that clicking "detach" on the ISO storage domain is a really bad= idea. This has gotten half way through the procedure and now can't be reco= vered from. Is there any advice for re-attaching the ISO storage domain man= ually? An NFS mount didn't add it back to the "pool" unfortunately. On a separate note, is it possible to migrate this storage to a new locatio= n? And if so how. Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users --_000_FB71A3FB6A03472B8BC9B2EB92C1C9C2welloxacuk_ Content-Type: text/html; charset="us-ascii" Content-ID: <AC7899520AA8A24A9C41467B3636FA8A@well.ox.ac.uk> Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
</head> <body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; line-break:= after-white-space;" class=3D""> <pre id=3D"line1" class=3D"">Attached, thank you for looking into this</pre=
<pre id=3D"line1" class=3D""><span class=3D"pi"><br class=3D""></span></pre=
> <span id=3D"line16" class=3D""></span> <<span class=3D"start-tag">bac= kup</span>>false</<span class=3D"end-tag">backup</span>> <span id=3D"line17" class=3D""></span> <<span class=3D"start-tag">com= mitted</span>>0</<span class=3D"end-tag">committed</span>> <span id=3D"line18" class=3D""></span> <<span class=3D"start-tag">cri= tical_space_action_blocker</span>>5</<span class=3D"end-tag">critical= _space_action_blocker</span>> <span id=3D"line19" class=3D""></span> <<span class=3D"start-tag">dis= card_after_delete</span>>false</<span class=3D"end-tag">discard_after= _delete</span>> <span id=3D"line20" class=3D""></span> <<span class=3D"start-tag">ext= ernal_status</span>>ok</<span class=3D"end-tag">external_status</span= > <span id=3D"line21" class=3D""></span> <<span class=3D"start-tag">mas= ter</span>>false</<span class=3D"end-tag">master</span>> <span id=3D"line22" class=3D""></span> <<span class=3D"start-tag">sto= rage</span>> <span id=3D"line23" class=3D""></span> <<span class=3D"start-tag"= address</span>>backoffice01.cluster</<span class=3D"end-tag">address= </span>> <span id=3D"line24" class=3D""></span> <<span class=3D"start-tag"= path</span>>/vm-iso</<span class=3D"end-tag">path</span>> <span id=3D"line25" class=3D""></span> <<span class=3D"start-tag"= type</span>>nfs</<span class=3D"end-tag">type</span>> <span id=3D"line26" class=3D""></span> </<span class=3D"end-tag">stor= age</span>> <span id=3D"line27" class=3D""></span> <<span class=3D"start-tag">sto= rage_format</span>>v1</<span class=3D"end-tag">storage_format</span>&= gt; <span id=3D"line28" class=3D""></span> <<span class=3D"start-tag">sup=
data_center</span> <span class=3D"attribute-name">href</span>=3D"<a c= lass=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-= engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304">/ovirt-engi= ne/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304</a>" <span = class=3D"attribute-name">id</span>=3D"<a class=3D"attribute-value">5a5= 4bf81-0228-02bc-0358-000000000304</a>"/> <span id=3D"line36" class=3D""></span> </<span class=3D"end-tag">data= _centers</span>> <span id=3D"line37" class=3D""></span></<span class=3D"end-tag">storage_= domain</span>> <span id=3D"line38" class=3D""></span></pre><div class=3D""><br class=3D"">= </div><div class=3D"">https://<i class=3D"">HOSTNAME</i>/ovirt-engine/api/v= 4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains</div></sp= an></pre> <pre id=3D"line1" class=3D""><pre id=3D"line1" class=3D""><span class=3D"pi= "><?xml version=3D"1.0" encoding=3D"UTF-8" standalon= e=3D"yes"?></span> <span id=3D"line2" class=3D""></span><<span class=3D"start-tag">storage_= domains</span>> <span id=3D"line3" class=3D""></span> <<span class=3D"start-tag">stor= age_domain</span> <span class=3D"attribute-name">href</span>=3D"<a cla= ss=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-en= gine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains= /d9f8093b-71cd-423d-9b7d-91a613ef98ba">/ovirt-engine/api/v4/datacenters/5a5= 4bf81-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91= a613ef98ba</a>" <span class=3D"attribute-name">id</span>=3D"<a cl= ass=3D"attribute-value">d9f8093b-71cd-423d-9b7d-91a613ef98ba</a>"> <span id=3D"line4" class=3D""></span> <<span class=3D"start-tag">= actions</span>> <span id=3D"line5" class=3D""></span> <<span class=3D"start-t= ag">link</span> <span class=3D"attribute-name">href</span>=3D"<a class= =3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engi= ne/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/d= 9f8093b-71cd-423d-9b7d-91a613ef98ba/activate">/ovirt-engine/api/v4/datacent= ers/5a54bf81-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-= 9b7d-91a613ef98ba/activate</a>" <span class=3D"attribute-name">rel</sp= an>=3D"<a class=3D"attribute-value">activate</a>"/> <span id=3D"line6" class=3D""></span> <<span class=3D"start-t= ag">link</span> <span class=3D"attribute-name">href</span>=3D"<a class= =3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engi= ne/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/d= 9f8093b-71cd-423d-9b7d-91a613ef98ba/deactivate">/ovirt-engine/api/v4/datace= nters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423= d-9b7d-91a613ef98ba/deactivate</a>" <span class=3D"attribute-name">rel= </span>=3D"<a class=3D"attribute-value">deactivate</a>"/> <span id=3D"line7" class=3D""></span> </<span class=3D"end-tag">a= ctions</span>> <span id=3D"line8" class=3D""></span> <<span class=3D"start-tag">= name</span>>tegile-virtman-backup</<span class=3D"end-tag">name</span= > <span id=3D"line9" class=3D""></span> <<span class=3D"start-tag">=
available</span>>17519171600384</<span class=3D"end-tag">available</= span>> <span id=3D"line11" class=3D""></span> <<span class=3D"start-tag"= backup</span>>false</<span class=3D"end-tag">backup</span>> <span id=3D"line12" class=3D""></span> <<span class=3D"start-tag"= committed</span>>0</<span class=3D"end-tag">committed</span>> <span id=3D"line13" class=3D""></span> <<span class=3D"start-tag"= critical_space_action_blocker</span>>5</<span class=3D"end-tag">crit= ical_space_action_blocker</span>> <span id=3D"line14" class=3D""></span> <<span class=3D"start-tag"= discard_after_delete</span>>false</<span class=3D"end-tag">discard_a= fter_delete</span>> <span id=3D"line15" class=3D""></span> <<span class=3D"start-tag"= external_status</span>>ok</<span class=3D"end-tag">external_status</= span>> <span id=3D"line16" class=3D""></span> <<span class=3D"start-tag"= master</span>>false</<span class=3D"end-tag">master</span>> <span id=3D"line17" class=3D""></span> <<span class=3D"start-tag"= status</span>>maintenance</<span class=3D"end-tag">status</span>> <span id=3D"line18" class=3D""></span> <<span class=3D"start-tag"= storage</span>> <span id=3D"line19" class=3D""></span> <<span class=3D"start-= tag">address</span>>192.168.64.248</<span class=3D"end-tag">address</= span>> <span id=3D"line20" class=3D""></span> <<span class=3D"start-= tag">nfs_version</span>>auto</<span class=3D"end-tag">nfs_version</sp= an>> <span id=3D"line21" class=3D""></span> <<span class=3D"start-= tag">path</span>>/export/virtman/backup</<span class=3D"end-tag">path= </span>> <span id=3D"line22" class=3D""></span> <<span class=3D"start-= tag">type</span>>nfs</<span class=3D"end-tag">type</span>> <span id=3D"line23" class=3D""></span> </<span class=3D"end-tag">= storage</span>> <span id=3D"line24" class=3D""></span> <<span class=3D"start-tag"= storage_format</span>>v1</<span class=3D"end-tag">storage_format</sp= an>> <span id=3D"line25" class=3D""></span> <<span class=3D"start-tag"= supports_discard</span>>false</<span class=3D"end-tag">supports_disc= ard</span>> <span id=3D"line26" class=3D""></span> <<span class=3D"start-tag"= supports_discard_zeroes_data</span>>false</<span class=3D"end-tag">s= upports_discard_zeroes_data</span>> <span id=3D"line27" class=3D""></span> <<span class=3D"start-tag"= type</span>>export</<span class=3D"end-tag">type</span>> <span id=3D"line28" class=3D""></span> <<span class=3D"start-tag"= used</span>>8589934592</<span class=3D"end-tag">used</span>> <span id=3D"line29" class=3D""></span> <<span class=3D"start-tag"= warning_low_space_indicator</span>>10</<span class=3D"end-tag">warni= ng_low_space_indicator</span>> <span id=3D"line30" class=3D""></span> <<span class=3D"start-tag"= wipe_after_delete</span>>false</<span class=3D"end-tag">wipe_after_d= elete</span>> <span id=3D"line31" class=3D""></span> <<span class=3D"start-tag"= data_center</span> <span class=3D"attribute-name">href</span>=3D"<a c= lass=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-= engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304">/ovirt-engi= ne/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304</a>" <span = class=3D"attribute-name">id</span>=3D"<a class=3D"attribute-value">5a5= 4bf81-0228-02bc-0358-000000000304</a>"/> <span id=3D"line32" class=3D""></span> <<span class=3D"start-tag"= data_centers</span>> <span id=3D"line33" class=3D""></span> <<span class=3D"start-= tag">data_center</span> <span class=3D"attribute-name">href</span>=3D"= <a class=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ov= irt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304">/ovirt-= engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304</a>" <s=
5a54bf81-0228-02bc-0358-000000000304</a>"/> <span id=3D"line34" class=3D""></span> </<span class=3D"end-tag">= data_centers</span>> <span id=3D"line35" class=3D""></span> </<span class=3D"end-tag">stor= age_domain</span>> <span id=3D"line36" class=3D""></span> <<span class=3D"start-tag">sto= rage_domain</span> <span class=3D"attribute-name">href</span>=3D"<a cl= ass=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-e= ngine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomain= s/017dd620-65ec-4fa4-8d46-67737dedd7a1">/ovirt-engine/api/v4/datacenters/5a= 54bf81-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-6= 7737dedd7a1</a>" <span class=3D"attribute-name">id</span>=3D"<a c= lass=3D"attribute-value">017dd620-65ec-4fa4-8d46-67737dedd7a1</a>"> <span id=3D"line37" class=3D""></span> <<span class=3D"start-tag"= actions</span>> <span id=3D"line38" class=3D""></span> <<span class=3D"start-= tag">link</span> <span class=3D"attribute-name">href</span>=3D"<a clas= s=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-eng= ine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/= 017dd620-65ec-4fa4-8d46-67737dedd7a1/activate">/ovirt-engine/api/v4/datacen= ters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4= -8d46-67737dedd7a1/activate</a>" <span class=3D"attribute-name">rel</s=
name</span>>VMStorage</<span class=3D"end-tag">name</span>> <span id=3D"line42" class=3D""></span> <<span class=3D"start-tag"= link</span> <span class=3D"attribute-name">href</span>=3D"<a class=3D= "attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engine/= api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/017d= d620-65ec-4fa4-8d46-67737dedd7a1/disks">/ovirt-engine/api/v4/datacenters/5a= 54bf81-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-6= 7737dedd7a1/disks</a>" <span class=3D"attribute-name">rel</span>=3D&qu= ot;<a class=3D"attribute-value">disks</a>"/> <span id=3D"line43" class=3D""></span> <<span class=3D"start-tag"= available</span>>11770357874688</<span class=3D"end-tag">available</= span>> <span id=3D"line44" class=3D""></span> <<span class=3D"start-tag"= backup</span>>false</<span class=3D"end-tag">backup</span>> <span id=3D"line45" class=3D""></span> <<span class=3D"start-tag"= committed</span>>118111600640</<span class=3D"end-tag">committed</sp= an>> <span id=3D"line46" class=3D""></span> <<span class=3D"start-tag"= critical_space_action_blocker</span>>5</<span class=3D"end-tag">crit= ical_space_action_blocker</span>> <span id=3D"line47" class=3D""></span> <<span class=3D"start-tag"= discard_after_delete</span>>false</<span class=3D"end-tag">discard_a= fter_delete</span>> <span id=3D"line48" class=3D""></span> <<span class=3D"start-tag"= external_status</span>>ok</<span class=3D"end-tag">external_status</= span>> <span id=3D"line49" class=3D""></span> <<span class=3D"start-tag"= master</span>>true</<span class=3D"end-tag">master</span>> <span id=3D"line50" class=3D""></span> <<span class=3D"start-tag"= status</span>>active</<span class=3D"end-tag">status</span>> <span id=3D"line51" class=3D""></span> <<span class=3D"start-tag"= storage</span>> <span id=3D"line52" class=3D""></span> <<span class=3D"start-= tag">address</span>>backoffice01.cluster</<span class=3D"end-tag">add= ress</span>> <span id=3D"line53" class=3D""></span> <<span class=3D"start-= tag">path</span>>/vm-storage2</<span class=3D"end-tag">path</span>>= ; <span id=3D"line54" class=3D""></span> <<span class=3D"start-= tag">type</span>>nfs</<span class=3D"end-tag">type</span>> <span id=3D"line55" class=3D""></span> </<span class=3D"end-tag">= storage</span>> <span id=3D"line56" class=3D""></span> <<span class=3D"start-tag"= storage_format</span>>v4</<span class=3D"end-tag">storage_format</sp= an>> <span id=3D"line57" class=3D""></span> <<span class=3D"start-tag"= supports_discard</span>>false</<span class=3D"end-tag">supports_disc= ard</span>> <span id=3D"line58" class=3D""></span> <<span class=3D"start-tag"= supports_discard_zeroes_data</span>>false</<span class=3D"end-tag">s= upports_discard_zeroes_data</span>> <span id=3D"line59" class=3D""></span> <<span class=3D"start-tag"= type</span>>data</<span class=3D"end-tag">type</span>> <span id=3D"line60" class=3D""></span> <<span class=3D"start-tag"= used</span>>38654705664</<span class=3D"end-tag">used</span>> <span id=3D"line61" class=3D""></span> <<span class=3D"start-tag"= warning_low_space_indicator</span>>10</<span class=3D"end-tag">warni= ng_low_space_indicator</span>> <span id=3D"line62" class=3D""></span> <<span class=3D"start-tag"= wipe_after_delete</span>>false</<span class=3D"end-tag">wipe_after_d= elete</span>> <span id=3D"line63" class=3D""></span> <<span class=3D"start-tag"= data_center</span> <span class=3D"attribute-name">href</span>=3D"<a c= lass=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-= engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304">/ovirt-engi= ne/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304</a>" <span = class=3D"attribute-name">id</span>=3D"<a class=3D"attribute-value">5a5= 4bf81-0228-02bc-0358-000000000304</a>"/> <span id=3D"line64" class=3D""></span> <<span class=3D"start-tag"= data_centers</span>> <span id=3D"line65" class=3D""></span> <<span class=3D"start-= tag">data_center</span> <span class=3D"attribute-name">href</span>=3D"= <a class=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ov= irt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304">/ovirt-= engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304</a>" <s=
5a54bf81-0228-02bc-0358-000000000304</a>"/> <span id=3D"line66" class=3D""></span> </<span class=3D"end-tag">= data_centers</span>> <span id=3D"line67" class=3D""></span> </<span class=3D"end-tag">stor= age_domain</span>> <span id=3D"line68" class=3D""></span> <<span class=3D"start-tag">sto= rage_domain</span> <span class=3D"attribute-name">href</span>=3D"<a cl= ass=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-e= ngine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomain= s/e572c874-17f8-4ad8-9bac-b6f563a9fc17">/ovirt-engine/api/v4/datacenters/5a= 54bf81-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b= 6f563a9fc17</a>" <span class=3D"attribute-name">id</span>=3D"<a c= lass=3D"attribute-value">e572c874-17f8-4ad8-9bac-b6f563a9fc17</a>"> <span id=3D"line69" class=3D""></span> <<span class=3D"start-tag"= actions</span>> <span id=3D"line70" class=3D""></span> <<span class=3D"start-= tag">link</span> <span class=3D"attribute-name">href</span>=3D"<a clas= s=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-eng= ine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/= e572c874-17f8-4ad8-9bac-b6f563a9fc17/activate">/ovirt-engine/api/v4/datacen= ters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8= -9bac-b6f563a9fc17/activate</a>" <span class=3D"attribute-name">rel</s=
name</span>>tegile-virtman</<span class=3D"end-tag">name</span>> <span id=3D"line74" class=3D""></span> <<span class=3D"start-tag"= link</span> <span class=3D"attribute-name">href</span>=3D"<a class=3D= "attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engine/= api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/e572= c874-17f8-4ad8-9bac-b6f563a9fc17/disks">/ovirt-engine/api/v4/datacenters/5a= 54bf81-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b= 6f563a9fc17/disks</a>" <span class=3D"attribute-name">rel</span>=3D&qu= ot;<a class=3D"attribute-value">disks</a>"/> <span id=3D"line75" class=3D""></span> <<span class=3D"start-tag"= available</span>>2190433320960</<span class=3D"end-tag">available</s=
backup</span>>false</<span class=3D"end-tag">backup</span>> <span id=3D"line77" class=3D""></span> <<span class=3D"start-tag"= committed</span>>226559524864</<span class=3D"end-tag">committed</sp= an>> <span id=3D"line78" class=3D""></span> <<span class=3D"start-tag"= critical_space_action_blocker</span>>5</<span class=3D"end-tag">crit= ical_space_action_blocker</span>> <span id=3D"line79" class=3D""></span> <<span class=3D"start-tag"= discard_after_delete</span>>false</<span class=3D"end-tag">discard_a= fter_delete</span>> <span id=3D"line80" class=3D""></span> <<span class=3D"start-tag"= external_status</span>>ok</<span class=3D"end-tag">external_status</= span>> <span id=3D"line81" class=3D""></span> <<span class=3D"start-tag"= master</span>>false</<span class=3D"end-tag">master</span>> <span id=3D"line82" class=3D""></span> <<span class=3D"start-tag"= status</span>>active</<span class=3D"end-tag">status</span>> <span id=3D"line83" class=3D""></span> <<span class=3D"start-tag"= storage</span>> <span id=3D"line84" class=3D""></span> <<span class=3D"start-= tag">address</span>>192.168.64.248</<span class=3D"end-tag">address</= span>> <span id=3D"line85" class=3D""></span> <<span class=3D"start-= tag">nfs_version</span>>auto</<span class=3D"end-tag">nfs_version</sp= an>> <span id=3D"line86" class=3D""></span> <<span class=3D"start-= tag">path</span>>/export/virtman/VirtualServerShare_1</<span class=3D= "end-tag">path</span>> <span id=3D"line87" class=3D""></span> <<span class=3D"start-= tag">type</span>>nfs</<span class=3D"end-tag">type</span>> <span id=3D"line88" class=3D""></span> </<span class=3D"end-tag">= storage</span>> <span id=3D"line89" class=3D""></span> <<span class=3D"start-tag"= storage_format</span>>v4</<span class=3D"end-tag">storage_format</sp= an>> <span id=3D"line90" class=3D""></span> <<span class=3D"start-tag"= supports_discard</span>>false</<span class=3D"end-tag">supports_disc= ard</span>> <span id=3D"line91" class=3D""></span> <<span class=3D"start-tag"= supports_discard_zeroes_data</span>>false</<span class=3D"end-tag">s= upports_discard_zeroes_data</span>> <span id=3D"line92" class=3D""></span> <<span class=3D"start-tag"= type</span>>data</<span class=3D"end-tag">type</span>> <span id=3D"line93" class=3D""></span> <<span class=3D"start-tag"= used</span>>8589934592</<span class=3D"end-tag">used</span>> <span id=3D"line94" class=3D""></span> <<span class=3D"start-tag"= warning_low_space_indicator</span>>10</<span class=3D"end-tag">warni= ng_low_space_indicator</span>> <span id=3D"line95" class=3D""></span> <<span class=3D"start-tag"= wipe_after_delete</span>>false</<span class=3D"end-tag">wipe_after_d= elete</span>> <span id=3D"line96" class=3D""></span> <<span class=3D"start-tag"= data_center</span> <span class=3D"attribute-name">href</span>=3D"<a c= lass=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-= engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304">/ovirt-engi= ne/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304</a>" <span = class=3D"attribute-name">id</span>=3D"<a class=3D"attribute-value">5a5= 4bf81-0228-02bc-0358-000000000304</a>"/> <span id=3D"line97" class=3D""></span> <<span class=3D"start-tag"= data_centers</span>> <span id=3D"line98" class=3D""></span> <<span class=3D"start-= tag">data_center</span> <span class=3D"attribute-name">href</span>=3D"= <a class=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ov= irt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304">/ovirt-= engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304</a>" <s=
5a54bf81-0228-02bc-0358-000000000304</a>"/> <span id=3D"line99" class=3D""></span> </<span class=3D"end-tag">= data_centers</span>> <span id=3D"line100" class=3D""></span> </<span class=3D"end-tag">sto= rage_domain</span>> <span id=3D"line101" class=3D""></span> <<span class=3D"start-tag">st= orage_domain</span> <span class=3D"attribute-name">href</span>=3D"<a c= lass=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-= engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomai= ns/f5914df0-f46c-4cc0-b666-c929aa0225ae">/ovirt-engine/api/v4/datacenters/5= a54bf81-0228-02bc-0358-000000000304/storagedomains/f5914df0-f46c-4cc0-b666-= c929aa0225ae</a>" <span class=3D"attribute-name">id</span>=3D"<a = class=3D"attribute-value">f5914df0-f46c-4cc0-b666-c929aa0225ae</a>">= ; <span id=3D"line102" class=3D""></span> <<span class=3D"start-tag= ">actions</span>> <span id=3D"line103" class=3D""></span> <<span class=3D"start= -tag">link</span> <span class=3D"attribute-name">href</span>=3D"<a cla= ss=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-en= gine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains= /f5914df0-f46c-4cc0-b666-c929aa0225ae/activate">/ovirt-engine/api/v4/datace= nters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/f5914df0-f46c-4cc= 0-b666-c929aa0225ae/activate</a>" <span class=3D"attribute-name">rel</= span>=3D"<a class=3D"attribute-value">activate</a>"/> <span id=3D"line104" class=3D""></span> <<span class=3D"start= -tag">link</span> <span class=3D"attribute-name">href</span>=3D"<a cla= ss=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-en= gine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains= /f5914df0-f46c-4cc0-b666-c929aa0225ae/deactivate">/ovirt-engine/api/v4/data=
actions</span>> <span id=3D"line106" class=3D""></span> <<span class=3D"start-tag= ">name</span>>VMISOs</<span class=3D"end-tag">name</span>> <span id=3D"line107" class=3D""></span> <<span class=3D"start-tag= ">available</span>>11770357874688</<span class=3D"end-tag">available<= /span>> <span id=3D"line108" class=3D""></span> <<span class=3D"start-tag= ">backup</span>>false</<span class=3D"end-tag">backup</span>> <span id=3D"line109" class=3D""></span> <<span class=3D"start-tag= ">committed</span>>0</<span class=3D"end-tag">committed</span>> <span id=3D"line110" class=3D""></span> <<span class=3D"start-tag= ">critical_space_action_blocker</span>>5</<span class=3D"end-tag">cri= tical_space_action_blocker</span>> <span id=3D"line111" class=3D""></span> <<span class=3D"start-tag= ">discard_after_delete</span>>false</<span class=3D"end-tag">discard_= after_delete</span>> <span id=3D"line112" class=3D""></span> <<span class=3D"start-tag= ">external_status</span>>ok</<span class=3D"end-tag">external_status<= /span>> <span id=3D"line113" class=3D""></span> <<span class=3D"start-tag= ">master</span>>false</<span class=3D"end-tag">master</span>> <span id=3D"line114" class=3D""></span> <<span class=3D"start-tag= ">status</span>>maintenance</<span class=3D"end-tag">status</span>>= ; <span id=3D"line115" class=3D""></span> <<span class=3D"start-tag= ">storage</span>> <span id=3D"line116" class=3D""></span> <<span class=3D"start= -tag">address</span>>backoffice01.cluster</<span class=3D"end-tag">ad= dress</span>> <span id=3D"line117" class=3D""></span> <<span class=3D"start= -tag">path</span>>/vm-iso</<span class=3D"end-tag">path</span>> <span id=3D"line118" class=3D""></span> <<span class=3D"start= -tag">type</span>>nfs</<span class=3D"end-tag">type</span>> <span id=3D"line119" class=3D""></span> </<span class=3D"end-tag"= storage</span>> <span id=3D"line120" class=3D""></span> <<span class=3D"start-tag= ">storage_format</span>>v1</<span class=3D"end-tag">storage_format</s=
data_centers</span>> <span id=3D"line131" class=3D""></span> </<span class=3D"end-tag">sto= rage_domain</span>> <span id=3D"line132" class=3D""></span></<span class=3D"end-tag">storage= _domains</span>> <span id=3D"line133" class=3D""></span></pre><div class=3D""><br class=3D""= </div> <span id=3D"line3" class=3D""></span></pre> <div class=3D""><img apple-inline=3D"yes" id=3D"36B0C1AD-FA39-4015-B2D0-7D1= 69F695CBB" src=3D"cid:921E5257-26F4-4E0D-83B0-68F996842D08@well.ox.ac.uk" c= lass=3D""></div> <div class=3D""> <div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; text-align: star= t; text-indent: 0px; text-transform: none; white-space: normal; word-spacin= g: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp= -mode: space; -webkit-line-break: after-white-space;" class=3D""> <div style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line= -break: after-white-space;" class=3D""> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Regards,</div> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Callum</div> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> <br class=3D""> </div> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> --<br class=3D""> <br class=3D""> Callum Smith</div> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Research Computing Core</div> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Wellcome Trust Centre for Human Genetics<br class=3D""> University of Oxford<br class=3D""> e. <a href=3D"mailto:callum@well.ox.ac.uk" class=3D"">callum@well.ox.ac.uk<= /a></div> </div> </div> </div> <div><br class=3D""> <blockquote type=3D"cite" class=3D""> <div class=3D"">On 2 May 2018, at 14:46, Fred Rolland <<a href=3D"mailto= :frolland@redhat.com" class=3D"">frolland@redhat.com</a>> wrote:</div> <br class=3D"Apple-interchange-newline"> <div class=3D""> <div dir=3D"ltr" class=3D""> <div class=3D"">Can you share the REST API data of the Storage domain and D= ata Center?<br class=3D""> </div> Here an example of the URLs, you will need to replace with correct ids.<br = class=3D""> <div class=3D""><br class=3D""> <a href=3D"http://my-server/ovirt-engine/api/v4/storagedomains/13461356-f6f= 7-4a58-9897-2fac61ff40af" class=3D"">http://MY-SERVER/ovirt-engine/api/v4/s= toragedomains/13461356-f6f7-4a58-9897-2fac61ff40af</a><br class=3D""> <br class=3D""> <a href=3D"http://my-server/ovirt-engine/api/v4/datacenters/5a5df553-022d-0= 36d-01e8-000000000071/storagedomains" class=3D"">http://MY-SERVER/ovirt-eng= ine/api/v4/datacenters/5a5df553-022d-036d-01e8-000000000071/storagedomains<= /a><br class=3D""> <br class=3D""> <br class=3D""> </div> </div> <div class=3D"gmail_extra"><br class=3D""> <div class=3D"gmail_quote">On Wed, May 2, 2018 at 12:53 PM, Callum Smith <s=
This is on 4.2.0.2-1, I've linked the main logs to dropbox simply because =
State is maintenance for the ISOs storage. I've extracted what is hopefull= y the relevant bits of the log. <div class=3D""> <div class=3D""><br class=3D""> </div> <div class=3D"">VDSM.log (SPM)</div> <div class=3D""><br class=3D""> </div> <div class=3D""> <div class=3D"">2018-05-02 09:16:03,455+0100 INFO (ioprocess comm= unication (179084)) [IOProcess] Starting ioprocess (__init__:447)</div> <div class=3D"">2018-05-02 09:16:03,456+0100 INFO (ioprocess comm= unication (179091)) [IOProcess] Starting ioprocess (__init__:447)</div> <div class=3D"">2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [vd= sm.api] FINISH activateStorageDomain error=3DStorage domain not in pool: u'= domain=3Df5914df0-f46c-4cc0-b6<wbr class=3D"">66-c929aa0225ae, pool=3D5a54b= f81-0228-02bc-0358-0<wbr class=3D"">00000000304' from=3D::ffff:192.168.64.2= 54,589<wbr class=3D"">68, flow_id=3D93433989-8e26-48a9-bd3<wbr class=3D"">a-2ab95f296c08, task_id=3D= 7f21f911-348f-45a3-b79<wbr class=3D"">c-e3cb11642035 (api:50)</div> <div class=3D"">2018-05-02 09:16:03,461+0100 ERROR (jsonrpc/0) [storage= .TaskManager.Task] (Task=3D'7f21f911-348f-45a3-b79c<wbr class=3D"">-e3cb116= 42035') Unexpected error (task:875)</div> <div class=3D"">Traceback (most recent call last):</div> <div class=3D""> File "/usr/lib/python2.7/site-packa<wbr class= =3D"">ges/vdsm/storage/task.py", line 882, in _run</div> <div class=3D""> return fn(*args, **kargs)</div> <div class=3D""> File "<string>", line 2, in activate= StorageDomain</div> <div class=3D""> File "/usr/lib/python2.7/site-packa<wbr class= =3D"">ges/vdsm/common/api.py", line 48, in method</div> <div class=3D""> ret =3D func(*args, **kwargs)</div> <div class=3D""> File "/usr/lib/python2.7/site-packa<wbr class= =3D"">ges/vdsm/storage/hsm.py", line 1256, in activateStorageDomain</d= iv> <div class=3D""> pool.activateSD(sdUUID)</div> <div class=3D""> File "/usr/lib/python2.7/site-packa<wbr class= =3D"">ges/vdsm/storage/securable.py"<wbr class=3D"">, line 79, in wrap=
m)</div> <div class=3D""> File "/usr/lib/python2.7/site-packa<wbr class= =3D"">ges/vdsm/storage/securable.py"<wbr class=3D"">, line 79, in wrap=
<wbr class=3D"">orage.domain.ActivateStorageDo<wbr class=3D"">mainCommand]= (default task-20) [93433989-8e26-48a9-bd3a-2ab95<wbr class=3D"">f296c08] Lock Acquired to object 'EngineLock:{exclusiveLocks=3D'[<wbr class=3D"">f5= 914df0-f46c-4cc0-b666-c929aa<wbr class=3D"">0225ae=3DSTORAGE]', sharedLocks= =3D''}'</div> <div class=3D"">2018-05-02 09:16:02,376+01 INFO [<a href=3D"http:= //org.ovirt.engine.core.bll.st/" class=3D"">org.ovirt.engine.core.bll.st</a= <wbr class=3D"">orage.domain.ActivateStorageDo<wbr class=3D"">mainCommand]= (EE-ManagedThreadFactory-engin<wbr class=3D"">e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95<wbr class=3D"">f296c08] Running command: Ac= tivateStorageDomainCommand internal: false. Entities affected : ID: f= 5914df0-f46c-4cc0-b666-c929aa<wbr class=3D"">0225ae Type: StorageAction gro= up MANIPULATE_STORAGE_DOMA</div> <div class=3D"">IN with role type ADMIN</div> <div class=3D"">2018-05-02 09:16:02,385+01 INFO [<a href=3D"http:= //org.ovirt.engine.core.bll.st/" class=3D"">org.ovirt.engine.core.bll.st</a= <wbr class=3D"">orage.domain.ActivateStorageDo<wbr class=3D"">mainCommand]= (EE-ManagedThreadFactory-engin<wbr class=3D"">e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95<wbr class=3D"">f296c08] Lock freed to objec= t 'EngineLock:{exclusiveLocks=3D'[<wbr class=3D"">f5914df0-f46c-4cc0-b666-c= 929aa<wbr class=3D"">0225ae=3DSTORAGE]', sharedLocks=3D''}'</div> <div class=3D"">2018-05-02 09:16:02,385+01 INFO [<a href=3D"http:= //org.ovirt.engine.core.bll.st/" class=3D"">org.ovirt.engine.core.bll.st</a= <wbr class=3D"">orage.domain.ActivateStorageDo<wbr class=3D"">mainCommand]= (EE-ManagedThreadFactory-engin<wbr class=3D"">e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95<wbr class=3D"">f296c08] ActivateStorage Dom= ain. Before Connect all hosts to pool. Time: Wed May 02 09:16:02 BST 2018</=
<wbr class=3D"">orage.connection.ConnectStorag<wbr class=3D"">eToVdsComman= d] (EE-ManagedThreadFactory-engin<wbr class=3D"">e-Thread-33456) [40a82b47] Running command: ConnectStorageToVdsCommand internal: true. Ent= ities affected : ID: aaa00000-0000-0000-0000-123456<wbr class=3D"">78= 9aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN</d= iv> <div class=3D"">2018-05-02 09:16:02,421+01 INFO [org.ovirt.engine= .core.vdsbro<wbr class=3D"">ker.vdsbroker.ConnectStorageSe<wbr class=3D"">r= verVDSCommand] (EE-ManagedThreadFactory-engin<wbr class=3D"">e-Thread-33456= ) [40a82b47] START, ConnectStorageServerVDSCommand<wbr class=3D"">(HostName =3D virtA003, StorageServerConnectionManagem<wbr class=3D"">entVDSParamete= rs:{hostId=3D'fe28<wbr class=3D"">61fc-2b47-4807-b054-470198eda4<wbr class= =3D"">73', storagePoolId=3D'00000000-0000-0<wbr class=3D"">000-0000-000000<= /div> <div class=3D"">000000', storageType=3D'NFS', connectionList=3D'[StorageSer= ver<wbr class=3D"">Connections:{id=3D'da392861-<wbr class=3D"">aedc-4f1e-97= f4-6919fb01f1e9', connection=3D'backoffice01.clust<wbr class=3D"">er:/vm-is= o', iqn=3D'null', vfsType=3D'null', mountOptions=3D'null', nfsVersion=3D'null', nfsRetrans=3D'null', nfsTimeo=3D'null', iface=3D'null= ', netIfaceName=3D'null'}]'}), log id: 23ce648f</div> <div class=3D"">2018-05-02 09:16:02,446+01 INFO [org.ovirt.engine= .core.vdsbro<wbr class=3D"">ker.vdsbroker.ConnectStorageSe<wbr class=3D"">r= verVDSCommand] (EE-ManagedThreadFactory-engin<wbr class=3D"">e-Thread-33456= ) [40a82b47] FINISH, ConnectStorageServerVDSCommand<wbr class=3D"">, return: {da392861-aedc-4f1e-97f4-6919f<wbr class=3D"">b01f1e9=3D0}, log id= : 23ce648f</div> <div class=3D"">2018-05-02 09:16:02,450+01 INFO [org.ovirt.engine= .core.vdsbro<wbr class=3D"">ker.irsbroker.ActivateStorageD<wbr class=3D"">o=
54bf81-0228-02bc-0358-00000000<wbr class=3D"">0304', ignoreFailoverLimit= =3D'false', stor</div> <div class=3D"">ageDomainId=3D'f5914df0-f46c-4cc<wbr class=3D"">0-b666-c929= aa0225ae'}), log id: 5c864594</div> <div class=3D"">2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core= .dal.dbb<wbr class=3D"">roker.auditloghandling.AuditLo<wbr class=3D"">gDire= ctor] (EE-ManagedThreadFactory-engin<wbr class=3D"">e-Thread-33455) [934339= 89-8e26-48a9-bd3a-2ab95<wbr class=3D"">f296c08] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,<wbr class=3D"">803), VDSM command= ActivateStorageDomainVDS failed: Storage domain not in pool: u'domain=3Df5= 914df0-f46c-4cc0-b6<wbr class=3D"">66-c929aa0225ae, pool=3D5a5</div> <div class=3D"">4bf81-0228-02bc-0358-000000000<wbr class=3D"">304'</div> <div class=3D"">2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core= .vdsbrok<wbr class=3D"">er.irsbroker.ActivateStorageDo<wbr class=3D"">mainV= DSCommand] (EE-ManagedThreadFactory-engin<wbr class=3D"">e-Thread-33455) [9= 3433989-8e26-48a9-bd3a-2ab95<wbr class=3D"">f296c08] Command 'ActivateStorageDomainVDSComma<wbr class=3D"">nd( ActivateStorageD= omainVDSComman<wbr class=3D"">dParameters:{storagePoolId=3D'5a<wbr class=3D= "">54bf81-0228-02bc-0358-00000000<wbr class=3D"">0304', ignoreFailoverLimit= =3D'false', st</div> <div class=3D"">orageDomainId=3D'f5914df0-f46c-4<wbr class=3D"">cc0-b666-c9= 29aa0225ae'})' execution failed: IRSGenericException: IRSErrorException: St= orage domain not in pool: u'domain=3Df5914df0-f46c-4cc0-b6<wbr class=3D"">6= 6-c929aa0225ae, pool=3D5a54bf81-0228-02bc-0358-0<wbr class=3D"">00000000304= '</div> <div class=3D"">2018-05-02 09:16:02,635+01 INFO [org.ovirt.engine= .core.vdsbro<wbr class=3D"">ker.irsbroker.ActivateStorageD<wbr class=3D"">o=
<wbr class=3D"">orage.domain.ActivateStorageDo<wbr class=3D"">mainCommand]= (EE-ManagedThreadFactory-engin<wbr class=3D"">e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95<wbr class=3D"">f296c08] Command [id=3D22b0f= 3c1-9a09-4e26-8096-d8<wbr class=3D"">3465c8f4ee]: Compensating CHANGED_STAT= US_ONLY of org.ovirt.engine.core.common.b<wbr class=3D"">usinessentities.St= oragePoolIso<wbr class=3D"">Map; snapshot: EntityStatus</div> <div class=3D"">Snapshot:{id=3D'StoragePoolIsoMa<wbr class=3D"">pId:{storag= ePoolId=3D'5a54bf81-<wbr class=3D"">0228-02bc-0358-000000000304', storageId= =3D'f5914df0-f46c-4cc0-<wbr class=3D"">b666-c929aa0225ae'}', status=3D'Main= tenance'}.</div> <div class=3D"">2018-05-02 09:16:02,660+01 ERROR [org.ovirt.engine.core= .dal.dbb<wbr class=3D"">roker.auditloghandling.AuditLo<wbr class=3D"">gDire= ctor] (EE-ManagedThreadFactory-engin<wbr class=3D"">e-Thread-33455) [934339= 89-8e26-48a9-bd3a-2ab95<wbr class=3D"">f296c08] EVENT_ID: USER_ACTIVATE_STORAGE_DOMAIN_F<wbr class=3D"">AILED(967), Failed= to activate Storage Domain VMISOs (Data Center Default) by admin@internal-= authz</div> </div> <div class=3D""><span class=3D""><br class=3D""> <div class=3D""> <div style=3D"letter-spacing:normal;text-align:start;text-indent:0px;text-t= ransform:none;white-space:normal;word-spacing:0px;word-wrap:break-word" cla= ss=3D""> <div style=3D"word-wrap:break-word" class=3D""> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> Regards,</div> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> Callum</div> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> <br class=3D""> </div> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> --<br class=3D""> <br class=3D""> Callum Smith</div> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> Research Computing Core</div> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> Wellcome Trust Centre for Human Genetics<br class=3D""> University of Oxford<br class=3D""> e. <a href=3D"mailto:callum@well.ox.ac.uk" target=3D"_blank" class=3D"">cal= lum@well.ox.ac.uk</a></div> </div> </div> </div> </span> <div class=3D""> <div class=3D"m_2571234484608545856h5"> <div class=3D""><br class=3D""> <blockquote type=3D"cite" class=3D""> <div class=3D"">On 2 May 2018, at 08:44, Fred Rolland <<a href=3D"mailto= :frolland@redhat.com" target=3D"_blank" class=3D"">frolland@redhat.com</a>&= gt; wrote:</div> <br class=3D"m_2571234484608545856m_-2093273250354586386Apple-interchange-n= ewline"> <div class=3D""> <div dir=3D"ltr" class=3D""> <div class=3D""> <div class=3D""> <div class=3D""> <div class=3D"">Hi,<br class=3D""> <br class=3D""> </div> Can you provide logs from engine and Vdsm(SPM)?<br class=3D""> </div> What is the state now?<br class=3D""> <br class=3D""> </div> Thanks,<br class=3D""> </div> Fred<br class=3D""> </div> <div class=3D"gmail_extra"><br class=3D""> <div class=3D"gmail_quote">On Tue, May 1, 2018 at 4:11 PM, Callum Smith <sp= an dir=3D"ltr" class=3D""> <<a href=3D"mailto:callum@well.ox.ac.uk" target=3D"_blank" class=3D"">ca= llum@well.ox.ac.uk</a>></span> wrote:<br class=3D""> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <div style=3D"word-wrap:break-word;line-break:after-white-space" class=3D""= Dear All, <div class=3D""><br class=3D""> </div> <div class=3D"">It appears that clicking "detach" on the ISO stor= age domain is a really bad idea. This has gotten half way through the proce= dure and now can't be recovered from. Is there any advice for re-attaching =
<pre id=3D"line1" class=3D""><span class=3D"pi">https://<i class=3D"">HOSTN= AME</i>/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa02= 25ae</span></pre> <pre id=3D"line1" class=3D""><span class=3D"pi"><pre id=3D"line1" class=3D"= "><span class=3D"pi"><?xml version=3D"1.0" encoding=3D"UT= F-8" standalone=3D"yes"?></span> <span id=3D"line2" class=3D""></span><<span class=3D"start-tag">storage_= domain</span> <span class=3D"attribute-name">href</span>=3D"<a class= =3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engi= ne/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae">/ovirt-engin= e/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae</a>" <spa= n class=3D"attribute-name">id</span>=3D"<a class=3D"attribute-value">f= 5914df0-f46c-4cc0-b666-c929aa0225ae</a>"> <span id=3D"line3" class=3D""></span> <<span class=3D"start-tag">acti= ons</span>> <span id=3D"line4" class=3D""></span> <<span class=3D"start-tag">= link</span> <span class=3D"attribute-name">href</span>=3D"<a class=3D"= attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engine/a= pi/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/isattached">/ovir= t-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/isattac= hed</a>" <span class=3D"attribute-name">rel</span>=3D"<a class=3D= "attribute-value">isattached</a>"/> <span id=3D"line5" class=3D""></span> <<span class=3D"start-tag">= link</span> <span class=3D"attribute-name">href</span>=3D"<a class=3D"= attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engine/a= pi/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/reduceluns">/ovir= t-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/reducel= uns</a>" <span class=3D"attribute-name">rel</span>=3D"<a class=3D= "attribute-value">reduceluns</a>"/> <span id=3D"line6" class=3D""></span> <<span class=3D"start-tag">= link</span> <span class=3D"attribute-name">href</span>=3D"<a class=3D"= attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engine/a= pi/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/refreshluns">/ovi= rt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/refres= hluns</a>" <span class=3D"attribute-name">rel</span>=3D"<a class= =3D"attribute-value">refreshluns</a>"/> <span id=3D"line7" class=3D""></span> <<span class=3D"start-tag">= link</span> <span class=3D"attribute-name">href</span>=3D"<a class=3D"= attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engine/a= pi/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/updateovfstore">/= ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/upd= ateovfstore</a>" <span class=3D"attribute-name">rel</span>=3D"<a = class=3D"attribute-value">updateovfstore</a>"/> <span id=3D"line8" class=3D""></span> </<span class=3D"end-tag">actio= ns</span>> <span id=3D"line9" class=3D""></span> <<span class=3D"start-tag">name= </span>>VMISOs</<span class=3D"end-tag">name</span>> <span id=3D"line10" class=3D""></span> <<span class=3D"start-tag">lin= k</span> <span class=3D"attribute-name">href</span>=3D"<a class=3D"att= ribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engine/api/= v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/disksnapshots">/ovir= t-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/disksna= pshots</a>" <span class=3D"attribute-name">rel</span>=3D"<a class= =3D"attribute-value">disksnapshots</a>"/> <span id=3D"line11" class=3D""></span> <<span class=3D"start-tag">lin= k</span> <span class=3D"attribute-name">href</span>=3D"<a class=3D"att= ribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engine/api/= v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/files">/ovirt-engine= /api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/files</a>"= <span class=3D"attribute-name">rel</span>=3D"<a class=3D"attribute-va= lue">files</a>"/> <span id=3D"line12" class=3D""></span> <<span class=3D"start-tag">lin= k</span> <span class=3D"attribute-name">href</span>=3D"<a class=3D"att= ribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engine/api/= v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/diskprofiles">/ovirt= -engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/diskprof= iles</a>" <span class=3D"attribute-name">rel</span>=3D"<a class= =3D"attribute-value">diskprofiles</a>"/> <span id=3D"line13" class=3D""></span> <<span class=3D"start-tag">lin= k</span> <span class=3D"attribute-name">href</span>=3D"<a class=3D"att= ribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engine/api/= v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/storageconnections">= /ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/st= orageconnections</a>" <span class=3D"attribute-name">rel</span>=3D&quo= t;<a class=3D"attribute-value">storageconnections</a>"/> <span id=3D"line14" class=3D""></span> <<span class=3D"start-tag">lin= k</span> <span class=3D"attribute-name">href</span>=3D"<a class=3D"att= ribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engine/api/= v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/permissions">/ovirt-= engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/permissio= ns</a>" <span class=3D"attribute-name">rel</span>=3D"<a class=3D"= attribute-value">permissions</a>"/> <span id=3D"line15" class=3D""></span> <<span class=3D"start-tag">ava= ilable</span>>11770357874688</<span class=3D"end-tag">available</span= ports_discard</span>>false</<span class=3D"end-tag">supports_discard<= /span>> <span id=3D"line29" class=3D""></span> <<span class=3D"start-tag">sup= ports_discard_zeroes_data</span>>false</<span class=3D"end-tag">suppo= rts_discard_zeroes_data</span>> <span id=3D"line30" class=3D""></span> <<span class=3D"start-tag">typ= e</span>>iso</<span class=3D"end-tag">type</span>> <span id=3D"line31" class=3D""></span> <<span class=3D"start-tag">use= d</span>>38654705664</<span class=3D"end-tag">used</span>> <span id=3D"line32" class=3D""></span> <<span class=3D"start-tag">war= ning_low_space_indicator</span>>10</<span class=3D"end-tag">warning_l= ow_space_indicator</span>> <span id=3D"line33" class=3D""></span> <<span class=3D"start-tag">wip= e_after_delete</span>>false</<span class=3D"end-tag">wipe_after_delet= e</span>> <span id=3D"line34" class=3D""></span> <<span class=3D"start-tag">dat= a_centers</span>> <span id=3D"line35" class=3D""></span> <<span class=3D"start-tag"= link</span> <span class=3D"attribute-name">href</span>=3D"<a class=3D"= attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engine/a= pi/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/d9f80= 93b-71cd-423d-9b7d-91a613ef98ba/disks">/ovirt-engine/api/v4/datacenters/5a5= 4bf81-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91= a613ef98ba/disks</a>" <span class=3D"attribute-name">rel</span>=3D&quo= t;<a class=3D"attribute-value">disks</a>"/> <span id=3D"line10" class=3D""></span> <<span class=3D"start-tag"= pan class=3D"attribute-name">id</span>=3D"<a class=3D"attribute-value"= pan>=3D"<a class=3D"attribute-value">activate</a>"/> <span id=3D"line39" class=3D""></span> <<span class=3D"start-= tag">link</span> <span class=3D"attribute-name">href</span>=3D"<a clas= s=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-eng= ine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/= 017dd620-65ec-4fa4-8d46-67737dedd7a1/deactivate">/ovirt-engine/api/v4/datac= enters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4f= a4-8d46-67737dedd7a1/deactivate</a>" <span class=3D"attribute-name">re= l</span>=3D"<a class=3D"attribute-value">deactivate</a>"/> <span id=3D"line40" class=3D""></span> </<span class=3D"end-tag">= actions</span>> <span id=3D"line41" class=3D""></span> <<span class=3D"start-tag"= pan class=3D"attribute-name">id</span>=3D"<a class=3D"attribute-value"= pan>=3D"<a class=3D"attribute-value">activate</a>"/> <span id=3D"line71" class=3D""></span> <<span class=3D"start-= tag">link</span> <span class=3D"attribute-name">href</span>=3D"<a clas= s=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-eng= ine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/= e572c874-17f8-4ad8-9bac-b6f563a9fc17/deactivate">/ovirt-engine/api/v4/datac= enters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4a= d8-9bac-b6f563a9fc17/deactivate</a>" <span class=3D"attribute-name">re= l</span>=3D"<a class=3D"attribute-value">deactivate</a>"/> <span id=3D"line72" class=3D""></span> </<span class=3D"end-tag">= actions</span>> <span id=3D"line73" class=3D""></span> <<span class=3D"start-tag"= pan>> <span id=3D"line76" class=3D""></span> <<span class=3D"start-tag"= pan class=3D"attribute-name">id</span>=3D"<a class=3D"attribute-value"= centers/5a54bf81-0228-02bc-0358-000000000304/storagedomains/f5914df0-f46c-4= cc0-b666-c929aa0225ae/deactivate</a>" <span class=3D"attribute-name">r= el</span>=3D"<a class=3D"attribute-value">deactivate</a>"/> <span id=3D"line105" class=3D""></span> </<span class=3D"end-tag"= pan>> <span id=3D"line121" class=3D""></span> <<span class=3D"start-tag= ">supports_discard</span>>false</<span class=3D"end-tag">supports_dis= card</span>> <span id=3D"line122" class=3D""></span> <<span class=3D"start-tag= ">supports_discard_zeroes_data</span>>false</<span class=3D"end-tag">= supports_discard_zeroes_data</span>> <span id=3D"line123" class=3D""></span> <<span class=3D"start-tag= ">type</span>>iso</<span class=3D"end-tag">type</span>> <span id=3D"line124" class=3D""></span> <<span class=3D"start-tag= ">used</span>>38654705664</<span class=3D"end-tag">used</span>> <span id=3D"line125" class=3D""></span> <<span class=3D"start-tag= ">warning_low_space_indicator</span>>10</<span class=3D"end-tag">warn= ing_low_space_indicator</span>> <span id=3D"line126" class=3D""></span> <<span class=3D"start-tag= ">wipe_after_delete</span>>false</<span class=3D"end-tag">wipe_after_= delete</span>> <span id=3D"line127" class=3D""></span> <<span class=3D"start-tag= ">data_center</span> <span class=3D"attribute-name">href</span>=3D"<a = class=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt= -engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304">/ovirt-eng= ine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304</a>" <span= class=3D"attribute-name">id</span>=3D"<a class=3D"attribute-value">5a= 54bf81-0228-02bc-0358-000000000304</a>"/> <span id=3D"line128" class=3D""></span> <<span class=3D"start-tag= ">data_centers</span>> <span id=3D"line129" class=3D""></span> <<span class=3D"start= -tag">data_center</span> <span class=3D"attribute-name">href</span>=3D"= ;<a class=3D"attribute-value" href=3D"view-source:https://virtman.cluster/o= virt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304">/ovirt= -engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304</a>" <= span class=3D"attribute-name">id</span>=3D"<a class=3D"attribute-value= ">5a54bf81-0228-02bc-0358-000000000304</a>"/> <span id=3D"line130" class=3D""></span> </<span class=3D"end-tag"= pan dir=3D"ltr" class=3D""> <<a href=3D"mailto:callum@well.ox.ac.uk" target=3D"_blank" class=3D"">ca= llum@well.ox.ac.uk</a>></span> wrote:<br class=3D""> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <div style=3D"word-wrap:break-word;line-break:after-white-space" class=3D""= they're big, full of noise right now. <div class=3D""><a href=3D"https://www.dropbox.com/s/f8q3m5amro2a1b2/engine= .log?dl=3D0" target=3D"_blank" class=3D"">https://www.dropbox.com/s/<wbr cl= ass=3D"">f8q3m5amro2a1b2/engine.log?dl=3D<wbr class=3D"">0</a></div> <div class=3D""><a href=3D"https://www.dropbox.com/s/uods85jk65halo3/vdsm.l= og?dl=3D0" target=3D"_blank" class=3D"">https://www.dropbox.com/s/<wbr clas= s=3D"">uods85jk65halo3/vdsm.log?dl=3D0</a></div> <div class=3D""><br class=3D""> <div class=3D""> <div class=3D""><span class=3D""> <div class=3D""> <div style=3D"letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: br= eak-word;" class=3D""> <div style=3D"word-wrap:break-word" class=3D""> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px;" class=3D""> Regards,</div> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px;" class=3D""> Callum</div> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px;" class=3D""> <br class=3D""> </div> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px;" class=3D""> --<br class=3D""> <br class=3D""> Callum Smith</div> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px;" class=3D""> Research Computing Core</div> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px;" class=3D""> Wellcome Trust Centre for Human Genetics<br class=3D""> University of Oxford<br class=3D""> e. <a href=3D"mailto:callum@well.ox.ac.uk" target=3D"_blank" class=3D"">cal= lum@well.ox.ac.uk</a></div> </div> </div> </div> </span> <div class=3D""> <div class=3D"h5"> <div class=3D""><br class=3D""> <blockquote type=3D"cite" class=3D""> <div class=3D"">On 2 May 2018, at 10:43, Fred Rolland <<a href=3D"mailto= :frolland@redhat.com" target=3D"_blank" class=3D"">frolland@redhat.com</a>&= gt; wrote:</div> <br class=3D"m_2571234484608545856Apple-interchange-newline"> <div class=3D""> <div dir=3D"ltr" class=3D""> <div class=3D"">Which version are you using?<br class=3D""> Can you provide the whole log?<br class=3D""> <br class=3D""> </div> For some reason, it looks like the Vdsm thinks that the Storage Domain is n= ot part of the pool.<br class=3D""> </div> <div class=3D"gmail_extra"><br class=3D""> <div class=3D"gmail_quote">On Wed, May 2, 2018 at 11:20 AM, Callum Smith <s= pan dir=3D"ltr" class=3D""> <<a href=3D"mailto:callum@well.ox.ac.uk" target=3D"_blank" class=3D"">ca= llum@well.ox.ac.uk</a>></span> wrote:<br class=3D""> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <div style=3D"word-wrap:break-word;line-break:after-white-space" class=3D""= per</div> <div class=3D""> return method(self, *args, **kwargs)</div> <div class=3D""> File "/usr/lib/python2.7/site-packa<wbr class= =3D"">ges/vdsm/storage/sp.py", line 1130, in activateSD</div> <div class=3D""> self.validateAttachedDomain(do<wbr class=3D""= per</div> <div class=3D""> return method(self, *args, **kwargs)</div> <div class=3D""> File "/usr/lib/python2.7/site-packa<wbr class= =3D"">ges/vdsm/storage/sp.py", line 557, in validateAttachedDomain</di= v> <div class=3D""> raise se.StorageDomainNotInPool(self<wbr clas= s=3D"">.spUUID, dom.sdUUID)</div> <div class=3D"">StorageDomainNotInPool: Storage domain not in pool: u'domai= n=3Df5914df0-f46c-4cc0-b6<wbr class=3D"">66-c929aa0225ae, pool=3D5a54bf81-0= 228-02bc-0358-0<wbr class=3D"">00000000304'</div> <div class=3D"">2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [st= orage.TaskManager.Task] (Task=3D'7f21f911-348f-45a3-b79c<wbr class=3D"">-e3= cb11642035') aborting: Task is aborted: "Storage domain not in pool: u= 'domain=3Df5914df0-f46c-4cc0-b6<wbr class=3D"">66-c929aa0225ae, pool=3D5a54bf81-0228-02bc-0358-0<wbr class=3D"">00000000304'" - code = 353 (task:1181)</div> <div class=3D"">2018-05-02 09:16:03,462+0100 ERROR (jsonrpc/0) [storage= .Dispatcher] FINISH activateStorageDomain error=3DStorage domain not in poo= l: u'domain=3Df5914df0-f46c-4cc0-b6<wbr class=3D"">66-c929aa0225ae, pool=3D= 5a54bf81-0228-02bc-0358-0<wbr class=3D"">00000000304' (dispatcher:82)</div> </div> <div class=3D""><br class=3D""> </div> <div class=3D"">engine.log</div> <div class=3D""><br class=3D""> </div> <div class=3D""> <div class=3D"">2018-05-02 09:16:02,326+01 INFO [<a href=3D"http:= //org.ovirt.engine.core.bll.st/" class=3D"">org.ovirt.engine.core.bll.st</a= div> <div class=3D"">2018-05-02 09:16:02,407+01 INFO [<a href=3D"http:= //org.ovirt.engine.core.bll.st/" class=3D"">org.ovirt.engine.core.bll.st</a= mainVDSCommand] (EE-ManagedThreadFactory-engin<wbr class=3D"">e-Thread-3345= 5) [93433989-8e26-48a9-bd3a-2ab95<wbr class=3D"">f296c08] START, ActivateStorageDomainVDSComman<wbr class=3D"">d( ActivateStorageDom= ainVDSComman<wbr class=3D"">dParameters:{storagePoolId=3D'5a<wbr class=3D""= mainVDSCommand] (EE-ManagedThreadFactory-engin<wbr class=3D"">e-Thread-3345= 5) [93433989-8e26-48a9-bd3a-2ab95<wbr class=3D"">f296c08] FINISH, ActivateStorageDomainVDSComman<wbr class=3D"">d, log id: 5c864594<= /div> <div class=3D"">2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core= .bll.sto<wbr class=3D"">rage.domain.ActivateStorageDom<wbr class=3D"">ainCo= mmand] (EE-ManagedThreadFactory-engin<wbr class=3D"">e-Thread-33455) [93433= 989-8e26-48a9-bd3a-2ab95<wbr class=3D"">f296c08] Command 'org.ovirt.engine.core.bll.sto<wbr class=3D"">rage.domain.Activate= StorageDom<wbr class=3D"">ainCommand' failed: EngineException: org.ovirt.en= gine.core.vdsbroke<wbr class=3D"">r.irsbroker.IrsOperationFailed<wbr class= =3D"">NoFailove</div> <div class=3D"">rException: IRSGenericException: IRSErrorException: Storage= domain not in pool: u'domain=3Df5914df0-f46c-4cc0-b6<wbr class=3D"">66-c92= 9aa0225ae, pool=3D5a54bf81-0228-02bc-0358-0<wbr class=3D"">00000000304' (Fa= iled with error StorageDomainNotInPool and code 353)</div> <div class=3D"">2018-05-02 09:16:02,636+01 INFO [<a href=3D"http:= //org.ovirt.engine.core.bll.st/" class=3D"">org.ovirt.engine.core.bll.st</a= the ISO storage domain manually? An NFS mount didn't add it back to the "pool" unfortunately.</div> <div class=3D""><br class=3D""> </div> <div class=3D"">On a separate note, is it possible to migrate this storage = to a new location? And if so how.</div> <div class=3D""><br class=3D""> <div class=3D""> <div style=3D"letter-spacing:normal;text-align:start;text-indent:0px;text-t= ransform:none;white-space:normal;word-spacing:0px;word-wrap:break-word" cla= ss=3D""> <div style=3D"word-wrap:break-word" class=3D""> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> Regards,</div> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> Callum</div> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> <br class=3D""> </div> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> --<br class=3D""> <br class=3D""> Callum Smith</div> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> Research Computing Core</div> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> Wellcome Trust Centre for Human Genetics<br class=3D""> University of Oxford<br class=3D""> e. <a href=3D"mailto:callum@well.ox.ac.uk" target=3D"_blank" class=3D"">cal= lum@well.ox.ac.uk</a></div> </div> </div> </div> <br class=3D""> </div> </div> <br class=3D""> ______________________________<wbr class=3D"">_________________<br class=3D= ""> Users mailing list<br class=3D""> <a href=3D"mailto:Users@ovirt.org" target=3D"_blank" class=3D"">Users@ovirt= .org</a><br class=3D""> <a href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer= " target=3D"_blank" class=3D"">http://lists.ovirt.org/mailman<wbr class=3D"= ">/listinfo/users</a><br class=3D""> <br class=3D""> </blockquote> </div> <br class=3D""> </div> </div> </blockquote> </div> <br class=3D""> </div> </div> </div> </div> </div> </blockquote> </div> <br class=3D""> </div> </div> </blockquote> </div> <br class=3D""> </div> </div> </div> </div> </div> </div> </blockquote> </div> <br class=3D""> </div> </div> </blockquote> </div> <br class=3D""> </body> </html> --_000_FB71A3FB6A03472B8BC9B2EB92C1C9C2welloxacuk_-- --_004_FB71A3FB6A03472B8BC9B2EB92C1C9C2welloxacuk_ Content-Type: image/png; name="Screen Shot 2018-05-02 at 14.54.42.png" Content-Description: Screen Shot 2018-05-02 at 14.54.42.png Content-Disposition: inline; filename="Screen Shot 2018-05-02 at 14.54.42.png"; size=24091; creation-date="Wed, 02 May 2018 14:09:14 GMT"; modification-date="Wed, 02 May 2018 14:09:14 GMT" Content-ID: <921E5257-26F4-4E0D-83B0-68F996842D08@well.ox.ac.uk> Content-Transfer-Encoding: base64 iVBORw0KGgoAAAANSUhEUgAAAaoAAACvCAYAAACsNZrxAAAKtWlDQ1BJQ0MgUHJvZmlsZQAASImV lgdUU9kWhs+96Y0WiICU0Jt0gQDSa+jSQVRCEkIoISYEFRsigyM4FkREQBlRQUTBUSky2LBgYRAs YJ8goqKOgwVRUXkXeISZ99Z7b73/rn3Pt3b22Wefk3vW2gCQn7KEwnRYAYAMQZYo3M+THhsXT8c9 ARCAAQGYAxUWWyz0CAsLAohmxr/rYx8SjeiW+WSuf//9v0qRwxWzAYDCEE7iiNkZCJ9E7BpbKMoC AJWL+PWWZwkn+QDCyiKkQIRbJ5k3zV2TnDTN0qmYyHAvhD8AgCezWCIeAOTJtejZbB6Sh0xH2ErA 4QsQnlzXlZ3C4iC8DeF5GRmZk9yOsHHSX/Lw/pYzSZaTxeLJeHovU8J788XCdNbK//M4/rcy0iUz a+ghRk4R+YcjI23y3NIyA2UsSAoJnWE+Zyp+ilMk/lEzzBZ7xc8wh+UdKJubHhI0w8l8X6YsTxYz coZFmeGy/FyxT8QMs0Sza0nSojxk63KZspw5KZExM5zNjw6ZYXFaROBsjJfML5KEy2pOFvnK9pgh /su++ExZfFZKpL9sj6zZ2rjiWFkNHK63j8wviJLFCLM8ZfmF6WGyeG66n8wvzo6Qzc1CPrbZuWGy 80llBYTNMPAGPiAIeeggCtgAB2AN7ABSYRZ3RdbkBrwyhStFfF5KFt0DuUFcOlPAtphHt7GyZgAw eR+n/+73d6fuGUTDz/oyewBwqEOgdtbHSgSgFTkZFb1Zn8FBAOSfAnCazZaIsqd96MkXBhCBPFAG akAL+Z6MkRtvA+yBM3BHKg4AoSASxIElgA1SQAYQgeVgNVgPCkAR2AZ2gnJQBfaDQ+AoOA5aQDs4 Dy6D66AH3AEPgBQMgVdgBHwE4xAE4SAKRIXUIG3IADKDbCAG5Ar5QEFQOBQHJUI8SABJoNXQBqgI KobKoX1QHfQLdAo6D12FeqF70AA0DL2DvsAomAwrw5qwIWwJM2APOBCOhBfDPHgZnAPnw1vgMrga PgI3w+fh6/AdWAq/gkdRAEVC0VA6KHMUA+WFCkXFo5JRItRaVCGqFFWNakC1oTpRt1BS1GvUZzQW TUXT0eZoZ7Q/OgrNRi9Dr0VvRpejD6Gb0RfRt9AD6BH0dwwFo4ExwzhhmJhYDA+zHFOAKcXUYJow lzB3MEOYj1gsloY1wjpg/bFx2FTsKuxm7B5sI/Ycthc7iB3F4XBqODOcCy4Ux8Jl4Qpwu3FHcGdx N3FDuE94El4bb4P3xcfjBfg8fCn+MP4M/ib+OX6coEAwIDgRQgkcwkrCVsIBQhvhBmGIME5UJBoR XYiRxFTiemIZsYF4ifiQ+J5EIumSHEkLSXxSLqmMdIx0hTRA+kxWIpuSvcgJZAl5C7mWfI58j/ye QqEYUtwp8ZQsyhZKHeUC5THlkxxVzkKOKceRWydXIdcsd1PujTxB3kDeQ36JfI58qfwJ+RvyrxUI CoYKXgoshbUKFQqnFPoVRhWpitaKoYoZipsVDyteVXyhhFMyVPJR4ijlK+1XuqA0SEVR9aheVDZ1 A/UA9RJ1SBmrbKTMVE5VLlI+qtytPKKipDJfJVplhUqFymkVKQ1FM6Qxaem0rbTjtD7alzmaczzm cOdsmtMw5+acMdW5qu6qXNVC1UbVO6pf1OhqPmppatvVWtQeqaPVTdUXqi9X36t+Sf31XOW5znPZ cwvnHp97XwPWMNUI11ilsV+jS2NUU0vTT1OouVvzguZrLZqWu1aqVonWGa1hbaq2qzZfu0T7rPZL ugrdg55OL6NfpI/oaOj460h09ul064zrGulG6ebpNuo+0iPqMfSS9Ur0OvRG9LX1g/VX69fr3zcg GDAMUgx2GXQajBkaGcYYbjRsMXxhpGrENMoxqjd6aEwxdjNeZlxtfNsEa8IwSTPZY9JjCpvamaaY VpjeMIPN7M34ZnvMeudh5jnOE8yrntdvTjb3MM82rzcfsKBZBFnkWbRYvLHUt4y33G7Zafndys4q 3eqA1QNrJesA6zzrNut3NqY2bJsKm9u2FFtf23W2rbZv55vN587fO/+uHdUu2G6jXYfdN3sHe5F9 g/2wg75DokOlQz9DmRHG2My44ohx9HRc59ju+NnJ3inL6bjTn87mzmnOh51fLDBawF1wYMGgi64L y2Wfi9SV7pro+rOr1E3HjeVW7fbEXc+d417j/tzDxCPV44jHG08rT5Fnk+eYl5PXGq9z3ihvP+9C 724fJZ8on3Kfx766vjzfet8RPzu/VX7n/DH+gf7b/fuZmkw2s445EuAQsCbgYiA5MCKwPPBJkGmQ KKgtGA4OCN4R/DDEIEQQ0hIKQpmhO0IfhRmFLQv7dSF2YdjCioXPwq3DV4d3RlAjlkYcjvgY6Rm5 NfJBlHGUJKojWj46IboueizGO6Y4RhprGbsm9nqcehw/rjUeFx8dXxM/ushn0c5FQwl2CQUJfYuN Fq9YfHWJ+pL0JaeXyi9lLT2RiEmMSTyc+JUVyqpmjSYxkyqTRthe7F3sVxx3TglnmOvCLeY+T3ZJ Lk5+wXPh7eANp7illKa85nvxy/lvU/1Tq1LH0kLTatMm0mPSGzPwGYkZpwRKgjTBxUytzBWZvUIz YYFQusxp2c5lI6JAUY0YEi8Wt2YpI41Pl8RY8oNkINs1uyL70/Lo5SdWKK4QrOhaabpy08rnOb45 B1ehV7FXdazWWb1+9cAajzX71kJrk9Z2rNNbl79uKNcv99B64vq09b/lWeUV533YELOhLV8zPzd/ 8Ae/H+oL5ApEBf0bnTdW/Yj+kf9j9ybbTbs3fS/kFF4rsioqLfq6mb352k/WP5X9NLEleUv3Vvut e7dhtwm29W13236oWLE4p3hwR/CO5hJ6SWHJh51Ld14tnV9atYu4S7JLWhZU1rpbf/e23V/LU8rv VHhWNFZqVG6qHNvD2XNzr/vehirNqqKqLz/zf767z29fc7Vhdel+7P7s/c8ORB/oPMg4WFejXlNU 861WUCs9FH7oYp1DXd1hjcNb6+F6Sf3wkYQjPUe9j7Y2mDfsa6Q1Fh0DxyTHXv6S+Evf8cDjHScY JxpOGpysbKI2FTZDzSubR1pSWqStca29pwJOdbQ5tzX9avFrbbtOe8VpldNbzxDP5J+ZOJtzdvSc 8Nzr87zzgx1LOx5ciL1w++LCi92XAi9duex7+UKnR+fZKy5X2q86XT11jXGt5br99eYuu66m3+x+ a+q2726+4XCjtcexp613Qe+Zm243z9/yvnX5NvP29Tshd3r7ovru9if0S+9y7r64l37v7f3s++MP ch9iHhY+UnhU+ljjcfXvJr83Su2lpwe8B7qeRDx5MMgefPVU/PTrUP4zyrPS59rP617YvGgf9h3u ebno5dAr4avx1wV/KP5R+cb4zck/3f/sGokdGXorejvxbvN7tfe1H+Z/6BgNG338MePj+FjhJ7VP hz4zPnd+ifnyfHz5V9zXsm8m39q+B35/OJExMSFkiVhTrQAKMTg5GYB3tQBQ4gCgIn0FcdF0vzwl aLrHnyLwn3i6p56SPQB17gBMtm3ByLgHGQ2RUR6xMMQi3QFsayuzf0qcbGsznYvUgrQmpRMT75E+ EWcCwLf+iYnxlomJbzVIsfcBOPdxuk+flMIRAHr4VuF2EbcHMLngX/QPVBQHsKaAZSMAAAGdaVRY dFhNTDpjb20uYWRvYmUueG1wAAAAAAA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEv IiB4OnhtcHRrPSJYTVAgQ29yZSA1LjQuMCI+CiAgIDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDov L3d3dy53My5vcmcvMTk5OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+CiAgICAgIDxyZGY6RGVzY3Jp cHRpb24gcmRmOmFib3V0PSIiCiAgICAgICAgICAgIHhtbG5zOmV4aWY9Imh0dHA6Ly9ucy5hZG9i ZS5jb20vZXhpZi8xLjAvIj4KICAgICAgICAgPGV4aWY6UGl4ZWxYRGltZW5zaW9uPjQyNjwvZXhp ZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj4xNzU8L2V4 aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICA8L3JkZjpEZXNjcmlwdGlvbj4KICAgPC9yZGY6UkRG Pgo8L3g6eG1wbWV0YT4Kleq4VAAAQABJREFUeAHtfQlgFEXW/y/XzOROJpATk3AkgHKEQwmrcijI 5QF8okFRUGFXwe8P0ZXAKnwsoBJ2FVgFVHQ5RAHdBURBjsilLnEFCWdCgpAEEkIgd0JmJtf/Vff0 ZCYXE5iEQF7BTHdXvXpV9avq+vV7VZO2q6KABsLx48fRqVMnC4nKykqYf4SK66ixyN/QRU1ddnZ2 EJ+mDEoZ9vb2MP80ZZmsmxFgBBgBRsA6BBytE7OUEhO7mNCVo0i1JVFZloZmISpRpkJY4siBEWAE GAFGoGUgcMNEJSZzhZyUo62apOhWjrbS25AehZyUY0OynMYIMAKMACPQfAjcEFEp1VMmdeWoxNvi qOhUjrbQyToYAUaAEWAEbj8ErCKqs2fP3n4t4xozAowAI8AI3BEIWEVUXbt2bfbGJiYm4laU2+wN NRbY2tp7q3BubeXyuLK+xxkr67FqbkmriMrWa1DWNvJWlWtt/Wwt19raa2v8WF/dCPC4qhuXumIZ q7pQufVx9re+ClwDRoARYAQYAUagfgTYoqofm2ZP4ae5Zoe8VRTI48r6bmasrMeqOSXZompOtLks RoARYAQYgUYjwBZVoyFrugz8NNd02LZmzTyurO99xsp6rJpTki2q5kSby2IEGAFGgBFoNAJsUTUa sqbLwE9zTYdta9bM48r63mesrMeqOSXZompOtLksRoARYAQYgUYjwBZVoyFrugwt/Wku49Q+nMr2 xf2D74Fr08HAmm2MwPXGlcFgAFQqqGxc7u2o7npY3eo2tdZ70CqiutWd02rLNyRj3rNv4rQZAJ6h 92LSlCm4P9zbLLY5TjOwat4KqS7OYd+gXzuD9RObWTucIqZg9ZuPyHkp/g1qX1roM1j9tzFMfs3R jaYySpCwZTn+9uWvKDPGOXneg+l//QvuC7p9KSs7+RSueYUi1NdGj1JmY1eB7va7B0uwdfokfJmp tMDsGEj33rKWf+9ZRVS36injVpVr1o3NelqrvVUG5Cs1cPSCY3k+ClJ/xbI3f8V///QPzHjYX0lt hmMgJv/lT/gpLRRBjo4o0evhZO18ZtaOsoRVWPPTYLxwvyMcKV6aJHMMuHxZjSBfnfXk1wwtvlOK qDWuqGGnvvgL3vlGnrkc3b2AonyUFWSgWO+JvGIdvFwrb7/mlyQg+s13gIELsfLFrnB3rmh0G2ph ZTZ2cdveg/S+QCcnQMz25cpjCV3SdbmdCpez6N7za9n3nlVE1eje5gw2Q4CGF4V2+N/FHyPMSY+U b/4PH8SdwKGP1+DhPnPQ3asMJRk/YxlZO8cKxCD0RN9npmH66Aia9EuwY94MbHW6HyN9U7Fhj7DN nDBi+l8QlvI1/rFDvu478f/wxqhwSsvH/jXL8eWeY5BUkWzfZ94w6Tq9ZxfiLvihY/+7EYRkvPPs X4Gh4xCYuh3fny6g/IEY/5c3MDoiiM4tg9wOOS5u+ed4uPsUdNCI2lBwpk8VvTamJBlrlq5C3LE0 mcCcSN8bRn30ZPvOC3J5bam8OFGe0934f38Zh5Svl5rKn7hwHkaG08RLIfXnNXhn2Q6ImgnZyf83 A0ONaSKq1YaSU1hmJKmBr/wT/3O3H0FRhIvn89HOhU4dK5Hx8xr81YRdCPXDDLlfG9kPJdRXf5q5 B0MmDkLal1/iNA1Rp7ufwOxxQfjXO2Shi+vAgZj99jTcYzSC6us3c12ZW7+k8S7y3os//2U6Ilwz yeL/mzxuDvwVLx9wxbiFSzA6/OYtq9v/HnTF6L9txIBsDVK2/gnvx12E76B38db/9ADKzmHlX/4H Z3WBmL78b7iXbh2Bc/Sba4GwZ/D2JCdEz9yKgRNH48rWtRLmnj2fweI3R0O+y0rw85q/GecSmn1C huKNeVMQdvOwW9yeVm2mEE8Zzf0RtWzuMm9lefW119RbZVXQaMtw/5RX0F96vPgNp9OrUFJwFNHR yySScvdtTw9NBTj85Tv486fJKKN+y8svQMGxHURSKfDzE7dcGb5f9ldpYPn5iaFWhsNr/47/XFQR 3s4wpBJJOXfH/YMehq8jpZGudb+WSbqyM1JRmJUBXYk9dGUGXCkrw7EdXxJJ2CHIXejOxIZ3NuFU vlOtvpPa4d6e5Ois/Hus25eLUnrKo3mqOrg4IZNIShMWifv79YZjmdD3Hg5dEfqqy4tLdoOfwKDs NP7x179K5fsZy1877ztcLHRA8clPECMmWsf2uP/+SLiT7KdvzcSPFxxq1e1W9ntTly3ArVlG8fkE mbzdn8QIIikHtR6+vnbo3a8tXJzLYJfyCV4zkpRvUDvCOY36IRqf/WJodD9UGchSo3Hx/dovkawN kh7qy05/g/l/JZJyDoI79WNZ5gEs+fQsCnR2Dfebma5jVe2NeX/Fu8uP4cqF04hLU0aTK7S+3tAX OSOHxkLN9td3XRdWQtYUbut7sBzebYvh7epgag4cyuAb5EsPb9QHoo+/y0J+qR2Sd2+VHlTVgX1Q fLmY+q8AcWuJpKrIq0O5C459iTkrU1BK2Jz6ZIY0lzi2E/dsdxSk7cFbU/6JC9SX9eF8I/FWEVV1 y/jsliIgeWM0aGt8Wim8Voaze7dJk457v/l4e86HeP+tGdJgunzoV1zMVZvWfYa89hXefGs97hdE QeG+V9bT9ccYIl3nIyFVh0KDCkNnr8X//b9pGPTAE3jwbvmZyaB3RGGRo0kXvQsZVVQX+UmzPf43 9gvEEKF0kjRfRF6eI65J5zW+PIbQ+tpEKTLxixVISnMwPpUpcqGYveoTTH96EgYNfhx3G+taUeyE wnKz8hZ/gjff/ivkZC9MpvLf/Ot8+Ao15b8hNQf4YW2cpDRo0AsYMXIiHqL5VliMJ2gyK9BJSa33 SyU33TG0A1xQCXdPZYIvh5t7MX4yYnf/lPV4a9bHeHNCpJThUPwF5BY5VPf7dftBBb2CctAkLHzr Eyx8ZYSx8BFY+PYnmDf1Sem66PhvyC2uwoEG+q1QUeb7JBa9/SHefn2KrOtiMop9xmLB/7SXrn2H zMFbcz7AoHZu9NZuWcRm37f7PVgHEH2fGifFZuw7iPxreYiLEz4I8rwM7gLYG4m60zT84+0vsPD/ PS7JZh/+L7KyMvEvSRa4/7FJGDF0PMJEavlJpF3SwJa3mSBIDrcVAmW4ZmQBDxcVrl3Jl2pfdI12 btmXw7dzG3LAAelFNKmU2oPmdwrt0bWdCo7OhQim2f3nIi/0DvWGvToHbcVsXyRk7FB2MR6vzV5C z7+1g/mDpUWq70AEuZC15+WGewOAs5dEqh3KqTq1FpwK86Dt8Dxe7LwW/zzzG/65WgXzJ6XMX1dh 1t9lgrEoQ1woSyaiPCcqz7sCWoouch+IUCpf7VwqeRDlfGXIFeVTyIibi7fMVJZLdaNhr5GRkaVa 2bcRm/LL2SijicjsGZuAKEO+kbcK6cpBrUPn7oIA4lF0+SJKy6iTRbCqH+xgMOry7dodLnYVcDJu 1HDv1x8e1KmOvkYB4f6tqECO8bLOfiuXR4tvj/4SwXp1C0EwZUt3pkemSiqr3DhIxOBzpHtBqzCb qLAtw218D9YBgyp0IB7y+BJ7C+Px009VOCZkgp5FVw+6hXXyOp8vPdSA+u+ue+8jzLfJmOtKkWfU d2Dlyzhg0l1MZ9Qf1A0a40ORKekGT6wiKgvz9wYLupFst6rcG6mrLfLUaq+Z10E8xor0U1tXIE66 mXujqx+RT0BbKjoD7h6ymVVVkie7dRz94OFkhwpJB2Ugt4UjTc6ySjdSVwUnTZnxWq59ctwmiaQc u8/E+38ciJR//wkf7L9obBq5f80baboQg7ISaiIPWmaqDlRXE7kpsjQZUTUwOPr/Yd3L/0B2Yny1 PLHlj6tkRuk+6Z+Y0scV//7L0zggkSiJKTpgLE+ZXWmScqTyaXe1RRDznghdn/4ArzzQjlwbtHGj lHDwcCNdpRKWssSd/11zXLl4tpEbnb0Vv1x6FI/5VMkPDNI2dUe0FYY0Pa14ELYC9pJs+dHF3ccP TkQIcldY1w+mQSDIw74STsZB4SwW9x0qoFIGSamoUhURkBzq7LcM4yAQuiivE7mDq4ccpRmT4UiD QVwquo06rTnUyqPoFJlv53vQ2PiakMjt9cTIZyOwd2UCvv1Cvt+7jxwgPVg4qmQAcjMo3r4jylNP IV3oEvcS9SFBIoWnYjbhAZqPxH1WSs+AztSRQnfN8ozijT5YRVSN1soZbIaAxEm4iJWLXoZL3kUU yhG4b8qr8KO70bXvADh+noCin2cjtughOCbulYjKUXpipQmo+k6utkqMtRNJ5i46F095ti+/fBD7 4w5js4mkbr45UrWlyYiq4fUHzH7k31iwO0NWbIx3EVxLXofLR3Yg7szhapKyonh7KsAIDUmTdffU vdiy7FckborFJv0j0OrO4eDOVLyy4j0EuLVia0pgGTQQL3T+DKvP5OOrhU/iP90i4F50HolpwJsf f4ruQ+4FTv+Kn//xJxT1D0XioQSpB3r0Fk/VqWbkIEVbfFn2g0WSFRdu6Df+Xnz9Xt39FuysmNV1 qwqIuBvYlobs/R9iZZoX/vDSqxgSevOr+vK4uv3vwbpRk2OD+j4OfyQgS7psj6Fd/WCv0pucIuWJ S7Dwg//A+ewvkoTj3WQhO3XG6D5O+MeRMny1OhaGwV2hy/wZP5yNwNJ3J4AcPjYLsi19HXUyMwp2 bL6PqFJzlnery6q7vY6mJ5bybJmkfDuNwMtvbcKEHvJAcvbtj/dinpfWazKO74VYT/btPg3zovrQ Uye5VqS+VZ57lMfD6mvavyBLkGC7PzyLcJEhOx6bv/kRkQ89bMwvyxgljReKLuVSua7WXY2psR3O ZM1QEPFdX3gV9xizQop3wwMvjZbKyz7xL2z71Q2DH+iuSIhcZufissY17bWVS6Zv+u/T5zXMfErU /yJ+3vpPfLtzPwo8OsJBp4a+vPnGcTUGt6ZMAVrtOjhh6LxP8PwD7SVML55MIJKiJwT3vigrcYZz X8JuTG9Ky8dxIikxUd839l08fbc3HBxrbPm+Tj+Yesn4SGwSN147VhltKGHpUmgjyq6n3wxK0SZd Ju1SXtfg/uhN7iqUE+mmHEXaFVfkF9/sZgplXJHa2/oeNI4/I9xOwuqkYBobLl0x8QF5Tdqx+7MI JQ+JyllvcZvlJv4izS/wHYGYp/tL61eRf34f4/rROMr+DVs3fYGdP6ZC0zYUuiIVKqmzTfpv8tyO FFn2tlT96q/jx48jNDS0OqKZzlJTU29Juc3UvFrF1Nvecg1y8mlHnnkOO3J1uejJNVNtQ+iKXGkT QynKnNzoSYeEab3KXXsNapqq86+60FoVret4FsHNqY7rHEqnAhw01+BJ5nzGZT0cXYR7kNQQ2VVW 2FNaKbzdqix12dGuKrHDDxXwbFNC8jV1m1Xa1A5FltIqqW25Stvk+MoiJ1zOc4CL9DhG6yfk4qOl C7nuNcsrr1G+qQyxZkZtdbRHcYEjcgvsyUuhEp4KUShcvYrhbJzszGp4R57WO66k1jqhINueNpeU ETbU30Z8nAkfV8KuiPo2j0x4J2UsOJXBy7OUhlZN3GteK2NW7ge1zpXKoI6kXWZtvGmMFltel9N1 vkinNRBPHzGO6u83xxq6ULMuZRpcvmRApbE9Ykx7W2lB14uVaVyZDZHb8R40Vr843x26cjvYORjg 461secjAZ6//GT9kAo+RG29osAZtaP4oSdmAKXO3Iejx9xAztIvk2hP3ksV9VKlCwRU7FJBbWLrP RDk0/3hRflvdZlYRVUhIiLGJzXdIS0vDrSi3+VpoWZJt2muPijIHVNqT/54I5saDPQ1IByKpMjhY ZXPfeEl15qx0QBmRk4NThcVmizplrYk06rMXGwduChdrCmtZMlaNqwrCu9Ie9jRuauEjYUdpt2Is 3ES/VZQR69bVnga6xyqsGsgvJ91+92DKtgWYtzFRrr7YUTnnBXiIh1ZX+o0mEdUf/+9b6XdXc57u Aq2nnh4c7WicKOZtNSCVNI5ErD09cNh63rAV4VXXls9uIQKVNLnT56ZrQIvepOeWBUG0tiRIW+u7 ZcA0UcFiY0IdE49UmoRd7UmpiWpiqfYm+s2BrL9bE26/e1Dl4kJ/pcILoX3G4emnR9OGlgq4EElJ wcmL/oQAudI9yPMhTQnUvnomGHsaQ7a8bc37zyqiuo530FyfTc9vVbk2bUQjlLW29jYCGha9CQR4 XFkPXmvEKvjh17DiXg/ZHUgPB87uRfTnzeRVYZeQ4Vi6ahyKyDVrZyd+8G2xCGE9sDcpaRVR3WQZ nJ0RYAQYAUagBSPg4lFo+mlAzWo6uRRBa9yEUTOtua6tIqpbxaK3qtzmAr9mOa2tvTXbz9dNgwCP K+txZaysx6o5JZvKpdicbeCyGAFGgBFgBO5gBNiiakGdy09zLagz7qCq8LiyvjMZK+uxak5Jq7an N2eFuCxGgBFgBBgBRsAcAassqh49epjnaZZz8UPjW1FuszSujkJaW3vrgICjmgABHlfWg8pYWY9V c0vyGlVzI87lMQKMACPACDQKASaqRsHFwowAI8AIMALNjQATVXMjzuUxAowAI8AINAoBJqpGwcXC jAAjwAgwAs2NABNVcyPO5TECjAAjwAg0CgEmqkbBxcKMACPACDACzY0AE1VzI87lMQKMACPACDQK Aat+R3U9jZUGA4oz0lFZXIyywnwYcq+gXJxfvQpDXi4qC/NQSumewx5D+PN/oheWMD9eD9ObSi8o hM7VA5ob7d2bzX9TlW/BmRmXFtw5XLU7GYEbncosMPlpTjSq4nbQm88r6U/Bixdn0dsj6Z89vbjE wcFROqrpVeG6zz5ChosHgp561iI/X9gSgUIs/eh3ZD3UEYvoT/fXGYpzkF/hQW9rlV7pWkPEivw1 crSOyxq4NIhh60CkdbRSj80TBsAw9yCiwtWto8ktsJU2IaoT//kJwfQa4m7tOkjWkr09kZOdeGOo +BBh2dO7TOhc/B2tyh92A0xUTTgUPDBpTFsgsB6SopITtqRjZ7dOmNWrLqK6fv4mrHwLVm2JS8MY tuBmcNUajYAhB9DDQPmYqBoNno0y2ISoHL29UaU3wNPTB2Xl9GZIQUjCtBKvgtS4QG8oReW1Erio nelNkjYp0kbNvw3UpJ/HrK/yESu9ZNUOGx4KRBRZSjtXJmJ/B18sGhYkN6I8G4vey0DomCB4/XAV +YM8EdUV2LryPPLDnZF0uBixZOXuuccBQzMpS+ZZzKZnhg2PdyI5dzMgihBvnn/V70hydUL8BQO+ Iakn7vLE0q4VeHl3MXbR9TB6Uc3GFzrDS3RrTjqWbshFdIn8crXpYV5YOra9pDv/aAomUR6ho6eD HfwrquDfpS3WPNEOWb8mYfjeUhyrqU/KqXwV1WrL0WkRiHDMxsbVmRhfKMoU+NxF+PgAuouY9VEu ugTa4YXz5ZIS8/pAR/k+p3y5cl1jqK6L6qiryLikbxBmPOxswmVQeip61cLQCQmbU9ArRS5L4LLm uc7w15CC5BSM/qECM/zKMJjSh/k5IusysHF6d3QR6RSyfjiF4YkqxL8aBmOUnMDfLQgBPXbPfwW/ hfRB8op1OEU1u2fqUsztU4S3XpqDFLoOGzUHy+Y+ChqBQFEylr81H+viRQowNGYFFo7tI53nnNyN NyiP0KENC4NPTgp8pm3CskdDkXlkM16aGotcSrPQJ+VsnV82WSxqG9wRdvSe4irx6XI3XOcuhNc7 S+D99+Vos3Ql0vv2xa7jvyI5K51cf3wbWj3UdBmYtCEfCYHuyHs5FIl9VRi/NwNbk4GIcAfEJuQj 36hMd+IqZtNL6CM7OCOLJu38cnkC1ukr8AKRVGhfLS6NCcQDD2rxLuV5t29b5E0Ixuj25iQlK7PI X1CF2RcqMGtMMPLGeMLrQgHa7y7BrMfpepwXNLnXMOOHbFOT/APdcGlyR1wa6YJlKfnYmU4PLuUZ mEEkFdnXF1Wvd8JHgSCSU2HRqHbAuRQEEEnNGuhL9WmL0QXX4L36nEmf+UnNtnRx02HnpxkYX6HB +cmhOC/hk479GVQmKpEk2p7tgMQJobj0iAv2U31m/UCPx9Bhq8hXoJLS8h53RxKlTdqeQWlE+FTX LhFtUSowf8gdkYHymFVw8Y+sjWHq9mQiqQrsI1xKJ/hgdMk1BHwqT1C6igp8U1iKwelUlzG+WDPU C1Eox5qfRV1EKMTOwwaE3uXCJCUD0mK/i5NPYcuKS5i2aQd2bFoK9xUz8PRLGzGFruO2rIBq+wLM 3y2eYuQQMHAKtsTFYcvnc7AndirkpFTMJ5IKX7wJv/xyEAuHAim5zyOGSEqfuRtjiKSmrNiEuB2b MDR5AUbOP6Soa7VHmxDV5cTjuHqtCL+ePYnCLl3hdk8POHfoAE1AAJzc3WHv6Q0dbbhQ00d1qboT Wy3qVjZcl1KAtTShb3ymE60neaPLw3djnxaYd/gy/Pt5Yhi5I/af05G2MsTH69GTrJ3QmgYrPeAP a98WLz8cAv9wX2g8PRBKObx8iXSCfKCx4rlhYjd/RIb7wCs8EMPJSO7Zvg0GdaXrDu0xj0gnIadU bpFPMKLGUl2pDl7B3lhCsakiTVeOBDofPjAIcKSJv68o1E6SSzpcQh4VZwzvRoTp5onhvShzro6o pI5Qsy1kFS2l7Bvu18KfxP37eiKGjgnZcn2Eji0jw9AlyBv+vTpjY3s7xJ6g59TibHxE+fa9cLeU 5tW1E9b0dcTakwVUroOka2dmKT0EuKHLvZ0Q2bWGG7UWhkXYf7ICMX3vknDRBAXj5ec8gZISJBQr 7bBD4stUXngQ/IPI6utCdTl6RW5nRjZeILFZDxA+HFo8ApELpqJPqA98QntjaBhZRNFzMZCu3QP7 4LXntUi+JGwhCu7hGDt2oGRduQf0xvMUdSm3iPyIxUiGFo8ODKUYNfoMHEtHN9AdgEsH15GZFoOB 3cgmUwVg4JRRAD0EUa5WHWpOazcERiWtR3mQS6+LhxZ2p4UxaxnsUAUVrVUFuHvS9EQuQQ7WISD1 jp3FU7a/D+Gno49bAGZ55GJG/FWMpqXBRYVEYMP8SK+wJqqDjlyGo2tMtDIJyBYXitMQsTxXcrsJ 8kic1rE6M52J/JHBzsY40k3XUaZr8uyqqS7lcp9m/XAaww/rZRceufeEa3C1aIObMybRYdbqJCwi q2fjbiISX7LGKA4iXV8K7+W/iytTSM1JQ9SnlvWqqy1Cx/jdwhKqDtMz6bbuKq7tyPXnZErwcqXn Mn0ZdI7yWoOX2ZKDlyRXSbI+mDGhGLqvcxHw0Un0dFVj43giO5o3zIMFhpQg6uGvrS4Lrip6kFCC 0KtCqNRgOS6UXLNIImu5AAglixdqN0TWKEPJzceWg4BYr+pNpKQEcT0wTFCMHNx8KE2vki4yD63F SzNWyC48cu+lUGy0SFITSWlzsWj+ZkSPDcD2N2KBMSskospRkcCpWIwcQHFK0I5Cjp54z2y8Kkmt 5WgTonJQq+FIGyhcVGqU17H13CO4Pc0ZYjOFmDo4WI2AtNxRJT11K3Ocjlxxx9SCZJwQ+YAax3YU IOEH4UpTY2sHIWVJVNctyy0Q8ZM9jBaMI7zcgPhamYykRvFigvaiSbhWEGtkRFLDaT0n4WFfSi7E otjfjXp9EdUtC9EnDRhOlkdEXx9UPRwsqxAuSq0nqqYQ21oEshKvVy/Km0p59k3uhUHVc4esRZdO xyokZZahSweZQHR6Igyxbkr5sihVEJ8SdMXiwjg6g0Iwa0YIZmRcxJqvr6DrV2moekU8BNQX5D4S rklTIELcReUvMkXUOPFsh+9d8/HRD2cRmVKFdwe2qSHAly0XAQNVTWYNcRbgTjeNMYhrOWRiBZFU /6VbMLc/uR3IJlrUbwhtyhAhEGOiI7FuyW+I9wlA77mfI6Z/uJRiMFwCRi3FL3P7S9f8JSNAj5g3 H7xD6ClcsJDxRhe/q9IXFqA06xKKaTNA9k/7YKCt6xfo91RXXFxuvsBWokET7Cq59xbtuii1OD+R NiTQctDKrt7StaZ7G1pvMqAXrW9Mj2hjYXnVD5EGoeQ+fIUsMZ1Eak7Q+HjDS/pUPxnWn1+kVBOX Sa7cIK2XeUkROiT9cIHWzGRLQ9Bb/PkKPNHeDVH3eyHyLgdk5cg2SWhXuuFzC7DxV2pYObn8MjKQ lJxHOa2ol5s3ZhHvDP7qDLIKSJ+uEFkn0pEvq5bw+OjnNFqvI3UZ5zGPCGFid29o3Dwwg6L6f5Ui pxVcxNK9BnKdukJTnoeEeNJRrIMmyA+Dgom8SswIiPKJVlli6IEIcivOPpCJJNEu0rFzs1g9VKNL 9Rwm5az+csKggWqsTSnCK+RujIqQ+7Q6nc9uHwSq6clUZ9pcVkQX8iOdHqkH1mELXcv0psfJ7fG4 Z+IojBkzCt0CVMgpkiksoPdEcvXNwOYjmURqehTlpCI5VWhq3cEmFlV+ajI0pSU4fjkD6oO7kZl0 BIarOaii3X52Oj3saTHZz80VleSbxRVLN03rhv86rfcMwdaRBozecQV2CVck4RjyXb/cS1kz8cXo sEzMJp/CpEhhxchB2FXGuVqarJVzOdUJERHkyttbAOfYAqwe2RGTuiv6rp9fQ8RgqY9cXmIUafww IyyXSDMDs+kDtSNiSFZ6eCECSaI1oW/OFyD1vFzGMdq0sIHKjup+NxIzT6ErbRIRG0VEmCh2A4bX nrjN2yVr8UDUlLbIWnWF3HSJchRN+onTgyETJkXlFMH7vaNSWk+tG+KHyThNerkIWR/lWKTtfyaE GpeBNQdysIw+crDDPtqEIkJ1+XVg+FQItqxKQ9dPlXrQRo3J8g6+mnjJeklfdz8s2ZGONbRZJlQo 59AiEZDJpvpbphS5qm7C02dWayHlI9hIHYopMUPJ9TdGIijcMwqjwugJUVCXPge/xZOHL34+XlpL Ubm5knsw5vM4jA2PwqYFOXh66hjapSuHyDm0GzDU2odIY6Y77GDVq+iv96bdjx/pB4+039FDq6W9 VuR8IutK+sGv+A0VXYsfADvQmZN4EvfWouN3P18Xxtb2ts2G20vuPHrCh4YGq00eLQh+Yb2QpaGx ZjfFdXvLTEBXRERG1pCZ3qQvj6JrgRe5z9qbBBNWH0UvRy2qniNyEIHqQxvhqH00Y99IG3VGOlDK Jdff8GW5WPp6BLoIai0n1tQ4SUVZfEn11VB9a6SRPgkftwYYpC4MrcmnVIC20I9edgVRY7rSj0kb KEeRv4Fjw+PqBhTewVmaBCu9sIuIt2h5RAnJa0fguZSF+GVhHyUKRxb1w1T3z/DLtG7GOMonZzRa YSbRVnliE9dfp8fHwYGsJlfy/2uIlNylD+BK7j5n+qgpzaGyDOXllaignV0cGosATaJuNiQpUTwR gjmZNLZG9coTmdbUq3El91nhNSSco7+IkZODpPgkyYW5JMzMLyYISpDMjZCUqIzIq5CUWeUk/pJ0 1yAiRUaqbx1ppEvTEEmJ/HVhaE0+XR6yztHPDFZfod+VqTG6iUhKaSIfbyECRFDmJCVqoqJ1KezZ g5OZOSgqykEy/W7qLfILPh9J8aYg56umN1NCqzy50WnBAqzBU19DPK1RVOzdCXsf2vnnqYU97QB0 onUPuLjDsa0vHImgVJ6ecBIbKzi0KgRCh92F70voN2Ffp0s7AnuSW3DeQ0EYLX6Y22TBniwpe8ld 12RF3Kji9KsYvoXc4LQpRXEP3qgqznf7IRA69G9YkLMEb4wZKbn8tOQWjF6xBY/0acr74fbDybzG NnH9mSu01XmTmOG2qlwT6Glt7W0CCFllHQjwuKoDlHqiGKt6gGkB0TZx/bWAdnAVGAFGgBFgBO5Q BJio7tCO5WYxAowAI3CnIMBEdaf0JLeDEWAEGIE7FAGr1qju0LZzsxgBRoARYARuAwSs2vXXqVOn Zm/K2bNncSvKbfaGGgtsbe29VTi3tnJ5XFnf44yV9Vg1tyS7/pobcS6PEWAEGAFGoFEIMFE1Ci4W ZgQYAUaAEWhuBJiomhtxLo8RYAQYAUagUQgwUTUKLhZmBBgBRoARaG4EmKiaG3EujxFgBBgBRqBR CDBRNQouFmYEGAFGgBFobgSYqJobcS6PEWAEGAFGoFEIMFE1Cq7bVFhfAp1eeq898hMzcDCxwHYN KSTdsmrb6WRNjAAjwAiYIcBEZQZGSzzVZeXeNBEc/+YifDbTq94pXE4sxogT12zU1BJ8uPYi3j5J r++tL5QUIL+Qmaw+eDi+pSNgwM6ZgzF4cPVn7MxPkJBVx+vn62yKAVm59EoXDjeFQJMS1b0/ud5U 5TjzNaz/+iqSK24OiR5D2uL0YOPL2R3s0N2RXmRok+CKCSO98XpY/f18fHsWPkuz9qa2SaVYCSNg UwQMZ4FhC9bi22+/xb83rMIk7U5Ejx+GbWevT0CG02swdU26TevTGpVZ9SeUbgQYQVKnHwHu3e2K Xx9o4In7RpS3kjx71l/AdNHWT87Qlz1++2MYOtOr1r7ecBWTiqoozg5rHvTDuAjx1mQ9ju+5iP5J svUyhAjpckUVXqeXFt6XlovlFR5YPNyF5MwD5dl+Af3PyUz4Wgd3LBgVaC5A53rsWZOKgyFaLBjc Vk4rz8V7K68geGRbeP2Yj/w/uGFcOPDtmgwUdFDjzLFSvE91+66zPR69TFkuX8Dc/cAaqS6ZePuq A/wKDXhf8JdKhd+GqPHdriLMFdWgt0QfGhuKHv40NHOz8OGWQsRcE20FplH9Fkv1u4Zv119EsrsT zqQb8IVIVDnit/Eh6OzRZENalMKhFSIghmmwvz/c3FT0pu1OeHzWZgRrxyJ6yjY8tO8ZuKEYez+Z jw83/Io8ku04OBqL5z4Ot6y9GDZtA8XsxNhv8xARsxZzh2vrlNW2Qlwb0+QmsagUkhIVkciKLavG 9IlJ9sH+ztL590QIGWP80FlNpLH+CiZVqHD62QCc7umEST9m4WAWkVPaZYmkvn+8HUpebIvHUIUT bdyJQFygu1aBg9cqTXqVk8v70omkqrDvcX9kjHTB8XNFeO9gzfUrNXp0sMf7JwuRb8yoozWuuUSc 94VocJkIs4AIUQSdvhJ/IpIK6emBcyPboH+kJ+ZT/Pye3sgY54/HQqguhkp8cdWAzn20yHnWGx/D gN47iqDu5o2ciW3xsUMF+m/PlPSJLz9/Z5x7th3ODdFgOdVvz0WZiHUlVZibXo4Jw9oiZ5w3llWU o/dO2b1pyswnjIDNELD0CkQ8/ipp3gt6TpKD9iEs3PBvfPvv5QjetwQf/pQFlf99WDCMkh97DZ9+ +2+8+oB/vbJGLXyoBwGbE5U5SSllMlkpSDTuqGmnwhCyTPzaaeHVzoOMmzx8SMtLa+7zgB+p8otw w2t0PH5FL5ERmRUYEEJuOFcthoaQe4+sKjkoR+OldLiGgycr8WgHN/RoYweNvyde9wbmZtVev/Lr 40b1KMePaXrKWY7/HjGge5AbQmoaL2QRDQn2xuQBAfDrqIXGwxXBlMOzrRu8SL9GbSxf5YwJfdtC o/XFg6KeUOPVAb4kr8WTf6CnViJWnRDV+mPcqLvgSeV4BrkjlqLS8kQdKFBZj3bxwYBwKsffF092 cyDLjXCQU/mbEWhaBLTB6GgqwQ0PPTkcnbQ0dlXBGD7eGwnpuZTqhuBOHeFN1pjWTUsfSqe4umVN yvikDgRqTjV1iFgfVRdJKbnZDagg0fijtGHPOMlrKPuk/VcslEwjctF0V6E7uSDmkCtvnHsl/kiW 0vz+NV19FtkEr+G7c4XwoY8SuruX0c7A8wiKUx4V1cj533Z43b0QM48UkFUEvFcEvDlYOCtkl6GS V0+Xj4VbrlfJtCJbXJIcGUTd/cUNKwedEGhTfU2cZQqXD57HE8cMOEExwpUZR8ePjSNWlDW0nUBD Dl7+RFTHlCs+MgJNi4Ah6zR+NxZhyErAoqnR2Ed+P++ORF+/58F/mjymlbtIqU1DsooMH2sjYDOi aoiklGKZrBQkrDyWV0Es8ZgCXafRxffPdsaAWk7tcrzpUoy3rxjgV2KPxUP8MaCrWLuqP+jpLnqt ZwAWDCBrzTyUX0OGn+xig1oNDRxxXz8VTsQV4fhBEGGosClEMEpt68tcTb3nlvyG7uo6DHuxDkYk NbRnW8RLjS3Bex9crGExVRMgW1L1os0JNkHA7GGK9P33yyW0GBWNYJUBe98iknogFrteu088+yHh /cGYrzCUQVhWSriOrCLGx1oI1DFD1JK5boREUj4kduT6n9MkJ+Q5WIGAqxpDaa3p6yM02MuJOFw9 8DoZDiO2peFyIZki9Puoy4lZyBdWSXkeviXeeLWbK8aRq87PwSDH11uMC+4NtsP7xy7j+EXa7FKu R/7vWUjLImWOLvDSesgfqoMIGiK9+eT263+sHNO6kRuvXr3mCSqQJxDTj+QTwRiJzzy5oXNacxJr YvJeRT3OHMyidTGqR0N5OI0RaAIEBPmk08Yeg6EYWekJ+JK2q8/ZBcTMHU6OPHJMmHFYcfpP+PBb 8lob47Sd7kPeqm1QdrM3JNsEVb9jVNrEouJdfU01HjwwrkMOkcMVIpSrOPRiOMZN8Mbl9XnosDbV WKjYDegPryydtPvti0MF5AKE5C4DTfW/vRhWb+U6P+GPrRuy0H/LRZPMmmEeCDGu+ZoipRMtrWdd xdxzwATaCKEEQWOKNWN+Lqc7okc3iv2xGD4fFOPjIe1wr5JROYoRWMPCErsZodbi1Q4FUtvnUvuh csBrRNJSGn3XLqs6TZxxYARshYCK7oddMRNB3CSFjoOnYPm/n8TdRja679VoeE+LwTAiKKArHhvW EWdlUWjvHk5jPhrjKbHjtFVY2oCsMQsf6kDAqlfR34o37ba2t2022F69cLHRI5ra7LlCL8woCuSa Ey649R9cwJme/uTGU9x9hfjwg0vAoHZ4tft1LFgLXZJW236RtaYjMtJIdW2kamq7Dg43lreRRd2J 4g2OqzuxwTfRppvDykAWl7CuzMwrs7oYKLE6rWFZs2x8akTAbOZjTFosAuo6NkVYTPoOUJO18X56 AV7NsiNroxzHE3IRQw3aFySI7DrBQtd1ZG8k2ZHWuW50pFHb2d13I6BznuZFQGXhAqxZdjVJiZSG ZWvm5WtajWAQ7gQE1Bg31gMFO4vwxNeXJLffo+4qHHo8AD203MV3Qg9zGxiB1owAz2J3Su/7B2Dy JPrcKe3hdjACjAAjYETAJrv+GE1GgBFgBBgBRqCpEGCiaipkWS8jwAgwAoyATRBgorIJjKyEEWAE GAFGoKkQsGp7elMVznoZAUaAEWAEGIHrIWDVZooePXpcT4/N048fP45bUa7NG2KlwtbWXithYbGb RIDHlfUAMlbWY9Xckuz6a27EuTxGgBFgBBiBRiHARNUouFiYEWAEGAFGoLkRYKJqbsS5PEaAEWAE GIFGIcBE1Si4WJgRYAQYAUaguRFgompuxLk8RoARYAQYgUYhwETVKLhYmBFgBBgBRqC5EbBqe3pj KvXkk3+RxCdPfhrDh/dsTFaWZQQYAUaAEWAEaiFgc4tKEJSrawA+/XQToqPX1SqQI24BArpC6HRl UsH5J85h/4kc21WigHQ38uW9tivcOk22bXMZkn5IQVKG8rpI6+rAUowAI3DjCNicqIQVNX78AFON du48Zjrnk8YjoMvIvmkiSPj6dzhvuCgVnnWiAIMTihtfkTpzFGLpR79j3tHCOlOlyOIc5BfIJFm/ UNOm2LbNOuw/XIyd2fSWPA6tAAE9vpveD/36mX8m4KTxvaXNA4AemTlFzVNUCy3FJkQl3H1MSE3R w0VYsz4TSTdpsUSMDML5R9rIFXS0Q08HetW7TYIHJo1pi1ldPerVlrAlHR+du8XWh2izeLOkjYJ4 keMNvwjSRnVgNc2HgCEZGLV4E+LidmDLli30WYkwK95Haqsa6k9+ipdWpdlK3W2px2ZrVLt2VVtO 4vyuu7rcloC0pErvXHUWr4gKLTtKXw5InN4DXZCNjaszMb6wiuLssOGhuxB1rw+d65Cw/Xf0Oik/ 6Q8jMsqqqMKsxzsh8txlLC33xtIgd5IzD5Rncwp6pchMGBPmhUVj25sL0LkOO1cmYn8HXywaFiSn lWdj0XsZCB0TBK8friJ/kCeiugJbV55HfrgzksjiiKW67bnHAUMzKUvmWczeDWyQ6pKKedkO8C/Q I1Y8ldLbhRNHOmPrtnzMptfVw8ERR8d3QUSQE5CTjqUbchFdItoKTKf6LZXqV4Stq35HkqcKSef1 WCsS1SokvhCOLp6Ur2agUX6M3n68aOkVzJaehKmMCWFUBlFOeR62briAeZkVECN4mIczNk7pAi/p zrDEp6eHG+Jf8bfQnn80CYN267B0wj0IPZyIedBizRPtZJlzKRi+oxJbX+0MTXIKRu82YLRrGV7I VvoukPrO10IfX7Q8BMQdFR4SAHd3NX2U+hVh84tP4beolVj4SChF6nFg0UtYGxCNf0b5YP4rnyL8 ERWWLNkuZRgaswJzxvaht28L0Ux8FzsTC7anSGmjKC3GmJa6ez7WFUei28l1iKX0/33/TXzwmlhC +Q4jtuSiz5xNWPhoqJSvNX3ZxKIS61IXLiRBEJQ5YbUmIJuirYMGuklq9xEh5I2/C100RBqfZmB8 hQbnJ4fifF8Vxu9Nx/4Mcq2duyCR1L5xHVE1LQijUYVjvl5EIO7QlZRjf0llrSpm7UomkqrCoXHB yBvjjoSUfCz6oeb6lQYR4Q6ITchHvlGD7sRVzCbijOzgjCwizPxymUh0+gq8QCQV2leLS2MC8cCD WrxLed7t2xZ5E4Ixuj3VhWTWZuvRJdIXpZPbYjXd4F235EPTqy1KXw7Casdy9NqSaqqrf6AbLk3u iEsjXbCM6rczXXYj6oi8Zp8vx6THg1A6oS1WlhvQddtFU75aJ3oD/LtTPSYH4fvACvRa/7vcHiIk DZHlGsKgdIIWXQpLMe9AtpQ9dbvApwLfPxKES+O0WNrLBcKaEkHjqCLMU+C9uxSTBgZiEBGraFtC iWBbOehKyrDLeK2rqMA3JQbsd/VA3suhONrNgfouQ+47JQMfWyQCKnoOPHnyJFJTU40fcY+4Y9Tc adgz52kcoMuiI0swcwvwGhEOYEDqqT1Ycigcm3bEYctnc3AkdipiD4l8euyeOQYLkodKaTs2LUYq pc3cnUppFIpTsT12Dr4LmYJNWzZhRN/BWDyK4sfMwnqy6KIHBkhire3LJkSl7O4TZKV8FCAFcbFb UEGjcUdNsBrDyDLxD/aFV7A3GTfZWFpClsn9Wojnev++noihY0J2KZGRmMDVGNSB3HBuvhjegdx7 jkr30nmtUIT9CRV4IswTEb720AT6YJYWmJ1Ze/3Kv58n1YMmWcmFV4b4eD163uWJUMnqMFNMhtmw 9m3x8sMh8A/3hcbTA6GU7OXrCa8gH2iUWV7thkmRQdD4tMOg9qJuzpjxcDuS90XUQHrmpLZIzkKf YESN7SRZN6L9S0gyNaeUvilQWU9088egrlROUDtE9aLKZBIOcmrtb1EmleHlQ9iMbyO1Z2eykPbG 8Kc6EwZkibl5YHQgsP+yKIPwOVmB6X0DMbyXL/w7hGAQ1VkEDXkRk5JTMfrrYizpG4QZRLpyEG0x w9oCH/Gg4IBFT3WAl6c3IkaFYQPFLD0sk6Kcn79bIgIqeibZs2Aqnn76afkz/wDRDd1toY/i85h7 MHPkCAyZugVTV6xEN3e5BcIKWxAThVAfdwR2exQro8OwfdUR6PVpWBdP/b5yopTmEzoQf1s6CvFz 4mnEiUA5w2KwcuJAhAaGwkftjpDwMGgDAuDj7kMfuj9aYbC4lW6m/f/61zsWhCR2/SnuP8XKUgjt ZsppjXnNd9WJuX787gwLGKZnFkEToUFPFGDW5rOI8qzAJLKU3jVaZBbC5hc05r9JyYUzfZTQ08OA /BOn4b1D8pFRtDNKYzpilkcuZsRfxegOwCLaOzFvmB+lydaNkldHxsToGutVMnHIFpckRwTTM7D6 ZtOJYnyrr00mC0Vn/XAaww/rZZccuTJ3Udxq44gVZQ0PdpZUii+vQOMaVHEaIpbnSnkEaSROi5BI rWdgtayI70J58snKQvolzNiQj2V0La3dkd5QiTyFViDCIp8cB6pDbJIgMzvMilBIyphW34HaDaik BwxFxMuViNfMAlPi+diyECi+BEz9/CAmhpuNU2MVw8fORWTs04gnconqY2QpKS2MiKi6HSotXVzK JRoKlSIF+SnBLaAbnRpodNA3DUnt0G70yFkdBOm19mAzohJA1iQiJqubHF7kUssyV0HXqXS9b3Iv DDK7CWSRMsxzLcC8y3r4l9hh6chgDOpeS8hcG7mqgJi+oVj0MFlr5qG8CHmB0sxKj43OxB1OiHxA jWM7CpDwA4gw1NjaQVCmJVGZq2jw3Khakalzo4NYByOSGk4WS8LDggwKsSj29xoWUzUBmiwpt0DE T/YwyjnCi7ynSTTKj5G1VR2qkEQXkY46bPyKSIosvqoXiIEpJKw+iklm9csqMLuQJOiLomIitBie nYv+n55CHj1VeylpJiuWLC8lThylO80g9ae/MT6fKu2vUaxec2E+v10QSN44H/G0LomUWGw8MhQT TWSVQtY/rW0Fyi0x0O5X+LgRGRlAZ5LhpLCRIYeYEG7Clqo76KUcdae1ktgmu0sEaSlrVwqW7AZU kLDy6OaM4bTWtPEXcg+VEym4eWMWGQ6DvzqDrAKa5ej3UVkn0iEmPNDEvpXcgjN6uSOKXHX+NAlL 8fUW5Y5IshxiD19AQjqZSOUkn5yOVPH7IEd3cpF5yx83ebrVdG9D600G9DpswPSINpaTcL1laBBK 9/ArZInpGktqtOYk1sRkAtDRb5cu0LpYjcm/znKdyKVorDu5XUyhpAT7E/PoUmw6ySWydcCgMA+L 3Xv5yWcxi6D2l0jFHRF3kSv0AO26FFuDdXlITZYnDGHNhQb7YdBzQRIm3qvPm4o5dp76RPQHuWnX 7KInAQflFhPHCmz84aIkm/XrOVprBF4mLDm0bASEpZN26RKKinKQkyM+RZLrT5/6HZ5bcgqLd3yP uBXPY8XUV3CShooIIs+6LQckd54+5wiWxKYg8vnetHcoBBPpnpi6ZLfs6itKxfKp66Cd2odWveoO PuH9kbtiMzJpOLXWIN2STdV4xcISltWMP7+AxDOXsGHDQak4Ja2pyr4z9HojKiyLyCGDCCUTR8mN FTWlLbJWXUHAR4nGJordgMHwIothLcWspZXdnnQUO9ho2xy5vup/6WWXp0Lw/eo09NrwuyQtvjY8 7o1QsRuuVvDF6LBMzE4BrS9Vu7uEpJiXRTA/l2OcEBFBLre9BXCOLcDqkR0RKSdUf4sRWMtooUld 44cZYblS22dT+6F2RIzk3ZPXgGqXJVQqpFCtXj6zp918ZGVuS8XgbXLMlsc7EiFpMGigM3ruLoBd 7FFKsMO7Wjt6QpZDxFPB2LAqHV0/PStHOJAb9M+0tkZXOiJS8lli1vhCzCbX4UdH8/DyMC1iPspB gLRLE5jo64hhBXJW5Xv/0SuIPnxFuozp1pZcqUIbh5aMgCoA2D7zaWw3VTIMnx1cjM1PL0Dkgk0Y KBwXPpNp08M6vPTWbhxcHCpLHlmEIf1mSufaUQvw1SOyeRX11edIfeo5SptTnTZRuP9kgpNOzL58 uo2i+2YqxgzYgrDoz7E+KtwstXWcWvUq+pt9067YTCHISgSxbjVsWM9absKacLe2t2022F6deEyj CU1DC/5K0BnpQdqhQL+3ij2LpL7B5MYTd40IeVgamwo80pGsLNpg0VCw0NWQ4A2mkbUm1tk0pt0U jdBDbdeR6/GG8tYqhqzSYsLNrcazq6gfRWuM1mOtbI2tQzH1l4bKMHsM1NG6n/MOe1TFdCGWE30n +rNWSTaPaHBc2by021uhzbDSJ2PCgPlY+Mt6WpHS0wYKtfgVRq2g1wsTqe60WsIUIeTVdSmqS/gO izO7lZquZTWtp5rXTVfyHaJZTHo1g8WkTxM5WRux52jDQxfawUdutoTDlxFNeQ6ZbTioqcJ0baHL FGu7E7JcbvgHstR2283nYmefGdkrLRT1k38JoMRYHhtbh5pEaNJWKVmftiFdk1I+aaEIFEs8VD8R NZZ0GivfQmG5oWo1C1GJmjE53VD/WJlJg6jxWuRvy8Pw9cWS2+8JDzWOjgtBhE8dE7OVWlnMdgho 6C9j9Kzjqdp2JbCmloOAitZmQ6V1qpZTp9u7Js1GVLc3TLdB7YNC8PIr9LkNqtoqqxjeGQmtb2mh VXY11KFY+P3C1tn2Jmp1favPTVQcq2UEGAFGgBFgBBqHABNV4/BiaUaAEWAEGIFmRoCJqpkB5+IY AUaAEWAEGoeAVdvTG6eSpRkBRoARYAQYAdshYNVmik6dOtmuRCs1nT17FreiXCurZ3Ox1tZemwPI CutEgMdVnbDUGclY1QlLi4hk11+L6AauBCPACDACjEB9CDBR1YcMxzMCjAAjwAi0CASYqFpEN3Al GAFGgBFgBOpDgImqPmQ4nhFgBBgBRqBFIMBE1SK6gSvBCDACjAAjUB8CTFT1IcPxjAAjwAgwAi0C ASaqFtENXAlGgBFgBBiB+hBgoqoPmTs+nl4QVXKNXnJT662FcsvL9dBRuq6+9DseH24gI1A/AgaD AfS/niDSGkqvJxtH14uAVT/4rTc3JzQpAn/84AwGDLkLE7q6VJeTlQHXr4ux79nO8DyUgt7nKvH9 mI4Y0M68K0vw9ccXMclgj99eDENn12v4esUF6AfLuvIT0jDxRx3iTFodcPqPnRAivYbiGo5vz0T/ c/SedCW4qPDbuPbofJ33LyrifGQE7hwEDNg2eRgMs3bhyU4q5CbsxMzoWFS/E3sY1u6ahWDx7nkY kLDtfUQv2VXd/HunYO3cZxDc0PvOqqX5rB4EzGe3ekSuH73e1RV9+vS5vqBR4siRI5hQUmK1fGsV nOANjDiSZ0FUZxIEbir00AJpqJKgWX7kKhGVfzVMF3OIpMSlnC7O9MQ7OnFdfgUziaQ6h3lg04Oe QFEJjieXwc/4rqT/fnUBgy8DGwe1xdBwV7K6CvCvLXnovTaFyCzMSGZCIwdGoHUgYMgVFCRuqCws IpIKjlmOpQ/RX+spzsLZ07nQSiQFJHwShegNwJzlG/BAJ7pBDVnYNn8iJj52lshsrpHMWgdmtm6l TYhKkI4gqz9aUbtPSIZJygqgSOS+PnQHxJXgDL0ptLNEJIXYk1KFZ7t5ym+9LbfDs23s8EV6MS6T B8/P2JvH/6NDd3dH+BVV0IuwawRy5R2nqE8eDICGeAiuLrhP4bjcTImk1pAV95hixal9MeElR5z5 4Are/k8uPhnsgeN70tA/SXEZ2uH7x0MwQDbHahTGl4zAHYSAoRhn4Y25D90NN0FO2mBEPBAsN7A4 gUgqD9GrduEhsrykoArGk4s34Ozg8fjwpz9i8UPKjSYn87f1CNhsjUqQjyChhgKTVEPo1E7TdPXE NLKC9iQWyolp+Yihs1f70NOaCGQlDejqhlg6+TbRaKGSxfTh5Sos+IMz/GQpy296oJhMMf3/mYKD ifSoaBZ0lwWt2eO+MDNXo5SuxQS6H4/nUfrFSxJJbR3mj5xn2+JQTw38vB3MtPApI3CHIiCIx5vI aNh8/HQ2S7KxlJYass7S6WBEKCSlJMAfT07zxtksy3vNlMwnViFgM6ISpTVEVkxSVvVHDSEiiCAg 5ki+FH/mSCl5/ZzRw7RWVAWdiyce62CH6f/NkWR0JwvxBdR4MMQR5MGrI3hg8ottsca7CiPirsD1 g2R8fdj8JnICZa0j2OGEnlyHLjIpHUyjjRgeWvQYEExrV3VmqEMHRzECtzMCbnhm4wbEDEtXadAA ABFsSURBVMvCnCnjMWzwZGxLyKpukHcE0VLtoIIWeVmG2gkcYzUCNiUqUWpdZMUkZXV/1BLs0ccZ uKbDmcJcfJcBzO9D60pmQad3QEh/IVNKLkI9fvylHM/29IamISPHVYtxE8JR8qI/vg+2w6RDV7Dn ouLKK0OacmpWDiqq0F1tR+6OQGQMckZyUiF8Vp7BHzdfQH5d8uZ5+ZwRuFMQUPlj+KwV2Lfr31gw TYsl0eOxV+GqvJ9oFat2MNAil7e/0R1YO5ljrEDA5kQlyjQnKyYpK3qhIZEQT3LtVeGzuDzMhR0e 7W5JVLQ7gsjDB2uImOYSacwx2JFrkGQqKhvSKqe5emLAE8H4mK7OkFtPQzsquqMSP6bU3OiSi/VE kgO85R0XXt2Dsel/OyNjmAsKMq7hHVq74sAItCoEVFo88ORCcs0DZ3OLofLvRKtXv+K/Z4trwJCF f63KQ4S/0V1fI5UvrUOgSYhKFK2QlThyuBkEyLUXZo/lGURIbVyNmypq6nPB0G6O+O5qBU54u6KH 2CRRX8jNwsGDWcjPpX7RX8Plw5n4E8l29ib3HVlL/6CFrT/FXcS3J4h8ysuhy83G+s+u4H1au5r2 IN1shVdwnNLE76u8wj3xJBFk2jWzrez1lcvxjMDtjkDxWfz037PILS6WfieVlbANy6lNncS2P7cI zB3vjeVTnsfe0/L6laE4Hf+aOR67aO1qEm+kuKneb9LFBSapm+obU+aQCBd0TynGqxFepjjlRGM8 8ernjkeP5eHJ+2hPu1mQbSDQqpUI5LrLM2DEMVrrOlZgkort5o2h7WSJ+566C4fE76j2XwHERwTx O6qJ7eW1q7xSSqP8SprKEacfbCvL8TcjcAciIDvtVDDkpmNOzAKLFo6fsxYPGd16EX/ciCX+9Duq aeOrZcTvqL6l31FVx/DZDSBg1avob8Wbdlvb2zabvb20nqUrr6At6jV3+BlHkfjLFOLHV44qaNQ1 n2fIutMb6HdZdaXdwCjkLE2GQLOPqyZrSdMrthYr8VcnRFCp6lt3Uv4qhYpkmr7eraGEmjNQa2gz t1EgoFYTATUAhSOl1zs6KIHIS7HmGtDCSYzAHYdA/QSlNJUJSkHCVscmW6OyVQVZDyPACDACjEDr RoCJqnX3P7eeEWAEGIEWjwATVYvvIq4gI8AIMAKtGwEmqtbd/9x6RoARYARaPAJMVC2+i7iCjAAj wAi0bgSs2p7euiHi1jMCjAAjwAjcSgTq3YBsXqkePXqYXzbL+fHjx3Erym2WxtVRSGtrbx0QcFQT IMDjynpQGSvrsWpuSXb9NTfiXB4jwAgwAoxAoxBgomoUXCzMCDACjAAj0NwIMFE1N+JcHiPACDAC jECjEGCiahRcLMwIMAKMACPQ3AgwUTU34lweI8AIMAKMQKMQYKJqFFwszAgwAowAI9DcCDBRNTfi XB4jwAgwAoxAoxBgomoUXCzMCDACjAAj0NwIMFE1N+K3srxyHXTFOnrF/K2sBJfNCDACjEDjEGCi ahxezSo9KfYo1pwosiwz4xzsKD4+B0jafFw6359eZimDQmxcepTSjiOpWCTpkPDNCdi9lwjn5Yl0 pLSVKRRLobwIWRmF4owDI8AI1EJAj++m90O/ftWfEdOX40imvpZk3RF6ZObUuIfrFqwRq8eB+f0w YvkRijfWYcQiZNaQgv4kpou6Tf+OpG4w6HNwQ1Wk4lI3v0hl777xsq2scpMS1T277aysBovVhcAk LfBC/BWLpKTDglTUiPAR0ZVS2tL4S9LR9JV+GeOlUSunIzkNvZLKseGRIJRO64S8cVrs6+UivaFX dyIdw3fnmbLyCSPACFgiYEgGRi3ehLi4OOzY8jmm+HyHqWMGYHPy9QlIf/JTvLQqzVKhFVf61D2Y uT0Mf5vcR5I20IMpcrfgYI0yMw9sRryQyDFIcjfydfLTCViVdmM0Fzp2LiLj52DjyetjcSN1U/I0 GVEJkjr9CMBkpUDd+GNkJL0rPrcQSZLpI/LnYWdSFSZGtJFfA19uj4m+DvjmfAGyzNx5CQdK0NND hWGUQ2TV6SvoW43RvXyhcXOHV4cQDIoMAs6lwHm3AceycxFBVtqMby6SHJD16xkMp2thudn9/QQS 0o0V0F3ErKUnsPOH03L635NIfxmStp+qlo8l+QyjhZeRhkV/N+qJTcBwOo/48IxsyemysXFlglwG pW38VdyJHBiBloeAoICQgAC4u7vDJzAcY+d+jxXPaxH73GbI03MRdi+fjhFGq2vCW5shRrM+czcG vLSO+OUNKe2t71Iptm5ZSrAIB5YsAJ6PRjeaAgCqgc89GDUqDEvWHTKTK8KeJdsxdMxQWUZKqV9/ zsndeNHMMlx+IBOZu+fjpXW52PLGaLIaR+C7VEFYehzZPN9kRc7ffMRoMemx+63p2HhgN+aPEBbm izipD8W0mDCseP9Qk1pVTUJUCkkJ3JispNFzQ1+a7m0wHVXYecJo8Zy7imjSNCPSV9ZH5DSomyeW 0KLT1hNG9115BpZmVmHRIBf4G0vVBDujJw0j55WnkZRu5ubr4Id9ZLXB1x37Jwdj3qA2QHIKAvZe w+i+bVH6chD2BVag14ZkpEpcVYkkfTlGHDZg1khfXBrrJxMmHDBLWGskv8WjHL22iSdIHXZuycVs H3eUvt4NlwaqsIv4cumYEMpDaZ9mYHyFBucnh+J8XxXG703HfoXgjPXmAyPQchCwtFj6jBF34h6Y DBGfofjblh2I2/EZQvfEYgmRgDqwPxaPIrExs7A+bgeiBwbIzalD1rKdmYgnM2nq0G6maENOIEY9 P4WKXIIjivGSeQgrcsfg+Ud7k0VlEiVSq10XkNNw+UtzELpgEw6SZbhpxWJEhrgjsP/zGENZx8xa iR071mNggJrIawGmxuZgBbVnx6bFSI6diiWH5AIMmfFYMnMOQmZ9hi2bFiKMiDSk91jg1MZqLMyq YqtTmxOVOUkplWSyUpBo7NEXk+4Con+5KmVMiqcFJ7UbIjwVPZXQuWkxOswOr/x8WYrUHc3HWjhj UHsnZClinu0RP84L75bo0XXD75KVFH9OjHYP+HvaoadaBS8fH3h5apCUUAJoPfHyw+2g8fTFoGfu QgwqsDOlmuBWP94Rg7oHwb+DN+lwQpdRXTCoqzPRjwaR4Sqg0EDnBmSRqiV9SY+jE/wjPSULT5QB sqaWUtqG+7USmfr39aQygITsUqXGfGQEWjYCPqEIM9XQHY9EPYowHxr7qhAiFC2OpOZSqjtCwsOg JWvMx92HPsI8qk/WpIwMmlwk02Woj2ROGRMy4R4QiTlhuVh3SF6pOrJuDrTRUQjRkogoTgr16VfD h9IP/ZZMnOaO0D4D0SfUnaoTgHDKHyDqSHOAu1qP+CV7cE/MFIRTe1QBvTGFyHbLb5ck7Qbi67Do TZg4sBsCQwPJT0NTUkA3wsIg7L4mC1a95sPa0usiKSWvQlanHqlSovhoBQIRkW7A1yVIKsjG1gvA uwPFqKwOOp0TQgeSzKdF5CLUIfVnAyb2DSZysJz0NR3aY9afyRrLuIiNW66g/9epuPR692pFpjPq H42D6QpwRQRd5RtjdLBDxF00wJUgXHiryToqFBF2REaU38GZzp0RSobf4F0piNC5ISshH7ugwkaq KrGYZImN350hMpnC9Ewiz14epms+YQRaKgL6SyeRYqycPvMIFrw0FXuILLRhRF8puQiIJtKiUHPy bkjWqI4OIm8YAsS9YhYMRAsDp43CghkHoX9kANZtoYfYHaFQF0mrVJJk/fp9MI2svaI3XsKYIURw kVOxcuFECK6S6yi+ZWJUkeF3KvYlDImtLlw7NJN8MkRHOcDQ3oLymjfYjKgaIimlSUxWChKNOHbQ kmuvGB9tv4JlRASJ8i4KMwU0wHz8sMGhCLM2pCBVb4c1/WgglaebyVSfaoLaYdLkCixdlossQRjl NR4cxFpXuXEThpStVLLMNGZrYNXaaOfhjkwiKRUuTb8H/sJY+vUUnPeK/E4YJAiUSDY+6Rq8fD2R 91wHeInMVGYqHfZN7oVBzT/mRQ04MAKNREAmHiVT/DqaxcNiEEIWyJ6ZRFIDl+LgrP7SVH9kUT+8 pexNoB111eE6siZBQRopuERZwwNNkdIakHvvKNyD5xC76BDiw6KxmO6f6iKuo9+nG2b98xdE56Ri 4xtP4+nYUPyyMNLMGpPLKibjadTSOMztTyxmEZRGWUTShahv0wabuP4kkhITzpHrf06TnJDnYC0C PhjdxQHLLtBg8PVAFyKD2sEdw3up8E12OY5pPRBR40ks/8R57D+ajfyCIqA4D/E7CnCM1pUEsfj7 OeLYBdqirqMNEERGoV3pqSo7HzsTxboYbZTYlUnrYnaI7FC3paNRmw2hgov4aC/V00GOy0okq47c iqPvb4vhpDc/27jW5uaNWWS0Df7qDLIKiC11hcii3Yf5dMqBEWhpCAiKSsu5BL2+CJmpR7CWtqvP 3A7MWTgKYipXmXFYUeoBLCFLR/Ha+YT3R+6KzVB2szcka2q3WotudJFaTPdrzaAOx5QxWmzfEo9R 0wbINpBZ+fXrL8LJI8ko0uuh9glF/0fvoQKKiPzUCCHX3oqNJyUiFFZV74lh2D4jlrbgi/L1yEml Neqc+kiKJC4lE62Sm7BmXW14bROLit15NuyROlSF9nVDz6QCzOhLmx1qBIW3vO73xBOHryDqgbYW EiI9P70Yg08qzjuR7IDvHw+Fv+h9YrWJh/MRsOwkhrX3wc6n7kZi5il03ZYKbBOydtgyMkReF6uD SEJp7Wn6SXJ1LDsqhBET6ICe9CQo/HtJ2bR7gnYcRn0tu/iOUewTYUXYOjYYUVPaImvVFQR8lCiE KTggcXqwbHHJEfzNCLQIBIQrbPuMp0HcJIWwoVPx2Y4odDOyUf/XYqB9aQYGEEHRYzjG0O68ZFkU Pt1GIRJTMYYSw6I/x8oGZI1Z6BCI/kOBmUQeE+f2r442nvWOmkKLRt8hqndgdZpW0IQa9daFzK6N U5+j7R9KuAeLNw2lHEC3UTG0c4PqTw2M/jwOUVErsSB1Om3BH6IIY86mg7RmJjsla1JWyoFVQGQ0 WZcmcZuf2FVRaEjrrXo9860qtyEsmjKt6dtLFpOuAjqymjRuCr0pLSqjv1hRYRkv/ooFWVliO/v1 g8hPWyk0JGt89NGdOA3nHRXIi+luIh/ZLWiH0pi7pTUqSS+tq0lBU7NO1y+VJa6PQNOPq+vX4XaR uDms9GRxEVWo656t9cKSMaU1LCvwkra2j5mDFXG/oI81t6AFyPXrF/UQobouSsY68pCskK4tq+Sh oz4ZEwY8h0cFwYU3uqJmiho+NfPbNCzIqbc7Ak7EUBpLMjI1yal2vKOQtXbgifzVJCXUatRiQ0a5 7HLMySPX3jnME27BQOdqkpIEiaCYpAQSHG5rBNQNTuiWk33DsgIGdeAj9Fst+t0U/Yap8aF+/aIe lnVRtNcRX6+skgdI3bMOOaMWY0wTkpQozSauv+pq8xkjYEQgPARHI4icaGef+J2GcCGu7OaF0lHt jQJ8YAQYgYYQ6DNtPdY3JNAC0kIfXYjvH236ijBRNT3GrbQEDSKG3Y2tw1pp87nZjAAjYDME2PVn MyhZESPACDACjEBTIMBE1RSosk5GgBFgBBgBmyHARGUzKFkRI8AIMAKMQFMgYNX29KYomHUyAowA I8AIMALWIHBdorJGCcswAowAI8AIMAJNhQC7/poKWdbLCDACjAAjYBMEmKhsAiMrYQQYAUaAEWgq BJiomgpZ1ssIMAKMACNgEwSYqGwCIythBBgBRoARaCoEmKiaClnWywgwAowAI2ATBJiobAIjK2EE GAFGgBFoKgSYqJoKWdbLCDACjAAjYBMEmKhsAiMrYQQYAUaAEWgqBJiomgpZ1ssIMAKMACNgEwSY qGwCIythBBgBRoARaCoE/j/GtLXr9aWHsgAAAABJRU5ErkJggg== --_004_FB71A3FB6A03472B8BC9B2EB92C1C9C2welloxacuk_--
--_000_EF464B7F02A647C7A11ED8E6837B3B2Ewelloxacuk_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Is there any sensible way to either clean up the existing ISOs storage or r= e-attach it? I'm struggling to even export VMs and migrate them elsewhere w= ith this and need to recover them asap. Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> On 2 May 2018, at 15:09, Callum Smith <callum@well.ox.ac.uk<mailto:callum@w= ell.ox.ac.uk>> wrote: Attached, thank you for looking into this https://HOSTNAME/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666= -c929aa0225ae <?xml version=3D"1.0" encoding=3D"UTF-8" standalone=3D"yes"?> <storage_domain href=3D"/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4= cc0-b666-c929aa0225ae<view-source:https://virtman.cluster/ovirt-engine/api/= v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae>" id=3D"f5914df0-f46= c-4cc0-b666-c929aa0225ae"> <actions> <link href=3D"/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc= 0-b666-c929aa0225ae/isattached<view-source:https://virtman.cluster/ovirt-en= gine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/isattached>= " rel=3D"isattached"/> <link href=3D"/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc= 0-b666-c929aa0225ae/reduceluns<view-source:https://virtman.cluster/ovirt-en= gine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/reduceluns>= " rel=3D"reduceluns"/> <link href=3D"/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc= 0-b666-c929aa0225ae/refreshluns<view-source:https://virtman.cluster/ovirt-e= ngine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/refreshlun= s>" rel=3D"refreshluns"/> <link href=3D"/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc= 0-b666-c929aa0225ae/updateovfstore<view-source:https://virtman.cluster/ovir= t-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/updateo= vfstore>" rel=3D"updateovfstore"/> </actions> <name>VMISOs</name> <link href=3D"/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b6= 66-c929aa0225ae/disksnapshots<view-source:https://virtman.cluster/ovirt-eng= ine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/disksnapshot= s>" rel=3D"disksnapshots"/> <link href=3D"/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b6= 66-c929aa0225ae/files<view-source:https://virtman.cluster/ovirt-engine/api/= v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/files>" rel=3D"files= "/> <link href=3D"/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b6= 66-c929aa0225ae/diskprofiles<view-source:https://virtman.cluster/ovirt-engi= ne/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/diskprofiles>= " rel=3D"diskprofiles"/> <link href=3D"/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b6= 66-c929aa0225ae/storageconnections<view-source:https://virtman.cluster/ovir= t-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/storage= connections>" rel=3D"storageconnections"/> <link href=3D"/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b6= 66-c929aa0225ae/permissions<view-source:https://virtman.cluster/ovirt-engin= e/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/permissions>" = rel=3D"permissions"/> <available>11770357874688</available> <backup>false</backup> <committed>0</committed> <critical_space_action_blocker>5</critical_space_action_blocker> <discard_after_delete>false</discard_after_delete> <external_status>ok</external_status> <master>false</master> <storage> <address>backoffice01.cluster</address> <path>/vm-iso</path> <type>nfs</type> </storage> <storage_format>v1</storage_format> <supports_discard>false</supports_discard> <supports_discard_zeroes_data>false</supports_discard_zeroes_data> <type>iso</type> <used>38654705664</used> <warning_low_space_indicator>10</warning_low_space_indicator> <wipe_after_delete>false</wipe_after_delete> <data_centers> <data_center href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228= -02bc-0358-000000000304<view-source:https://virtman.cluster/ovirt-engine/ap= i/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304>" id=3D"5a54bf81-0228= -02bc-0358-000000000304"/> </data_centers> </storage_domain> https://HOSTNAME/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-00= 0000000304/storagedomains <?xml version=3D"1.0" encoding=3D"UTF-8" standalone=3D"yes"?> <storage_domains> <storage_domain href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-= 02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91a613ef98ba<= view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenters/5a54bf8= 1-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91a613= ef98ba>" id=3D"d9f8093b-71cd-423d-9b7d-91a613ef98ba"> <actions> <link href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02= bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91a613ef98ba/ac= tivate<view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenters/= 5a54bf81-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d= -91a613ef98ba/activate>" rel=3D"activate"/> <link href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02= bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91a613ef98ba/de= activate<view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenter= s/5a54bf81-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b= 7d-91a613ef98ba/deactivate>" rel=3D"deactivate"/> </actions> <name>tegile-virtman-backup</name> <link href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0= 358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91a613ef98ba/disks<= view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenters/5a54bf8= 1-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91a613= ef98ba/disks>" rel=3D"disks"/> <available>17519171600384</available> <backup>false</backup> <committed>0</committed> <critical_space_action_blocker>5</critical_space_action_blocker> <discard_after_delete>false</discard_after_delete> <external_status>ok</external_status> <master>false</master> <status>maintenance</status> <storage> <address>192.168.64.248</address> <nfs_version>auto</nfs_version> <path>/export/virtman/backup</path> <type>nfs</type> </storage> <storage_format>v1</storage_format> <supports_discard>false</supports_discard> <supports_discard_zeroes_data>false</supports_discard_zeroes_data> <type>export</type> <used>8589934592</used> <warning_low_space_indicator>10</warning_low_space_indicator> <wipe_after_delete>false</wipe_after_delete> <data_center href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228= -02bc-0358-000000000304<view-source:https://virtman.cluster/ovirt-engine/ap= i/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304>" id=3D"5a54bf81-0228= -02bc-0358-000000000304"/> <data_centers> <data_center href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-= 0228-02bc-0358-000000000304<view-source:https://virtman.cluster/ovirt-engin= e/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304>" id=3D"5a54bf81-= 0228-02bc-0358-000000000304"/> </data_centers> </storage_domain> <storage_domain href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-= 02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-67737dedd7a1<= view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenters/5a54bf8= 1-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-67737d= edd7a1>" id=3D"017dd620-65ec-4fa4-8d46-67737dedd7a1"> <actions> <link href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02= bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-67737dedd7a1/ac= tivate<view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenters/= 5a54bf81-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46= -67737dedd7a1/activate>" rel=3D"activate"/> <link href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02= bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-67737dedd7a1/de= activate<view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenter= s/5a54bf81-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d= 46-67737dedd7a1/deactivate>" rel=3D"deactivate"/> </actions> <name>VMStorage</name> <link href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0= 358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-67737dedd7a1/disks<= view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenters/5a54bf8= 1-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-67737d= edd7a1/disks>" rel=3D"disks"/> <available>11770357874688</available> <backup>false</backup> <committed>118111600640</committed> <critical_space_action_blocker>5</critical_space_action_blocker> <discard_after_delete>false</discard_after_delete> <external_status>ok</external_status> <master>true</master> <status>active</status> <storage> <address>backoffice01.cluster</address> <path>/vm-storage2</path> <type>nfs</type> </storage> <storage_format>v4</storage_format> <supports_discard>false</supports_discard> <supports_discard_zeroes_data>false</supports_discard_zeroes_data> <type>data</type> <used>38654705664</used> <warning_low_space_indicator>10</warning_low_space_indicator> <wipe_after_delete>false</wipe_after_delete> <data_center href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228= -02bc-0358-000000000304<view-source:https://virtman.cluster/ovirt-engine/ap= i/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304>" id=3D"5a54bf81-0228= -02bc-0358-000000000304"/> <data_centers> <data_center href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-= 0228-02bc-0358-000000000304<view-source:https://virtman.cluster/ovirt-engin= e/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304>" id=3D"5a54bf81-= 0228-02bc-0358-000000000304"/> </data_centers> </storage_domain> <storage_domain href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-= 02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b6f563a9fc17<= view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenters/5a54bf8= 1-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b6f563= a9fc17>" id=3D"e572c874-17f8-4ad8-9bac-b6f563a9fc17"> <actions> <link href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02= bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b6f563a9fc17/ac= tivate<view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenters/= 5a54bf81-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac= -b6f563a9fc17/activate>" rel=3D"activate"/> <link href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02= bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b6f563a9fc17/de= activate<view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenter= s/5a54bf81-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9b= ac-b6f563a9fc17/deactivate>" rel=3D"deactivate"/> </actions> <name>tegile-virtman</name> <link href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0= 358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b6f563a9fc17/disks<= view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenters/5a54bf8= 1-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b6f563= a9fc17/disks>" rel=3D"disks"/> <available>2190433320960</available> <backup>false</backup> <committed>226559524864</committed> <critical_space_action_blocker>5</critical_space_action_blocker> <discard_after_delete>false</discard_after_delete> <external_status>ok</external_status> <master>false</master> <status>active</status> <storage> <address>192.168.64.248</address> <nfs_version>auto</nfs_version> <path>/export/virtman/VirtualServerShare_1</path> <type>nfs</type> </storage> <storage_format>v4</storage_format> <supports_discard>false</supports_discard> <supports_discard_zeroes_data>false</supports_discard_zeroes_data> <type>data</type> <used>8589934592</used> <warning_low_space_indicator>10</warning_low_space_indicator> <wipe_after_delete>false</wipe_after_delete> <data_center href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228= -02bc-0358-000000000304<view-source:https://virtman.cluster/ovirt-engine/ap= i/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304>" id=3D"5a54bf81-0228= -02bc-0358-000000000304"/> <data_centers> <data_center href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-= 0228-02bc-0358-000000000304<view-source:https://virtman.cluster/ovirt-engin= e/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304>" id=3D"5a54bf81-= 0228-02bc-0358-000000000304"/> </data_centers> </storage_domain> <storage_domain href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-= 02bc-0358-000000000304/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae<= view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenters/5a54bf8= 1-0228-02bc-0358-000000000304/storagedomains/f5914df0-f46c-4cc0-b666-c929aa= 0225ae>" id=3D"f5914df0-f46c-4cc0-b666-c929aa0225ae"> <actions> <link href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02= bc-0358-000000000304/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/ac= tivate<view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenters/= 5a54bf81-0228-02bc-0358-000000000304/storagedomains/f5914df0-f46c-4cc0-b666= -c929aa0225ae/activate>" rel=3D"activate"/> <link href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02= bc-0358-000000000304/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/de= activate<view-source:https://virtman.cluster/ovirt-engine/api/v4/datacenter= s/5a54bf81-0228-02bc-0358-000000000304/storagedomains/f5914df0-f46c-4cc0-b6= 66-c929aa0225ae/deactivate>" rel=3D"deactivate"/> </actions> <name>VMISOs</name> <available>11770357874688</available> <backup>false</backup> <committed>0</committed> <critical_space_action_blocker>5</critical_space_action_blocker> <discard_after_delete>false</discard_after_delete> <external_status>ok</external_status> <master>false</master> <status>maintenance</status> <storage> <address>backoffice01.cluster</address> <path>/vm-iso</path> <type>nfs</type> </storage> <storage_format>v1</storage_format> <supports_discard>false</supports_discard> <supports_discard_zeroes_data>false</supports_discard_zeroes_data> <type>iso</type> <used>38654705664</used> <warning_low_space_indicator>10</warning_low_space_indicator> <wipe_after_delete>false</wipe_after_delete> <data_center href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-0228= -02bc-0358-000000000304<view-source:https://virtman.cluster/ovirt-engine/ap= i/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304>" id=3D"5a54bf81-0228= -02bc-0358-000000000304"/> <data_centers> <data_center href=3D"/ovirt-engine/api/v4/datacenters/5a54bf81-= 0228-02bc-0358-000000000304<view-source:https://virtman.cluster/ovirt-engin= e/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304>" id=3D"5a54bf81-= 0228-02bc-0358-000000000304"/> </data_centers> </storage_domain> </storage_domains> <Screen Shot 2018-05-02 at 14.54.42.png> Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> On 2 May 2018, at 14:46, Fred Rolland <frolland@redhat.com<mailto:frolland@= redhat.com>> wrote: Can you share the REST API data of the Storage domain and Data Center? Here an example of the URLs, you will need to replace with correct ids. http://MY-SERVER/ovirt-engine/api/v4/storagedomains/13461356-f6f7-4a58-9897= -2fac61ff40af<http://my-server/ovirt-engine/api/v4/storagedomains/13461356-= f6f7-4a58-9897-2fac61ff40af> http://MY-SERVER/ovirt-engine/api/v4/datacenters/5a5df553-022d-036d-01e8-00= 0000000071/storagedomains<http://my-server/ovirt-engine/api/v4/datacenters/= 5a5df553-022d-036d-01e8-000000000071/storagedomains> On Wed, May 2, 2018 at 12:53 PM, Callum Smith <callum@well.ox.ac.uk<mailto:= callum@well.ox.ac.uk>> wrote: This is on 4.2.0.2-1, I've linked the main logs to dropbox simply because t= hey're big, full of noise right now. https://www.dropbox.com/s/f8q3m5amro2a1b2/engine.log?dl=3D0 https://www.dropbox.com/s/uods85jk65halo3/vdsm.log?dl=3D0 Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> On 2 May 2018, at 10:43, Fred Rolland <frolland@redhat.com<mailto:frolland@= redhat.com>> wrote: Which version are you using? Can you provide the whole log? For some reason, it looks like the Vdsm thinks that the Storage Domain is n= ot part of the pool. On Wed, May 2, 2018 at 11:20 AM, Callum Smith <callum@well.ox.ac.uk<mailto:= callum@well.ox.ac.uk>> wrote: State is maintenance for the ISOs storage. I've extracted what is hopefully= the relevant bits of the log. VDSM.log (SPM) 2018-05-02 09:16:03,455+0100 INFO (ioprocess communication (179084)) [IOPr= ocess] Starting ioprocess (__init__:447) 2018-05-02 09:16:03,456+0100 INFO (ioprocess communication (179091)) [IOPr= ocess] Starting ioprocess (__init__:447) 2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [vdsm.api] FINISH activateSt= orageDomain error=3DStorage domain not in pool: u'domain=3Df5914df0-f46c-4c= c0-b666-c929aa0225ae, pool=3D5a54bf81-0228-02bc-0358-000000000304' from=3D:= :ffff:192.168.64.254,58968, flow_id=3D93433989-8e26-48a9-bd3a-2ab95f296c08,= task_id=3D7f21f911-348f-45a3-b79c-e3cb11642035 (api:50) 2018-05-02 09:16:03,461+0100 ERROR (jsonrpc/0) [storage.TaskManager.Task] (= Task=3D'7f21f911-348f-45a3-b79c-e3cb11642035') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, i= n _run return fn(*args, **kargs) File "<string>", line 2, in activateStorageDomain File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in m= ethod ret =3D func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1256, i= n activateStorageDomain pool.activateSD(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 7= 9, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1130, in= activateSD self.validateAttachedDomain(dom) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 7= 9, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 557, in = validateAttachedDomain raise se.StorageDomainNotInPool(self.spUUID, dom.sdUUID) StorageDomainNotInPool: Storage domain not in pool: u'domain=3Df5914df0-f46= c-4cc0-b666-c929aa0225ae, pool=3D5a54bf81-0228-02bc-0358-000000000304' 2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [storage.TaskManager.Task] (= Task=3D'7f21f911-348f-45a3-b79c-e3cb11642035') aborting: Task is aborted: "= Storage domain not in pool: u'domain=3Df5914df0-f46c-4cc0-b666-c929aa0225ae= , pool=3D5a54bf81-0228-02bc-0358-000000000304'" - code 353 (task:1181) 2018-05-02 09:16:03,462+0100 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH = activateStorageDomain error=3DStorage domain not in pool: u'domain=3Df5914d= f0-f46c-4cc0-b666-c929aa0225ae, pool=3D5a54bf81-0228-02bc-0358-000000000304= ' (dispatcher:82) engine.log 2018-05-02 09:16:02,326+01 INFO [org.ovirt.engine.core.bll.st<http://org.o= virt.engine.core.bll.st/>orage.domain.ActivateStorageDomainCommand] (defaul= t task-20) [93433989-8e26-48a9-bd3a-2ab95f296c08] Lock Acquired to object '= EngineLock:{exclusiveLocks=3D'[f5914df0-f46c-4cc0-b666-c929aa0225ae=3DSTORA= GE]', sharedLocks=3D''}' 2018-05-02 09:16:02,376+01 INFO [org.ovirt.engine.core.bll.st<http://org.o= virt.engine.core.bll.st/>orage.domain.ActivateStorageDomainCommand] (EE-Man= agedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c0= 8] Running command: ActivateStorageDomainCommand internal: false. Entities = affected : ID: f5914df0-f46c-4cc0-b666-c929aa0225ae Type: StorageAction gr= oup MANIPULATE_STORAGE_DOMA IN with role type ADMIN 2018-05-02 09:16:02,385+01 INFO [org.ovirt.engine.core.bll.st<http://org.o= virt.engine.core.bll.st/>orage.domain.ActivateStorageDomainCommand] (EE-Man= agedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c0= 8] Lock freed to object 'EngineLock:{exclusiveLocks=3D'[f5914df0-f46c-4cc0-= b666-c929aa0225ae=3DSTORAGE]', sharedLocks=3D''}' 2018-05-02 09:16:02,385+01 INFO [org.ovirt.engine.core.bll.st<http://org.o= virt.engine.core.bll.st/>orage.domain.ActivateStorageDomainCommand] (EE-Man= agedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c0= 8] ActivateStorage Domain. Before Connect all hosts to pool. Time: Wed May = 02 09:16:02 BST 2018 2018-05-02 09:16:02,407+01 INFO [org.ovirt.engine.core.bll.st<http://org.o= virt.engine.core.bll.st/>orage.connection.ConnectStorageToVdsCommand] (EE-M= anagedThreadFactory-engine-Thread-33456) [40a82b47] Running command: Connec= tStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000= -0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with= role type ADMIN 2018-05-02 09:16:02,421+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker= .ConnectStorageServerVDSCommand] (EE-ManagedThreadFactory-engine-Thread-334= 56) [40a82b47] START, ConnectStorageServerVDSCommand(HostName =3D virtA003,= StorageServerConnectionManagementVDSParameters:{hostId=3D'fe2861fc-2b47-48= 07-b054-470198eda473', storagePoolId=3D'00000000-0000-0000-0000-000000 000000', storageType=3D'NFS', connectionList=3D'[StorageServerConnections:{= id=3D'da392861-aedc-4f1e-97f4-6919fb01f1e9', connection=3D'backoffice01.clu= ster:/vm-iso', iqn=3D'null', vfsType=3D'null', mountOptions=3D'null', nfsVe= rsion=3D'null', nfsRetrans=3D'null', nfsTimeo=3D'null', iface=3D'null', net= IfaceName=3D'null'}]'}), log id: 23ce648f 2018-05-02 09:16:02,446+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker= .ConnectStorageServerVDSCommand] (EE-ManagedThreadFactory-engine-Thread-334= 56) [40a82b47] FINISH, ConnectStorageServerVDSCommand, return: {da392861-ae= dc-4f1e-97f4-6919fb01f1e9=3D0}, log id: 23ce648f 2018-05-02 09:16:02,450+01 INFO [org.ovirt.engine.core.vdsbroker.irsbroker= .ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33= 455) [93433989-8e26-48a9-bd3a-2ab95f296c08] START, ActivateStorageDomainVDS= Command( ActivateStorageDomainVDSCommandParameters:{storagePoolId=3D'5a54bf= 81-0228-02bc-0358-000000000304', ignoreFailoverLimit=3D'false', stor ageDomainId=3D'f5914df0-f46c-4cc0-b666-c929aa0225ae'}), log id: 5c864594 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditl= oghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-33455) = [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID: IRS_BROKER_COMMAND_FAILURE= (10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain not = in pool: u'domain=3Df5914df0-f46c-4cc0-b666-c929aa0225ae, pool=3D5a5 4bf81-0228-02bc-0358-000000000304' 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker= .ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33= 455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command 'ActivateStorageDomainV= DSCommand( ActivateStorageDomainVDSCommandParameters:{storagePoolId=3D'5a54= bf81-0228-02bc-0358-000000000304', ignoreFailoverLimit=3D'false', st orageDomainId=3D'f5914df0-f46c-4cc0-b666-c929aa0225ae'})' execution failed:= IRSGenericException: IRSErrorException: Storage domain not in pool: u'doma= in=3Df5914df0-f46c-4cc0-b666-c929aa0225ae, pool=3D5a54bf81-0228-02bc-0358-0= 00000000304' 2018-05-02 09:16:02,635+01 INFO [org.ovirt.engine.core.vdsbroker.irsbroker= .ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33= 455) [93433989-8e26-48a9-bd3a-2ab95f296c08] FINISH, ActivateStorageDomainVD= SCommand, log id: 5c864594 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.bll.storage.domain.= ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455)= [93433989-8e26-48a9-bd3a-2ab95f296c08] Command 'org.ovirt.engine.core.bll.= storage.domain.ActivateStorageDomainCommand' failed: EngineException: org.o= virt.engine.core.vdsbroker.irsbroker.IrsOperationFailedNoFailove rException: IRSGenericException: IRSErrorException: Storage domain not in p= ool: u'domain=3Df5914df0-f46c-4cc0-b666-c929aa0225ae, pool=3D5a54bf81-0228-= 02bc-0358-000000000304' (Failed with error StorageDomainNotInPool and code = 353) 2018-05-02 09:16:02,636+01 INFO [org.ovirt.engine.core.bll.st<http://org.o= virt.engine.core.bll.st/>orage.domain.ActivateStorageDomainCommand] (EE-Man= agedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c0= 8] Command [id=3D22b0f3c1-9a09-4e26-8096-d83465c8f4ee]: Compensating CHANGE= D_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.StoragePoolI= soMap; snapshot: EntityStatus Snapshot:{id=3D'StoragePoolIsoMapId:{storagePoolId=3D'5a54bf81-0228-02bc-03= 58-000000000304', storageId=3D'f5914df0-f46c-4cc0-b666-c929aa0225ae'}', sta= tus=3D'Maintenance'}. 2018-05-02 09:16:02,660+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditl= oghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-33455) = [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID: USER_ACTIVATE_STORAGE_DOMA= IN_FAILED(967), Failed to activate Storage Domain VMISOs (Data Center Defau= lt) by admin@internal-authz Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> On 2 May 2018, at 08:44, Fred Rolland <frolland@redhat.com<mailto:frolland@= redhat.com>> wrote: Hi, Can you provide logs from engine and Vdsm(SPM)? What is the state now? Thanks, Fred On Tue, May 1, 2018 at 4:11 PM, Callum Smith <callum@well.ox.ac.uk<mailto:c= allum@well.ox.ac.uk>> wrote: Dear All, It appears that clicking "detach" on the ISO storage domain is a really bad= idea. This has gotten half way through the procedure and now can't be reco= vered from. Is there any advice for re-attaching the ISO storage domain man= ually? An NFS mount didn't add it back to the "pool" unfortunately. On a separate note, is it possible to migrate this storage to a new locatio= n? And if so how. Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users --_000_EF464B7F02A647C7A11ED8E6837B3B2Ewelloxacuk_ Content-Type: text/html; charset="us-ascii" Content-ID: <47102BA196711145A5EC026558D4D487@well.ox.ac.uk> Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
</head> <body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; line-break:= after-white-space;" class=3D""> Is there any sensible way to either clean up the existing ISOs storage or r= e-attach it? I'm struggling to even export VMs and migrate them elsewhere w= ith this and need to recover them asap. <div class=3D""><br class=3D""> <div class=3D""> <div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; text-align: star= t; text-indent: 0px; text-transform: none; white-space: normal; word-spacin= g: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp= -mode: space; -webkit-line-break: after-white-space;" class=3D""> <div style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line= -break: after-white-space;" class=3D""> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Regards,</div> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Callum</div> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> <br class=3D""> </div> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> --<br class=3D""> <br class=3D""> Callum Smith</div> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Research Computing Core</div> <div style=3D"color: rgb(0, 0, 0); font-family: Arial; font-size: 13px; fon= t-style: normal; font-variant-caps: normal; font-weight: normal; letter-spa= cing: normal; text-align: start; text-indent: 0px; text-transform: none; wh= ite-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"> Wellcome Trust Centre for Human Genetics<br class=3D""> University of Oxford<br class=3D""> e. <a href=3D"mailto:callum@well.ox.ac.uk" class=3D"">callum@well.ox.ac.uk<= /a></div> </div> </div> </div> <div><br class=3D""> <blockquote type=3D"cite" class=3D""> <div class=3D"">On 2 May 2018, at 15:09, Callum Smith <<a href=3D"mailto= :callum@well.ox.ac.uk" class=3D"">callum@well.ox.ac.uk</a>> wrote:</div> <br class=3D"Apple-interchange-newline"> <div class=3D""> <div style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; line-break: = after-white-space;" class=3D""> <pre id=3D"line1" class=3D"">Attached, thank you for looking into this</pre=
<pre id=3D"line1" class=3D""><span class=3D"pi"><br class=3D""></span></pre=
> <span id=3D"line16" class=3D""></span> <<span class=3D"start-tag">bac= kup</span>>false</<span class=3D"end-tag">backup</span>> <span id=3D"line17" class=3D""></span> <<span class=3D"start-tag">com= mitted</span>>0</<span class=3D"end-tag">committed</span>> <span id=3D"line18" class=3D""></span> <<span class=3D"start-tag">cri= tical_space_action_blocker</span>>5</<span class=3D"end-tag">critical= _space_action_blocker</span>> <span id=3D"line19" class=3D""></span> <<span class=3D"start-tag">dis= card_after_delete</span>>false</<span class=3D"end-tag">discard_after= _delete</span>> <span id=3D"line20" class=3D""></span> <<span class=3D"start-tag">ext= ernal_status</span>>ok</<span class=3D"end-tag">external_status</span= > <span id=3D"line21" class=3D""></span> <<span class=3D"start-tag">mas= ter</span>>false</<span class=3D"end-tag">master</span>> <span id=3D"line22" class=3D""></span> <<span class=3D"start-tag">sto= rage</span>> <span id=3D"line23" class=3D""></span> <<span class=3D"start-tag"= address</span>>backoffice01.cluster</<span class=3D"end-tag">address= </span>> <span id=3D"line24" class=3D""></span> <<span class=3D"start-tag"= path</span>>/vm-iso</<span class=3D"end-tag">path</span>> <span id=3D"line25" class=3D""></span> <<span class=3D"start-tag"= type</span>>nfs</<span class=3D"end-tag">type</span>> <span id=3D"line26" class=3D""></span> </<span class=3D"end-tag">stor= age</span>> <span id=3D"line27" class=3D""></span> <<span class=3D"start-tag">sto= rage_format</span>>v1</<span class=3D"end-tag">storage_format</span>&= gt; <span id=3D"line28" class=3D""></span> <<span class=3D"start-tag">sup=
data_center</span> <span class=3D"attribute-name">href</span>=3D"<a c= lass=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-= engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304">/ovirt-engi= ne/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304</a>" <span = class=3D"attribute-name">id</span>=3D"<a class=3D"attribute-value">5a5= 4bf81-0228-02bc-0358-000000000304</a>"/> <span id=3D"line36" class=3D""></span> </<span class=3D"end-tag">data= _centers</span>> <span id=3D"line37" class=3D""></span></<span class=3D"end-tag">storage_= domain</span>> <span id=3D"line38" class=3D""></span></pre><div class=3D""><br class=3D"">= </div><div class=3D"">https://<i class=3D"">HOSTNAME</i>/ovirt-engine/api/v= 4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains</div></sp= an></pre> <pre id=3D"line1" class=3D""><pre id=3D"line1" class=3D""><span class=3D"pi= "><?xml version=3D"1.0" encoding=3D"UTF-8" standalon= e=3D"yes"?></span> <span id=3D"line2" class=3D""></span><<span class=3D"start-tag">storage_= domains</span>> <span id=3D"line3" class=3D""></span> <<span class=3D"start-tag">stor= age_domain</span> <span class=3D"attribute-name">href</span>=3D"<a cla= ss=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-en= gine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains= /d9f8093b-71cd-423d-9b7d-91a613ef98ba">/ovirt-engine/api/v4/datacenters/5a5= 4bf81-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91= a613ef98ba</a>" <span class=3D"attribute-name">id</span>=3D"<a cl= ass=3D"attribute-value">d9f8093b-71cd-423d-9b7d-91a613ef98ba</a>"> <span id=3D"line4" class=3D""></span> <<span class=3D"start-tag">= actions</span>> <span id=3D"line5" class=3D""></span> <<span class=3D"start-t= ag">link</span> <span class=3D"attribute-name">href</span>=3D"<a class= =3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engi= ne/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/d= 9f8093b-71cd-423d-9b7d-91a613ef98ba/activate">/ovirt-engine/api/v4/datacent= ers/5a54bf81-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-= 9b7d-91a613ef98ba/activate</a>" <span class=3D"attribute-name">rel</sp= an>=3D"<a class=3D"attribute-value">activate</a>"/> <span id=3D"line6" class=3D""></span> <<span class=3D"start-t= ag">link</span> <span class=3D"attribute-name">href</span>=3D"<a class= =3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engi= ne/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/d= 9f8093b-71cd-423d-9b7d-91a613ef98ba/deactivate">/ovirt-engine/api/v4/datace= nters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423= d-9b7d-91a613ef98ba/deactivate</a>" <span class=3D"attribute-name">rel= </span>=3D"<a class=3D"attribute-value">deactivate</a>"/> <span id=3D"line7" class=3D""></span> </<span class=3D"end-tag">a= ctions</span>> <span id=3D"line8" class=3D""></span> <<span class=3D"start-tag">= name</span>>tegile-virtman-backup</<span class=3D"end-tag">name</span= > <span id=3D"line9" class=3D""></span> <<span class=3D"start-tag">=
available</span>>17519171600384</<span class=3D"end-tag">available</= span>> <span id=3D"line11" class=3D""></span> <<span class=3D"start-tag"= backup</span>>false</<span class=3D"end-tag">backup</span>> <span id=3D"line12" class=3D""></span> <<span class=3D"start-tag"= committed</span>>0</<span class=3D"end-tag">committed</span>> <span id=3D"line13" class=3D""></span> <<span class=3D"start-tag"= critical_space_action_blocker</span>>5</<span class=3D"end-tag">crit= ical_space_action_blocker</span>> <span id=3D"line14" class=3D""></span> <<span class=3D"start-tag"= discard_after_delete</span>>false</<span class=3D"end-tag">discard_a= fter_delete</span>> <span id=3D"line15" class=3D""></span> <<span class=3D"start-tag"= external_status</span>>ok</<span class=3D"end-tag">external_status</= span>> <span id=3D"line16" class=3D""></span> <<span class=3D"start-tag"= master</span>>false</<span class=3D"end-tag">master</span>> <span id=3D"line17" class=3D""></span> <<span class=3D"start-tag"= status</span>>maintenance</<span class=3D"end-tag">status</span>> <span id=3D"line18" class=3D""></span> <<span class=3D"start-tag"= storage</span>> <span id=3D"line19" class=3D""></span> <<span class=3D"start-= tag">address</span>>192.168.64.248</<span class=3D"end-tag">address</= span>> <span id=3D"line20" class=3D""></span> <<span class=3D"start-= tag">nfs_version</span>>auto</<span class=3D"end-tag">nfs_version</sp= an>> <span id=3D"line21" class=3D""></span> <<span class=3D"start-= tag">path</span>>/export/virtman/backup</<span class=3D"end-tag">path= </span>> <span id=3D"line22" class=3D""></span> <<span class=3D"start-= tag">type</span>>nfs</<span class=3D"end-tag">type</span>> <span id=3D"line23" class=3D""></span> </<span class=3D"end-tag">= storage</span>> <span id=3D"line24" class=3D""></span> <<span class=3D"start-tag"= storage_format</span>>v1</<span class=3D"end-tag">storage_format</sp= an>> <span id=3D"line25" class=3D""></span> <<span class=3D"start-tag"= supports_discard</span>>false</<span class=3D"end-tag">supports_disc= ard</span>> <span id=3D"line26" class=3D""></span> <<span class=3D"start-tag"= supports_discard_zeroes_data</span>>false</<span class=3D"end-tag">s= upports_discard_zeroes_data</span>> <span id=3D"line27" class=3D""></span> <<span class=3D"start-tag"= type</span>>export</<span class=3D"end-tag">type</span>> <span id=3D"line28" class=3D""></span> <<span class=3D"start-tag"= used</span>>8589934592</<span class=3D"end-tag">used</span>> <span id=3D"line29" class=3D""></span> <<span class=3D"start-tag"= warning_low_space_indicator</span>>10</<span class=3D"end-tag">warni= ng_low_space_indicator</span>> <span id=3D"line30" class=3D""></span> <<span class=3D"start-tag"= wipe_after_delete</span>>false</<span class=3D"end-tag">wipe_after_d= elete</span>> <span id=3D"line31" class=3D""></span> <<span class=3D"start-tag"= data_center</span> <span class=3D"attribute-name">href</span>=3D"<a c= lass=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-= engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304">/ovirt-engi= ne/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304</a>" <span = class=3D"attribute-name">id</span>=3D"<a class=3D"attribute-value">5a5= 4bf81-0228-02bc-0358-000000000304</a>"/> <span id=3D"line32" class=3D""></span> <<span class=3D"start-tag"= data_centers</span>> <span id=3D"line33" class=3D""></span> <<span class=3D"start-= tag">data_center</span> <span class=3D"attribute-name">href</span>=3D"= <a class=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ov= irt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304">/ovirt-= engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304</a>" <s=
5a54bf81-0228-02bc-0358-000000000304</a>"/> <span id=3D"line34" class=3D""></span> </<span class=3D"end-tag">= data_centers</span>> <span id=3D"line35" class=3D""></span> </<span class=3D"end-tag">stor= age_domain</span>> <span id=3D"line36" class=3D""></span> <<span class=3D"start-tag">sto= rage_domain</span> <span class=3D"attribute-name">href</span>=3D"<a cl= ass=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-e= ngine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomain= s/017dd620-65ec-4fa4-8d46-67737dedd7a1">/ovirt-engine/api/v4/datacenters/5a= 54bf81-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-6= 7737dedd7a1</a>" <span class=3D"attribute-name">id</span>=3D"<a c= lass=3D"attribute-value">017dd620-65ec-4fa4-8d46-67737dedd7a1</a>"> <span id=3D"line37" class=3D""></span> <<span class=3D"start-tag"= actions</span>> <span id=3D"line38" class=3D""></span> <<span class=3D"start-= tag">link</span> <span class=3D"attribute-name">href</span>=3D"<a clas= s=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-eng= ine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/= 017dd620-65ec-4fa4-8d46-67737dedd7a1/activate">/ovirt-engine/api/v4/datacen= ters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4= -8d46-67737dedd7a1/activate</a>" <span class=3D"attribute-name">rel</s=
name</span>>VMStorage</<span class=3D"end-tag">name</span>> <span id=3D"line42" class=3D""></span> <<span class=3D"start-tag"= link</span> <span class=3D"attribute-name">href</span>=3D"<a class=3D= "attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engine/= api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/017d= d620-65ec-4fa4-8d46-67737dedd7a1/disks">/ovirt-engine/api/v4/datacenters/5a= 54bf81-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-6= 7737dedd7a1/disks</a>" <span class=3D"attribute-name">rel</span>=3D&qu= ot;<a class=3D"attribute-value">disks</a>"/> <span id=3D"line43" class=3D""></span> <<span class=3D"start-tag"= available</span>>11770357874688</<span class=3D"end-tag">available</= span>> <span id=3D"line44" class=3D""></span> <<span class=3D"start-tag"= backup</span>>false</<span class=3D"end-tag">backup</span>> <span id=3D"line45" class=3D""></span> <<span class=3D"start-tag"= committed</span>>118111600640</<span class=3D"end-tag">committed</sp= an>> <span id=3D"line46" class=3D""></span> <<span class=3D"start-tag"= critical_space_action_blocker</span>>5</<span class=3D"end-tag">crit= ical_space_action_blocker</span>> <span id=3D"line47" class=3D""></span> <<span class=3D"start-tag"= discard_after_delete</span>>false</<span class=3D"end-tag">discard_a= fter_delete</span>> <span id=3D"line48" class=3D""></span> <<span class=3D"start-tag"= external_status</span>>ok</<span class=3D"end-tag">external_status</= span>> <span id=3D"line49" class=3D""></span> <<span class=3D"start-tag"= master</span>>true</<span class=3D"end-tag">master</span>> <span id=3D"line50" class=3D""></span> <<span class=3D"start-tag"= status</span>>active</<span class=3D"end-tag">status</span>> <span id=3D"line51" class=3D""></span> <<span class=3D"start-tag"= storage</span>> <span id=3D"line52" class=3D""></span> <<span class=3D"start-= tag">address</span>>backoffice01.cluster</<span class=3D"end-tag">add= ress</span>> <span id=3D"line53" class=3D""></span> <<span class=3D"start-= tag">path</span>>/vm-storage2</<span class=3D"end-tag">path</span>>= ; <span id=3D"line54" class=3D""></span> <<span class=3D"start-= tag">type</span>>nfs</<span class=3D"end-tag">type</span>> <span id=3D"line55" class=3D""></span> </<span class=3D"end-tag">= storage</span>> <span id=3D"line56" class=3D""></span> <<span class=3D"start-tag"= storage_format</span>>v4</<span class=3D"end-tag">storage_format</sp= an>> <span id=3D"line57" class=3D""></span> <<span class=3D"start-tag"= supports_discard</span>>false</<span class=3D"end-tag">supports_disc= ard</span>> <span id=3D"line58" class=3D""></span> <<span class=3D"start-tag"= supports_discard_zeroes_data</span>>false</<span class=3D"end-tag">s= upports_discard_zeroes_data</span>> <span id=3D"line59" class=3D""></span> <<span class=3D"start-tag"= type</span>>data</<span class=3D"end-tag">type</span>> <span id=3D"line60" class=3D""></span> <<span class=3D"start-tag"= used</span>>38654705664</<span class=3D"end-tag">used</span>> <span id=3D"line61" class=3D""></span> <<span class=3D"start-tag"= warning_low_space_indicator</span>>10</<span class=3D"end-tag">warni= ng_low_space_indicator</span>> <span id=3D"line62" class=3D""></span> <<span class=3D"start-tag"= wipe_after_delete</span>>false</<span class=3D"end-tag">wipe_after_d= elete</span>> <span id=3D"line63" class=3D""></span> <<span class=3D"start-tag"= data_center</span> <span class=3D"attribute-name">href</span>=3D"<a c= lass=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-= engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304">/ovirt-engi= ne/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304</a>" <span = class=3D"attribute-name">id</span>=3D"<a class=3D"attribute-value">5a5= 4bf81-0228-02bc-0358-000000000304</a>"/> <span id=3D"line64" class=3D""></span> <<span class=3D"start-tag"= data_centers</span>> <span id=3D"line65" class=3D""></span> <<span class=3D"start-= tag">data_center</span> <span class=3D"attribute-name">href</span>=3D"= <a class=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ov= irt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304">/ovirt-= engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304</a>" <s=
5a54bf81-0228-02bc-0358-000000000304</a>"/> <span id=3D"line66" class=3D""></span> </<span class=3D"end-tag">= data_centers</span>> <span id=3D"line67" class=3D""></span> </<span class=3D"end-tag">stor= age_domain</span>> <span id=3D"line68" class=3D""></span> <<span class=3D"start-tag">sto= rage_domain</span> <span class=3D"attribute-name">href</span>=3D"<a cl= ass=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-e= ngine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomain= s/e572c874-17f8-4ad8-9bac-b6f563a9fc17">/ovirt-engine/api/v4/datacenters/5a= 54bf81-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b= 6f563a9fc17</a>" <span class=3D"attribute-name">id</span>=3D"<a c= lass=3D"attribute-value">e572c874-17f8-4ad8-9bac-b6f563a9fc17</a>"> <span id=3D"line69" class=3D""></span> <<span class=3D"start-tag"= actions</span>> <span id=3D"line70" class=3D""></span> <<span class=3D"start-= tag">link</span> <span class=3D"attribute-name">href</span>=3D"<a clas= s=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-eng= ine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/= e572c874-17f8-4ad8-9bac-b6f563a9fc17/activate">/ovirt-engine/api/v4/datacen= ters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8= -9bac-b6f563a9fc17/activate</a>" <span class=3D"attribute-name">rel</s=
name</span>>tegile-virtman</<span class=3D"end-tag">name</span>> <span id=3D"line74" class=3D""></span> <<span class=3D"start-tag"= link</span> <span class=3D"attribute-name">href</span>=3D"<a class=3D= "attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engine/= api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/e572= c874-17f8-4ad8-9bac-b6f563a9fc17/disks">/ovirt-engine/api/v4/datacenters/5a= 54bf81-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b= 6f563a9fc17/disks</a>" <span class=3D"attribute-name">rel</span>=3D&qu= ot;<a class=3D"attribute-value">disks</a>"/> <span id=3D"line75" class=3D""></span> <<span class=3D"start-tag"= available</span>>2190433320960</<span class=3D"end-tag">available</s=
backup</span>>false</<span class=3D"end-tag">backup</span>> <span id=3D"line77" class=3D""></span> <<span class=3D"start-tag"= committed</span>>226559524864</<span class=3D"end-tag">committed</sp= an>> <span id=3D"line78" class=3D""></span> <<span class=3D"start-tag"= critical_space_action_blocker</span>>5</<span class=3D"end-tag">crit= ical_space_action_blocker</span>> <span id=3D"line79" class=3D""></span> <<span class=3D"start-tag"= discard_after_delete</span>>false</<span class=3D"end-tag">discard_a= fter_delete</span>> <span id=3D"line80" class=3D""></span> <<span class=3D"start-tag"= external_status</span>>ok</<span class=3D"end-tag">external_status</= span>> <span id=3D"line81" class=3D""></span> <<span class=3D"start-tag"= master</span>>false</<span class=3D"end-tag">master</span>> <span id=3D"line82" class=3D""></span> <<span class=3D"start-tag"= status</span>>active</<span class=3D"end-tag">status</span>> <span id=3D"line83" class=3D""></span> <<span class=3D"start-tag"= storage</span>> <span id=3D"line84" class=3D""></span> <<span class=3D"start-= tag">address</span>>192.168.64.248</<span class=3D"end-tag">address</= span>> <span id=3D"line85" class=3D""></span> <<span class=3D"start-= tag">nfs_version</span>>auto</<span class=3D"end-tag">nfs_version</sp= an>> <span id=3D"line86" class=3D""></span> <<span class=3D"start-= tag">path</span>>/export/virtman/VirtualServerShare_1</<span class=3D= "end-tag">path</span>> <span id=3D"line87" class=3D""></span> <<span class=3D"start-= tag">type</span>>nfs</<span class=3D"end-tag">type</span>> <span id=3D"line88" class=3D""></span> </<span class=3D"end-tag">= storage</span>> <span id=3D"line89" class=3D""></span> <<span class=3D"start-tag"= storage_format</span>>v4</<span class=3D"end-tag">storage_format</sp= an>> <span id=3D"line90" class=3D""></span> <<span class=3D"start-tag"= supports_discard</span>>false</<span class=3D"end-tag">supports_disc= ard</span>> <span id=3D"line91" class=3D""></span> <<span class=3D"start-tag"= supports_discard_zeroes_data</span>>false</<span class=3D"end-tag">s= upports_discard_zeroes_data</span>> <span id=3D"line92" class=3D""></span> <<span class=3D"start-tag"= type</span>>data</<span class=3D"end-tag">type</span>> <span id=3D"line93" class=3D""></span> <<span class=3D"start-tag"= used</span>>8589934592</<span class=3D"end-tag">used</span>> <span id=3D"line94" class=3D""></span> <<span class=3D"start-tag"= warning_low_space_indicator</span>>10</<span class=3D"end-tag">warni= ng_low_space_indicator</span>> <span id=3D"line95" class=3D""></span> <<span class=3D"start-tag"= wipe_after_delete</span>>false</<span class=3D"end-tag">wipe_after_d= elete</span>> <span id=3D"line96" class=3D""></span> <<span class=3D"start-tag"= data_center</span> <span class=3D"attribute-name">href</span>=3D"<a c= lass=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-= engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304">/ovirt-engi= ne/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304</a>" <span = class=3D"attribute-name">id</span>=3D"<a class=3D"attribute-value">5a5= 4bf81-0228-02bc-0358-000000000304</a>"/> <span id=3D"line97" class=3D""></span> <<span class=3D"start-tag"= data_centers</span>> <span id=3D"line98" class=3D""></span> <<span class=3D"start-= tag">data_center</span> <span class=3D"attribute-name">href</span>=3D"= <a class=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ov= irt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304">/ovirt-= engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304</a>" <s=
5a54bf81-0228-02bc-0358-000000000304</a>"/> <span id=3D"line99" class=3D""></span> </<span class=3D"end-tag">= data_centers</span>> <span id=3D"line100" class=3D""></span> </<span class=3D"end-tag">sto= rage_domain</span>> <span id=3D"line101" class=3D""></span> <<span class=3D"start-tag">st= orage_domain</span> <span class=3D"attribute-name">href</span>=3D"<a c= lass=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-= engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomai= ns/f5914df0-f46c-4cc0-b666-c929aa0225ae">/ovirt-engine/api/v4/datacenters/5= a54bf81-0228-02bc-0358-000000000304/storagedomains/f5914df0-f46c-4cc0-b666-= c929aa0225ae</a>" <span class=3D"attribute-name">id</span>=3D"<a = class=3D"attribute-value">f5914df0-f46c-4cc0-b666-c929aa0225ae</a>">= ; <span id=3D"line102" class=3D""></span> <<span class=3D"start-tag= ">actions</span>> <span id=3D"line103" class=3D""></span> <<span class=3D"start= -tag">link</span> <span class=3D"attribute-name">href</span>=3D"<a cla= ss=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-en= gine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains= /f5914df0-f46c-4cc0-b666-c929aa0225ae/activate">/ovirt-engine/api/v4/datace= nters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/f5914df0-f46c-4cc= 0-b666-c929aa0225ae/activate</a>" <span class=3D"attribute-name">rel</= span>=3D"<a class=3D"attribute-value">activate</a>"/> <span id=3D"line104" class=3D""></span> <<span class=3D"start= -tag">link</span> <span class=3D"attribute-name">href</span>=3D"<a cla= ss=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-en= gine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains= /f5914df0-f46c-4cc0-b666-c929aa0225ae/deactivate">/ovirt-engine/api/v4/data=
actions</span>> <span id=3D"line106" class=3D""></span> <<span class=3D"start-tag= ">name</span>>VMISOs</<span class=3D"end-tag">name</span>> <span id=3D"line107" class=3D""></span> <<span class=3D"start-tag= ">available</span>>11770357874688</<span class=3D"end-tag">available<= /span>> <span id=3D"line108" class=3D""></span> <<span class=3D"start-tag= ">backup</span>>false</<span class=3D"end-tag">backup</span>> <span id=3D"line109" class=3D""></span> <<span class=3D"start-tag= ">committed</span>>0</<span class=3D"end-tag">committed</span>> <span id=3D"line110" class=3D""></span> <<span class=3D"start-tag= ">critical_space_action_blocker</span>>5</<span class=3D"end-tag">cri= tical_space_action_blocker</span>> <span id=3D"line111" class=3D""></span> <<span class=3D"start-tag= ">discard_after_delete</span>>false</<span class=3D"end-tag">discard_= after_delete</span>> <span id=3D"line112" class=3D""></span> <<span class=3D"start-tag= ">external_status</span>>ok</<span class=3D"end-tag">external_status<= /span>> <span id=3D"line113" class=3D""></span> <<span class=3D"start-tag= ">master</span>>false</<span class=3D"end-tag">master</span>> <span id=3D"line114" class=3D""></span> <<span class=3D"start-tag= ">status</span>>maintenance</<span class=3D"end-tag">status</span>>= ; <span id=3D"line115" class=3D""></span> <<span class=3D"start-tag= ">storage</span>> <span id=3D"line116" class=3D""></span> <<span class=3D"start= -tag">address</span>>backoffice01.cluster</<span class=3D"end-tag">ad= dress</span>> <span id=3D"line117" class=3D""></span> <<span class=3D"start= -tag">path</span>>/vm-iso</<span class=3D"end-tag">path</span>> <span id=3D"line118" class=3D""></span> <<span class=3D"start= -tag">type</span>>nfs</<span class=3D"end-tag">type</span>> <span id=3D"line119" class=3D""></span> </<span class=3D"end-tag"= storage</span>> <span id=3D"line120" class=3D""></span> <<span class=3D"start-tag= ">storage_format</span>>v1</<span class=3D"end-tag">storage_format</s=
data_centers</span>> <span id=3D"line131" class=3D""></span> </<span class=3D"end-tag">sto= rage_domain</span>> <span id=3D"line132" class=3D""></span></<span class=3D"end-tag">storage= _domains</span>> <span id=3D"line133" class=3D""></span></pre><div class=3D""><br class=3D""= </div> <span id=3D"line3" class=3D""></span></pre> <div class=3D""><span id=3D"cid:921E5257-26F4-4E0D-83B0-68F996842D08@well.o= x.ac.uk"><Screen Shot 2018-05-02 at 14.54.42.png></span></div> <div class=3D""> <div style=3D"letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-= stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-br= eak: after-white-space;" class=3D""> <div style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line= -break: after-white-space;" class=3D""> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""> Regards,</div> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""> Callum</div> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""> <br class=3D""> </div> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""> --<br class=3D""> <br class=3D""> Callum Smith</div> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""> Research Computing Core</div> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""> Wellcome Trust Centre for Human Genetics<br class=3D""> University of Oxford<br class=3D""> e. <a href=3D"mailto:callum@well.ox.ac.uk" class=3D"">callum@well.ox.ac.uk<= /a></div> </div> </div> </div> <div class=3D""><br class=3D""> <blockquote type=3D"cite" class=3D""> <div class=3D"">On 2 May 2018, at 14:46, Fred Rolland <<a href=3D"mailto= :frolland@redhat.com" class=3D"">frolland@redhat.com</a>> wrote:</div> <br class=3D"Apple-interchange-newline"> <div class=3D""> <div dir=3D"ltr" class=3D""> <div class=3D"">Can you share the REST API data of the Storage domain and D= ata Center?<br class=3D""> </div> Here an example of the URLs, you will need to replace with correct ids.<br = class=3D""> <div class=3D""><br class=3D""> <a href=3D"http://my-server/ovirt-engine/api/v4/storagedomains/13461356-f6f= 7-4a58-9897-2fac61ff40af" class=3D"">http://MY-SERVER/ovirt-engine/api/v4/s= toragedomains/13461356-f6f7-4a58-9897-2fac61ff40af</a><br class=3D""> <br class=3D""> <a href=3D"http://my-server/ovirt-engine/api/v4/datacenters/5a5df553-022d-0= 36d-01e8-000000000071/storagedomains" class=3D"">http://MY-SERVER/ovirt-eng= ine/api/v4/datacenters/5a5df553-022d-036d-01e8-000000000071/storagedomains<= /a><br class=3D""> <br class=3D""> <br class=3D""> </div> </div> <div class=3D"gmail_extra"><br class=3D""> <div class=3D"gmail_quote">On Wed, May 2, 2018 at 12:53 PM, Callum Smith <s=
This is on 4.2.0.2-1, I've linked the main logs to dropbox simply because =
State is maintenance for the ISOs storage. I've extracted what is hopefull= y the relevant bits of the log. <div class=3D""> <div class=3D""><br class=3D""> </div> <div class=3D"">VDSM.log (SPM)</div> <div class=3D""><br class=3D""> </div> <div class=3D""> <div class=3D"">2018-05-02 09:16:03,455+0100 INFO (ioprocess comm= unication (179084)) [IOProcess] Starting ioprocess (__init__:447)</div> <div class=3D"">2018-05-02 09:16:03,456+0100 INFO (ioprocess comm= unication (179091)) [IOProcess] Starting ioprocess (__init__:447)</div> <div class=3D"">2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [vd= sm.api] FINISH activateStorageDomain error=3DStorage domain not in pool: u'= domain=3Df5914df0-f46c-4cc0-b6<wbr class=3D"">66-c929aa0225ae, pool=3D5a54b= f81-0228-02bc-0358-0<wbr class=3D"">00000000304' from=3D::ffff:192.168.64.2= 54,589<wbr class=3D"">68, flow_id=3D93433989-8e26-48a9-bd3<wbr class=3D"">a-2ab95f296c08, task_id=3D= 7f21f911-348f-45a3-b79<wbr class=3D"">c-e3cb11642035 (api:50)</div> <div class=3D"">2018-05-02 09:16:03,461+0100 ERROR (jsonrpc/0) [storage= .TaskManager.Task] (Task=3D'7f21f911-348f-45a3-b79c<wbr class=3D"">-e3cb116= 42035') Unexpected error (task:875)</div> <div class=3D"">Traceback (most recent call last):</div> <div class=3D""> File "/usr/lib/python2.7/site-packa<wbr class= =3D"">ges/vdsm/storage/task.py", line 882, in _run</div> <div class=3D""> return fn(*args, **kargs)</div> <div class=3D""> File "<string>", line 2, in activate= StorageDomain</div> <div class=3D""> File "/usr/lib/python2.7/site-packa<wbr class= =3D"">ges/vdsm/common/api.py", line 48, in method</div> <div class=3D""> ret =3D func(*args, **kwargs)</div> <div class=3D""> File "/usr/lib/python2.7/site-packa<wbr class= =3D"">ges/vdsm/storage/hsm.py", line 1256, in activateStorageDomain</d= iv> <div class=3D""> pool.activateSD(sdUUID)</div> <div class=3D""> File "/usr/lib/python2.7/site-packa<wbr class= =3D"">ges/vdsm/storage/securable.py"<wbr class=3D"">, line 79, in wrap=
m)</div> <div class=3D""> File "/usr/lib/python2.7/site-packa<wbr class= =3D"">ges/vdsm/storage/securable.py"<wbr class=3D"">, line 79, in wrap=
<wbr class=3D"">orage.domain.ActivateStorageDo<wbr class=3D"">mainCommand]= (default task-20) [93433989-8e26-48a9-bd3a-2ab95<wbr class=3D"">f296c08] Lock Acquired to object 'EngineLock:{exclusiveLocks=3D'[<wbr class=3D"">f5= 914df0-f46c-4cc0-b666-c929aa<wbr class=3D"">0225ae=3DSTORAGE]', sharedLocks= =3D''}'</div> <div class=3D"">2018-05-02 09:16:02,376+01 INFO [<a href=3D"http:= //org.ovirt.engine.core.bll.st/" class=3D"">org.ovirt.engine.core.bll.st</a= <wbr class=3D"">orage.domain.ActivateStorageDo<wbr class=3D"">mainCommand]= (EE-ManagedThreadFactory-engin<wbr class=3D"">e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95<wbr class=3D"">f296c08] Running command: Ac= tivateStorageDomainCommand internal: false. Entities affected : ID: f= 5914df0-f46c-4cc0-b666-c929aa<wbr class=3D"">0225ae Type: StorageAction gro= up MANIPULATE_STORAGE_DOMA</div> <div class=3D"">IN with role type ADMIN</div> <div class=3D"">2018-05-02 09:16:02,385+01 INFO [<a href=3D"http:= //org.ovirt.engine.core.bll.st/" class=3D"">org.ovirt.engine.core.bll.st</a= <wbr class=3D"">orage.domain.ActivateStorageDo<wbr class=3D"">mainCommand]= (EE-ManagedThreadFactory-engin<wbr class=3D"">e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95<wbr class=3D"">f296c08] Lock freed to objec= t 'EngineLock:{exclusiveLocks=3D'[<wbr class=3D"">f5914df0-f46c-4cc0-b666-c= 929aa<wbr class=3D"">0225ae=3DSTORAGE]', sharedLocks=3D''}'</div> <div class=3D"">2018-05-02 09:16:02,385+01 INFO [<a href=3D"http:= //org.ovirt.engine.core.bll.st/" class=3D"">org.ovirt.engine.core.bll.st</a= <wbr class=3D"">orage.domain.ActivateStorageDo<wbr class=3D"">mainCommand]= (EE-ManagedThreadFactory-engin<wbr class=3D"">e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95<wbr class=3D"">f296c08] ActivateStorage Dom= ain. Before Connect all hosts to pool. Time: Wed May 02 09:16:02 BST 2018</=
<wbr class=3D"">orage.connection.ConnectStorag<wbr class=3D"">eToVdsComman= d] (EE-ManagedThreadFactory-engin<wbr class=3D"">e-Thread-33456) [40a82b47] Running command: ConnectStorageToVdsCommand internal: true. Ent= ities affected : ID: aaa00000-0000-0000-0000-123456<wbr class=3D"">78= 9aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN</d= iv> <div class=3D"">2018-05-02 09:16:02,421+01 INFO [org.ovirt.engine= .core.vdsbro<wbr class=3D"">ker.vdsbroker.ConnectStorageSe<wbr class=3D"">r= verVDSCommand] (EE-ManagedThreadFactory-engin<wbr class=3D"">e-Thread-33456= ) [40a82b47] START, ConnectStorageServerVDSCommand<wbr class=3D"">(HostName =3D virtA003, StorageServerConnectionManagem<wbr class=3D"">entVDSParamete= rs:{hostId=3D'fe28<wbr class=3D"">61fc-2b47-4807-b054-470198eda4<wbr class= =3D"">73', storagePoolId=3D'00000000-0000-0<wbr class=3D"">000-0000-000000<= /div> <div class=3D"">000000', storageType=3D'NFS', connectionList=3D'[StorageSer= ver<wbr class=3D"">Connections:{id=3D'da392861-<wbr class=3D"">aedc-4f1e-97= f4-6919fb01f1e9', connection=3D'backoffice01.clust<wbr class=3D"">er:/vm-is= o', iqn=3D'null', vfsType=3D'null', mountOptions=3D'null', nfsVersion=3D'null', nfsRetrans=3D'null', nfsTimeo=3D'null', iface=3D'null= ', netIfaceName=3D'null'}]'}), log id: 23ce648f</div> <div class=3D"">2018-05-02 09:16:02,446+01 INFO [org.ovirt.engine= .core.vdsbro<wbr class=3D"">ker.vdsbroker.ConnectStorageSe<wbr class=3D"">r= verVDSCommand] (EE-ManagedThreadFactory-engin<wbr class=3D"">e-Thread-33456= ) [40a82b47] FINISH, ConnectStorageServerVDSCommand<wbr class=3D"">, return: {da392861-aedc-4f1e-97f4-6919f<wbr class=3D"">b01f1e9=3D0}, log id= : 23ce648f</div> <div class=3D"">2018-05-02 09:16:02,450+01 INFO [org.ovirt.engine= .core.vdsbro<wbr class=3D"">ker.irsbroker.ActivateStorageD<wbr class=3D"">o=
54bf81-0228-02bc-0358-00000000<wbr class=3D"">0304', ignoreFailoverLimit= =3D'false', stor</div> <div class=3D"">ageDomainId=3D'f5914df0-f46c-4cc<wbr class=3D"">0-b666-c929= aa0225ae'}), log id: 5c864594</div> <div class=3D"">2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core= .dal.dbb<wbr class=3D"">roker.auditloghandling.AuditLo<wbr class=3D"">gDire= ctor] (EE-ManagedThreadFactory-engin<wbr class=3D"">e-Thread-33455) [934339= 89-8e26-48a9-bd3a-2ab95<wbr class=3D"">f296c08] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,<wbr class=3D"">803), VDSM command= ActivateStorageDomainVDS failed: Storage domain not in pool: u'domain=3Df5= 914df0-f46c-4cc0-b6<wbr class=3D"">66-c929aa0225ae, pool=3D5a5</div> <div class=3D"">4bf81-0228-02bc-0358-000000000<wbr class=3D"">304'</div> <div class=3D"">2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core= .vdsbrok<wbr class=3D"">er.irsbroker.ActivateStorageDo<wbr class=3D"">mainV= DSCommand] (EE-ManagedThreadFactory-engin<wbr class=3D"">e-Thread-33455) [9= 3433989-8e26-48a9-bd3a-2ab95<wbr class=3D"">f296c08] Command 'ActivateStorageDomainVDSComma<wbr class=3D"">nd( ActivateStorageD= omainVDSComman<wbr class=3D"">dParameters:{storagePoolId=3D'5a<wbr class=3D= "">54bf81-0228-02bc-0358-00000000<wbr class=3D"">0304', ignoreFailoverLimit= =3D'false', st</div> <div class=3D"">orageDomainId=3D'f5914df0-f46c-4<wbr class=3D"">cc0-b666-c9= 29aa0225ae'})' execution failed: IRSGenericException: IRSErrorException: St= orage domain not in pool: u'domain=3Df5914df0-f46c-4cc0-b6<wbr class=3D"">6= 6-c929aa0225ae, pool=3D5a54bf81-0228-02bc-0358-0<wbr class=3D"">00000000304= '</div> <div class=3D"">2018-05-02 09:16:02,635+01 INFO [org.ovirt.engine= .core.vdsbro<wbr class=3D"">ker.irsbroker.ActivateStorageD<wbr class=3D"">o=
<wbr class=3D"">orage.domain.ActivateStorageDo<wbr class=3D"">mainCommand]= (EE-ManagedThreadFactory-engin<wbr class=3D"">e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95<wbr class=3D"">f296c08] Command [id=3D22b0f= 3c1-9a09-4e26-8096-d8<wbr class=3D"">3465c8f4ee]: Compensating CHANGED_STAT= US_ONLY of org.ovirt.engine.core.common.b<wbr class=3D"">usinessentities.St= oragePoolIso<wbr class=3D"">Map; snapshot: EntityStatus</div> <div class=3D"">Snapshot:{id=3D'StoragePoolIsoMa<wbr class=3D"">pId:{storag= ePoolId=3D'5a54bf81-<wbr class=3D"">0228-02bc-0358-000000000304', storageId= =3D'f5914df0-f46c-4cc0-<wbr class=3D"">b666-c929aa0225ae'}', status=3D'Main= tenance'}.</div> <div class=3D"">2018-05-02 09:16:02,660+01 ERROR [org.ovirt.engine.core= .dal.dbb<wbr class=3D"">roker.auditloghandling.AuditLo<wbr class=3D"">gDire= ctor] (EE-ManagedThreadFactory-engin<wbr class=3D"">e-Thread-33455) [934339= 89-8e26-48a9-bd3a-2ab95<wbr class=3D"">f296c08] EVENT_ID: USER_ACTIVATE_STORAGE_DOMAIN_F<wbr class=3D"">AILED(967), Failed= to activate Storage Domain VMISOs (Data Center Default) by admin@internal-= authz</div> </div> <div class=3D""><span class=3D""><br class=3D""> <div class=3D""> <div style=3D"letter-spacing:normal;text-align:start;text-indent:0px;text-t= ransform:none;white-space:normal;word-spacing:0px;word-wrap:break-word" cla= ss=3D""> <div style=3D"word-wrap:break-word" class=3D""> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> Regards,</div> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> Callum</div> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> <br class=3D""> </div> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> --<br class=3D""> <br class=3D""> Callum Smith</div> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> Research Computing Core</div> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> Wellcome Trust Centre for Human Genetics<br class=3D""> University of Oxford<br class=3D""> e. <a href=3D"mailto:callum@well.ox.ac.uk" target=3D"_blank" class=3D"">cal= lum@well.ox.ac.uk</a></div> </div> </div> </div> </span> <div class=3D""> <div class=3D"m_2571234484608545856h5"> <div class=3D""><br class=3D""> <blockquote type=3D"cite" class=3D""> <div class=3D"">On 2 May 2018, at 08:44, Fred Rolland <<a href=3D"mailto= :frolland@redhat.com" target=3D"_blank" class=3D"">frolland@redhat.com</a>&= gt; wrote:</div> <br class=3D"m_2571234484608545856m_-2093273250354586386Apple-interchange-n= ewline"> <div class=3D""> <div dir=3D"ltr" class=3D""> <div class=3D""> <div class=3D""> <div class=3D""> <div class=3D"">Hi,<br class=3D""> <br class=3D""> </div> Can you provide logs from engine and Vdsm(SPM)?<br class=3D""> </div> What is the state now?<br class=3D""> <br class=3D""> </div> Thanks,<br class=3D""> </div> Fred<br class=3D""> </div> <div class=3D"gmail_extra"><br class=3D""> <div class=3D"gmail_quote">On Tue, May 1, 2018 at 4:11 PM, Callum Smith <sp= an dir=3D"ltr" class=3D""> <<a href=3D"mailto:callum@well.ox.ac.uk" target=3D"_blank" class=3D"">ca= llum@well.ox.ac.uk</a>></span> wrote:<br class=3D""> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <div style=3D"word-wrap:break-word;line-break:after-white-space" class=3D""= Dear All, <div class=3D""><br class=3D""> </div> <div class=3D"">It appears that clicking "detach" on the ISO stor= age domain is a really bad idea. This has gotten half way through the proce= dure and now can't be recovered from. Is there any advice for re-attaching =
<pre id=3D"line1" class=3D""><span class=3D"pi">https://<i class=3D"">HOSTN= AME</i>/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa02= 25ae</span></pre> <pre id=3D"line1" class=3D""><span class=3D"pi"><pre id=3D"line1" class=3D"= "><span class=3D"pi"><?xml version=3D"1.0" encoding=3D"UT= F-8" standalone=3D"yes"?></span> <span id=3D"line2" class=3D""></span><<span class=3D"start-tag">storage_= domain</span> <span class=3D"attribute-name">href</span>=3D"<a class= =3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engi= ne/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae">/ovirt-engin= e/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae</a>" <spa= n class=3D"attribute-name">id</span>=3D"<a class=3D"attribute-value">f= 5914df0-f46c-4cc0-b666-c929aa0225ae</a>"> <span id=3D"line3" class=3D""></span> <<span class=3D"start-tag">acti= ons</span>> <span id=3D"line4" class=3D""></span> <<span class=3D"start-tag">= link</span> <span class=3D"attribute-name">href</span>=3D"<a class=3D"= attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engine/a= pi/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/isattached">/ovir= t-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/isattac= hed</a>" <span class=3D"attribute-name">rel</span>=3D"<a class=3D= "attribute-value">isattached</a>"/> <span id=3D"line5" class=3D""></span> <<span class=3D"start-tag">= link</span> <span class=3D"attribute-name">href</span>=3D"<a class=3D"= attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engine/a= pi/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/reduceluns">/ovir= t-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/reducel= uns</a>" <span class=3D"attribute-name">rel</span>=3D"<a class=3D= "attribute-value">reduceluns</a>"/> <span id=3D"line6" class=3D""></span> <<span class=3D"start-tag">= link</span> <span class=3D"attribute-name">href</span>=3D"<a class=3D"= attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engine/a= pi/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/refreshluns">/ovi= rt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/refres= hluns</a>" <span class=3D"attribute-name">rel</span>=3D"<a class= =3D"attribute-value">refreshluns</a>"/> <span id=3D"line7" class=3D""></span> <<span class=3D"start-tag">= link</span> <span class=3D"attribute-name">href</span>=3D"<a class=3D"= attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engine/a= pi/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/updateovfstore">/= ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/upd= ateovfstore</a>" <span class=3D"attribute-name">rel</span>=3D"<a = class=3D"attribute-value">updateovfstore</a>"/> <span id=3D"line8" class=3D""></span> </<span class=3D"end-tag">actio= ns</span>> <span id=3D"line9" class=3D""></span> <<span class=3D"start-tag">name= </span>>VMISOs</<span class=3D"end-tag">name</span>> <span id=3D"line10" class=3D""></span> <<span class=3D"start-tag">lin= k</span> <span class=3D"attribute-name">href</span>=3D"<a class=3D"att= ribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engine/api/= v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/disksnapshots">/ovir= t-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/disksna= pshots</a>" <span class=3D"attribute-name">rel</span>=3D"<a class= =3D"attribute-value">disksnapshots</a>"/> <span id=3D"line11" class=3D""></span> <<span class=3D"start-tag">lin= k</span> <span class=3D"attribute-name">href</span>=3D"<a class=3D"att= ribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engine/api/= v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/files">/ovirt-engine= /api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/files</a>"= <span class=3D"attribute-name">rel</span>=3D"<a class=3D"attribute-va= lue">files</a>"/> <span id=3D"line12" class=3D""></span> <<span class=3D"start-tag">lin= k</span> <span class=3D"attribute-name">href</span>=3D"<a class=3D"att= ribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engine/api/= v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/diskprofiles">/ovirt= -engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/diskprof= iles</a>" <span class=3D"attribute-name">rel</span>=3D"<a class= =3D"attribute-value">diskprofiles</a>"/> <span id=3D"line13" class=3D""></span> <<span class=3D"start-tag">lin= k</span> <span class=3D"attribute-name">href</span>=3D"<a class=3D"att= ribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engine/api/= v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/storageconnections">= /ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/st= orageconnections</a>" <span class=3D"attribute-name">rel</span>=3D&quo= t;<a class=3D"attribute-value">storageconnections</a>"/> <span id=3D"line14" class=3D""></span> <<span class=3D"start-tag">lin= k</span> <span class=3D"attribute-name">href</span>=3D"<a class=3D"att= ribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engine/api/= v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/permissions">/ovirt-= engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/permissio= ns</a>" <span class=3D"attribute-name">rel</span>=3D"<a class=3D"= attribute-value">permissions</a>"/> <span id=3D"line15" class=3D""></span> <<span class=3D"start-tag">ava= ilable</span>>11770357874688</<span class=3D"end-tag">available</span= ports_discard</span>>false</<span class=3D"end-tag">supports_discard<= /span>> <span id=3D"line29" class=3D""></span> <<span class=3D"start-tag">sup= ports_discard_zeroes_data</span>>false</<span class=3D"end-tag">suppo= rts_discard_zeroes_data</span>> <span id=3D"line30" class=3D""></span> <<span class=3D"start-tag">typ= e</span>>iso</<span class=3D"end-tag">type</span>> <span id=3D"line31" class=3D""></span> <<span class=3D"start-tag">use= d</span>>38654705664</<span class=3D"end-tag">used</span>> <span id=3D"line32" class=3D""></span> <<span class=3D"start-tag">war= ning_low_space_indicator</span>>10</<span class=3D"end-tag">warning_l= ow_space_indicator</span>> <span id=3D"line33" class=3D""></span> <<span class=3D"start-tag">wip= e_after_delete</span>>false</<span class=3D"end-tag">wipe_after_delet= e</span>> <span id=3D"line34" class=3D""></span> <<span class=3D"start-tag">dat= a_centers</span>> <span id=3D"line35" class=3D""></span> <<span class=3D"start-tag"= link</span> <span class=3D"attribute-name">href</span>=3D"<a class=3D"= attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-engine/a= pi/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/d9f80= 93b-71cd-423d-9b7d-91a613ef98ba/disks">/ovirt-engine/api/v4/datacenters/5a5= 4bf81-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91= a613ef98ba/disks</a>" <span class=3D"attribute-name">rel</span>=3D&quo= t;<a class=3D"attribute-value">disks</a>"/> <span id=3D"line10" class=3D""></span> <<span class=3D"start-tag"= pan class=3D"attribute-name">id</span>=3D"<a class=3D"attribute-value"= pan>=3D"<a class=3D"attribute-value">activate</a>"/> <span id=3D"line39" class=3D""></span> <<span class=3D"start-= tag">link</span> <span class=3D"attribute-name">href</span>=3D"<a clas= s=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-eng= ine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/= 017dd620-65ec-4fa4-8d46-67737dedd7a1/deactivate">/ovirt-engine/api/v4/datac= enters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4f= a4-8d46-67737dedd7a1/deactivate</a>" <span class=3D"attribute-name">re= l</span>=3D"<a class=3D"attribute-value">deactivate</a>"/> <span id=3D"line40" class=3D""></span> </<span class=3D"end-tag">= actions</span>> <span id=3D"line41" class=3D""></span> <<span class=3D"start-tag"= pan class=3D"attribute-name">id</span>=3D"<a class=3D"attribute-value"= pan>=3D"<a class=3D"attribute-value">activate</a>"/> <span id=3D"line71" class=3D""></span> <<span class=3D"start-= tag">link</span> <span class=3D"attribute-name">href</span>=3D"<a clas= s=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt-eng= ine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/= e572c874-17f8-4ad8-9bac-b6f563a9fc17/deactivate">/ovirt-engine/api/v4/datac= enters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4a= d8-9bac-b6f563a9fc17/deactivate</a>" <span class=3D"attribute-name">re= l</span>=3D"<a class=3D"attribute-value">deactivate</a>"/> <span id=3D"line72" class=3D""></span> </<span class=3D"end-tag">= actions</span>> <span id=3D"line73" class=3D""></span> <<span class=3D"start-tag"= pan>> <span id=3D"line76" class=3D""></span> <<span class=3D"start-tag"= pan class=3D"attribute-name">id</span>=3D"<a class=3D"attribute-value"= centers/5a54bf81-0228-02bc-0358-000000000304/storagedomains/f5914df0-f46c-4= cc0-b666-c929aa0225ae/deactivate</a>" <span class=3D"attribute-name">r= el</span>=3D"<a class=3D"attribute-value">deactivate</a>"/> <span id=3D"line105" class=3D""></span> </<span class=3D"end-tag"= pan>> <span id=3D"line121" class=3D""></span> <<span class=3D"start-tag= ">supports_discard</span>>false</<span class=3D"end-tag">supports_dis= card</span>> <span id=3D"line122" class=3D""></span> <<span class=3D"start-tag= ">supports_discard_zeroes_data</span>>false</<span class=3D"end-tag">= supports_discard_zeroes_data</span>> <span id=3D"line123" class=3D""></span> <<span class=3D"start-tag= ">type</span>>iso</<span class=3D"end-tag">type</span>> <span id=3D"line124" class=3D""></span> <<span class=3D"start-tag= ">used</span>>38654705664</<span class=3D"end-tag">used</span>> <span id=3D"line125" class=3D""></span> <<span class=3D"start-tag= ">warning_low_space_indicator</span>>10</<span class=3D"end-tag">warn= ing_low_space_indicator</span>> <span id=3D"line126" class=3D""></span> <<span class=3D"start-tag= ">wipe_after_delete</span>>false</<span class=3D"end-tag">wipe_after_= delete</span>> <span id=3D"line127" class=3D""></span> <<span class=3D"start-tag= ">data_center</span> <span class=3D"attribute-name">href</span>=3D"<a = class=3D"attribute-value" href=3D"view-source:https://virtman.cluster/ovirt= -engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304">/ovirt-eng= ine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304</a>" <span= class=3D"attribute-name">id</span>=3D"<a class=3D"attribute-value">5a= 54bf81-0228-02bc-0358-000000000304</a>"/> <span id=3D"line128" class=3D""></span> <<span class=3D"start-tag= ">data_centers</span>> <span id=3D"line129" class=3D""></span> <<span class=3D"start= -tag">data_center</span> <span class=3D"attribute-name">href</span>=3D"= ;<a class=3D"attribute-value" href=3D"view-source:https://virtman.cluster/o= virt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304">/ovirt= -engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304</a>" <= span class=3D"attribute-name">id</span>=3D"<a class=3D"attribute-value= ">5a54bf81-0228-02bc-0358-000000000304</a>"/> <span id=3D"line130" class=3D""></span> </<span class=3D"end-tag"= pan dir=3D"ltr" class=3D""> <<a href=3D"mailto:callum@well.ox.ac.uk" target=3D"_blank" class=3D"">ca= llum@well.ox.ac.uk</a>></span> wrote:<br class=3D""> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <div style=3D"word-wrap:break-word;line-break:after-white-space" class=3D""= they're big, full of noise right now. <div class=3D""><a href=3D"https://www.dropbox.com/s/f8q3m5amro2a1b2/engine= .log?dl=3D0" target=3D"_blank" class=3D"">https://www.dropbox.com/s/<wbr cl= ass=3D"">f8q3m5amro2a1b2/engine.log?dl=3D<wbr class=3D"">0</a></div> <div class=3D""><a href=3D"https://www.dropbox.com/s/uods85jk65halo3/vdsm.l= og?dl=3D0" target=3D"_blank" class=3D"">https://www.dropbox.com/s/<wbr clas= s=3D"">uods85jk65halo3/vdsm.log?dl=3D0</a></div> <div class=3D""><br class=3D""> <div class=3D""> <div class=3D""><span class=3D""> <div class=3D""> <div style=3D"letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: br= eak-word;" class=3D""> <div style=3D"word-wrap:break-word" class=3D""> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px;" class=3D""> Regards,</div> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px;" class=3D""> Callum</div> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px;" class=3D""> <br class=3D""> </div> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px;" class=3D""> --<br class=3D""> <br class=3D""> Callum Smith</div> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px;" class=3D""> Research Computing Core</div> <div style=3D"font-family: Arial; font-size: 13px; font-style: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; text-al= ign: start; text-indent: 0px; text-transform: none; white-space: normal; wo= rd-spacing: 0px;" class=3D""> Wellcome Trust Centre for Human Genetics<br class=3D""> University of Oxford<br class=3D""> e. <a href=3D"mailto:callum@well.ox.ac.uk" target=3D"_blank" class=3D"">cal= lum@well.ox.ac.uk</a></div> </div> </div> </div> </span> <div class=3D""> <div class=3D"h5"> <div class=3D""><br class=3D""> <blockquote type=3D"cite" class=3D""> <div class=3D"">On 2 May 2018, at 10:43, Fred Rolland <<a href=3D"mailto= :frolland@redhat.com" target=3D"_blank" class=3D"">frolland@redhat.com</a>&= gt; wrote:</div> <br class=3D"m_2571234484608545856Apple-interchange-newline"> <div class=3D""> <div dir=3D"ltr" class=3D""> <div class=3D"">Which version are you using?<br class=3D""> Can you provide the whole log?<br class=3D""> <br class=3D""> </div> For some reason, it looks like the Vdsm thinks that the Storage Domain is n= ot part of the pool.<br class=3D""> </div> <div class=3D"gmail_extra"><br class=3D""> <div class=3D"gmail_quote">On Wed, May 2, 2018 at 11:20 AM, Callum Smith <s= pan dir=3D"ltr" class=3D""> <<a href=3D"mailto:callum@well.ox.ac.uk" target=3D"_blank" class=3D"">ca= llum@well.ox.ac.uk</a>></span> wrote:<br class=3D""> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <div style=3D"word-wrap:break-word;line-break:after-white-space" class=3D""= per</div> <div class=3D""> return method(self, *args, **kwargs)</div> <div class=3D""> File "/usr/lib/python2.7/site-packa<wbr class= =3D"">ges/vdsm/storage/sp.py", line 1130, in activateSD</div> <div class=3D""> self.validateAttachedDomain(do<wbr class=3D""= per</div> <div class=3D""> return method(self, *args, **kwargs)</div> <div class=3D""> File "/usr/lib/python2.7/site-packa<wbr class= =3D"">ges/vdsm/storage/sp.py", line 557, in validateAttachedDomain</di= v> <div class=3D""> raise se.StorageDomainNotInPool(self<wbr clas= s=3D"">.spUUID, dom.sdUUID)</div> <div class=3D"">StorageDomainNotInPool: Storage domain not in pool: u'domai= n=3Df5914df0-f46c-4cc0-b6<wbr class=3D"">66-c929aa0225ae, pool=3D5a54bf81-0= 228-02bc-0358-0<wbr class=3D"">00000000304'</div> <div class=3D"">2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [st= orage.TaskManager.Task] (Task=3D'7f21f911-348f-45a3-b79c<wbr class=3D"">-e3= cb11642035') aborting: Task is aborted: "Storage domain not in pool: u= 'domain=3Df5914df0-f46c-4cc0-b6<wbr class=3D"">66-c929aa0225ae, pool=3D5a54bf81-0228-02bc-0358-0<wbr class=3D"">00000000304'" - code = 353 (task:1181)</div> <div class=3D"">2018-05-02 09:16:03,462+0100 ERROR (jsonrpc/0) [storage= .Dispatcher] FINISH activateStorageDomain error=3DStorage domain not in poo= l: u'domain=3Df5914df0-f46c-4cc0-b6<wbr class=3D"">66-c929aa0225ae, pool=3D= 5a54bf81-0228-02bc-0358-0<wbr class=3D"">00000000304' (dispatcher:82)</div> </div> <div class=3D""><br class=3D""> </div> <div class=3D"">engine.log</div> <div class=3D""><br class=3D""> </div> <div class=3D""> <div class=3D"">2018-05-02 09:16:02,326+01 INFO [<a href=3D"http:= //org.ovirt.engine.core.bll.st/" class=3D"">org.ovirt.engine.core.bll.st</a= div> <div class=3D"">2018-05-02 09:16:02,407+01 INFO [<a href=3D"http:= //org.ovirt.engine.core.bll.st/" class=3D"">org.ovirt.engine.core.bll.st</a= mainVDSCommand] (EE-ManagedThreadFactory-engin<wbr class=3D"">e-Thread-3345= 5) [93433989-8e26-48a9-bd3a-2ab95<wbr class=3D"">f296c08] START, ActivateStorageDomainVDSComman<wbr class=3D"">d( ActivateStorageDom= ainVDSComman<wbr class=3D"">dParameters:{storagePoolId=3D'5a<wbr class=3D""= mainVDSCommand] (EE-ManagedThreadFactory-engin<wbr class=3D"">e-Thread-3345= 5) [93433989-8e26-48a9-bd3a-2ab95<wbr class=3D"">f296c08] FINISH, ActivateStorageDomainVDSComman<wbr class=3D"">d, log id: 5c864594<= /div> <div class=3D"">2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core= .bll.sto<wbr class=3D"">rage.domain.ActivateStorageDom<wbr class=3D"">ainCo= mmand] (EE-ManagedThreadFactory-engin<wbr class=3D"">e-Thread-33455) [93433= 989-8e26-48a9-bd3a-2ab95<wbr class=3D"">f296c08] Command 'org.ovirt.engine.core.bll.sto<wbr class=3D"">rage.domain.Activate= StorageDom<wbr class=3D"">ainCommand' failed: EngineException: org.ovirt.en= gine.core.vdsbroke<wbr class=3D"">r.irsbroker.IrsOperationFailed<wbr class= =3D"">NoFailove</div> <div class=3D"">rException: IRSGenericException: IRSErrorException: Storage= domain not in pool: u'domain=3Df5914df0-f46c-4cc0-b6<wbr class=3D"">66-c92= 9aa0225ae, pool=3D5a54bf81-0228-02bc-0358-0<wbr class=3D"">00000000304' (Fa= iled with error StorageDomainNotInPool and code 353)</div> <div class=3D"">2018-05-02 09:16:02,636+01 INFO [<a href=3D"http:= //org.ovirt.engine.core.bll.st/" class=3D"">org.ovirt.engine.core.bll.st</a= the ISO storage domain manually? An NFS mount didn't add it back to the "pool" unfortunately.</div> <div class=3D""><br class=3D""> </div> <div class=3D"">On a separate note, is it possible to migrate this storage = to a new location? And if so how.</div> <div class=3D""><br class=3D""> <div class=3D""> <div style=3D"letter-spacing:normal;text-align:start;text-indent:0px;text-t= ransform:none;white-space:normal;word-spacing:0px;word-wrap:break-word" cla= ss=3D""> <div style=3D"word-wrap:break-word" class=3D""> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> Regards,</div> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> Callum</div> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> <br class=3D""> </div> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> --<br class=3D""> <br class=3D""> Callum Smith</div> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> Research Computing Core</div> <div style=3D"font-family:Arial;font-size:13px;font-style:normal;font-varia= nt-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;te= xt-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" clas= s=3D""> Wellcome Trust Centre for Human Genetics<br class=3D""> University of Oxford<br class=3D""> e. <a href=3D"mailto:callum@well.ox.ac.uk" target=3D"_blank" class=3D"">cal= lum@well.ox.ac.uk</a></div> </div> </div> </div> <br class=3D""> </div> </div> <br class=3D""> ______________________________<wbr class=3D"">_________________<br class=3D= ""> Users mailing list<br class=3D""> <a href=3D"mailto:Users@ovirt.org" target=3D"_blank" class=3D"">Users@ovirt= .org</a><br class=3D""> <a href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer= " target=3D"_blank" class=3D"">http://lists.ovirt.org/mailman<wbr class=3D"= ">/listinfo/users</a><br class=3D""> <br class=3D""> </blockquote> </div> <br class=3D""> </div> </div> </blockquote> </div> <br class=3D""> </div> </div> </div> </div> </div> </blockquote> </div> <br class=3D""> </div> </div> </blockquote> </div> <br class=3D""> </div> </div> </div> </div> </div> </div> </blockquote> </div> <br class=3D""> </div> </div> </blockquote> </div> <br class=3D""> </div> _______________________________________________<br class=3D""> Users mailing list<br class=3D""> <a href=3D"mailto:Users@ovirt.org" class=3D"">Users@ovirt.org</a><br class= =3D""> http://lists.ovirt.org/mailman/listinfo/users<br class=3D""> </div> </blockquote> </div> <br class=3D""> </div> </body> </html> --_000_EF464B7F02A647C7A11ED8E6837B3B2Ewelloxacuk_--
I am trying to reproduce this on my setup, with no success for now. Can you share the engine log corresponding to the time you tried to detach? What are the options available currently? I understand that "Activate" does not work, what about "Detach"? Can your hosts access the ISO domain? On Fri, May 4, 2018 at 2:09 PM, Callum Smith <callum@well.ox.ac.uk> wrote:
Is there any sensible way to either clean up the existing ISOs storage or re-attach it? I'm struggling to even export VMs and migrate them elsewhere with this and need to recover them asap.
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk
On 2 May 2018, at 15:09, Callum Smith <callum@well.ox.ac.uk> wrote:
Attached, thank you for looking into this
https://*HOSTNAME*/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b66...
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><storage_domain href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae" id="f5914df0-f46c-4cc0-b666-c929aa0225ae"> <actions> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/isattached" rel="isattached"/> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/reduceluns" rel="reduceluns"/> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/refreshluns" rel="refreshluns"/> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/updateovfstore" rel="updateovfstore"/> </actions> <name>VMISOs</name> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/disksnapshots" rel="disksnapshots"/> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/files" rel="files"/> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/diskprofiles" rel="diskprofiles"/> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/storageconnections" rel="storageconnections"/> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/permissions" rel="permissions"/> <available>11770357874688</available> <backup>false</backup> <committed>0</committed> <critical_space_action_blocker>5</critical_space_action_blocker> <discard_after_delete>false</discard_after_delete> <external_status>ok</external_status> <master>false</master> <storage> <address>backoffice01.cluster</address> <path>/vm-iso</path> <type>nfs</type> </storage> <storage_format>v1</storage_format> <supports_discard>false</supports_discard> <supports_discard_zeroes_data>false</supports_discard_zeroes_data> <type>iso</type> <used>38654705664</used> <warning_low_space_indicator>10</warning_low_space_indicator> <wipe_after_delete>false</wipe_after_delete> <data_centers> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304" id="5a54bf81-0228-02bc-0358-000000000304"/> </data_centers></storage_domain>
https://*HOSTNAME*/ovirt-engine/api/v4/datacenters/5a54bf81- 0228-02bc-0358-000000000304/storagedomains
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><storage_domains> <storage_domain href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91a613ef98ba" id="d9f8093b-71cd-423d-9b7d-91a613ef98ba"> <actions> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91a613ef98ba/activate" rel="activate"/> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91a613ef98ba/deactivate" rel="deactivate"/> </actions> <name>tegile-virtman-backup</name> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91a613ef98ba/disks" rel="disks"/> <available>17519171600384</available> <backup>false</backup> <committed>0</committed> <critical_space_action_blocker>5</critical_space_action_blocker> <discard_after_delete>false</discard_after_delete> <external_status>ok</external_status> <master>false</master> <status>maintenance</status> <storage> <address>192.168.64.248</address> <nfs_version>auto</nfs_version> <path>/export/virtman/backup</path> <type>nfs</type> </storage> <storage_format>v1</storage_format> <supports_discard>false</supports_discard> <supports_discard_zeroes_data>false</supports_discard_zeroes_data> <type>export</type> <used>8589934592</used> <warning_low_space_indicator>10</warning_low_space_indicator> <wipe_after_delete>false</wipe_after_delete> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304" id="5a54bf81-0228-02bc-0358-000000000304"/> <data_centers> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304" id="5a54bf81-0228-02bc-0358-000000000304"/> </data_centers> </storage_domain> <storage_domain href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-67737dedd7a1" id="017dd620-65ec-4fa4-8d46-67737dedd7a1"> <actions> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-67737dedd7a1/activate" rel="activate"/> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-67737dedd7a1/deactivate" rel="deactivate"/> </actions> <name>VMStorage</name> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-67737dedd7a1/disks" rel="disks"/> <available>11770357874688</available> <backup>false</backup> <committed>118111600640</committed> <critical_space_action_blocker>5</critical_space_action_blocker> <discard_after_delete>false</discard_after_delete> <external_status>ok</external_status> <master>true</master> <status>active</status> <storage> <address>backoffice01.cluster</address> <path>/vm-storage2</path> <type>nfs</type> </storage> <storage_format>v4</storage_format> <supports_discard>false</supports_discard> <supports_discard_zeroes_data>false</supports_discard_zeroes_data> <type>data</type> <used>38654705664</used> <warning_low_space_indicator>10</warning_low_space_indicator> <wipe_after_delete>false</wipe_after_delete> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304" id="5a54bf81-0228-02bc-0358-000000000304"/> <data_centers> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304" id="5a54bf81-0228-02bc-0358-000000000304"/> </data_centers> </storage_domain> <storage_domain href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b6f563a9fc17" id="e572c874-17f8-4ad8-9bac-b6f563a9fc17"> <actions> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b6f563a9fc17/activate" rel="activate"/> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b6f563a9fc17/deactivate" rel="deactivate"/> </actions> <name>tegile-virtman</name> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b6f563a9fc17/disks" rel="disks"/> <available>2190433320960</available> <backup>false</backup> <committed>226559524864</committed> <critical_space_action_blocker>5</critical_space_action_blocker> <discard_after_delete>false</discard_after_delete> <external_status>ok</external_status> <master>false</master> <status>active</status> <storage> <address>192.168.64.248</address> <nfs_version>auto</nfs_version> <path>/export/virtman/VirtualServerShare_1</path> <type>nfs</type> </storage> <storage_format>v4</storage_format> <supports_discard>false</supports_discard> <supports_discard_zeroes_data>false</supports_discard_zeroes_data> <type>data</type> <used>8589934592</used> <warning_low_space_indicator>10</warning_low_space_indicator> <wipe_after_delete>false</wipe_after_delete> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304" id="5a54bf81-0228-02bc-0358-000000000304"/> <data_centers> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304" id="5a54bf81-0228-02bc-0358-000000000304"/> </data_centers> </storage_domain> <storage_domain href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae" id="f5914df0-f46c-4cc0-b666-c929aa0225ae"> <actions> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/activate" rel="activate"/> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/deactivate" rel="deactivate"/> </actions> <name>VMISOs</name> <available>11770357874688</available> <backup>false</backup> <committed>0</committed> <critical_space_action_blocker>5</critical_space_action_blocker> <discard_after_delete>false</discard_after_delete> <external_status>ok</external_status> <master>false</master> <status>maintenance</status> <storage> <address>backoffice01.cluster</address> <path>/vm-iso</path> <type>nfs</type> </storage> <storage_format>v1</storage_format> <supports_discard>false</supports_discard> <supports_discard_zeroes_data>false</supports_discard_zeroes_data> <type>iso</type> <used>38654705664</used> <warning_low_space_indicator>10</warning_low_space_indicator> <wipe_after_delete>false</wipe_after_delete> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304" id="5a54bf81-0228-02bc-0358-000000000304"/> <data_centers> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304" id="5a54bf81-0228-02bc-0358-000000000304"/> </data_centers> </storage_domain></storage_domains>
<Screen Shot 2018-05-02 at 14.54.42.png> Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk
On 2 May 2018, at 14:46, Fred Rolland <frolland@redhat.com> wrote:
Can you share the REST API data of the Storage domain and Data Center? Here an example of the URLs, you will need to replace with correct ids.
http://MY-SERVER/ovirt-engine/api/v4/storagedomains/ 13461356-f6f7-4a58-9897-2fac61ff40af <http://my-server/ovirt-engine/api/v4/storagedomains/13461356-f6f7-4a58-9897-2fac61ff40af>
http://MY-SERVER/ovirt-engine/api/v4/datacenters/5a5df553- 022d-036d-01e8-000000000071/storagedomains <http://my-server/ovirt-engine/api/v4/datacenters/5a5df553-022d-036d-01e8-000000000071/storagedomains>
On Wed, May 2, 2018 at 12:53 PM, Callum Smith <callum@well.ox.ac.uk> wrote:
This is on 4.2.0.2-1, I've linked the main logs to dropbox simply because they're big, full of noise right now. https://www.dropbox.com/s/f8q3m5amro2a1b2/engine.log?dl=0 https://www.dropbox.com/s/uods85jk65halo3/vdsm.log?dl=0
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk
On 2 May 2018, at 10:43, Fred Rolland <frolland@redhat.com> wrote:
Which version are you using? Can you provide the whole log?
For some reason, it looks like the Vdsm thinks that the Storage Domain is not part of the pool.
On Wed, May 2, 2018 at 11:20 AM, Callum Smith <callum@well.ox.ac.uk> wrote:
State is maintenance for the ISOs storage. I've extracted what is hopefully the relevant bits of the log.
VDSM.log (SPM)
2018-05-02 09:16:03,455+0100 INFO (ioprocess communication (179084)) [IOProcess] Starting ioprocess (__init__:447) 2018-05-02 09:16:03,456+0100 INFO (ioprocess communication (179091)) [IOProcess] Starting ioprocess (__init__:447) 2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [vdsm.api] FINISH activateStorageDomain error=Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' from=::ffff:192.168.64.254,58968, flow_id=93433989-8e26-48a9-bd3a-2ab95f296c08, task_id=7f21f911-348f-45a3-b79c-e3cb11642035 (api:50) 2018-05-02 09:16:03,461+0100 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='7f21f911-348f-45a3-b79c-e3cb11642035') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in activateStorageDomain File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1256, in activateStorageDomain pool.activateSD(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1130, in activateSD self.validateAttachedDomain(dom) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 557, in validateAttachedDomain raise se.StorageDomainNotInPool(self.spUUID, dom.sdUUID) StorageDomainNotInPool: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' 2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [storage.TaskManager.Task] (Task='7f21f911-348f-45a3-b79c-e3cb11642035') aborting: Task is aborted: "Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304'" - code 353 (task:1181) 2018-05-02 09:16:03,462+0100 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH activateStorageDomain error=Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' (dispatcher:82)
engine.log
2018-05-02 09:16:02,326+01 INFO [org.ovirt.engine.core.bll.st orage.domain.ActivateStorageDomainCommand] (default task-20) [93433989-8e26-48a9-bd3a-2ab95f296c08] Lock Acquired to object 'EngineLock:{exclusiveLocks='[f5914df0-f46c-4cc0-b666-c929aa0225ae=STORAGE]', sharedLocks=''}' 2018-05-02 09:16:02,376+01 INFO [org.ovirt.engine.core.bll.st orage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Running command: ActivateStorageDomainCommand internal: false. Entities affected : ID: f5914df0-f46c-4cc0-b666-c929aa0225ae Type: StorageAction group MANIPULATE_STORAGE_DOMA IN with role type ADMIN 2018-05-02 09:16:02,385+01 INFO [org.ovirt.engine.core.bll.st orage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Lock freed to object 'EngineLock:{exclusiveLocks='[f5914df0-f46c-4cc0-b666-c929aa0225ae=STORAGE]', sharedLocks=''}' 2018-05-02 09:16:02,385+01 INFO [org.ovirt.engine.core.bll.st orage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] ActivateStorage Domain. Before Connect all hosts to pool. Time: Wed May 02 09:16:02 BST 2018 2018-05-02 09:16:02,407+01 INFO [org.ovirt.engine.core.bll.st orage.connection.ConnectStorageToVdsCommand] (EE-ManagedThreadFactory-engine-Thread-33456) [40a82b47] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2018-05-02 09:16:02,421+01 INFO [org.ovirt.engine.core.vdsbro ker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33456) [40a82b47] START, ConnectStorageServerVDSCommand(HostName = virtA003, StorageServerConnectionManagementVDSParameters:{hostId='fe28 61fc-2b47-4807-b054-470198eda473', storagePoolId='00000000-0000-0 000-0000-000000 000000', storageType='NFS', connectionList='[StorageServer Connections:{id='da392861-aedc-4f1e-97f4-6919fb01f1e9', connection='backoffice01.cluster:/vm-iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 23ce648f 2018-05-02 09:16:02,446+01 INFO [org.ovirt.engine.core.vdsbro ker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33456) [40a82b47] FINISH, ConnectStorageServerVDSCommand, return: {da392861-aedc-4f1e-97f4-6919fb01f1e9=0}, log id: 23ce648f 2018-05-02 09:16:02,450+01 INFO [org.ovirt.engine.core.vdsbro ker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] START, ActivateStorageDomainVDSCommand( ActivateStorageDomainVDSComman dParameters:{storagePoolId='5a54bf81-0228-02bc-0358-000000000304', ignoreFailoverLimit='false', stor ageDomainId='f5914df0-f46c-4cc0-b666-c929aa0225ae'}), log id: 5c864594 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a5 4bf81-0228-02bc-0358-000000000304' 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.vdsbrok er.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command 'ActivateStorageDomainVDSCommand( ActivateStorageDomainVDSComman dParameters:{storagePoolId='5a54bf81-0228-02bc-0358-000000000304', ignoreFailoverLimit='false', st orageDomainId='f5914df0-f46c-4cc0-b666-c929aa0225ae'})' execution failed: IRSGenericException: IRSErrorException: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' 2018-05-02 09:16:02,635+01 INFO [org.ovirt.engine.core.vdsbro ker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] FINISH, ActivateStorageDomainVDSCommand, log id: 5c864594 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.bll.sto rage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command 'org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroke r.irsbroker.IrsOperationFailedNoFailove rException: IRSGenericException: IRSErrorException: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' (Failed with error StorageDomainNotInPool and code 353) 2018-05-02 09:16:02,636+01 INFO [org.ovirt.engine.core.bll.st orage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command [id=22b0f3c1-9a09-4e26-8096-d83465c8f4ee]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.b usinessentities.StoragePoolIsoMap; snapshot: EntityStatus Snapshot:{id='StoragePoolIsoMapId:{storagePoolId='5a54bf81-0228-02bc-0358-000000000304', storageId='f5914df0-f46c-4cc0-b666-c929aa0225ae'}', status='Maintenance'}. 2018-05-02 09:16:02,660+01 ERROR [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID: USER_ACTIVATE_STORAGE_DOMAIN_FAILED(967), Failed to activate Storage Domain VMISOs (Data Center Default) by admin@internal-authz
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk
On 2 May 2018, at 08:44, Fred Rolland <frolland@redhat.com> wrote:
Hi,
Can you provide logs from engine and Vdsm(SPM)? What is the state now?
Thanks, Fred
On Tue, May 1, 2018 at 4:11 PM, Callum Smith <callum@well.ox.ac.uk> wrote:
Dear All,
It appears that clicking "detach" on the ISO storage domain is a really bad idea. This has gotten half way through the procedure and now can't be recovered from. Is there any advice for re-attaching the ISO storage domain manually? An NFS mount didn't add it back to the "pool" unfortunately.
On a separate note, is it possible to migrate this storage to a new location? And if so how.
Regards, Callum
--
Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Dear Fred, Hosts can indeed access the share, in fact i can manually mount it on the hosts and engine. I've attached the day's logs from when the ISO domain was detached, again expect lots of noise. The time pressure on this issue has gone as we now have a functional fresh install on another node, and the ability to export the VMs across using a shared export domain. Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> On 6 May 2018, at 15:38, Fred Rolland <frolland@redhat.com<mailto:frolland@redhat.com>> wrote: I am trying to reproduce this on my setup, with no success for now. Can you share the engine log corresponding to the time you tried to detach? What are the options available currently? I understand that "Activate" does not work, what about "Detach"? Can your hosts access the ISO domain? On Fri, May 4, 2018 at 2:09 PM, Callum Smith <callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>> wrote: Is there any sensible way to either clean up the existing ISOs storage or re-attach it? I'm struggling to even export VMs and migrate them elsewhere with this and need to recover them asap. Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> On 2 May 2018, at 15:09, Callum Smith <callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>> wrote: Attached, thank you for looking into this https://HOSTNAME/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-... <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <storage_domain href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae" id="f5914df0-f46c-4cc0-b666-c929aa0225ae"> <actions> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/isattached" rel="isattached"/> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/reduceluns" rel="reduceluns"/> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/refreshluns" rel="refreshluns"/> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/updateovfstore" rel="updateovfstore"/> </actions> <name>VMISOs</name> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/disksnapshots" rel="disksnapshots"/> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/files" rel="files"/> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/diskprofiles" rel="diskprofiles"/> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/storageconnections" rel="storageconnections"/> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/permissions" rel="permissions"/> <available>11770357874688</available> <backup>false</backup> <committed>0</committed> <critical_space_action_blocker>5</critical_space_action_blocker> <discard_after_delete>false</discard_after_delete> <external_status>ok</external_status> <master>false</master> <storage> <address>backoffice01.cluster</address> <path>/vm-iso</path> <type>nfs</type> </storage> <storage_format>v1</storage_format> <supports_discard>false</supports_discard> <supports_discard_zeroes_data>false</supports_discard_zeroes_data> <type>iso</type> <used>38654705664</used> <warning_low_space_indicator>10</warning_low_space_indicator> <wipe_after_delete>false</wipe_after_delete> <data_centers> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304" id="5a54bf81-0228-02bc-0358-000000000304"/> </data_centers> </storage_domain> https://HOSTNAME/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000... <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <storage_domains> <storage_domain href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91a613ef98ba" id="d9f8093b-71cd-423d-9b7d-91a613ef98ba"> <actions> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91a613ef98ba/activate" rel="activate"/> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91a613ef98ba/deactivate" rel="deactivate"/> </actions> <name>tegile-virtman-backup</name> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91a613ef98ba/disks" rel="disks"/> <available>17519171600384</available> <backup>false</backup> <committed>0</committed> <critical_space_action_blocker>5</critical_space_action_blocker> <discard_after_delete>false</discard_after_delete> <external_status>ok</external_status> <master>false</master> <status>maintenance</status> <storage> <address>192.168.64.248</address> <nfs_version>auto</nfs_version> <path>/export/virtman/backup</path> <type>nfs</type> </storage> <storage_format>v1</storage_format> <supports_discard>false</supports_discard> <supports_discard_zeroes_data>false</supports_discard_zeroes_data> <type>export</type> <used>8589934592</used> <warning_low_space_indicator>10</warning_low_space_indicator> <wipe_after_delete>false</wipe_after_delete> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304" id="5a54bf81-0228-02bc-0358-000000000304"/> <data_centers> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304" id="5a54bf81-0228-02bc-0358-000000000304"/> </data_centers> </storage_domain> <storage_domain href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-67737dedd7a1" id="017dd620-65ec-4fa4-8d46-67737dedd7a1"> <actions> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-67737dedd7a1/activate" rel="activate"/> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-67737dedd7a1/deactivate" rel="deactivate"/> </actions> <name>VMStorage</name> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-67737dedd7a1/disks" rel="disks"/> <available>11770357874688</available> <backup>false</backup> <committed>118111600640</committed> <critical_space_action_blocker>5</critical_space_action_blocker> <discard_after_delete>false</discard_after_delete> <external_status>ok</external_status> <master>true</master> <status>active</status> <storage> <address>backoffice01.cluster</address> <path>/vm-storage2</path> <type>nfs</type> </storage> <storage_format>v4</storage_format> <supports_discard>false</supports_discard> <supports_discard_zeroes_data>false</supports_discard_zeroes_data> <type>data</type> <used>38654705664</used> <warning_low_space_indicator>10</warning_low_space_indicator> <wipe_after_delete>false</wipe_after_delete> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304" id="5a54bf81-0228-02bc-0358-000000000304"/> <data_centers> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304" id="5a54bf81-0228-02bc-0358-000000000304"/> </data_centers> </storage_domain> <storage_domain href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b6f563a9fc17" id="e572c874-17f8-4ad8-9bac-b6f563a9fc17"> <actions> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b6f563a9fc17/activate" rel="activate"/> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b6f563a9fc17/deactivate" rel="deactivate"/> </actions> <name>tegile-virtman</name> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b6f563a9fc17/disks" rel="disks"/> <available>2190433320960</available> <backup>false</backup> <committed>226559524864</committed> <critical_space_action_blocker>5</critical_space_action_blocker> <discard_after_delete>false</discard_after_delete> <external_status>ok</external_status> <master>false</master> <status>active</status> <storage> <address>192.168.64.248</address> <nfs_version>auto</nfs_version> <path>/export/virtman/VirtualServerShare_1</path> <type>nfs</type> </storage> <storage_format>v4</storage_format> <supports_discard>false</supports_discard> <supports_discard_zeroes_data>false</supports_discard_zeroes_data> <type>data</type> <used>8589934592</used> <warning_low_space_indicator>10</warning_low_space_indicator> <wipe_after_delete>false</wipe_after_delete> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304" id="5a54bf81-0228-02bc-0358-000000000304"/> <data_centers> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304" id="5a54bf81-0228-02bc-0358-000000000304"/> </data_centers> </storage_domain> <storage_domain href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae" id="f5914df0-f46c-4cc0-b666-c929aa0225ae"> <actions> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/activate" rel="activate"/> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/deactivate" rel="deactivate"/> </actions> <name>VMISOs</name> <available>11770357874688</available> <backup>false</backup> <committed>0</committed> <critical_space_action_blocker>5</critical_space_action_blocker> <discard_after_delete>false</discard_after_delete> <external_status>ok</external_status> <master>false</master> <status>maintenance</status> <storage> <address>backoffice01.cluster</address> <path>/vm-iso</path> <type>nfs</type> </storage> <storage_format>v1</storage_format> <supports_discard>false</supports_discard> <supports_discard_zeroes_data>false</supports_discard_zeroes_data> <type>iso</type> <used>38654705664</used> <warning_low_space_indicator>10</warning_low_space_indicator> <wipe_after_delete>false</wipe_after_delete> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304" id="5a54bf81-0228-02bc-0358-000000000304"/> <data_centers> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304" id="5a54bf81-0228-02bc-0358-000000000304"/> </data_centers> </storage_domain> </storage_domains> <Screen Shot 2018-05-02 at 14.54.42.png> Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> On 2 May 2018, at 14:46, Fred Rolland <frolland@redhat.com<mailto:frolland@redhat.com>> wrote: Can you share the REST API data of the Storage domain and Data Center? Here an example of the URLs, you will need to replace with correct ids. http://MY-SERVER/ovirt-engine/api/v4/storagedomains/13461356-f6f7-4a58-9897-2fac61ff40af<http://my-server/ovirt-engine/api/v4/storagedomains/13461356-f6f7-4a58-9897-2fac61ff40af> http://MY-SERVER/ovirt-engine/api/v4/datacenters/5a5df553-022d-036d-01e8-000000000071/storagedomains<http://my-server/ovirt-engine/api/v4/datacenters/5a5df553-022d-036d-01e8-000000000071/storagedomains> On Wed, May 2, 2018 at 12:53 PM, Callum Smith <callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>> wrote: This is on 4.2.0.2-1, I've linked the main logs to dropbox simply because they're big, full of noise right now. https://www.dropbox.com/s/f8q3m5amro2a1b2/engine.log?dl=0 https://www.dropbox.com/s/uods85jk65halo3/vdsm.log?dl=0 Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> On 2 May 2018, at 10:43, Fred Rolland <frolland@redhat.com<mailto:frolland@redhat.com>> wrote: Which version are you using? Can you provide the whole log? For some reason, it looks like the Vdsm thinks that the Storage Domain is not part of the pool. On Wed, May 2, 2018 at 11:20 AM, Callum Smith <callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>> wrote: State is maintenance for the ISOs storage. I've extracted what is hopefully the relevant bits of the log. VDSM.log (SPM) 2018-05-02 09:16:03,455+0100 INFO (ioprocess communication (179084)) [IOProcess] Starting ioprocess (__init__:447) 2018-05-02 09:16:03,456+0100 INFO (ioprocess communication (179091)) [IOProcess] Starting ioprocess (__init__:447) 2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [vdsm.api] FINISH activateStorageDomain error=Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' from=::ffff:192.168.64.254,58968, flow_id=93433989-8e26-48a9-bd3a-2ab95f296c08, task_id=7f21f911-348f-45a3-b79c-e3cb11642035 (api:50) 2018-05-02 09:16:03,461+0100 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='7f21f911-348f-45a3-b79c-e3cb11642035') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in activateStorageDomain File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1256, in activateStorageDomain pool.activateSD(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1130, in activateSD self.validateAttachedDomain(dom) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 557, in validateAttachedDomain raise se.StorageDomainNotInPool(self.spUUID, dom.sdUUID) StorageDomainNotInPool: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' 2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [storage.TaskManager.Task] (Task='7f21f911-348f-45a3-b79c-e3cb11642035') aborting: Task is aborted: "Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304'" - code 353 (task:1181) 2018-05-02 09:16:03,462+0100 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH activateStorageDomain error=Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' (dispatcher:82) engine.log 2018-05-02 09:16:02,326+01 INFO [org.ovirt.engine.core.bll.st<http://org.ovirt.engine.core.bll.st/>orage.domain.ActivateStorageDomainCommand] (default task-20) [93433989-8e26-48a9-bd3a-2ab95f296c08] Lock Acquired to object 'EngineLock:{exclusiveLocks='[f5914df0-f46c-4cc0-b666-c929aa0225ae=STORAGE]', sharedLocks=''}' 2018-05-02 09:16:02,376+01 INFO [org.ovirt.engine.core.bll.st<http://org.ovirt.engine.core.bll.st/>orage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Running command: ActivateStorageDomainCommand internal: false. Entities affected : ID: f5914df0-f46c-4cc0-b666-c929aa0225ae Type: StorageAction group MANIPULATE_STORAGE_DOMA IN with role type ADMIN 2018-05-02 09:16:02,385+01 INFO [org.ovirt.engine.core.bll.st<http://org.ovirt.engine.core.bll.st/>orage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Lock freed to object 'EngineLock:{exclusiveLocks='[f5914df0-f46c-4cc0-b666-c929aa0225ae=STORAGE]', sharedLocks=''}' 2018-05-02 09:16:02,385+01 INFO [org.ovirt.engine.core.bll.st<http://org.ovirt.engine.core.bll.st/>orage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] ActivateStorage Domain. Before Connect all hosts to pool. Time: Wed May 02 09:16:02 BST 2018 2018-05-02 09:16:02,407+01 INFO [org.ovirt.engine.core.bll.st<http://org.ovirt.engine.core.bll.st/>orage.connection.ConnectStorageToVdsCommand] (EE-ManagedThreadFactory-engine-Thread-33456) [40a82b47] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2018-05-02 09:16:02,421+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33456) [40a82b47] START, ConnectStorageServerVDSCommand(HostName = virtA003, StorageServerConnectionManagementVDSParameters:{hostId='fe2861fc-2b47-4807-b054-470198eda473', storagePoolId='00000000-0000-0000-0000-000000 000000', storageType='NFS', connectionList='[StorageServerConnections:{id='da392861-aedc-4f1e-97f4-6919fb01f1e9', connection='backoffice01.cluster:/vm-iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 23ce648f 2018-05-02 09:16:02,446+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33456) [40a82b47] FINISH, ConnectStorageServerVDSCommand, return: {da392861-aedc-4f1e-97f4-6919fb01f1e9=0}, log id: 23ce648f 2018-05-02 09:16:02,450+01 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] START, ActivateStorageDomainVDSCommand( ActivateStorageDomainVDSCommandParameters:{storagePoolId='5a54bf81-0228-02bc-0358-000000000304', ignoreFailoverLimit='false', stor ageDomainId='f5914df0-f46c-4cc0-b666-c929aa0225ae'}), log id: 5c864594 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a5 4bf81-0228-02bc-0358-000000000304' 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command 'ActivateStorageDomainVDSCommand( ActivateStorageDomainVDSCommandParameters:{storagePoolId='5a54bf81-0228-02bc-0358-000000000304', ignoreFailoverLimit='false', st orageDomainId='f5914df0-f46c-4cc0-b666-c929aa0225ae'})' execution failed: IRSGenericException: IRSErrorException: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' 2018-05-02 09:16:02,635+01 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] FINISH, ActivateStorageDomainVDSCommand, log id: 5c864594 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command 'org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.irsbroker.IrsOperationFailedNoFailove rException: IRSGenericException: IRSErrorException: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' (Failed with error StorageDomainNotInPool and code 353) 2018-05-02 09:16:02,636+01 INFO [org.ovirt.engine.core.bll.st<http://org.ovirt.engine.core.bll.st/>orage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command [id=22b0f3c1-9a09-4e26-8096-d83465c8f4ee]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: EntityStatus Snapshot:{id='StoragePoolIsoMapId:{storagePoolId='5a54bf81-0228-02bc-0358-000000000304', storageId='f5914df0-f46c-4cc0-b666-c929aa0225ae'}', status='Maintenance'}. 2018-05-02 09:16:02,660+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID: USER_ACTIVATE_STORAGE_DOMAIN_FAILED(967), Failed to activate Storage Domain VMISOs (Data Center Default) by admin@internal-authz Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> On 2 May 2018, at 08:44, Fred Rolland <frolland@redhat.com<mailto:frolland@redhat.com>> wrote: Hi, Can you provide logs from engine and Vdsm(SPM)? What is the state now? Thanks, Fred On Tue, May 1, 2018 at 4:11 PM, Callum Smith <callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>> wrote: Dear All, It appears that clicking "detach" on the ISO storage domain is a really bad idea. This has gotten half way through the procedure and now can't be recovered from. Is there any advice for re-attaching the ISO storage domain manually? An NFS mount didn't add it back to the "pool" unfortunately. On a separate note, is it possible to migrate this storage to a new location? And if so how. Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users
Dear Fred, Hosts can indeed access the share, in fact i can manually mount it on the hosts and engine. I've attached the day's logs from when the ISO domain was detached, again expect lots of noise. The time pressure on this issue has gone as we now have a functional fresh install on another node, and the ability to export the VMs across using a shared export domain. https://www.dropbox.com/s/rdcyzoef3j3cjgx/engine.log-20180501.gz?dl=0 Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> On 6 May 2018, at 15:38, Fred Rolland <frolland@redhat.com<mailto:frolland@redhat.com>> wrote: I am trying to reproduce this on my setup, with no success for now. Can you share the engine log corresponding to the time you tried to detach? What are the options available currently? I understand that "Activate" does not work, what about "Detach"? Can your hosts access the ISO domain? On Fri, May 4, 2018 at 2:09 PM, Callum Smith <callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>> wrote: Is there any sensible way to either clean up the existing ISOs storage or re-attach it? I'm struggling to even export VMs and migrate them elsewhere with this and need to recover them asap. Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> On 2 May 2018, at 15:09, Callum Smith <callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>> wrote: Attached, thank you for looking into this https://HOSTNAME/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-... <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <storage_domain href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae" id="f5914df0-f46c-4cc0-b666-c929aa0225ae"> <actions> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/isattached" rel="isattached"/> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/reduceluns" rel="reduceluns"/> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/refreshluns" rel="refreshluns"/> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/updateovfstore" rel="updateovfstore"/> </actions> <name>VMISOs</name> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/disksnapshots" rel="disksnapshots"/> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/files" rel="files"/> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/diskprofiles" rel="diskprofiles"/> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/storageconnections" rel="storageconnections"/> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/permissions" rel="permissions"/> <available>11770357874688</available> <backup>false</backup> <committed>0</committed> <critical_space_action_blocker>5</critical_space_action_blocker> <discard_after_delete>false</discard_after_delete> <external_status>ok</external_status> <master>false</master> <storage> <address>backoffice01.cluster</address> <path>/vm-iso</path> <type>nfs</type> </storage> <storage_format>v1</storage_format> <supports_discard>false</supports_discard> <supports_discard_zeroes_data>false</supports_discard_zeroes_data> <type>iso</type> <used>38654705664</used> <warning_low_space_indicator>10</warning_low_space_indicator> <wipe_after_delete>false</wipe_after_delete> <data_centers> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304" id="5a54bf81-0228-02bc-0358-000000000304"/> </data_centers> </storage_domain> https://HOSTNAME/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000... <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <storage_domains> <storage_domain href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91a613ef98ba" id="d9f8093b-71cd-423d-9b7d-91a613ef98ba"> <actions> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91a613ef98ba/activate" rel="activate"/> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91a613ef98ba/deactivate" rel="deactivate"/> </actions> <name>tegile-virtman-backup</name> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/d9f8093b-71cd-423d-9b7d-91a613ef98ba/disks" rel="disks"/> <available>17519171600384</available> <backup>false</backup> <committed>0</committed> <critical_space_action_blocker>5</critical_space_action_blocker> <discard_after_delete>false</discard_after_delete> <external_status>ok</external_status> <master>false</master> <status>maintenance</status> <storage> <address>192.168.64.248</address> <nfs_version>auto</nfs_version> <path>/export/virtman/backup</path> <type>nfs</type> </storage> <storage_format>v1</storage_format> <supports_discard>false</supports_discard> <supports_discard_zeroes_data>false</supports_discard_zeroes_data> <type>export</type> <used>8589934592</used> <warning_low_space_indicator>10</warning_low_space_indicator> <wipe_after_delete>false</wipe_after_delete> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304" id="5a54bf81-0228-02bc-0358-000000000304"/> <data_centers> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304" id="5a54bf81-0228-02bc-0358-000000000304"/> </data_centers> </storage_domain> <storage_domain href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-67737dedd7a1" id="017dd620-65ec-4fa4-8d46-67737dedd7a1"> <actions> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-67737dedd7a1/activate" rel="activate"/> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-67737dedd7a1/deactivate" rel="deactivate"/> </actions> <name>VMStorage</name> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/017dd620-65ec-4fa4-8d46-67737dedd7a1/disks" rel="disks"/> <available>11770357874688</available> <backup>false</backup> <committed>118111600640</committed> <critical_space_action_blocker>5</critical_space_action_blocker> <discard_after_delete>false</discard_after_delete> <external_status>ok</external_status> <master>true</master> <status>active</status> <storage> <address>backoffice01.cluster</address> <path>/vm-storage2</path> <type>nfs</type> </storage> <storage_format>v4</storage_format> <supports_discard>false</supports_discard> <supports_discard_zeroes_data>false</supports_discard_zeroes_data> <type>data</type> <used>38654705664</used> <warning_low_space_indicator>10</warning_low_space_indicator> <wipe_after_delete>false</wipe_after_delete> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304" id="5a54bf81-0228-02bc-0358-000000000304"/> <data_centers> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304" id="5a54bf81-0228-02bc-0358-000000000304"/> </data_centers> </storage_domain> <storage_domain href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b6f563a9fc17" id="e572c874-17f8-4ad8-9bac-b6f563a9fc17"> <actions> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b6f563a9fc17/activate" rel="activate"/> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b6f563a9fc17/deactivate" rel="deactivate"/> </actions> <name>tegile-virtman</name> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/e572c874-17f8-4ad8-9bac-b6f563a9fc17/disks" rel="disks"/> <available>2190433320960</available> <backup>false</backup> <committed>226559524864</committed> <critical_space_action_blocker>5</critical_space_action_blocker> <discard_after_delete>false</discard_after_delete> <external_status>ok</external_status> <master>false</master> <status>active</status> <storage> <address>192.168.64.248</address> <nfs_version>auto</nfs_version> <path>/export/virtman/VirtualServerShare_1</path> <type>nfs</type> </storage> <storage_format>v4</storage_format> <supports_discard>false</supports_discard> <supports_discard_zeroes_data>false</supports_discard_zeroes_data> <type>data</type> <used>8589934592</used> <warning_low_space_indicator>10</warning_low_space_indicator> <wipe_after_delete>false</wipe_after_delete> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304" id="5a54bf81-0228-02bc-0358-000000000304"/> <data_centers> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304" id="5a54bf81-0228-02bc-0358-000000000304"/> </data_centers> </storage_domain> <storage_domain href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae" id="f5914df0-f46c-4cc0-b666-c929aa0225ae"> <actions> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/activate" rel="activate"/> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304/storagedomains/f5914df0-f46c-4cc0-b666-c929aa0225ae/deactivate" rel="deactivate"/> </actions> <name>VMISOs</name> <available>11770357874688</available> <backup>false</backup> <committed>0</committed> <critical_space_action_blocker>5</critical_space_action_blocker> <discard_after_delete>false</discard_after_delete> <external_status>ok</external_status> <master>false</master> <status>maintenance</status> <storage> <address>backoffice01.cluster</address> <path>/vm-iso</path> <type>nfs</type> </storage> <storage_format>v1</storage_format> <supports_discard>false</supports_discard> <supports_discard_zeroes_data>false</supports_discard_zeroes_data> <type>iso</type> <used>38654705664</used> <warning_low_space_indicator>10</warning_low_space_indicator> <wipe_after_delete>false</wipe_after_delete> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304" id="5a54bf81-0228-02bc-0358-000000000304"/> <data_centers> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228-02bc-0358-000000000304" id="5a54bf81-0228-02bc-0358-000000000304"/> </data_centers> </storage_domain> </storage_domains> <Screen Shot 2018-05-02 at 14.54.42.png> Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> On 2 May 2018, at 14:46, Fred Rolland <frolland@redhat.com<mailto:frolland@redhat.com>> wrote: Can you share the REST API data of the Storage domain and Data Center? Here an example of the URLs, you will need to replace with correct ids. http://MY-SERVER/ovirt-engine/api/v4/storagedomains/13461356-f6f7-4a58-9897-2fac61ff40af<http://my-server/ovirt-engine/api/v4/storagedomains/13461356-f6f7-4a58-9897-2fac61ff40af> http://MY-SERVER/ovirt-engine/api/v4/datacenters/5a5df553-022d-036d-01e8-000000000071/storagedomains<http://my-server/ovirt-engine/api/v4/datacenters/5a5df553-022d-036d-01e8-000000000071/storagedomains> On Wed, May 2, 2018 at 12:53 PM, Callum Smith <callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>> wrote: This is on 4.2.0.2-1, I've linked the main logs to dropbox simply because they're big, full of noise right now. https://www.dropbox.com/s/f8q3m5amro2a1b2/engine.log?dl=0 https://www.dropbox.com/s/uods85jk65halo3/vdsm.log?dl=0 Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> On 2 May 2018, at 10:43, Fred Rolland <frolland@redhat.com<mailto:frolland@redhat.com>> wrote: Which version are you using? Can you provide the whole log? For some reason, it looks like the Vdsm thinks that the Storage Domain is not part of the pool. On Wed, May 2, 2018 at 11:20 AM, Callum Smith <callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>> wrote: State is maintenance for the ISOs storage. I've extracted what is hopefully the relevant bits of the log. VDSM.log (SPM) 2018-05-02 09:16:03,455+0100 INFO (ioprocess communication (179084)) [IOProcess] Starting ioprocess (__init__:447) 2018-05-02 09:16:03,456+0100 INFO (ioprocess communication (179091)) [IOProcess] Starting ioprocess (__init__:447) 2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [vdsm.api] FINISH activateStorageDomain error=Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' from=::ffff:192.168.64.254,58968, flow_id=93433989-8e26-48a9-bd3a-2ab95f296c08, task_id=7f21f911-348f-45a3-b79c-e3cb11642035 (api:50) 2018-05-02 09:16:03,461+0100 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='7f21f911-348f-45a3-b79c-e3cb11642035') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in activateStorageDomain File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1256, in activateStorageDomain pool.activateSD(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1130, in activateSD self.validateAttachedDomain(dom) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 557, in validateAttachedDomain raise se.StorageDomainNotInPool(self.spUUID, dom.sdUUID) StorageDomainNotInPool: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' 2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [storage.TaskManager.Task] (Task='7f21f911-348f-45a3-b79c-e3cb11642035') aborting: Task is aborted: "Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304'" - code 353 (task:1181) 2018-05-02 09:16:03,462+0100 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH activateStorageDomain error=Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' (dispatcher:82) engine.log 2018-05-02 09:16:02,326+01 INFO [org.ovirt.engine.core.bll.st<http://org.ovirt.engine.core.bll.st/>orage.domain.ActivateStorageDomainCommand] (default task-20) [93433989-8e26-48a9-bd3a-2ab95f296c08] Lock Acquired to object 'EngineLock:{exclusiveLocks='[f5914df0-f46c-4cc0-b666-c929aa0225ae=STORAGE]', sharedLocks=''}' 2018-05-02 09:16:02,376+01 INFO [org.ovirt.engine.core.bll.st<http://org.ovirt.engine.core.bll.st/>orage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Running command: ActivateStorageDomainCommand internal: false. Entities affected : ID: f5914df0-f46c-4cc0-b666-c929aa0225ae Type: StorageAction group MANIPULATE_STORAGE_DOMA IN with role type ADMIN 2018-05-02 09:16:02,385+01 INFO [org.ovirt.engine.core.bll.st<http://org.ovirt.engine.core.bll.st/>orage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Lock freed to object 'EngineLock:{exclusiveLocks='[f5914df0-f46c-4cc0-b666-c929aa0225ae=STORAGE]', sharedLocks=''}' 2018-05-02 09:16:02,385+01 INFO [org.ovirt.engine.core.bll.st<http://org.ovirt.engine.core.bll.st/>orage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] ActivateStorage Domain. Before Connect all hosts to pool. Time: Wed May 02 09:16:02 BST 2018 2018-05-02 09:16:02,407+01 INFO [org.ovirt.engine.core.bll.st<http://org.ovirt.engine.core.bll.st/>orage.connection.ConnectStorageToVdsCommand] (EE-ManagedThreadFactory-engine-Thread-33456) [40a82b47] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2018-05-02 09:16:02,421+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33456) [40a82b47] START, ConnectStorageServerVDSCommand(HostName = virtA003, StorageServerConnectionManagementVDSParameters:{hostId='fe2861fc-2b47-4807-b054-470198eda473', storagePoolId='00000000-0000-0000-0000-000000 000000', storageType='NFS', connectionList='[StorageServerConnections:{id='da392861-aedc-4f1e-97f4-6919fb01f1e9', connection='backoffice01.cluster:/vm-iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 23ce648f 2018-05-02 09:16:02,446+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33456) [40a82b47] FINISH, ConnectStorageServerVDSCommand, return: {da392861-aedc-4f1e-97f4-6919fb01f1e9=0}, log id: 23ce648f 2018-05-02 09:16:02,450+01 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] START, ActivateStorageDomainVDSCommand( ActivateStorageDomainVDSCommandParameters:{storagePoolId='5a54bf81-0228-02bc-0358-000000000304', ignoreFailoverLimit='false', stor ageDomainId='f5914df0-f46c-4cc0-b666-c929aa0225ae'}), log id: 5c864594 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a5 4bf81-0228-02bc-0358-000000000304' 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command 'ActivateStorageDomainVDSCommand( ActivateStorageDomainVDSCommandParameters:{storagePoolId='5a54bf81-0228-02bc-0358-000000000304', ignoreFailoverLimit='false', st orageDomainId='f5914df0-f46c-4cc0-b666-c929aa0225ae'})' execution failed: IRSGenericException: IRSErrorException: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' 2018-05-02 09:16:02,635+01 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] FINISH, ActivateStorageDomainVDSCommand, log id: 5c864594 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command 'org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.irsbroker.IrsOperationFailedNoFailove rException: IRSGenericException: IRSErrorException: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' (Failed with error StorageDomainNotInPool and code 353) 2018-05-02 09:16:02,636+01 INFO [org.ovirt.engine.core.bll.st<http://org.ovirt.engine.core.bll.st/>orage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command [id=22b0f3c1-9a09-4e26-8096-d83465c8f4ee]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: EntityStatus Snapshot:{id='StoragePoolIsoMapId:{storagePoolId='5a54bf81-0228-02bc-0358-000000000304', storageId='f5914df0-f46c-4cc0-b666-c929aa0225ae'}', status='Maintenance'}. 2018-05-02 09:16:02,660+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID: USER_ACTIVATE_STORAGE_DOMAIN_FAILED(967), Failed to activate Storage Domain VMISOs (Data Center Default) by admin@internal-authz Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> On 2 May 2018, at 08:44, Fred Rolland <frolland@redhat.com<mailto:frolland@redhat.com>> wrote: Hi, Can you provide logs from engine and Vdsm(SPM)? What is the state now? Thanks, Fred On Tue, May 1, 2018 at 4:11 PM, Callum Smith <callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>> wrote: Dear All, It appears that clicking "detach" on the ISO storage domain is a really bad idea. This has gotten half way through the procedure and now can't be recovered from. Is there any advice for re-attaching the ISO storage domain manually? An NFS mount didn't add it back to the "pool" unfortunately. On a separate note, is it possible to migrate this storage to a new location? And if so how. Regards, Callum -- Callum Smith Research Computing Core Wellcome Trust Centre for Human Genetics University of Oxford e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk> _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users
participants (2)
-
Callum Smith -
Fred Rolland