Is there any sensible way to either clean up the existing ISOs storage or re-attach it? I'm struggling to even export VMs and migrate them elsewhere with this and need to recover them asap.
Regards,Callum
--
Callum SmithResearch Computing Core
On 2 May 2018, at 15:09, Callum Smith <callum@well.ox.ac.uk> wrote:
Attached, thank you for looking into this
https://HOSTNAME/ovirt-engine/
api/v4/storagedomains/ f5914df0-f46c-4cc0-b666- c929aa0225ae
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <storage_domain href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c- " id="f5914df0-f46c-4cc0-b666-4cc0-b666-c929aa0225ae c929aa0225ae "> <actions> <link href="/ovirt-engine/api/v4/storagedomains/f5914df0-f46c- " rel="isattached"/> <link href="/ovirt-engine/api/v4/4cc0-b666-c929aa0225ae/ isattached storagedomains/f5914df0-f46c- " rel="reduceluns"/> <link href="/ovirt-engine/api/v4/4cc0-b666-c929aa0225ae/ reduceluns storagedomains/f5914df0-f46c- " rel="refreshluns"/> <link href="/ovirt-engine/api/v4/4cc0-b666-c929aa0225ae/ refreshluns storagedomains/f5914df0-f46c- " rel="updateovfstore"/> </actions> <name>VMISOs</name> <link href="/ovirt-engine/api/v4/4cc0-b666-c929aa0225ae/ updateovfstore storagedomains/f5914df0-f46c- " rel="disksnapshots"/> <link href="/ovirt-engine/api/v4/4cc0-b666-c929aa0225ae/ disksnapshots storagedomains/f5914df0-f46c- " rel="files"/> <link href="/ovirt-engine/api/v4/4cc0-b666-c929aa0225ae/files storagedomains/f5914df0-f46c- " rel="diskprofiles"/> <link href="/ovirt-engine/api/v4/4cc0-b666-c929aa0225ae/ diskprofiles storagedomains/f5914df0-f46c- " rel="storageconnections"/> <link href="/ovirt-engine/api/v4/4cc0-b666-c929aa0225ae/ storageconnections storagedomains/f5914df0-f46c- " rel="permissions"/> <available>11770357874688</ava4cc0-b666-c929aa0225ae/ permissions ilable > <backup>false</backup> <committed>0</committed> <critical_space_action_blocker>5</critical_space_action_ blocker > <discard_after_delete>false</discard_after_delete > <external_status>ok</external_status > <master>false</master> <storage> <address>backoffice01.cluster</address> <path>/vm-iso</path> <type>nfs</type> </storage> <storage_format>v1</storage_ format > <supports_discard>false</supports_discard > <supports_discard_zeroes_data>false</supports_discard_ zeroes_data > <type>iso</type> <used>38654705664</used> <warning_low_space_indicator>10</warning_low_space_ indicator > <wipe_after_delete>false</wipe_after_delete > <data_centers> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228- " id="5a54bf81-0228-02bc-0358-02bc-0358-000000000304 000000000304 "/> </data_centers> </storage_domain>https://HOSTNAME/ovirt-engine/api/v4/datacenters/5a54bf81- 0228-02bc-0358-000000000304/ storagedomains <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <storage_domains> <storage_domain href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228- " id="d9f8093b-71cd-423d-9b7d-02bc-0358-000000000304/ storagedomains/d9f8093b-71cd- 423d-9b7d-91a613ef98ba 91a613ef98ba "> <actions> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228- " rel="activate"/> <link href="/ovirt-engine/api/v4/02bc-0358-000000000304/ storagedomains/d9f8093b-71cd- 423d-9b7d-91a613ef98ba/ activate datacenters/5a54bf81-0228- " rel="deactivate"/> </actions> <name>tegile-virtman-backup</n02bc-0358-000000000304/ storagedomains/d9f8093b-71cd- 423d-9b7d-91a613ef98ba/ deactivate ame > <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228- " rel="disks"/> <available>17519171600384</ava02bc-0358-000000000304/ storagedomains/d9f8093b-71cd- 423d-9b7d-91a613ef98ba/disks ilable > <backup>false</backup> <committed>0</committed> <critical_space_action_blocker>5</critical_space_action_ blocker > <discard_after_delete>false</discard_after_delete > <external_status>ok</external_status > <master>false</master> <status>maintenance</status> <storage> <address>192.168.64.248</address > <nfs_version>auto</nfs_version> <path>/export/virtman/backup</ path > <type>nfs</type> </storage> <storage_format>v1</storage_format > <supports_discard>false</supports_discard > <supports_discard_zeroes_data>false</supports_discard_ zeroes_data > <type>export</type> <used>8589934592</used> <warning_low_space_indicator>10</warning_low_space_ indicator > <wipe_after_delete>false</wipe_after_delete > <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228- " id="5a54bf81-0228-02bc-0358-02bc-0358-000000000304 000000000304 "/> <data_centers> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228- " id="5a54bf81-0228-02bc-0358-02bc-0358-000000000304 000000000304 "/> </data_centers> </storage_domain> <storage_domain href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228- " id="017dd620-65ec-4fa4-8d46-02bc-0358-000000000304/ storagedomains/017dd620-65ec- 4fa4-8d46-67737dedd7a1 67737dedd7a1 "> <actions> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228- " rel="activate"/> <link href="/ovirt-engine/api/v4/02bc-0358-000000000304/ storagedomains/017dd620-65ec- 4fa4-8d46-67737dedd7a1/ activate datacenters/5a54bf81-0228- " rel="deactivate"/> </actions> <name>VMStorage</name> <link href="/ovirt-engine/api/v4/02bc-0358-000000000304/ storagedomains/017dd620-65ec- 4fa4-8d46-67737dedd7a1/ deactivate datacenters/5a54bf81-0228- " rel="disks"/> <available>11770357874688</ava02bc-0358-000000000304/ storagedomains/017dd620-65ec- 4fa4-8d46-67737dedd7a1/disks ilable > <backup>false</backup> <committed>118111600640</committed > <critical_space_action_blocker>5</critical_space_action_ blocker > <discard_after_delete>false</discard_after_delete > <external_status>ok</external_status > <master>true</master> <status>active</status> <storage> <address>backoffice01.cluster</address> <path>/vm-storage2</path> <type>nfs</type> </storage> <storage_format>v4</storage_ format > <supports_discard>false</supports_discard > <supports_discard_zeroes_data>false</supports_discard_ zeroes_data > <type>data</type> <used>38654705664</used> <warning_low_space_indicator>10</warning_low_space_ indicator > <wipe_after_delete>false</wipe_after_delete > <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228- " id="5a54bf81-0228-02bc-0358-02bc-0358-000000000304 000000000304 "/> <data_centers> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228- " id="5a54bf81-0228-02bc-0358-02bc-0358-000000000304 000000000304 "/> </data_centers> </storage_domain> <storage_domain href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228- " id="e572c874-17f8-4ad8-9bac-02bc-0358-000000000304/ storagedomains/e572c874-17f8- 4ad8-9bac-b6f563a9fc17 b6f563a9fc17 "> <actions> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228- " rel="activate"/> <link href="/ovirt-engine/api/v4/02bc-0358-000000000304/ storagedomains/e572c874-17f8- 4ad8-9bac-b6f563a9fc17/ activate datacenters/5a54bf81-0228- " rel="deactivate"/> </actions> <name>tegile-virtman</name> <link href="/ovirt-engine/api/v4/02bc-0358-000000000304/ storagedomains/e572c874-17f8- 4ad8-9bac-b6f563a9fc17/ deactivate datacenters/5a54bf81-0228- " rel="disks"/> <available>2190433320960</avai02bc-0358-000000000304/ storagedomains/e572c874-17f8- 4ad8-9bac-b6f563a9fc17/disks lable > <backup>false</backup> <committed>226559524864</committed > <critical_space_action_blocker>5</critical_space_action_ blocker > <discard_after_delete>false</discard_after_delete > <external_status>ok</external_status > <master>false</master> <status>active</status> <storage> <address>192.168.64.248</address > <nfs_version>auto</nfs_version> <path>/export/virtman/ VirtualServerShare_1</path> <type>nfs</type> </storage> <storage_format>v4</storage_ format > <supports_discard>false</supports_discard > <supports_discard_zeroes_data>false</supports_discard_ zeroes_data > <type>data</type> <used>8589934592</used> <warning_low_space_indicator>10</warning_low_space_ indicator > <wipe_after_delete>false</wipe_after_delete > <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228- " id="5a54bf81-0228-02bc-0358-02bc-0358-000000000304 000000000304 "/> <data_centers> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228- " id="5a54bf81-0228-02bc-0358-02bc-0358-000000000304 000000000304 "/> </data_centers> </storage_domain> <storage_domain href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228- " id="f5914df0-f46c-4cc0-b666-02bc-0358-000000000304/ storagedomains/f5914df0-f46c- 4cc0-b666-c929aa0225ae c929aa0225ae "> <actions> <link href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228- " rel="activate"/> <link href="/ovirt-engine/api/v4/02bc-0358-000000000304/ storagedomains/f5914df0-f46c- 4cc0-b666-c929aa0225ae/ activate datacenters/5a54bf81-0228- " rel="deactivate"/> </actions> <name>VMISOs</name> <available>11770357874688</ava02bc-0358-000000000304/ storagedomains/f5914df0-f46c- 4cc0-b666-c929aa0225ae/ deactivate ilable > <backup>false</backup> <committed>0</committed> <critical_space_action_blocker>5</critical_space_action_ blocker > <discard_after_delete>false</discard_after_delete > <external_status>ok</external_status > <master>false</master> <status>maintenance</status> <storage> <address>backoffice01.cluster</address> <path>/vm-iso</path> <type>nfs</type> </storage> <storage_format>v1</storage_ format > <supports_discard>false</supports_discard > <supports_discard_zeroes_data>false</supports_discard_ zeroes_data > <type>iso</type> <used>38654705664</used> <warning_low_space_indicator>10</warning_low_space_ indicator > <wipe_after_delete>false</wipe_after_delete > <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228- " id="5a54bf81-0228-02bc-0358-02bc-0358-000000000304 000000000304 "/> <data_centers> <data_center href="/ovirt-engine/api/v4/datacenters/5a54bf81-0228- " id="5a54bf81-0228-02bc-0358-02bc-0358-000000000304 000000000304 "/> </data_centers> </storage_domain> </storage_domains><Screen Shot 2018-05-02 at 14.54.42.png>Regards,Callum
--
Callum SmithResearch Computing Core
On 2 May 2018, at 14:46, Fred Rolland <frolland@redhat.com> wrote:
Can you share the REST API data of the Storage domain and Data Center?Here an example of the URLs, you will need to replace with correct ids.
On Wed, May 2, 2018 at 12:53 PM, Callum Smith <callum@well.ox.ac.uk> wrote:
This is on 4.2.0.2-1, I've linked the main logs to dropbox simply because they're big, full of noise right now.
Regards,Callum
--
Callum SmithResearch Computing Core
On 2 May 2018, at 10:43, Fred Rolland <frolland@redhat.com> wrote:
Which version are you using?For some reason, it looks like the Vdsm thinks that the Storage Domain is not part of the pool.
Can you provide the whole log?
On Wed, May 2, 2018 at 11:20 AM, Callum Smith <callum@well.ox.ac.uk> wrote:
State is maintenance for the ISOs storage. I've extracted what is hopefully the relevant bits of the log.
VDSM.log (SPM)
2018-05-02 09:16:03,455+0100 INFO (ioprocess communication (179084)) [IOProcess] Starting ioprocess (__init__:447)2018-05-02 09:16:03,456+0100 INFO (ioprocess communication (179091)) [IOProcess] Starting ioprocess (__init__:447)2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [vdsm.api] FINISH activateStorageDomain error=Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-0 00000000304' from=::ffff:192.168.64.254,589 68, flow_id=93433989-8e26-48a9-bd3 a-2ab95f296c08, task_id=7f21f911-348f-45a3-b79 c-e3cb11642035 (api:50) 2018-05-02 09:16:03,461+0100 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='7f21f911-348f-45a3-b79c-e3cb11642035') Unexpected error (task:875) Traceback (most recent call last):File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs)File "<string>", line 2, in activateStorageDomainFile "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs)File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1256, in activateStorageDomain pool.activateSD(sdUUID)File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py" , line 79, in wrapper return method(self, *args, **kwargs)File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1130, in activateSD self.validateAttachedDomain(dom) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py" , line 79, in wrapper return method(self, *args, **kwargs)File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 557, in validateAttachedDomain raise se.StorageDomainNotInPool(self.spUUID, dom.sdUUID) StorageDomainNotInPool: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-0 00000000304' 2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [storage.TaskManager.Task] (Task='7f21f911-348f-45a3-b79c-e3cb11642035') aborting: Task is aborted: "Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b6 66-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-0 00000000304'" - code 353 (task:1181) 2018-05-02 09:16:03,462+0100 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH activateStorageDomain error=Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-0 00000000304' (dispatcher:82)
engine.log
2018-05-02 09:16:02,326+01 INFO [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDo mainCommand] (default task-20) [93433989-8e26-48a9-bd3a-2ab95 f296c08] Lock Acquired to object 'EngineLock:{exclusiveLocks='[ f5914df0-f46c-4cc0-b666-c929aa 0225ae=STORAGE]', sharedLocks=''}' 2018-05-02 09:16:02,376+01 INFO [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDo mainCommand] (EE-ManagedThreadFactory-engin e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95 f296c08] Running command: ActivateStorageDomainCommand internal: false. Entities affected : ID: f5914df0-f46c-4cc0-b666-c929aa 0225ae Type: StorageAction group MANIPULATE_STORAGE_DOMA IN with role type ADMIN2018-05-02 09:16:02,385+01 INFO [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDo mainCommand] (EE-ManagedThreadFactory-engin e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95 f296c08] Lock freed to object 'EngineLock:{exclusiveLocks='[ f5914df0-f46c-4cc0-b666-c929aa 0225ae=STORAGE]', sharedLocks=''}' 2018-05-02 09:16:02,385+01 INFO [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDo mainCommand] (EE-ManagedThreadFactory-engin e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95 f296c08] ActivateStorage Domain. Before Connect all hosts to pool. Time: Wed May 02 09:16:02 BST 2018 2018-05-02 09:16:02,407+01 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorag eToVdsCommand] (EE-ManagedThreadFactory-engin e-Thread-33456) [40a82b47] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456 789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2018-05-02 09:16:02,421+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageSe rverVDSCommand] (EE-ManagedThreadFactory-engin e-Thread-33456) [40a82b47] START, ConnectStorageServerVDSCommand (HostName = virtA003, StorageServerConnectionManagem entVDSParameters:{hostId='fe28 61fc-2b47-4807-b054-470198eda4 73', storagePoolId='00000000-0000-0 000-0000-000000 000000', storageType='NFS', connectionList='[StorageServerConnections:{id='da392861-aedc -4f1e-97f4-6919fb01f1e9', connection='backoffice01.clust er:/vm-iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 23ce648f 2018-05-02 09:16:02,446+01 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageSe rverVDSCommand] (EE-ManagedThreadFactory-engin e-Thread-33456) [40a82b47] FINISH, ConnectStorageServerVDSCommand , return: {da392861-aedc-4f1e-97f4-6919f b01f1e9=0}, log id: 23ce648f 2018-05-02 09:16:02,450+01 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageD omainVDSCommand] (EE-ManagedThreadFactory-engin e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95 f296c08] START, ActivateStorageDomainVDSComman d( ActivateStorageDomainVDSComman dParameters:{storagePoolId='5a 54bf81-0228-02bc-0358-00000000 0304', ignoreFailoverLimit='false', stor ageDomainId='f5914df0-f46c-4cc0-b666-c929aa0225ae'}), log id: 5c864594 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLo gDirector] (EE-ManagedThreadFactory-engin e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95 f296c08] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10, 803), VDSM command ActivateStorageDomainVDS failed: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b6 66-c929aa0225ae, pool=5a5 4bf81-0228-02bc-0358-000000000304' 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDo mainVDSCommand] (EE-ManagedThreadFactory-engin e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95 f296c08] Command 'ActivateStorageDomainVDSComma nd( ActivateStorageDomainVDSComman dParameters:{storagePoolId='5a 54bf81-0228-02bc-0358-00000000 0304', ignoreFailoverLimit='false', st orageDomainId='f5914df0-f46c-4cc0-b666-c929aa0225ae'})' execution failed: IRSGenericException: IRSErrorException: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b6 66-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-0 00000000304' 2018-05-02 09:16:02,635+01 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageD omainVDSCommand] (EE-ManagedThreadFactory-engin e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95 f296c08] FINISH, ActivateStorageDomainVDSComman d, log id: 5c864594 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDom ainCommand] (EE-ManagedThreadFactory-engin e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95 f296c08] Command 'org.ovirt.engine.core.bll.sto rage.domain.ActivateStorageDom ainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroke r.irsbroker.IrsOperationFailed NoFailove rException: IRSGenericException: IRSErrorException: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-0 00000000304' (Failed with error StorageDomainNotInPool and code 353) 2018-05-02 09:16:02,636+01 INFO [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDo mainCommand] (EE-ManagedThreadFactory-engin e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95 f296c08] Command [id=22b0f3c1-9a09-4e26-8096-d8 3465c8f4ee]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.b usinessentities.StoragePoolIso Map; snapshot: EntityStatus Snapshot:{id='StoragePoolIsoMapId:{storagePoolId='5a54bf81-0 228-02bc-0358-000000000304', storageId='f5914df0-f46c-4cc0- b666-c929aa0225ae'}', status='Maintenance'}. 2018-05-02 09:16:02,660+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLo gDirector] (EE-ManagedThreadFactory-engin e-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95 f296c08] EVENT_ID: USER_ACTIVATE_STORAGE_DOMAIN_F AILED(967), Failed to activate Storage Domain VMISOs (Data Center Default) by admin@internal-authz
Regards,Callum
--
Callum SmithResearch Computing Core
On 2 May 2018, at 08:44, Fred Rolland <frolland@redhat.com> wrote:
FredThanks,What is the state now?Hi,Can you provide logs from engine and Vdsm(SPM)?
On Tue, May 1, 2018 at 4:11 PM, Callum Smith <callum@well.ox.ac.uk> wrote:
Dear All,
It appears that clicking "detach" on the ISO storage domain is a really bad idea. This has gotten half way through the procedure and now can't be recovered from. Is there any advice for re-attaching the ISO storage domain manually? An NFS mount didn't add it back to the "pool" unfortunately.
On a separate note, is it possible to migrate this storage to a new location? And if so how.
Regards,Callum
--
Callum SmithResearch Computing Core
_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users