[ovirt-users] Re-attaching ISOs and moving ISOs storage

Wed May 2 08:20:20 UTC 2018

State is maintenance for the ISOs storage. I've extracted what is hopefully the relevant bits of the log.

VDSM.log (SPM)

2018-05-02 09:16:03,455+0100 INFO  (ioprocess communication (179084)) [IOProcess] Starting ioprocess (__init__:447)
2018-05-02 09:16:03,456+0100 INFO  (ioprocess communication (179091)) [IOProcess] Starting ioprocess (__init__:447)
2018-05-02 09:16:03,461+0100 INFO  (jsonrpc/0) [vdsm.api] FINISH activateStorageDomain error=Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' from=::ffff:192.168.64.254,58968, flow_id=93433989-8e26-48a9-bd3a-2ab95f296c08, task_id=7f21f911-348f-45a3-b79c-e3cb11642035 (api:50)
2018-05-02 09:16:03,461+0100 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='7f21f911-348f-45a3-b79c-e3cb11642035') Unexpected error (task:875)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run
    return fn(*args, **kargs)
  File "<string>", line 2, in activateStorageDomain
  File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method
    ret = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1256, in activateStorageDomain
    pool.activateSD(sdUUID)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1130, in activateSD
    self.validateAttachedDomain(dom)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 557, in validateAttachedDomain
    raise se.StorageDomainNotInPool(self.spUUID, dom.sdUUID)
StorageDomainNotInPool: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304'
2018-05-02 09:16:03,461+0100 INFO  (jsonrpc/0) [storage.TaskManager.Task] (Task='7f21f911-348f-45a3-b79c-e3cb11642035') aborting: Task is aborted: "Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304'" - code 353 (task:1181)
2018-05-02 09:16:03,462+0100 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH activateStorageDomain error=Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' (dispatcher:82)

engine.log

2018-05-02 09:16:02,326+01 INFO  [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (default task-20) [93433989-8e26-48a9-bd3a-2ab95f296c08] Lock Acquired to object 'EngineLock:{exclusiveLocks='[f5914df0-f46c-4cc0-b666-c929aa0225ae=STORAGE]', sharedLocks=''}'
2018-05-02 09:16:02,376+01 INFO  [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Running command: ActivateStorageDomainCommand internal: false. Entities affected :  ID: f5914df0-f46c-4cc0-b666-c929aa0225ae Type: StorageAction group MANIPULATE_STORAGE_DOMA
IN with role type ADMIN
2018-05-02 09:16:02,385+01 INFO  [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Lock freed to object 'EngineLock:{exclusiveLocks='[f5914df0-f46c-4cc0-b666-c929aa0225ae=STORAGE]', sharedLocks=''}'
2018-05-02 09:16:02,385+01 INFO  [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] ActivateStorage Domain. Before Connect all hosts to pool. Time: Wed May 02 09:16:02 BST 2018
2018-05-02 09:16:02,407+01 INFO  [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (EE-ManagedThreadFactory-engine-Thread-33456) [40a82b47] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN
2018-05-02 09:16:02,421+01 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33456) [40a82b47] START, ConnectStorageServerVDSCommand(HostName = virtA003, StorageServerConnectionManagementVDSParameters:{hostId='fe2861fc-2b47-4807-b054-470198eda473', storagePoolId='00000000-0000-0000-0000-000000
000000', storageType='NFS', connectionList='[StorageServerConnections:{id='da392861-aedc-4f1e-97f4-6919fb01f1e9', connection='backoffice01.cluster:/vm-iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 23ce648f
2018-05-02 09:16:02,446+01 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33456) [40a82b47] FINISH, ConnectStorageServerVDSCommand, return: {da392861-aedc-4f1e-97f4-6919fb01f1e9=0}, log id: 23ce648f
2018-05-02 09:16:02,450+01 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] START, ActivateStorageDomainVDSCommand( ActivateStorageDomainVDSCommandParameters:{storagePoolId='5a54bf81-0228-02bc-0358-000000000304', ignoreFailoverLimit='false', stor
ageDomainId='f5914df0-f46c-4cc0-b666-c929aa0225ae'}), log id: 5c864594
2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a5
4bf81-0228-02bc-0358-000000000304'
2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command 'ActivateStorageDomainVDSCommand( ActivateStorageDomainVDSCommandParameters:{storagePoolId='5a54bf81-0228-02bc-0358-000000000304', ignoreFailoverLimit='false', st
orageDomainId='f5914df0-f46c-4cc0-b666-c929aa0225ae'})' execution failed: IRSGenericException: IRSErrorException: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304'
2018-05-02 09:16:02,635+01 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] FINISH, ActivateStorageDomainVDSCommand, log id: 5c864594
2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command 'org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.irsbroker.IrsOperationFailedNoFailove
rException: IRSGenericException: IRSErrorException: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' (Failed with error StorageDomainNotInPool and code 353)
2018-05-02 09:16:02,636+01 INFO  [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command [id=22b0f3c1-9a09-4e26-8096-d83465c8f4ee]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: EntityStatus
Snapshot:{id='StoragePoolIsoMapId:{storagePoolId='5a54bf81-0228-02bc-0358-000000000304', storageId='f5914df0-f46c-4cc0-b666-c929aa0225ae'}', status='Maintenance'}.
2018-05-02 09:16:02,660+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID: USER_ACTIVATE_STORAGE_DOMAIN_FAILED(967), Failed to activate Storage Domain VMISOs (Data Center Default) by admin at internal-authz

Regards,
Callum

--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum at well.ox.ac.uk<mailto:callum at well.ox.ac.uk>

On 2 May 2018, at 08:44, Fred Rolland <frolland at redhat.com<mailto:frolland at redhat.com>> wrote:

Hi,

Can you provide logs from engine and Vdsm(SPM)?
What is the state now?

Thanks,
Fred

On Tue, May 1, 2018 at 4:11 PM, Callum Smith <callum at well.ox.ac.uk<mailto:callum at well.ox.ac.uk>> wrote:
Dear All,

It appears that clicking "detach" on the ISO storage domain is a really bad idea. This has gotten half way through the procedure and now can't be recovered from. Is there any advice for re-attaching the ISO storage domain manually? An NFS mount didn't add it back to the "pool" unfortunately.

On a separate note, is it possible to migrate this storage to a new location? And if so how.

Regards,
Callum

--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum at well.ox.ac.uk<mailto:callum at well.ox.ac.uk>

_______________________________________________
Users mailing list
Users at ovirt.org<mailto:Users at ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20180502/4ed68bbb/attachment.html>