[ovirt-users] Re-attaching ISOs and moving ISOs storage

Callum Smith callum at well.ox.ac.uk
Wed May 2 09:53:15 UTC 2018 

This is on 4.2.0.2-1, I've linked the main logs to dropbox simply because they're big, full of noise right now.
https://www.dropbox.com/s/f8q3m5amro2a1b2/engine.log?dl=0
https://www.dropbox.com/s/uods85jk65halo3/vdsm.log?dl=0

Regards,
Callum

--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum at well.ox.ac.uk<mailto:callum at well.ox.ac.uk>

On 2 May 2018, at 10:43, Fred Rolland <frolland at redhat.com<mailto:frolland at redhat.com>> wrote:

Which version are you using?
Can you provide the whole log?

For some reason, it looks like the Vdsm thinks that the Storage Domain is not part of the pool.

On Wed, May 2, 2018 at 11:20 AM, Callum Smith <callum at well.ox.ac.uk<mailto:callum at well.ox.ac.uk>> wrote:
State is maintenance for the ISOs storage. I've extracted what is hopefully the relevant bits of the log.

VDSM.log (SPM)

2018-05-02 09:16:03,455+0100 INFO  (ioprocess communication (179084)) [IOProcess] Starting ioprocess (__init__:447)
2018-05-02 09:16:03,456+0100 INFO  (ioprocess communication (179091)) [IOProcess] Starting ioprocess (__init__:447)
2018-05-02 09:16:03,461+0100 INFO  (jsonrpc/0) [vdsm.api] FINISH activateStorageDomain error=Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' from=::ffff:192.168.64.254,58968, flow_id=93433989-8e26-48a9-bd3a-2ab95f296c08, task_id=7f21f911-348f-45a3-b79c-e3cb11642035 (api:50)
2018-05-02 09:16:03,461+0100 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='7f21f911-348f-45a3-b79c-e3cb11642035') Unexpected error (task:875)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run
    return fn(*args, **kargs)
  File "<string>", line 2, in activateStorageDomain
  File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method
    ret = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1256, in activateStorageDomain
    pool.activateSD(sdUUID)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1130, in activateSD
    self.validateAttachedDomain(dom)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 557, in validateAttachedDomain
    raise se.StorageDomainNotInPool(self.spUUID, dom.sdUUID)
StorageDomainNotInPool: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304'
2018-05-02 09:16:03,461+0100 INFO  (jsonrpc/0) [storage.TaskManager.Task] (Task='7f21f911-348f-45a3-b79c-e3cb11642035') aborting: Task is aborted: "Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304'" - code 353 (task:1181)
2018-05-02 09:16:03,462+0100 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH activateStorageDomain error=Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' (dispatcher:82)

engine.log

2018-05-02 09:16:02,326+01 INFO  [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (default task-20) [93433989-8e26-48a9-bd3a-2ab95f296c08] Lock Acquired to object 'EngineLock:{exclusiveLocks='[f5914df0-f46c-4cc0-b666-c929aa0225ae=STORAGE]', sharedLocks=''}'
2018-05-02 09:16:02,376+01 INFO  [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Running command: ActivateStorageDomainCommand internal: false. Entities affected :  ID: f5914df0-f46c-4cc0-b666-c929aa0225ae Type: StorageAction group MANIPULATE_STORAGE_DOMA
IN with role type ADMIN
2018-05-02 09:16:02,385+01 INFO  [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Lock freed to object 'EngineLock:{exclusiveLocks='[f5914df0-f46c-4cc0-b666-c929aa0225ae=STORAGE]', sharedLocks=''}'
2018-05-02 09:16:02,385+01 INFO  [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] ActivateStorage Domain. Before Connect all hosts to pool. Time: Wed May 02 09:16:02 BST 2018
2018-05-02 09:16:02,407+01 INFO  [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (EE-ManagedThreadFactory-engine-Thread-33456) [40a82b47] Running command: ConnectStorageToVdsCommand internal: true. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN
2018-05-02 09:16:02,421+01 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33456) [40a82b47] START, ConnectStorageServerVDSCommand(HostName = virtA003, StorageServerConnectionManagementVDSParameters:{hostId='fe2861fc-2b47-4807-b054-470198eda473', storagePoolId='00000000-0000-0000-0000-000000
000000', storageType='NFS', connectionList='[StorageServerConnections:{id='da392861-aedc-4f1e-97f4-6919fb01f1e9', connection='backoffice01.cluster:/vm-iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 23ce648f
2018-05-02 09:16:02,446+01 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33456) [40a82b47] FINISH, ConnectStorageServerVDSCommand, return: {da392861-aedc-4f1e-97f4-6919fb01f1e9=0}, log id: 23ce648f
2018-05-02 09:16:02,450+01 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] START, ActivateStorageDomainVDSCommand( ActivateStorageDomainVDSCommandParameters:{storagePoolId='5a54bf81-0228-02bc-0358-000000000304', ignoreFailoverLimit='false', stor
ageDomainId='f5914df0-f46c-4cc0-b666-c929aa0225ae'}), log id: 5c864594
2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a5
4bf81-0228-02bc-0358-000000000304'
2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command 'ActivateStorageDomainVDSCommand( ActivateStorageDomainVDSCommandParameters:{storagePoolId='5a54bf81-0228-02bc-0358-000000000304', ignoreFailoverLimit='false', st
orageDomainId='f5914df0-f46c-4cc0-b666-c929aa0225ae'})' execution failed: IRSGenericException: IRSErrorException: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304'
2018-05-02 09:16:02,635+01 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] FINISH, ActivateStorageDomainVDSCommand, log id: 5c864594
2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command 'org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.irsbroker.IrsOperationFailedNoFailove
rException: IRSGenericException: IRSErrorException: Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a54bf81-0228-02bc-0358-000000000304' (Failed with error StorageDomainNotInPool and code 353)
2018-05-02 09:16:02,636+01 INFO  [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] Command [id=22b0f3c1-9a09-4e26-8096-d83465c8f4ee]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: EntityStatus
Snapshot:{id='StoragePoolIsoMapId:{storagePoolId='5a54bf81-0228-02bc-0358-000000000304', storageId='f5914df0-f46c-4cc0-b666-c929aa0225ae'}', status='Maintenance'}.
2018-05-02 09:16:02,660+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-33455) [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID: USER_ACTIVATE_STORAGE_DOMAIN_FAILED(967), Failed to activate Storage Domain VMISOs (Data Center Default) by admin at internal-authz

Regards,
Callum

--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum at well.ox.ac.uk<mailto:callum at well.ox.ac.uk>

On 2 May 2018, at 08:44, Fred Rolland <frolland at redhat.com<mailto:frolland at redhat.com>> wrote:

Hi,

Can you provide logs from engine and Vdsm(SPM)?
What is the state now?

Thanks,
Fred

On Tue, May 1, 2018 at 4:11 PM, Callum Smith <callum at well.ox.ac.uk<mailto:callum at well.ox.ac.uk>> wrote:
Dear All,

It appears that clicking "detach" on the ISO storage domain is a really bad idea. This has gotten half way through the procedure and now can't be recovered from. Is there any advice for re-attaching the ISO storage domain manually? An NFS mount didn't add it back to the "pool" unfortunately.

On a separate note, is it possible to migrate this storage to a new location? And if so how.

Regards,
Callum

--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum at well.ox.ac.uk<mailto:callum at well.ox.ac.uk>


_______________________________________________
Users mailing list
Users at ovirt.org<mailto:Users at ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20180502/f4f0dca4/attachment.html>

More information about the Users mailing list