[ovirt-users] Re-attaching ISOs and moving ISOs storage

Wed May 2 13:46:55 UTC 2018

Can you share the REST API data of the Storage domain and Data Center?
Here an example of the URLs, you will need to replace with correct ids.

http://MY-SERVER/ovirt-engine/api/v4/storagedomains/13461356-f6f7-4a58-9897-2fac61ff40af

http://MY-SERVER/ovirt-engine/api/v4/datacenters/5a5df553-022d-036d-01e8-000000000071/storagedomains

On Wed, May 2, 2018 at 12:53 PM, Callum Smith <callum at well.ox.ac.uk> wrote:

> This is on 4.2.0.2-1, I've linked the main logs to dropbox simply because
> they're big, full of noise right now.
> https://www.dropbox.com/s/f8q3m5amro2a1b2/engine.log?dl=0
> https://www.dropbox.com/s/uods85jk65halo3/vdsm.log?dl=0
>
> Regards,
> Callum
>
> --
>
> Callum Smith
> Research Computing Core
> Wellcome Trust Centre for Human Genetics
> University of Oxford
> e. callum at well.ox.ac.uk
>
> On 2 May 2018, at 10:43, Fred Rolland <frolland at redhat.com> wrote:
>
> Which version are you using?
> Can you provide the whole log?
>
> For some reason, it looks like the Vdsm thinks that the Storage Domain is
> not part of the pool.
>
> On Wed, May 2, 2018 at 11:20 AM, Callum Smith <callum at well.ox.ac.uk>
> wrote:
>
>> State is maintenance for the ISOs storage. I've extracted what is
>> hopefully the relevant bits of the log.
>>
>> VDSM.log (SPM)
>>
>> 2018-05-02 09:16:03,455+0100 INFO  (ioprocess communication (179084))
>> [IOProcess] Starting ioprocess (__init__:447)
>> 2018-05-02 09:16:03,456+0100 INFO  (ioprocess communication (179091))
>> [IOProcess] Starting ioprocess (__init__:447)
>> 2018-05-02 09:16:03,461+0100 INFO  (jsonrpc/0) [vdsm.api] FINISH
>> activateStorageDomain error=Storage domain not in pool:
>> u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae,
>> pool=5a54bf81-0228-02bc-0358-000000000304' from=::ffff:192.168.64.254,58968,
>> flow_id=93433989-8e26-48a9-bd3a-2ab95f296c08,
>> task_id=7f21f911-348f-45a3-b79c-e3cb11642035 (api:50)
>> 2018-05-02 09:16:03,461+0100 ERROR (jsonrpc/0) [storage.TaskManager.Task]
>> (Task='7f21f911-348f-45a3-b79c-e3cb11642035') Unexpected error (task:875)
>> Traceback (most recent call last):
>>   File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line
>> 882, in _run
>>     return fn(*args, **kargs)
>>   File "<string>", line 2, in activateStorageDomain
>>   File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48,
>> in method
>>     ret = func(*args, **kwargs)
>>   File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line
>> 1256, in activateStorageDomain
>>     pool.activateSD(sdUUID)
>>   File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py",
>> line 79, in wrapper
>>     return method(self, *args, **kwargs)
>>   File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1130,
>> in activateSD
>>     self.validateAttachedDomain(dom)
>>   File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py",
>> line 79, in wrapper
>>     return method(self, *args, **kwargs)
>>   File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 557,
>> in validateAttachedDomain
>>     raise se.StorageDomainNotInPool(self.spUUID, dom.sdUUID)
>> StorageDomainNotInPool: Storage domain not in pool:
>> u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae,
>> pool=5a54bf81-0228-02bc-0358-000000000304'
>> 2018-05-02 09:16:03,461+0100 INFO  (jsonrpc/0) [storage.TaskManager.Task]
>> (Task='7f21f911-348f-45a3-b79c-e3cb11642035') aborting: Task is aborted:
>> "Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae,
>> pool=5a54bf81-0228-02bc-0358-000000000304'" - code 353 (task:1181)
>> 2018-05-02 09:16:03,462+0100 ERROR (jsonrpc/0) [storage.Dispatcher]
>> FINISH activateStorageDomain error=Storage domain not in pool:
>> u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae,
>> pool=5a54bf81-0228-02bc-0358-000000000304' (dispatcher:82)
>>
>> engine.log
>>
>> 2018-05-02 09:16:02,326+01 INFO  [org.ovirt.engine.core.bll.st
>> orage.domain.ActivateStorageDomainCommand] (default task-20)
>> [93433989-8e26-48a9-bd3a-2ab95f296c08] Lock Acquired to object
>> 'EngineLock:{exclusiveLocks='[f5914df0-f46c-4cc0-b666-c929aa0225ae=STORAGE]',
>> sharedLocks=''}'
>> 2018-05-02 09:16:02,376+01 INFO  [org.ovirt.engine.core.bll.st
>> orage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455)
>> [93433989-8e26-48a9-bd3a-2ab95f296c08] Running command:
>> ActivateStorageDomainCommand internal: false. Entities affected :  ID:
>> f5914df0-f46c-4cc0-b666-c929aa0225ae Type: StorageAction group
>> MANIPULATE_STORAGE_DOMA
>> IN with role type ADMIN
>> 2018-05-02 09:16:02,385+01 INFO  [org.ovirt.engine.core.bll.st
>> orage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455)
>> [93433989-8e26-48a9-bd3a-2ab95f296c08] Lock freed to object
>> 'EngineLock:{exclusiveLocks='[f5914df0-f46c-4cc0-b666-c929aa0225ae=STORAGE]',
>> sharedLocks=''}'
>> 2018-05-02 09:16:02,385+01 INFO  [org.ovirt.engine.core.bll.st
>> orage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455)
>> [93433989-8e26-48a9-bd3a-2ab95f296c08] ActivateStorage Domain. Before
>> Connect all hosts to pool. Time: Wed May 02 09:16:02 BST 2018
>> 2018-05-02 09:16:02,407+01 INFO  [org.ovirt.engine.core.bll.st
>> orage.connection.ConnectStorageToVdsCommand]
>> (EE-ManagedThreadFactory-engine-Thread-33456) [40a82b47] Running
>> command: ConnectStorageToVdsCommand internal: true. Entities affected :
>>  ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group
>> CREATE_STORAGE_DOMAIN with role type ADMIN
>> 2018-05-02 09:16:02,421+01 INFO  [org.ovirt.engine.core.vdsbro
>> ker.vdsbroker.ConnectStorageServerVDSCommand]
>> (EE-ManagedThreadFactory-engine-Thread-33456) [40a82b47] START,
>> ConnectStorageServerVDSCommand(HostName = virtA003,
>> StorageServerConnectionManagementVDSParameters:{hostId='fe28
>> 61fc-2b47-4807-b054-470198eda473', storagePoolId='00000000-0000-0
>> 000-0000-000000
>> 000000', storageType='NFS', connectionList='[StorageServer
>> Connections:{id='da392861-aedc-4f1e-97f4-6919fb01f1e9',
>> connection='backoffice01.cluster:/vm-iso', iqn='null', vfsType='null',
>> mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null',
>> iface='null', netIfaceName='null'}]'}), log id: 23ce648f
>> 2018-05-02 09:16:02,446+01 INFO  [org.ovirt.engine.core.vdsbro
>> ker.vdsbroker.ConnectStorageServerVDSCommand]
>> (EE-ManagedThreadFactory-engine-Thread-33456) [40a82b47] FINISH,
>> ConnectStorageServerVDSCommand, return: {da392861-aedc-4f1e-97f4-6919fb01f1e9=0},
>> log id: 23ce648f
>> 2018-05-02 09:16:02,450+01 INFO  [org.ovirt.engine.core.vdsbro
>> ker.irsbroker.ActivateStorageDomainVDSCommand]
>> (EE-ManagedThreadFactory-engine-Thread-33455)
>> [93433989-8e26-48a9-bd3a-2ab95f296c08] START,
>> ActivateStorageDomainVDSCommand( ActivateStorageDomainVDSComman
>> dParameters:{storagePoolId='5a54bf81-0228-02bc-0358-000000000304',
>> ignoreFailoverLimit='false', stor
>> ageDomainId='f5914df0-f46c-4cc0-b666-c929aa0225ae'}), log id: 5c864594
>> 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.dal.dbb
>> roker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-33455)
>> [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID:
>> IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command
>> ActivateStorageDomainVDS failed: Storage domain not in pool:
>> u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a5
>> 4bf81-0228-02bc-0358-000000000304'
>> 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.vdsbrok
>> er.irsbroker.ActivateStorageDomainVDSCommand]
>> (EE-ManagedThreadFactory-engine-Thread-33455)
>> [93433989-8e26-48a9-bd3a-2ab95f296c08] Command
>> 'ActivateStorageDomainVDSCommand( ActivateStorageDomainVDSComman
>> dParameters:{storagePoolId='5a54bf81-0228-02bc-0358-000000000304',
>> ignoreFailoverLimit='false', st
>> orageDomainId='f5914df0-f46c-4cc0-b666-c929aa0225ae'})' execution
>> failed: IRSGenericException: IRSErrorException: Storage domain not in pool:
>> u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae,
>> pool=5a54bf81-0228-02bc-0358-000000000304'
>> 2018-05-02 09:16:02,635+01 INFO  [org.ovirt.engine.core.vdsbro
>> ker.irsbroker.ActivateStorageDomainVDSCommand]
>> (EE-ManagedThreadFactory-engine-Thread-33455)
>> [93433989-8e26-48a9-bd3a-2ab95f296c08] FINISH,
>> ActivateStorageDomainVDSCommand, log id: 5c864594
>> 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.bll.sto
>> rage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455)
>> [93433989-8e26-48a9-bd3a-2ab95f296c08] Command
>> 'org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand'
>> failed: EngineException: org.ovirt.engine.core.vdsbroke
>> r.irsbroker.IrsOperationFailedNoFailove
>> rException: IRSGenericException: IRSErrorException: Storage domain not in
>> pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae,
>> pool=5a54bf81-0228-02bc-0358-000000000304' (Failed with error
>> StorageDomainNotInPool and code 353)
>> 2018-05-02 09:16:02,636+01 INFO  [org.ovirt.engine.core.bll.st
>> orage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-33455)
>> [93433989-8e26-48a9-bd3a-2ab95f296c08] Command
>> [id=22b0f3c1-9a09-4e26-8096-d83465c8f4ee]: Compensating
>> CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.b
>> usinessentities.StoragePoolIsoMap; snapshot: EntityStatus
>> Snapshot:{id='StoragePoolIsoMapId:{storagePoolId='5a54bf81-0228-02bc-0358-000000000304',
>> storageId='f5914df0-f46c-4cc0-b666-c929aa0225ae'}',
>> status='Maintenance'}.
>> 2018-05-02 09:16:02,660+01 ERROR [org.ovirt.engine.core.dal.dbb
>> roker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-33455)
>> [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID:
>> USER_ACTIVATE_STORAGE_DOMAIN_FAILED(967), Failed to activate Storage
>> Domain VMISOs (Data Center Default) by admin at internal-authz
>>
>> Regards,
>> Callum
>>
>> --
>>
>> Callum Smith
>> Research Computing Core
>> Wellcome Trust Centre for Human Genetics
>> University of Oxford
>> e. callum at well.ox.ac.uk
>>
>> On 2 May 2018, at 08:44, Fred Rolland <frolland at redhat.com> wrote:
>>
>> Hi,
>>
>> Can you provide logs from engine and Vdsm(SPM)?
>> What is the state now?
>>
>> Thanks,
>> Fred
>>
>> On Tue, May 1, 2018 at 4:11 PM, Callum Smith <callum at well.ox.ac.uk>
>> wrote:
>>
>>> Dear All,
>>>
>>> It appears that clicking "detach" on the ISO storage domain is a really
>>> bad idea. This has gotten half way through the procedure and now can't be
>>> recovered from. Is there any advice for re-attaching the ISO storage domain
>>> manually? An NFS mount didn't add it back to the "pool" unfortunately.
>>>
>>> On a separate note, is it possible to migrate this storage to a new
>>> location? And if so how.
>>>
>>> Regards,
>>> Callum
>>>
>>> --
>>>
>>> Callum Smith
>>> Research Computing Core
>>> Wellcome Trust Centre for Human Genetics
>>> University of Oxford
>>> e. callum at well.ox.ac.uk
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20180502/070c7b6f/attachment.html>