6:51 a.m.
The issue was found.
We are not looking for a directory with the exact name of the Storage Domain UUID, but we
look for ALL directories that contain the Storage Domain UUID.
For example:
[root@ovirt3 gluster1:_data__fast4]# ll
общо 1
drwxrwxr-x. 5 vdsm kvm 48 10 ное 13,42 578bca3d-6540-41cd-8e0e-9e3047026484
drwxrwxr-x. 5 vdsm kvm 48 10 ное 13,42 578bca3d-6540-41cd-8e0e-9e3047026484-NEW
[root@ovirt3 gluster1:_data__fast]# ll
общо 1
drwxrwxrwx. 6 vdsm kvm 59 11 апр 6,30 396604d9-2a9e-49cd-9563-fdc79981f67b
drwxr-xr-x. 5 vdsm kvm 48 19 ное 21,32 396604d9-2a9e-49cd-9563-fdc79981f67b-OLD
As 578bca3d-6540-41cd-8e0e-9e3047026484-NEW & 396604d9-2a9e-49cd-9563-fdc79981f67b-OLD
are not valid SDuuids - the activation fails.
I have opened a bug for it (not sure on the correct Product/Component) :
https://bugzilla.redhat.com/show_bug.cgi?id=1823033
Hey Sandro,
is this expected, as I never saw it as a requirement ?
If it is expected (already in the docs), please mark 1823033 "notabug".
Best Regards,
Strahil Nikolov
В вторник, 7 април 2020 г., 23:13:43 Гринуич+3, Strahil Nikolov
<hunter86_bg(a)yahoo.com> написа:
Hi Shani,
thanks for the asitance.
First atempt to activate storage domain "data_fast" (for today) which is ontop
gluster volume "data_fast":
So Far I see the following when I filter "INFO" on the engine:
2020-04-07 22:44:55,760+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [7de0d120-fde0-4471-9032-f10e69d2e54
0] Failed in 'ActivateStorageDomainVDS' method
2020-04-07 22:44:55,781+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100115) [7de0d120-fde0-4471-9032-f10e69d2e540] EV
ENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command ActivateStorageDomainVDS failed:
Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:44:55,781+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [7de0d120-fde0-4471-9032-f10e69d2e540]
IrsBroker::F
ailed::ActivateStorageDomainVDS: IRSGenericException: IRSErrorException: Failed to
ActivateStorageDomainVDS, error = Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',),
code = 358
2020-04-07 22:44:56,223+03 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100115) [174bb3e2] EVENT_ID: SYSTEM_CHANGE_STORAG
E_POOL_STATUS_PROBLEMATIC_SEARCHING_NEW_SPM(986), Data Center is being initialized, please
wait for initialization to complete.
2020-04-07 22:44:58,086+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [174bb3e2] Failed in 'ActivateStorag
eDomainVDS' method
2020-04-07 22:44:58,106+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100115) [174bb3e2] EVENT_ID: IRS_BROKER_COMMAND_F
AILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not
exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:44:58,106+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [174bb3e2]
IrsBroker::Failed::ActivateStorageDomain
VDS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error =
Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code =
358
2020-04-07 22:44:58,345+03 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] EVENT_ID: SYSTEM_CHANGE_STORAGE
_POOL_STATUS_PROBLEMATIC_SEARCHING_NEW_SPM(986), Data Center is being initialized, please
wait for initialization to complete.
2020-04-07 22:45:00,222+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] Failed in 'ActivateStorage
DomainVDS' method
2020-04-07 22:45:00,240+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] EVENT_ID: IRS_BROKER_COMMAND_FA
ILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not
exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:45:00,241+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6]
IrsBroker::Failed::ActivateStorageDomainV
DS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error =
Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code =
358
2020-04-07 22:45:00,241+03 ERROR
[org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] Command
'org.ovirt.engine.core
.bll.storage.domain.ActivateStorageDomainCommand' failed: EngineException:
org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException:
IRSErrorException: Failed to Activa
teStorageDomainVDS, error = Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code = 358 (Failed with error
StorageDomainDoesNotExist and code 358)
2020-04-07 22:45:00,298+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] EVENT_ID: USER_ACTIVATE_STORAGE
_DOMAIN_FAILED(967), Failed to activate Storage Domain data_fast (Data Center Default) by
admin@internal-authz
Second attempt looks the same:
2020-04-07 22:46:10,899+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [424b86ad-e2f7-4563-a44b-43b8c31855d
1] Failed in 'ActivateStorageDomainVDS' method
2020-04-07 22:46:10,923+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100149) [424b86ad-e2f7-4563-a44b-43b8c31855d1] EV
ENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command ActivateStorageDomainVDS failed:
Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:46:10,923+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [424b86ad-e2f7-4563-a44b-43b8c31855d1]
IrsBroker::F
ailed::ActivateStorageDomainVDS: IRSGenericException: IRSErrorException: Failed to
ActivateStorageDomainVDS, error = Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',),
code = 358
2020-04-07 22:46:11,226+03 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100149) [88d4fb2] EVENT_ID: SYSTEM_CHANGE_STORAGE
_POOL_STATUS_PROBLEMATIC_SEARCHING_NEW_SPM(986), Data Center is being initialized, please
wait for initialization to complete.
2020-04-07 22:46:13,088+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [88d4fb2] Failed in 'ActivateStorage
DomainVDS' method
2020-04-07 22:46:13,107+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100149) [88d4fb2] EVENT_ID: IRS_BROKER_COMMAND_FA
ILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not
exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:46:13,107+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [88d4fb2]
IrsBroker::Failed::ActivateStorageDomainV
DS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error =
Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code =
358
2020-04-07 22:46:13,358+03 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] EVENT_ID: SYSTEM_CHANGE_STORAG
E_POOL_STATUS_PROBLEMATIC_SEARCHING_NEW_SPM(986), Data Center is being initialized, please
wait for initialization to complete.
2020-04-07 22:46:15,131+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] Failed in 'ActivateStorag
eDomainVDS' method
2020-04-07 22:46:15,151+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] EVENT_ID: IRS_BROKER_COMMAND_F
AILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not
exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:46:15,151+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45]
IrsBroker::Failed::ActivateStorageDomain
VDS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error =
Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code =
358
2020-04-07 22:46:15,151+03 ERROR
[org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] Command
'org.ovirt.engine.cor
e.bll.storage.domain.ActivateStorageDomainCommand' failed: EngineException:
org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException:
IRSErrorException: Failed to Activ
ateStorageDomainVDS, error = Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code = 358 (Failed with error
StorageDomainDoesNotExist and code 358)
2020-04-07 22:46:15,192+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] EVENT_ID: USER_ACTIVATE_STORAG
E_DOMAIN_FAILED(967), Failed to activate Storage Domain data_fast (Data Center Default) by
admin@internal-authz
SPM is ovirt1 and vdsm log shows:
2020-04-07 22:44:55,759+0300 ERROR (jsonrpc/0) [storage.TaskManager.Task]
(Task='3bf6603f-5fda-4595-acd6-37176dca3fd4') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run
return fn(*args, **kargs)
File "<string>", line 2, in activateStorageDomain
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1261, in
activateStorageDomain
pool.activateSD(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in
wrapper
return method(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1138, in
activateSD
dom = sdCache.produce(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in
produce
domain.getRealDomain()
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in
getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in
_realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in
_findDomain
return findMethod(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in
_findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:44:55,759+0300 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH
activateStorageDomain error=Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',) (dispatcher:83)
2020-04-07 22:46:15,129+0300 ERROR (jsonrpc/0) [storage.TaskManager.Task]
(Task='9c0ea753-e00c-4bee-87c1-9d5269c90b23') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run
return fn(*args, **kargs)
File "<string>", line 2, in activateStorageDomain
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1261, in
activateStorageDomain
pool.activateSD(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in
wrapper
return method(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1138, in
activateSD
dom = sdCache.produce(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in
produce
domain.getRealDomain()
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in
getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in
_realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in
_findDomain
return findMethod(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in
_findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:46:15,130+0300 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH
activateStorageDomain error=Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',) (dispatcher:83)
Metadata shows the following:
[root@ovirt1 dom_md]# cat
/rhev/data-center/mnt/glusterSD/gluster1\:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/dom_md/metadata
CLASS=Data
DESCRIPTION=data_fast
IOOPTIMEOUTSEC=10
LEASERETRIES=3
LEASETIMESEC=60
LOCKPOLICY=
LOCKRENEWALINTERVALSEC=5
POOL_UUID=b803f7e4-2543-11e9-ba9a-00163e6272c8
REMOTE_PATH=gluster1:/data_fast
ROLE=Regular
SDUUID=396604d9-2a9e-49cd-9563-fdc79981f67b
TYPE=GLUSTERFS
VERSION=4
_SHA_CKSUM=9b3a87a67ea2da9ea10997fbe46e4b8ab565208b
[root@ovirt1 dom_md]# ls -l
/rhev/data-center/mnt/glusterSD/gluster1\:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/dom_md/metadata
-rw-r--r--. 1 vdsm kvm 332 9 мар 2,25
/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/dom_md/metadata
[root@ovirt1 dom_md]# vdsm-client StorageDomain validate
"storagedomainID"="396604d9-2a9e-49cd-9563-fdc79981f67b"
vdsm-client: Command StorageDomain.validate with args {'storagedomainID':
'396604d9-2a9e-49cd-9563-fdc79981f67b'} failed:
(code=358, message=Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',))
[root@ovirt1 dom_md]# vdsm-client StorageDomain getStats
"storagedomainID"="396604d9-2a9e-49cd-9563-fdc79981f67b"
vdsm-client: Command StorageDomain.getStats with args {'storagedomainID':
'396604d9-2a9e-49cd-9563-fdc79981f67b'} failed:
(code=358, message=Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',))
[root@ovirt1 dom_md]# vdsm-client StorageDomain activate
"storagedomainID"="396604d9-2a9e-49cd-9563-fdc79981f67b"
"storagepoolID"="b803f7e4-2543-11e9-ba9a-00163e6272c8"
vdsm-client: Command StorageDomain.activate with args {'storagepoolID':
'b803f7e4-2543-11e9-ba9a-00163e6272c8', 'storagedomainID':
'396604d9-2a9e-49cd-9563-fdc79981f67b'} failed:
(code=358, message=Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',))
I had issues with my Gluster ACL xlator and I have reverted to Gluster v7.0 (v6.6+ &
v7.1+ are causing me headaches).
My main suspect is storage layer, yet I need to verify that.In this situation 4 out of 6
volumes are in maintenance mode and I cannot detach , nor to activate them :)
What is the flow that the engine is implementing (what files are checked with which user,
what is being read, extracted,etc) in order to activate a storage domain?
I'm attaching the engine's log , SPM's log and some output from the DB.
Affected storage domains are "data_fast" , "data_fast2" ,
"data_fast3" & "data_fast4" (gluster volumes are with the same
name).
Best Regards,
Strahil Nikolov