11:13 p.m.
Hi Shani,
thanks for the asitance.
First atempt to activate storage domain "data_fast" (for today) which is ontop
gluster volume "data_fast":
So Far I see the following when I filter "INFO" on the engine:
2020-04-07 22:44:55,760+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [7de0d120-fde0-4471-9032-f10e69d2e54
0] Failed in 'ActivateStorageDomainVDS' method
2020-04-07 22:44:55,781+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100115) [7de0d120-fde0-4471-9032-f10e69d2e540] EV
ENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command ActivateStorageDomainVDS failed:
Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:44:55,781+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [7de0d120-fde0-4471-9032-f10e69d2e540]
IrsBroker::F
ailed::ActivateStorageDomainVDS: IRSGenericException: IRSErrorException: Failed to
ActivateStorageDomainVDS, error = Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',),
code = 358
2020-04-07 22:44:56,223+03 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100115) [174bb3e2] EVENT_ID: SYSTEM_CHANGE_STORAG
E_POOL_STATUS_PROBLEMATIC_SEARCHING_NEW_SPM(986), Data Center is being initialized, please
wait for initialization to complete.
2020-04-07 22:44:58,086+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [174bb3e2] Failed in 'ActivateStorag
eDomainVDS' method
2020-04-07 22:44:58,106+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100115) [174bb3e2] EVENT_ID: IRS_BROKER_COMMAND_F
AILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not
exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:44:58,106+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [174bb3e2]
IrsBroker::Failed::ActivateStorageDomain
VDS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error =
Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code =
358
2020-04-07 22:44:58,345+03 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] EVENT_ID: SYSTEM_CHANGE_STORAGE
_POOL_STATUS_PROBLEMATIC_SEARCHING_NEW_SPM(986), Data Center is being initialized, please
wait for initialization to complete.
2020-04-07 22:45:00,222+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] Failed in 'ActivateStorage
DomainVDS' method
2020-04-07 22:45:00,240+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] EVENT_ID: IRS_BROKER_COMMAND_FA
ILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not
exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:45:00,241+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6]
IrsBroker::Failed::ActivateStorageDomainV
DS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error =
Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code =
358
2020-04-07 22:45:00,241+03 ERROR
[org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] Command
'org.ovirt.engine.core
.bll.storage.domain.ActivateStorageDomainCommand' failed: EngineException:
org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException:
IRSErrorException: Failed to Activa
teStorageDomainVDS, error = Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code = 358 (Failed with error
StorageDomainDoesNotExist and code 358)
2020-04-07 22:45:00,298+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] EVENT_ID: USER_ACTIVATE_STORAGE
_DOMAIN_FAILED(967), Failed to activate Storage Domain data_fast (Data Center Default) by
admin@internal-authz
Second attempt looks the same:
2020-04-07 22:46:10,899+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [424b86ad-e2f7-4563-a44b-43b8c31855d
1] Failed in 'ActivateStorageDomainVDS' method
2020-04-07 22:46:10,923+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100149) [424b86ad-e2f7-4563-a44b-43b8c31855d1] EV
ENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command ActivateStorageDomainVDS failed:
Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:46:10,923+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [424b86ad-e2f7-4563-a44b-43b8c31855d1]
IrsBroker::F
ailed::ActivateStorageDomainVDS: IRSGenericException: IRSErrorException: Failed to
ActivateStorageDomainVDS, error = Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',),
code = 358
2020-04-07 22:46:11,226+03 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100149) [88d4fb2] EVENT_ID: SYSTEM_CHANGE_STORAGE
_POOL_STATUS_PROBLEMATIC_SEARCHING_NEW_SPM(986), Data Center is being initialized, please
wait for initialization to complete.
2020-04-07 22:46:13,088+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [88d4fb2] Failed in 'ActivateStorage
DomainVDS' method
2020-04-07 22:46:13,107+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100149) [88d4fb2] EVENT_ID: IRS_BROKER_COMMAND_FA
ILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not
exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:46:13,107+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [88d4fb2]
IrsBroker::Failed::ActivateStorageDomainV
DS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error =
Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code =
358
2020-04-07 22:46:13,358+03 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] EVENT_ID: SYSTEM_CHANGE_STORAG
E_POOL_STATUS_PROBLEMATIC_SEARCHING_NEW_SPM(986), Data Center is being initialized, please
wait for initialization to complete.
2020-04-07 22:46:15,131+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] Failed in 'ActivateStorag
eDomainVDS' method
2020-04-07 22:46:15,151+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] EVENT_ID: IRS_BROKER_COMMAND_F
AILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not
exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:46:15,151+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45]
IrsBroker::Failed::ActivateStorageDomain
VDS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error =
Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code =
358
2020-04-07 22:46:15,151+03 ERROR
[org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] Command
'org.ovirt.engine.cor
e.bll.storage.domain.ActivateStorageDomainCommand' failed: EngineException:
org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException:
IRSErrorException: Failed to Activ
ateStorageDomainVDS, error = Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code = 358 (Failed with error
StorageDomainDoesNotExist and code 358)
2020-04-07 22:46:15,192+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] EVENT_ID: USER_ACTIVATE_STORAG
E_DOMAIN_FAILED(967), Failed to activate Storage Domain data_fast (Data Center Default) by
admin@internal-authz
SPM is ovirt1 and vdsm log shows:
2020-04-07 22:44:55,759+0300 ERROR (jsonrpc/0) [storage.TaskManager.Task]
(Task='3bf6603f-5fda-4595-acd6-37176dca3fd4') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run
return fn(*args, **kargs)
File "<string>", line 2, in activateStorageDomain
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1261, in
activateStorageDomain
pool.activateSD(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in
wrapper
return method(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1138, in
activateSD
dom = sdCache.produce(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in
produce
domain.getRealDomain()
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in
getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in
_realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in
_findDomain
return findMethod(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in
_findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:44:55,759+0300 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH
activateStorageDomain error=Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',) (dispatcher:83)
2020-04-07 22:46:15,129+0300 ERROR (jsonrpc/0) [storage.TaskManager.Task]
(Task='9c0ea753-e00c-4bee-87c1-9d5269c90b23') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run
return fn(*args, **kargs)
File "<string>", line 2, in activateStorageDomain
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1261, in
activateStorageDomain
pool.activateSD(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in
wrapper
return method(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1138, in
activateSD
dom = sdCache.produce(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in
produce
domain.getRealDomain()
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in
getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in
_realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in
_findDomain
return findMethod(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in
_findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:46:15,130+0300 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH
activateStorageDomain error=Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',) (dispatcher:83)
Metadata shows the following:
[root@ovirt1 dom_md]# cat
/rhev/data-center/mnt/glusterSD/gluster1\:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/dom_md/metadata
CLASS=Data
DESCRIPTION=data_fast
IOOPTIMEOUTSEC=10
LEASERETRIES=3
LEASETIMESEC=60
LOCKPOLICY=
LOCKRENEWALINTERVALSEC=5
POOL_UUID=b803f7e4-2543-11e9-ba9a-00163e6272c8
REMOTE_PATH=gluster1:/data_fast
ROLE=Regular
SDUUID=396604d9-2a9e-49cd-9563-fdc79981f67b
TYPE=GLUSTERFS
VERSION=4
_SHA_CKSUM=9b3a87a67ea2da9ea10997fbe46e4b8ab565208b
[root@ovirt1 dom_md]# ls -l
/rhev/data-center/mnt/glusterSD/gluster1\:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/dom_md/metadata
-rw-r--r--. 1 vdsm kvm 332 9 мар 2,25
/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/dom_md/metadata
[root@ovirt1 dom_md]# vdsm-client StorageDomain validate
"storagedomainID"="396604d9-2a9e-49cd-9563-fdc79981f67b"
vdsm-client: Command StorageDomain.validate with args {'storagedomainID':
'396604d9-2a9e-49cd-9563-fdc79981f67b'} failed:
(code=358, message=Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',))
[root@ovirt1 dom_md]# vdsm-client StorageDomain getStats
"storagedomainID"="396604d9-2a9e-49cd-9563-fdc79981f67b"
vdsm-client: Command StorageDomain.getStats with args {'storagedomainID':
'396604d9-2a9e-49cd-9563-fdc79981f67b'} failed:
(code=358, message=Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',))
[root@ovirt1 dom_md]# vdsm-client StorageDomain activate
"storagedomainID"="396604d9-2a9e-49cd-9563-fdc79981f67b"
"storagepoolID"="b803f7e4-2543-11e9-ba9a-00163e6272c8"
vdsm-client: Command StorageDomain.activate with args {'storagepoolID':
'b803f7e4-2543-11e9-ba9a-00163e6272c8', 'storagedomainID':
'396604d9-2a9e-49cd-9563-fdc79981f67b'} failed:
(code=358, message=Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',))
I had issues with my Gluster ACL xlator and I have reverted to Gluster v7.0 (v6.6+ &
v7.1+ are causing me headaches).
My main suspect is storage layer, yet I need to verify that.In this situation 4 out of 6
volumes are in maintenance mode and I cannot detach , nor to activate them :)
What is the flow that the engine is implementing (what files are checked with which user,
what is being read, extracted,etc) in order to activate a storage domain?
I'm attaching the engine's log , SPM's log and some output from the DB.
Affected storage domains are "data_fast" , "data_fast2" ,
"data_fast3" & "data_fast4" (gluster volumes are with the same
name).
Best Regards,
Strahil Nikolov