Hi Shani, thanks for the asitance. First atempt to activate storage domain "data_fast" (for today) which is ontop gluster volume "data_fast": So Far I see the following when I filter "INFO" on the engine: 2020-04-07 22:44:55,760+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-100115) [7de0d120-fde0-4471-9032-f10e69d2e54 0] Failed in 'ActivateStorageDomainVDS' method 2020-04-07 22:44:55,781+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-100115) [7de0d120-fde0-4471-9032-f10e69d2e540] EV ENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',) 2020-04-07 22:44:55,781+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedThreadFactory-engine-Thread-100115) [7de0d120-fde0-4471-9032-f10e69d2e540] IrsBroker::F ailed::ActivateStorageDomainVDS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error = Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code = 358 2020-04-07 22:44:56,223+03 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-100115) [174bb3e2] EVENT_ID: SYSTEM_CHANGE_STORAG E_POOL_STATUS_PROBLEMATIC_SEARCHING_NEW_SPM(986), Data Center is being initialized, please wait for initialization to complete. 2020-04-07 22:44:58,086+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-100115) [174bb3e2] Failed in 'ActivateStorag eDomainVDS' method 2020-04-07 22:44:58,106+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-100115) [174bb3e2] EVENT_ID: IRS_BROKER_COMMAND_F AILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',) 2020-04-07 22:44:58,106+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedThreadFactory-engine-Thread-100115) [174bb3e2] IrsBroker::Failed::ActivateStorageDomain VDS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error = Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code = 358 2020-04-07 22:44:58,345+03 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] EVENT_ID: SYSTEM_CHANGE_STORAGE _POOL_STATUS_PROBLEMATIC_SEARCHING_NEW_SPM(986), Data Center is being initialized, please wait for initialization to complete. 2020-04-07 22:45:00,222+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] Failed in 'ActivateStorage DomainVDS' method 2020-04-07 22:45:00,240+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] EVENT_ID: IRS_BROKER_COMMAND_FA ILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',) 2020-04-07 22:45:00,241+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] IrsBroker::Failed::ActivateStorageDomainV DS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error = Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code = 358 2020-04-07 22:45:00,241+03 ERROR [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] Command 'org.ovirt.engine.core .bll.storage.domain.ActivateStorageDomainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException: IRSErrorException: Failed to Activa teStorageDomainVDS, error = Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code = 358 (Failed with error StorageDomainDoesNotExist and code 358) 2020-04-07 22:45:00,298+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] EVENT_ID: USER_ACTIVATE_STORAGE _DOMAIN_FAILED(967), Failed to activate Storage Domain data_fast (Data Center Default) by admin@internal-authz Second attempt looks the same: 2020-04-07 22:46:10,899+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-100149) [424b86ad-e2f7-4563-a44b-43b8c31855d 1] Failed in 'ActivateStorageDomainVDS' method 2020-04-07 22:46:10,923+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-100149) [424b86ad-e2f7-4563-a44b-43b8c31855d1] EV ENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',) 2020-04-07 22:46:10,923+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedThreadFactory-engine-Thread-100149) [424b86ad-e2f7-4563-a44b-43b8c31855d1] IrsBroker::F ailed::ActivateStorageDomainVDS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error = Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code = 358 2020-04-07 22:46:11,226+03 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-100149) [88d4fb2] EVENT_ID: SYSTEM_CHANGE_STORAGE _POOL_STATUS_PROBLEMATIC_SEARCHING_NEW_SPM(986), Data Center is being initialized, please wait for initialization to complete. 2020-04-07 22:46:13,088+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-100149) [88d4fb2] Failed in 'ActivateStorage DomainVDS' method 2020-04-07 22:46:13,107+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-100149) [88d4fb2] EVENT_ID: IRS_BROKER_COMMAND_FA ILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',) 2020-04-07 22:46:13,107+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedThreadFactory-engine-Thread-100149) [88d4fb2] IrsBroker::Failed::ActivateStorageDomainV DS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error = Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code = 358 2020-04-07 22:46:13,358+03 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] EVENT_ID: SYSTEM_CHANGE_STORAG E_POOL_STATUS_PROBLEMATIC_SEARCHING_NEW_SPM(986), Data Center is being initialized, please wait for initialization to complete. 2020-04-07 22:46:15,131+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] Failed in 'ActivateStorag eDomainVDS' method 2020-04-07 22:46:15,151+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] EVENT_ID: IRS_BROKER_COMMAND_F AILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',) 2020-04-07 22:46:15,151+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] IrsBroker::Failed::ActivateStorageDomain VDS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error = Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code = 358 2020-04-07 22:46:15,151+03 ERROR [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] Command 'org.ovirt.engine.cor e.bll.storage.domain.ActivateStorageDomainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException: IRSErrorException: Failed to Activ ateStorageDomainVDS, error = Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code = 358 (Failed with error StorageDomainDoesNotExist and code 358) 2020-04-07 22:46:15,192+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] EVENT_ID: USER_ACTIVATE_STORAG E_DOMAIN_FAILED(967), Failed to activate Storage Domain data_fast (Data Center Default) by admin@internal-authz SPM is ovirt1 and vdsm log shows: 2020-04-07 22:44:55,759+0300 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='3bf6603f-5fda-4595-acd6-37176dca3fd4') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in activateStorageDomain File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1261, in activateStorageDomain pool.activateSD(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1138, in activateSD dom = sdCache.produce(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',) 2020-04-07 22:44:55,759+0300 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH activateStorageDomain error=Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',) (dispatcher:83) 2020-04-07 22:46:15,129+0300 ERROR (jsonrpc/0) [storage.TaskManager.Task] (Task='9c0ea753-e00c-4bee-87c1-9d5269c90b23') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in activateStorageDomain File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1261, in activateStorageDomain pool.activateSD(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1138, in activateSD dom = sdCache.produce(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',) 2020-04-07 22:46:15,130+0300 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH activateStorageDomain error=Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',) (dispatcher:83) Metadata shows the following: [root@ovirt1 dom_md]# cat /rhev/data-center/mnt/glusterSD/gluster1\:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/dom_md/metadata CLASS=Data DESCRIPTION=data_fast IOOPTIMEOUTSEC=10 LEASERETRIES=3 LEASETIMESEC=60 LOCKPOLICY= LOCKRENEWALINTERVALSEC=5 POOL_UUID=b803f7e4-2543-11e9-ba9a-00163e6272c8 REMOTE_PATH=gluster1:/data_fast ROLE=Regular SDUUID=396604d9-2a9e-49cd-9563-fdc79981f67b TYPE=GLUSTERFS VERSION=4 _SHA_CKSUM=9b3a87a67ea2da9ea10997fbe46e4b8ab565208b [root@ovirt1 dom_md]# ls -l /rhev/data-center/mnt/glusterSD/gluster1\:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/dom_md/metadata -rw-r--r--. 1 vdsm kvm 332 9 мар 2,25 /rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/dom_md/metadata [root@ovirt1 dom_md]# vdsm-client StorageDomain validate "storagedomainID"="396604d9-2a9e-49cd-9563-fdc79981f67b" vdsm-client: Command StorageDomain.validate with args {'storagedomainID': '396604d9-2a9e-49cd-9563-fdc79981f67b'} failed: (code=358, message=Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)) [root@ovirt1 dom_md]# vdsm-client StorageDomain getStats "storagedomainID"="396604d9-2a9e-49cd-9563-fdc79981f67b" vdsm-client: Command StorageDomain.getStats with args {'storagedomainID': '396604d9-2a9e-49cd-9563-fdc79981f67b'} failed: (code=358, message=Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)) [root@ovirt1 dom_md]# vdsm-client StorageDomain activate "storagedomainID"="396604d9-2a9e-49cd-9563-fdc79981f67b" "storagepoolID"="b803f7e4-2543-11e9-ba9a-00163e6272c8" vdsm-client: Command StorageDomain.activate with args {'storagepoolID': 'b803f7e4-2543-11e9-ba9a-00163e6272c8', 'storagedomainID': '396604d9-2a9e-49cd-9563-fdc79981f67b'} failed: (code=358, message=Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)) I had issues with my Gluster ACL xlator and I have reverted to Gluster v7.0 (v6.6+ & v7.1+ are causing me headaches). My main suspect is storage layer, yet I need to verify that.In this situation 4 out of 6 volumes are in maintenance mode and I cannot detach , nor to activate them :) What is the flow that the engine is implementing (what files are checked with which user, what is being read, extracted,etc) in order to activate a storage domain? I'm attaching the engine's log , SPM's log and some output from the DB. Affected storage domains are "data_fast" , "data_fast2" , "data_fast3" & "data_fast4" (gluster volumes are with the same name). Best Regards, Strahil Nikolov