1:21 a.m.
Hello All,
I have several storage domains in maintenance mode and I can't neither detach, nor
activate.
Can someone explain to me what is actually doing the engine whwn I try to activate - so I
can debug it?
The error from vdsm-client is that the storage domain does not exist, yet it is in the DB
.
I assume that there is somw storage issue, but in order to debug it - I need to know what
is gooing on under the hood.
Thanks in advance
Best Regards,
Strahil Nikolov
5:25 a.m.
Hi Strahil,
You may find some more information on the logs:
On the engine machine: /var/log/ovirt-engine/engine.log
On the relevant host: /var/log/vdsm/vdsm.log.
You can also attach them/send privately so we can also have a look.
*Regards,*
*Shani Leviim*
On Sun, Apr 5, 2020 at 9:29 AM Strahil Nikolov <hunter86_bg(a)yahoo.com>
wrote:
Hello All,
I have several storage domains in maintenance mode and I can't neither
detach, nor activate.
Can someone explain to me what is actually doing the engine whwn I try to
activate - so I can debug it?
The error from vdsm-client is that the storage domain does not exist, yet
it is in the DB .
I assume that there is somw storage issue, but in order to debug it - I
need to know what is gooing on under the hood.
Thanks in advance
Best Regards,
Strahil Nikolov
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/6FDVRUEC5HR...
3:13 p.m.
Hi Shani,
thanks for the asitance.
First atempt to activate storage domain "data_fast" (for today) which is ontop
gluster volume "data_fast":
So Far I see the following when I filter "INFO" on the engine:
2020-04-07 22:44:55,760+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [7de0d120-fde0-4471-9032-f10e69d2e54
0] Failed in 'ActivateStorageDomainVDS' method
2020-04-07 22:44:55,781+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100115) [7de0d120-fde0-4471-9032-f10e69d2e540] EV
ENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command ActivateStorageDomainVDS failed:
Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:44:55,781+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [7de0d120-fde0-4471-9032-f10e69d2e540]
IrsBroker::F
ailed::ActivateStorageDomainVDS: IRSGenericException: IRSErrorException: Failed to
ActivateStorageDomainVDS, error = Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',),
code = 358
2020-04-07 22:44:56,223+03 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100115) [174bb3e2] EVENT_ID: SYSTEM_CHANGE_STORAG
E_POOL_STATUS_PROBLEMATIC_SEARCHING_NEW_SPM(986), Data Center is being initialized, please
wait for initialization to complete.
2020-04-07 22:44:58,086+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [174bb3e2] Failed in 'ActivateStorag
eDomainVDS' method
2020-04-07 22:44:58,106+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100115) [174bb3e2] EVENT_ID: IRS_BROKER_COMMAND_F
AILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not
exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:44:58,106+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [174bb3e2]
IrsBroker::Failed::ActivateStorageDomain
VDS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error =
Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code =
358
2020-04-07 22:44:58,345+03 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] EVENT_ID: SYSTEM_CHANGE_STORAGE
_POOL_STATUS_PROBLEMATIC_SEARCHING_NEW_SPM(986), Data Center is being initialized, please
wait for initialization to complete.
2020-04-07 22:45:00,222+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] Failed in 'ActivateStorage
DomainVDS' method
2020-04-07 22:45:00,240+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] EVENT_ID: IRS_BROKER_COMMAND_FA
ILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not
exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:45:00,241+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6]
IrsBroker::Failed::ActivateStorageDomainV
DS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error =
Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code =
358
2020-04-07 22:45:00,241+03 ERROR
[org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] Command
'org.ovirt.engine.core
.bll.storage.domain.ActivateStorageDomainCommand' failed: EngineException:
org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException:
IRSErrorException: Failed to Activa
teStorageDomainVDS, error = Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code = 358 (Failed with error
StorageDomainDoesNotExist and code 358)
2020-04-07 22:45:00,298+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] EVENT_ID: USER_ACTIVATE_STORAGE
_DOMAIN_FAILED(967), Failed to activate Storage Domain data_fast (Data Center Default) by
admin@internal-authz
Second attempt looks the same:
2020-04-07 22:46:10,899+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [424b86ad-e2f7-4563-a44b-43b8c31855d
1] Failed in 'ActivateStorageDomainVDS' method
2020-04-07 22:46:10,923+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100149) [424b86ad-e2f7-4563-a44b-43b8c31855d1] EV
ENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command ActivateStorageDomainVDS failed:
Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:46:10,923+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [424b86ad-e2f7-4563-a44b-43b8c31855d1]
IrsBroker::F
ailed::ActivateStorageDomainVDS: IRSGenericException: IRSErrorException: Failed to
ActivateStorageDomainVDS, error = Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',),
code = 358
2020-04-07 22:46:11,226+03 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100149) [88d4fb2] EVENT_ID: SYSTEM_CHANGE_STORAGE
_POOL_STATUS_PROBLEMATIC_SEARCHING_NEW_SPM(986), Data Center is being initialized, please
wait for initialization to complete.
2020-04-07 22:46:13,088+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [88d4fb2] Failed in 'ActivateStorage
DomainVDS' method
2020-04-07 22:46:13,107+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100149) [88d4fb2] EVENT_ID: IRS_BROKER_COMMAND_FA
ILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not
exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:46:13,107+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [88d4fb2]
IrsBroker::Failed::ActivateStorageDomainV
DS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error =
Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code =
358
2020-04-07 22:46:13,358+03 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] EVENT_ID: SYSTEM_CHANGE_STORAG
E_POOL_STATUS_PROBLEMATIC_SEARCHING_NEW_SPM(986), Data Center is being initialized, please
wait for initialization to complete.
2020-04-07 22:46:15,131+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] Failed in 'ActivateStorag
eDomainVDS' method
2020-04-07 22:46:15,151+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] EVENT_ID: IRS_BROKER_COMMAND_F
AILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not
exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:46:15,151+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45]
IrsBroker::Failed::ActivateStorageDomain
VDS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error =
Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code =
358
2020-04-07 22:46:15,151+03 ERROR
[org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] Command
'org.ovirt.engine.cor
e.bll.storage.domain.ActivateStorageDomainCommand' failed: EngineException:
org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException:
IRSErrorException: Failed to Activ
ateStorageDomainVDS, error = Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code = 358 (Failed with error
StorageDomainDoesNotExist and code 358)
2020-04-07 22:46:15,192+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] EVENT_ID: USER_ACTIVATE_STORAG
E_DOMAIN_FAILED(967), Failed to activate Storage Domain data_fast (Data Center Default) by
admin@internal-authz
SPM is ovirt1 and vdsm log shows:
2020-04-07 22:44:55,759+0300 ERROR (jsonrpc/0) [storage.TaskManager.Task]
(Task='3bf6603f-5fda-4595-acd6-37176dca3fd4') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run
return fn(*args, **kargs)
File "<string>", line 2, in activateStorageDomain
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1261, in
activateStorageDomain
pool.activateSD(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in
wrapper
return method(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1138, in
activateSD
dom = sdCache.produce(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in
produce
domain.getRealDomain()
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in
getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in
_realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in
_findDomain
return findMethod(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in
_findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:44:55,759+0300 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH
activateStorageDomain error=Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',) (dispatcher:83)
2020-04-07 22:46:15,129+0300 ERROR (jsonrpc/0) [storage.TaskManager.Task]
(Task='9c0ea753-e00c-4bee-87c1-9d5269c90b23') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run
return fn(*args, **kargs)
File "<string>", line 2, in activateStorageDomain
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1261, in
activateStorageDomain
pool.activateSD(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in
wrapper
return method(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1138, in
activateSD
dom = sdCache.produce(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in
produce
domain.getRealDomain()
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in
getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in
_realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in
_findDomain
return findMethod(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in
_findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:46:15,130+0300 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH
activateStorageDomain error=Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',) (dispatcher:83)
Metadata shows the following:
[root@ovirt1 dom_md]# cat
/rhev/data-center/mnt/glusterSD/gluster1\:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/dom_md/metadata
CLASS=Data
DESCRIPTION=data_fast
IOOPTIMEOUTSEC=10
LEASERETRIES=3
LEASETIMESEC=60
LOCKPOLICY=
LOCKRENEWALINTERVALSEC=5
POOL_UUID=b803f7e4-2543-11e9-ba9a-00163e6272c8
REMOTE_PATH=gluster1:/data_fast
ROLE=Regular
SDUUID=396604d9-2a9e-49cd-9563-fdc79981f67b
TYPE=GLUSTERFS
VERSION=4
_SHA_CKSUM=9b3a87a67ea2da9ea10997fbe46e4b8ab565208b
[root@ovirt1 dom_md]# ls -l
/rhev/data-center/mnt/glusterSD/gluster1\:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/dom_md/metadata
-rw-r--r--. 1 vdsm kvm 332 9 мар 2,25
/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/dom_md/metadata
[root@ovirt1 dom_md]# vdsm-client StorageDomain validate
"storagedomainID"="396604d9-2a9e-49cd-9563-fdc79981f67b"
vdsm-client: Command StorageDomain.validate with args {'storagedomainID':
'396604d9-2a9e-49cd-9563-fdc79981f67b'} failed:
(code=358, message=Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',))
[root@ovirt1 dom_md]# vdsm-client StorageDomain getStats
"storagedomainID"="396604d9-2a9e-49cd-9563-fdc79981f67b"
vdsm-client: Command StorageDomain.getStats with args {'storagedomainID':
'396604d9-2a9e-49cd-9563-fdc79981f67b'} failed:
(code=358, message=Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',))
[root@ovirt1 dom_md]# vdsm-client StorageDomain activate
"storagedomainID"="396604d9-2a9e-49cd-9563-fdc79981f67b"
"storagepoolID"="b803f7e4-2543-11e9-ba9a-00163e6272c8"
vdsm-client: Command StorageDomain.activate with args {'storagepoolID':
'b803f7e4-2543-11e9-ba9a-00163e6272c8', 'storagedomainID':
'396604d9-2a9e-49cd-9563-fdc79981f67b'} failed:
(code=358, message=Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',))
I had issues with my Gluster ACL xlator and I have reverted to Gluster v7.0 (v6.6+ &
v7.1+ are causing me headaches).
My main suspect is storage layer, yet I need to verify that.In this situation 4 out of 6
volumes are in maintenance mode and I cannot detach , nor to activate them :)
What is the flow that the engine is implementing (what files are checked with which user,
what is being read, extracted,etc) in order to activate a storage domain?
I'm attaching the engine's log , SPM's log and some output from the DB.
Affected storage domains are "data_fast" , "data_fast2" ,
"data_fast3" & "data_fast4" (gluster volumes are with the same
name).
Best Regards,
Strahil Nikolov
10:51 p.m.
The issue was found.
We are not looking for a directory with the exact name of the Storage Domain UUID, but we
look for ALL directories that contain the Storage Domain UUID.
For example:
[root@ovirt3 gluster1:_data__fast4]# ll
общо 1
drwxrwxr-x. 5 vdsm kvm 48 10 ное 13,42 578bca3d-6540-41cd-8e0e-9e3047026484
drwxrwxr-x. 5 vdsm kvm 48 10 ное 13,42 578bca3d-6540-41cd-8e0e-9e3047026484-NEW
[root@ovirt3 gluster1:_data__fast]# ll
общо 1
drwxrwxrwx. 6 vdsm kvm 59 11 апр 6,30 396604d9-2a9e-49cd-9563-fdc79981f67b
drwxr-xr-x. 5 vdsm kvm 48 19 ное 21,32 396604d9-2a9e-49cd-9563-fdc79981f67b-OLD
As 578bca3d-6540-41cd-8e0e-9e3047026484-NEW & 396604d9-2a9e-49cd-9563-fdc79981f67b-OLD
are not valid SDuuids - the activation fails.
I have opened a bug for it (not sure on the correct Product/Component) :
https://bugzilla.redhat.com/show_bug.cgi?id=1823033
Hey Sandro,
is this expected, as I never saw it as a requirement ?
If it is expected (already in the docs), please mark 1823033 "notabug".
Best Regards,
Strahil Nikolov
В вторник, 7 април 2020 г., 23:13:43 Гринуич+3, Strahil Nikolov
<hunter86_bg(a)yahoo.com> написа:
Hi Shani,
thanks for the asitance.
First atempt to activate storage domain "data_fast" (for today) which is ontop
gluster volume "data_fast":
So Far I see the following when I filter "INFO" on the engine:
2020-04-07 22:44:55,760+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [7de0d120-fde0-4471-9032-f10e69d2e54
0] Failed in 'ActivateStorageDomainVDS' method
2020-04-07 22:44:55,781+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100115) [7de0d120-fde0-4471-9032-f10e69d2e540] EV
ENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command ActivateStorageDomainVDS failed:
Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:44:55,781+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [7de0d120-fde0-4471-9032-f10e69d2e540]
IrsBroker::F
ailed::ActivateStorageDomainVDS: IRSGenericException: IRSErrorException: Failed to
ActivateStorageDomainVDS, error = Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',),
code = 358
2020-04-07 22:44:56,223+03 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100115) [174bb3e2] EVENT_ID: SYSTEM_CHANGE_STORAG
E_POOL_STATUS_PROBLEMATIC_SEARCHING_NEW_SPM(986), Data Center is being initialized, please
wait for initialization to complete.
2020-04-07 22:44:58,086+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [174bb3e2] Failed in 'ActivateStorag
eDomainVDS' method
2020-04-07 22:44:58,106+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100115) [174bb3e2] EVENT_ID: IRS_BROKER_COMMAND_F
AILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not
exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:44:58,106+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [174bb3e2]
IrsBroker::Failed::ActivateStorageDomain
VDS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error =
Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code =
358
2020-04-07 22:44:58,345+03 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] EVENT_ID: SYSTEM_CHANGE_STORAGE
_POOL_STATUS_PROBLEMATIC_SEARCHING_NEW_SPM(986), Data Center is being initialized, please
wait for initialization to complete.
2020-04-07 22:45:00,222+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] Failed in 'ActivateStorage
DomainVDS' method
2020-04-07 22:45:00,240+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] EVENT_ID: IRS_BROKER_COMMAND_FA
ILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not
exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:45:00,241+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6]
IrsBroker::Failed::ActivateStorageDomainV
DS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error =
Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code =
358
2020-04-07 22:45:00,241+03 ERROR
[org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand]
(EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] Command
'org.ovirt.engine.core
.bll.storage.domain.ActivateStorageDomainCommand' failed: EngineException:
org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException:
IRSErrorException: Failed to Activa
teStorageDomainVDS, error = Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code = 358 (Failed with error
StorageDomainDoesNotExist and code 358)
2020-04-07 22:45:00,298+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100115) [3ac80a6] EVENT_ID: USER_ACTIVATE_STORAGE
_DOMAIN_FAILED(967), Failed to activate Storage Domain data_fast (Data Center Default) by
admin@internal-authz
Second attempt looks the same:
2020-04-07 22:46:10,899+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [424b86ad-e2f7-4563-a44b-43b8c31855d
1] Failed in 'ActivateStorageDomainVDS' method
2020-04-07 22:46:10,923+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100149) [424b86ad-e2f7-4563-a44b-43b8c31855d1] EV
ENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command ActivateStorageDomainVDS failed:
Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:46:10,923+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [424b86ad-e2f7-4563-a44b-43b8c31855d1]
IrsBroker::F
ailed::ActivateStorageDomainVDS: IRSGenericException: IRSErrorException: Failed to
ActivateStorageDomainVDS, error = Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',),
code = 358
2020-04-07 22:46:11,226+03 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100149) [88d4fb2] EVENT_ID: SYSTEM_CHANGE_STORAGE
_POOL_STATUS_PROBLEMATIC_SEARCHING_NEW_SPM(986), Data Center is being initialized, please
wait for initialization to complete.
2020-04-07 22:46:13,088+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [88d4fb2] Failed in 'ActivateStorage
DomainVDS' method
2020-04-07 22:46:13,107+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100149) [88d4fb2] EVENT_ID: IRS_BROKER_COMMAND_FA
ILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not
exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:46:13,107+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [88d4fb2]
IrsBroker::Failed::ActivateStorageDomainV
DS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error =
Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code =
358
2020-04-07 22:46:13,358+03 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] EVENT_ID: SYSTEM_CHANGE_STORAG
E_POOL_STATUS_PROBLEMATIC_SEARCHING_NEW_SPM(986), Data Center is being initialized, please
wait for initialization to complete.
2020-04-07 22:46:15,131+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] Failed in 'ActivateStorag
eDomainVDS' method
2020-04-07 22:46:15,151+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] EVENT_ID: IRS_BROKER_COMMAND_F
AILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage domain does not
exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:46:15,151+03 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45]
IrsBroker::Failed::ActivateStorageDomain
VDS: IRSGenericException: IRSErrorException: Failed to ActivateStorageDomainVDS, error =
Storage domain does not exist: (u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code =
358
2020-04-07 22:46:15,151+03 ERROR
[org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand]
(EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] Command
'org.ovirt.engine.cor
e.bll.storage.domain.ActivateStorageDomainCommand' failed: EngineException:
org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException:
IRSErrorException: Failed to Activ
ateStorageDomainVDS, error = Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',), code = 358 (Failed with error
StorageDomainDoesNotExist and code 358)
2020-04-07 22:46:15,192+03 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-100149) [2a490b45] EVENT_ID: USER_ACTIVATE_STORAG
E_DOMAIN_FAILED(967), Failed to activate Storage Domain data_fast (Data Center Default) by
admin@internal-authz
SPM is ovirt1 and vdsm log shows:
2020-04-07 22:44:55,759+0300 ERROR (jsonrpc/0) [storage.TaskManager.Task]
(Task='3bf6603f-5fda-4595-acd6-37176dca3fd4') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run
return fn(*args, **kargs)
File "<string>", line 2, in activateStorageDomain
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1261, in
activateStorageDomain
pool.activateSD(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in
wrapper
return method(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1138, in
activateSD
dom = sdCache.produce(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in
produce
domain.getRealDomain()
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in
getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in
_realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in
_findDomain
return findMethod(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in
_findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:44:55,759+0300 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH
activateStorageDomain error=Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',) (dispatcher:83)
2020-04-07 22:46:15,129+0300 ERROR (jsonrpc/0) [storage.TaskManager.Task]
(Task='9c0ea753-e00c-4bee-87c1-9d5269c90b23') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run
return fn(*args, **kargs)
File "<string>", line 2, in activateStorageDomain
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1261, in
activateStorageDomain
pool.activateSD(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in
wrapper
return method(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1138, in
activateSD
dom = sdCache.produce(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in
produce
domain.getRealDomain()
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in
getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in
_realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in
_findDomain
return findMethod(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in
_findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',)
2020-04-07 22:46:15,130+0300 ERROR (jsonrpc/0) [storage.Dispatcher] FINISH
activateStorageDomain error=Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',) (dispatcher:83)
Metadata shows the following:
[root@ovirt1 dom_md]# cat
/rhev/data-center/mnt/glusterSD/gluster1\:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/dom_md/metadata
CLASS=Data
DESCRIPTION=data_fast
IOOPTIMEOUTSEC=10
LEASERETRIES=3
LEASETIMESEC=60
LOCKPOLICY=
LOCKRENEWALINTERVALSEC=5
POOL_UUID=b803f7e4-2543-11e9-ba9a-00163e6272c8
REMOTE_PATH=gluster1:/data_fast
ROLE=Regular
SDUUID=396604d9-2a9e-49cd-9563-fdc79981f67b
TYPE=GLUSTERFS
VERSION=4
_SHA_CKSUM=9b3a87a67ea2da9ea10997fbe46e4b8ab565208b
[root@ovirt1 dom_md]# ls -l
/rhev/data-center/mnt/glusterSD/gluster1\:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/dom_md/metadata
-rw-r--r--. 1 vdsm kvm 332 9 мар 2,25
/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/dom_md/metadata
[root@ovirt1 dom_md]# vdsm-client StorageDomain validate
"storagedomainID"="396604d9-2a9e-49cd-9563-fdc79981f67b"
vdsm-client: Command StorageDomain.validate with args {'storagedomainID':
'396604d9-2a9e-49cd-9563-fdc79981f67b'} failed:
(code=358, message=Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',))
[root@ovirt1 dom_md]# vdsm-client StorageDomain getStats
"storagedomainID"="396604d9-2a9e-49cd-9563-fdc79981f67b"
vdsm-client: Command StorageDomain.getStats with args {'storagedomainID':
'396604d9-2a9e-49cd-9563-fdc79981f67b'} failed:
(code=358, message=Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',))
[root@ovirt1 dom_md]# vdsm-client StorageDomain activate
"storagedomainID"="396604d9-2a9e-49cd-9563-fdc79981f67b"
"storagepoolID"="b803f7e4-2543-11e9-ba9a-00163e6272c8"
vdsm-client: Command StorageDomain.activate with args {'storagepoolID':
'b803f7e4-2543-11e9-ba9a-00163e6272c8', 'storagedomainID':
'396604d9-2a9e-49cd-9563-fdc79981f67b'} failed:
(code=358, message=Storage domain does not exist:
(u'396604d9-2a9e-49cd-9563-fdc79981f67b',))
I had issues with my Gluster ACL xlator and I have reverted to Gluster v7.0 (v6.6+ &
v7.1+ are causing me headaches).
My main suspect is storage layer, yet I need to verify that.In this situation 4 out of 6
volumes are in maintenance mode and I cannot detach , nor to activate them :)
What is the flow that the engine is implementing (what files are checked with which user,
what is being read, extracted,etc) in order to activate a storage domain?
I'm attaching the engine's log , SPM's log and some output from the DB.
Affected storage domains are "data_fast" , "data_fast2" ,
"data_fast3" & "data_fast4" (gluster volumes are with the same
name).
Best Regards,
Strahil Nikolov
5:14 a.m.
On Sat, Apr 11, 2020 at 6:55 AM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote:
I have opened a bug for it (not sure on the correct
Product/Component) :
https://bugzilla.redhat.com/1823033
This is expected behavior. We don't support arbitrary modifications of
storage domain
directory structure.
I explained in the bug how you can do the same modifications safely:
https://bugzilla.redhat.com/1823033#c1
Nir
6:19 a.m.
On April 11, 2020 1:14:52 PM GMT+03:00, Nir Soffer <nsoffer(a)redhat.com> wrote:
On Sat, Apr 11, 2020 at 6:55 AM Strahil Nikolov
<hunter86_bg(a)yahoo.com>
wrote:
> I have opened a bug for it (not sure on the correct
Product/Component) :
> https://bugzilla.redhat.com/1823033
This is expected behavior. We don't support arbitrary modifications of
storage domain
directory structure.
I explained in the bug how you can do the same modifications safely:
https://bugzilla.redhat.com/1823033#c1
Nir
Hey Nir,
Sadly I didn't know that , but I couldn't also find anything related in the
docs.
Am I missing it or it's just not there ?
A Feature request from my side - can oVirt create the storage domain structure + a file
named 'DO_NOT_PUT_ANYTHING_HERE' with the info from the bug inside ?
Would the dev consider that ?
I think that the implementation is quite simple.
Best Regards,
Strahil Nikolov
7:46 a.m.
On Sat, Apr 11, 2020, 14:19 Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote:
On April 11, 2020 1:14:52 PM GMT+03:00, Nir Soffer
<nsoffer(a)redhat.com>
wrote:
>On Sat, Apr 11, 2020 at 6:55 AM Strahil Nikolov <hunter86_bg(a)yahoo.com>
>wrote:
>
>> I have opened a bug for it (not sure on the correct
>Product/Component) :
>> https://bugzilla.redhat.com/1823033
>
>This is expected behavior. We don't support arbitrary modifications of
>storage domain
>directory structure.
>
>I explained in the bug how you can do the same modifications safely:
>https://bugzilla.redhat.com/1823033#c1
>
>Nir
Hey Nir,
Sadly I didn't know that , but I couldn't also find anything related in
the docs.
Am I missing it or it's just not there ?
A Feature request from my side - can oVirt create the storage domain
structure + a file named 'DO_NOT_PUT_ANYTHING_HERE' with the info from
the bug inside ?
Would the dev consider that ?
I think that the implementation is quite simple.
You can file a RFE about that.
Best Regards,
Strahil Nikolov
8:37 a.m.
On April 11, 2020 3:46:49 PM GMT+03:00, Nir Soffer <nsoffer(a)redhat.com> wrote:
On Sat, Apr 11, 2020, 14:19 Strahil Nikolov
<hunter86_bg(a)yahoo.com>
wrote:
> On April 11, 2020 1:14:52 PM GMT+03:00, Nir Soffer
<nsoffer(a)redhat.com>
> wrote:
> >On Sat, Apr 11, 2020 at 6:55 AM Strahil Nikolov
<hunter86_bg(a)yahoo.com>
> >wrote:
> >
> >> I have opened a bug for it (not sure on the correct
> >Product/Component) :
> >> https://bugzilla.redhat.com/1823033
> >
> >This is expected behavior. We don't support arbitrary modifications
of
> >storage domain
> >directory structure.
> >
> >I explained in the bug how you can do the same modifications safely:
> >https://bugzilla.redhat.com/1823033#c1
> >
> >Nir
>
> Hey Nir,
>
> Sadly I didn't know that , but I couldn't also find anything
related in
> the docs.
> Am I missing it or it's just not there ?
>
> A Feature request from my side - can oVirt create the storage domain
> structure + a file named 'DO_NOT_PUT_ANYTHING_HERE' with the info
from
> the bug inside ?
>
> Would the dev consider that ?
> I think that the implementation is quite simple.
>
You can file a RFE about that.
> Best Regards,
> Strahil Nikolov
>
>
Done -> https://bugzilla.redhat.com/show_bug.cgi?id=1823082
1487
days inactive
1493
days old
7 comments
3 participants
participants (3)
-
Nir Soffer -
Shani Leviim -
Strahil Nikolov