3:16 a.m.
Hi,
For the first time in many months I have run into some trouble with oVirt (4.3.4.3) and I
need some help.
Yesterday, I noticed one of my iSCSI storage domains was almost full, and tried to move a
disk image off of it, to another domain. This failed, and somewhere in the process, the
whole storage domain went to status 'Inactive'.
From engine.log:
2019-07-17 16:30:35,319+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy]
(EE-ManagedThreadFactory-engine-Thread-1836383) [] starting processDomainRecovery for
domain '875847b6-29a4-4419-be92-9315f4435429:HQST0_ISCSI02'.
2019-07-17 16:30:35,337+02 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy]
(EE-ManagedThreadFactory-engine-Thread-1836383) [] Domain
'875847b6-29a4-4419-be92-9315f4435429:HQST0_ISCSI02' was reported by all hosts in
status UP as problematic. Moving the domain to NonOperational.
2019-07-17 16:30:35,410+02 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-1836383) [5f6fd35e] EVENT_ID:
SYSTEM_DEACTIVATED_STORAGE_DOMAIN(970), Storage Domain HQST0_ISCSI02 (Data Center ISAAC01)
was deactivated by system because it's not visible by any of the hosts.
The thing is, the domain is still functional on all my hosts. It carries over 50 disks,
and all involved VMs are up and running, and don't seem to have any problems. Also,
'iscsiadm' on all hosts seems to indiciate that everything is fine with this
specific target and reading from the device with dd, or getting its size with
'blockdev' all works without issue.
When I try to reactivate the domain, these errors are logged:
2019-07-18 09:34:53,631+02 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-43475) [79e386e] EVENT_ID:
IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command ActivateStorageDomainVDS failed: Storage
domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',)
2019-07-18 09:34:53,631+02 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-engine-Thread-43475) [79e386e]
IrsBroker::Failed::ActivateStorageDomainVDS: IRSGenericException: IRSErrorException:
Failed to ActivateStorageDomainVDS, error = Storage domain does not exist:
(u'875847b6-29a4-4419-be92-9315f4435429',), code = 358
2019-07-18 09:34:53,648+02 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-43475) [79e386e] EVENT_ID:
USER_ACTIVATE_STORAGE_DOMAIN_FAILED(967), Failed to activate Storage Domain HQST0_ISCSI02
(Data Center ISAAC01) by martijn@-authz
On the SPM host, there are errors that indicate problems with the LVM volume group:
2019-07-18 09:34:50,462+0200 INFO (jsonrpc/2) [vdsm.api] START
activateStorageDomain(sdUUID=u'875847b6-29a4-4419-be92-9315f4435429',
spUUID=u'aefd5844-6e01-4070-b3b9-c0d73cc40c78', options=None)
from=::ffff:172.17.1.140,56570, flow_id=197dadec,
task_id=51107845-d80b-47f4-aed8-345aaa49f0f8 (api:48)
2019-07-18 09:34:50,464+0200 INFO (jsonrpc/2) [storage.StoragePool]
sdUUID=875847b6-29a4-4419-be92-9315f4435429 spUUID=aefd5844-6e01-4070-b3b9-c0d73cc40c78
(sp:1125)
2019-07-18 09:34:50,629+0200 WARN (jsonrpc/2) [storage.LVM] Reloading VGs failed
(vgs=[u'875847b6-29a4-4419-be92-9315f4435429'] rc=5 out=[] err=['
/dev/mapper/23536316636393463: Checksum error at offset 2748693688832', "
Couldn't read volume group metadata from /dev/mapper/23536316636393463.", '
Metadata location on /dev/mapper/23536316636393463 at 2748693688832 has invalid summary
for VG.', ' Failed to read metadata summary from
/dev/mapper/23536316636393463', ' Failed to scan VG from
/dev/mapper/23536316636393463', ' Volume group
"875847b6-29a4-4419-be92-9315f4435429" not found', ' Cannot process
volume group 875847b6-29a4-4419-be92-9315f4435429']) (lvm:442)
2019-07-18 09:34:50,629+0200 INFO (jsonrpc/2) [vdsm.api] FINISH activateStorageDomain
error=Storage domain does not exist: (u'875847b6-29a4-4419-be92-9315f4435429',)
from=::ffff:172.17.1.140,56570, flow_id=197dadec,
task_id=51107845-d80b-47f4-aed8-345aaa49f0f8 (api:52)
2019-07-18 09:34:50,629+0200 ERROR (jsonrpc/2) [storage.TaskManager.Task]
(Task='51107845-d80b-47f4-aed8-345aaa49f0f8') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run
return fn(*args, **kargs)
File "<string>", line 2, in activateStorageDomain
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1262, in
activateStorageDomain
pool.activateSD(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in
wrapper
return method(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1127, in
activateSD
dom = sdCache.produce(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in
produce
domain.getRealDomain()
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in
getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in
_realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in
_findDomain
return findMethod(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1807, in
findDomain
return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1665, in
findDomainPath
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
(u'875847b6-29a4-4419-be92-9315f4435429',)
2019-07-18 09:34:50,629+0200 INFO (jsonrpc/2) [storage.TaskManager.Task]
(Task='51107845-d80b-47f4-aed8-345aaa49f0f8') aborting: Task is aborted:
"Storage domain does not exist:
(u'875847b6-29a4-4419-be92-9315f4435429',)" - code 358 (task:1181)
2019-07-18 09:34:50,629+0200 ERROR (jsonrpc/2) [storage.Dispatcher] FINISH
activateStorageDomain error=Storage domain does not exist:
(u'875847b6-29a4-4419-be92-9315f4435429',) (dispatcher:83)
I need help getting this storage domain back online. Can anyone here help me? If you need
any additional information, please let me know!
Best regards,
Martijn Grendelman
ISAAC