3:16 a.m.
Oct 25, 2022, 4:10:01 PM - Failed to connect Host <hostname> to Storage Pool
<storage pool>
Oct 25, 2022, 4:10:01 PM - Host <hostname> cannot access the Storage Domain(s)
VM_Data attached to the <data center>. Setting Host state to Non-Operational.
Oct 25, 2022, 4:10:01 PM - Host <hostname> reports about one of the Active Storage
Domains as Problematic.
Oct 25, 2022, 4:09:52 PM - Activation of host <hostname> initiated by <user>.
Hi All,
Today morning, noticed that my 1st and 2nd host is in non-operational status and saw that
my storage pool isn't being mounted and whenever I try to press Activate under
Management, the above error happens.
I have multiple VM running and I can't bring it up if it goes down, and it is my
production server for my company.
Any advise on how to check on this or what logs to look & provide to resolve this?
The person who deployed this setup is no longer in my company and I have no one else to
seek help for this issue.
3:40 a.m.
New subject: URGENT HELP NEEDED!! Hosts are in non-operational state.
Hi Tomas,
[root@host1 vdsm]# tail -f vdsm.log
StorageUnavailableError: Unable to get volume size for domain
107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume 581f65d2-3db5-4fde-960f-4c4ffe27341e
2022-10-25 16:39:51,785+0800 INFO (periodic/3) [virt.vm]
(vmId='a8d6fcaf-a673-46df-b00a-925a8b9791ef') Requesting extension for volume
581f65d2-3db5-4fde-960f-4c4ffe27341e on domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19
(apparent: 91268055040, capacity: 322122547200, allocated: 90872676352, physical:
91268055040 threshold_state: exceeded) (vm:1289)
2022-10-25 16:39:51,787+0800 INFO (periodic/3) [vdsm.api] START
sendExtendMsg(spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef',
volDict={'newSize': 92341796864, 'domainID':
'107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda',
'poolID': '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock':
<Clock(total=0.00*, extend-volume=0.00*)>, 'internal': False,
'volumeID': '581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID':
'2d1375ba-ce52-4da1-9873-d9181d9e775b'}, newSize=92341796864,
callbackFunc=<bound method Vm.__afterVolumeExtension of <vdsm.virt.vm.Vm object at
0x7fea007336d0>>) from=internal, task_id=2b0c7a67-0f31-4f62-af70-a29e0b84dafb
(api:48)
2022-10-25 16:39:51,787+0800 INFO (periodic/3) [vdsm.api] FINISH sendExtendMsg
return=None from=internal, task_id=2b0c7a67-0f31-4f62-af70-a29e0b84dafb (api:54)
2022-10-25 16:39:52,635+0800 INFO (mailbox-hsm) [storage.MailBox.HsmMailMonitor]
HSM_MailMonitor sending mail to SPM - ['/usr/bin/dd',
'of=/rhev/data-center/81f9e45a-5879-11ea-a3bd-00163e27dfef/mastersd/dom_md/inbox',
'iflag=fullblock', 'oflag=direct', 'conv=notrunc',
'bs=4096', 'count=1', 'seek=1'] (mailbox:380)
2022-10-25 16:39:52,710+0800 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.ping2
succeeded in 0.00 seconds (__init__:312)
2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [api.virt] START getStats() from=::1,39184,
vmId=6579a9c4-d55e-4d61-9c3a-c745cdcfa430 (api:48)
2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'6579a9c4-d55e-4d61-9c3a-c745cdcfa430'}
(api:129)
2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [api.virt] FINISH getStats
return={'status': {'message': "Virtual machine does not exist:
{'vmId': u'6579a9c4-d55e-4d61-9c3a-c745cdcfa430'}", 'code':
1}} from=::1,39184, vmId=6579a9c4-d55e-4d61-9c3a-c745cdcfa430 (api:54)
2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2022-10-25 16:39:53,786+0800 INFO (periodic/5) [virt.vm]
(vmId='a8d6fcaf-a673-46df-b00a-925a8b9791ef') Requesting extension for volume
581f65d2-3db5-4fde-960f-4c4ffe27341e on domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19
(apparent: 91268055040, capacity: 322122547200, allocated: 90872676352, physical:
91268055040 threshold_state: exceeded) (vm:1289)
2022-10-25 16:39:53,787+0800 INFO (periodic/5) [vdsm.api] START
sendExtendMsg(spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef',
volDict={'newSize': 92341796864, 'domainID':
'107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda',
'poolID': '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock':
<Clock(total=0.00*, extend-volume=0.00*)>, 'internal': False,
'volumeID': '581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID':
'2d1375ba-ce52-4da1-9873-d9181d9e775b'}, newSize=92341796864,
callbackFunc=<bound method Vm.__afterVolumeExtension of <vdsm.virt.vm.Vm object at
0x7fea007336d0>>) from=internal, task_id=197234ba-5cab-4098-86c8-7f2c979171d2
(api:48)
2022-10-25 16:39:53,787+0800 INFO (periodic/5) [vdsm.api] FINISH sendExtendMsg
return=None from=internal, task_id=197234ba-5cab-4098-86c8-7f2c979171d2 (api:54)
2022-10-25 16:39:54,167+0800 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.ping2
succeeded in 0.00 seconds (__init__:312)
2022-10-25 16:39:54,658+0800 INFO (mailbox-hsm) [storage.MailBox.HsmMailMonitor]
HSM_MailMonitor sending mail to SPM - ['/usr/bin/dd',
'of=/rhev/data-center/81f9e45a-5879-11ea-a3bd-00163e27dfef/mastersd/dom_md/inbox',
'iflag=fullblock', 'oflag=direct', 'conv=notrunc',
'bs=4096', 'count=1', 'seek=1'] (mailbox:380)
2022-10-25 16:39:54,661+0800 INFO (mailbox-hsm/3) [storage.ThreadPool.WorkerThread] START
task b81cf2c1-c795-4f97-9c5a-6b06a6a3173c (cmd=<function runTask at 0x7fea630c8500>,
args=(<bound method Vm.__afterVolumeExtension of <vdsm.virt.vm.Vm object at
0x7fea007336d0>>, {'newSize': 92341796864, 'domainID':
'107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda',
'poolID': '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock':
<Clock(total=2.88*, extend-volume=2.88, refresh-volume=0.00*)>, 'internal':
False, 'volumeID': '581f65d2-3db5-4fde-960f-4c4ffe27341e',
'imageID': '2d1375ba-ce52-4da1-9873-d9181d9e775b'})) (threadPool:208)
2022-10-25 16:39:54,663+0800 INFO (mailbox-hsm/3) [vdsm.api] START
refreshVolume(sdUUID='107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',
spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef',
imgUUID='2d1375ba-ce52-4da1-9873-d9181d9e775b',
volUUID='581f65d2-3db5-4fde-960f-4c4ffe27341e') from=internal,
task_id=918b7e3f-d730-4226-a941-fc0e60ee36b0 (api:48)
2022-10-25 16:39:54,809+0800 WARN (mailbox-hsm/3) [storage.LVM] Command
['/sbin/lvm', 'vgs', '--config', 'devices {
preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
write_cache_state=0 disable_after_error_count=3
filter=["a|^/dev/mapper/3600c0ff00051441d1e145f5e01000000$|^/dev/mapper/3600c0ff00051441de4f6555e01000000$|^/dev/mapper/364cd98f0bb99140025dd822d083dd183$|^/dev/mapper/364cd98f0bb99140025dd82520a7280ae$|",
"r|.*|"] } global { locking_type=4 prioritise_write_locks=1 wait_for_locks=1
use_lvmetad=0 } backup { retain_min=50 retain_days=0 }', '--noheadings',
'--units', 'b', '--nosuffix', '--separator', '|',
'--ignoreskippedcluster', '-o',
'uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name',
'--select', 'vg_name = 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19'] succeeded
with warnings: [' /dev/mapper/3600c0ff00051441d1e145f5e01000000: Checksum error at
offset 30099960109568', " Couldn't read volume
group metadata from /dev/mapper/3600c0ff00051441d1e145f5e01000000.", ' Metadata
location on /dev/mapper/3600c0ff00051441d1e145f5e01000000 at 30099960109568 has invalid
summary for VG.', ' Failed to read metadata summary from
/dev/mapper/3600c0ff00051441d1e145f5e01000000', ' Failed to scan VG from
/dev/mapper/3600c0ff00051441d1e145f5e01000000'] (lvm:466)
2022-10-25 16:39:54,824+0800 INFO (mailbox-hsm/3) [vdsm.api] FINISH refreshVolume
error=Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)
from=internal, task_id=918b7e3f-d730-4226-a941-fc0e60ee36b0 (api:52)
2022-10-25 16:39:54,824+0800 ERROR (mailbox-hsm/3) [storage.TaskManager.Task]
(Task='918b7e3f-d730-4226-a941-fc0e60ee36b0') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run
return fn(*args, **kargs)
File "<string>", line 2, in refreshVolume
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3033, in
refreshVolume
sdUUID=sdUUID).produceVolume(imgUUID=imgUUID,
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in
produce
domain.getRealDomain()
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in
getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in
_realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in
_findDomain
return findMethod(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1843, in
findDomain
return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1701, in
findDomainPath
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)
2022-10-25 16:39:54,825+0800 INFO (mailbox-hsm/3) [storage.TaskManager.Task]
(Task='918b7e3f-d730-4226-a941-fc0e60ee36b0') aborting: Task is aborted:
"Storage domain does not exist:
('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)" - code 358 (task:1181)
2022-10-25 16:39:54,825+0800 ERROR (mailbox-hsm/3) [storage.Dispatcher] FINISH
refreshVolume error=Storage domain does not exist:
('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) (dispatcher:83)
2022-10-25 16:39:54,826+0800 INFO (mailbox-hsm/3) [vdsm.api] START
getVolumeSize(sdUUID='107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',
spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef',
imgUUID='2d1375ba-ce52-4da1-9873-d9181d9e775b',
volUUID='581f65d2-3db5-4fde-960f-4c4ffe27341e', options=None) from=internal,
task_id=c5a7364d-3b39-4e12-b75b-22d5c4487cad (api:48)
2022-10-25 16:39:54,970+0800 WARN (mailbox-hsm/3) [storage.LVM] Command
['/sbin/lvm', 'vgs', '--config', 'devices {
preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
write_cache_state=0 disable_after_error_count=3
filter=["a|^/dev/mapper/3600c0ff00051441d1e145f5e01000000$|^/dev/mapper/3600c0ff00051441de4f6555e01000000$|^/dev/mapper/364cd98f0bb99140025dd822d083dd183$|^/dev/mapper/364cd98f0bb99140025dd82520a7280ae$|",
"r|.*|"] } global { locking_type=4 prioritise_write_locks=1 wait_for_locks=1
use_lvmetad=0 } backup { retain_min=50 retain_days=0 }', '--noheadings',
'--units', 'b', '--nosuffix', '--separator', '|',
'--ignoreskippedcluster', '-o',
'uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name',
'--select', 'vg_name = 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19'] succeeded
with warnings: [' /dev/mapper/3600c0ff00051441d1e145f5e01000000: Checksum error at
offset 30099960109568', " Couldn't read volume
group metadata from /dev/mapper/3600c0ff00051441d1e145f5e01000000.", ' Metadata
location on /dev/mapper/3600c0ff00051441d1e145f5e01000000 at 30099960109568 has invalid
summary for VG.', ' Failed to read metadata summary from
/dev/mapper/3600c0ff00051441d1e145f5e01000000', ' Failed to scan VG from
/dev/mapper/3600c0ff00051441d1e145f5e01000000'] (lvm:466)
2022-10-25 16:39:54,983+0800 INFO (mailbox-hsm/3) [vdsm.api] FINISH getVolumeSize
error=Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)
from=internal, task_id=c5a7364d-3b39-4e12-b75b-22d5c4487cad (api:52)
2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.TaskManager.Task]
(Task='c5a7364d-3b39-4e12-b75b-22d5c4487cad') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run
return fn(*args, **kargs)
File "<string>", line 2, in getVolumeSize
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3071, in
getVolumeSize
dom = sdCache.produce(sdUUID=sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in
produce
domain.getRealDomain()
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in
getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in
_realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in
_findDomain
return findMethod(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1843, in
findDomain
return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1701, in
findDomainPath
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)
2022-10-25 16:39:54,983+0800 INFO (mailbox-hsm/3) [storage.TaskManager.Task]
(Task='c5a7364d-3b39-4e12-b75b-22d5c4487cad') aborting: Task is aborted:
"Storage domain does not exist:
('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)" - code 358 (task:1181)
2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.Dispatcher] FINISH
getVolumeSize error=Storage domain does not exist:
('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) (dispatcher:83)
2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.TaskManager.Task]
(Task='033cec9a-5b65-4413-97c3-a273aafa6114') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run
return fn(*args, **kargs)
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1407, in
__afterVolumeExtension
volSize = self.__verifyVolumeExtension(volInfo)
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1325, in
__verifyVolumeExtension
volInfo['imageID'], volInfo['volumeID'])
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 6147, in
_getVolumeSize
(domainID, volumeID))
StorageUnavailableError: Unable to get volume size for domain
107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume 581f65d2-3db5-4fde-960f-4c4ffe27341e
2022-10-25 16:39:54,983+0800 INFO (mailbox-hsm/3) [storage.TaskManager.Task]
(Task='033cec9a-5b65-4413-97c3-a273aafa6114') aborting: Task is aborted:
u'Unable to get volume size for domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume
581f65d2-3db5-4fde-960f-4c4ffe27341e' - code 100 (task:1181)
2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.ThreadPool.WorkerThread]
FINISH task b81cf2c1-c795-4f97-9c5a-6b06a6a3173c failed (callback=None, cmd=<function
runTask at 0x7fea630c8500>, args=(<bound method Vm.__afterVolumeExtension of
<vdsm.virt.vm.Vm object at 0x7fea007336d0>>, {'newSize': 92341796864,
'domainID': '107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name':
'vda', 'poolID': '81f9e45a-5879-11ea-a3bd-00163e27dfef',
'clock': <Clock(total=3.20*, extend-volume=2.88, refresh-volume=0.16)>,
'internal': False, 'volumeID':
'581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID':
'2d1375ba-ce52-4da1-9873-d9181d9e775b'})) (threadPool:222)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/threadPool.py", line 209,
in _processNextTask
cmd(args)
File "/usr/lib/python2.7/site-packages/vdsm/storage/mailbox.py", line 85, in
runTask
ctask.prepare(cmd, *args)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in
wrapper
return m(self, *a, **kw)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in
prepare
raise self.error
StorageUnavailableError: Unable to get volume size for domain
107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume 581f65d2-3db5-4fde-960f-4c4ffe27341e
^C
This is what I get from vdsm when I ran tail -f.
11:28 a.m.
New subject: URGENT HELP NEEDED!! Hosts are in non-operational state.
Both the initial messages and the additional detail in these vdsm logs show that it can’t
mount a storage volume. In a multi node environment, it’s usually iscsi, gluster, etc, so
start by looking at your network and the storage devices and figuring out why it can’t
mount them. Comparison to any remaining running nodes may be helpful.
12:57 a.m.
New subject: URGENT HELP NEEDED!! Hosts are in non-operational state.
On Tue, Oct 25, 2022 at 7:38 PM Darrell Budic <budic(a)onholyground.com> wrote:
Indeed, but in this particular case, it also provides more details:
[' /dev/mapper/3600c0ff00051441d1
e145f5e01000000: Checksum error at offset 30099960109568', " Couldn't
read volume
group metadata from /dev/mapper/3600c0ff00051441d1e145f5e01000000.",
' Metadata location on /dev/mapper/3600c0ff00051441d1e145f5e01000000
at 30099960109568 has invalid summary for VG.', ' Failed to read
metadata summary from /dev/mapper/3600c0ff00051441d1e145f5e01000000',
' Failed to scan VG from
/dev/mapper/3600c0ff00051441d1e145f5e01000000']
If this VG does work well from some other host, perhaps it's a local
issue. Otherwise, it's probably corrupted - if you have good and
recent backups, it might be best to restore them, otherwise you can
try fixing the VG - I'd start by searching the net for relevant parts
of this error message and see how people handled similar cases.
Good luck and best regards,
--
Didi
925
days inactive
926
days old
4 comments
4 participants
participants (4)
-
Darrell Budic -
eugene＠knorydev.com -
Tomáš Golembiovský -
Yedidyah Bar David