11:28 a.m.
Both the initial messages and the additional detail in these vdsm logs show that it can’t
mount a storage volume. In a multi node environment, it’s usually iscsi, gluster, etc, so
start by looking at your network and the storage devices and figuring out why it can’t
mount them. Comparison to any remaining running nodes may be helpful.
On Oct 25, 2022, at 3:40 AM, eugene(a)knorydev.com wrote:
Hi Tomas,
[root@host1 vdsm]# tail -f vdsm.log
StorageUnavailableError: Unable to get volume size for domain
107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume 581f65d2-3db5-4fde-960f-4c4ffe27341e
2022-10-25 16:39:51,785+0800 INFO (periodic/3) [virt.vm]
(vmId='a8d6fcaf-a673-46df-b00a-925a8b9791ef') Requesting extension for volume
581f65d2-3db5-4fde-960f-4c4ffe27341e on domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19
(apparent: 91268055040, capacity: 322122547200, allocated: 90872676352, physical:
91268055040 threshold_state: exceeded) (vm:1289)
2022-10-25 16:39:51,787+0800 INFO (periodic/3) [vdsm.api] START
sendExtendMsg(spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef',
volDict={'newSize': 92341796864, 'domainID':
'107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda',
'poolID': '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock':
<Clock(total=0.00*, extend-volume=0.00*)>, 'internal': False,
'volumeID': '581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID':
'2d1375ba-ce52-4da1-9873-d9181d9e775b'}, newSize=92341796864,
callbackFunc=<bound method Vm.__afterVolumeExtension of <vdsm.virt.vm.Vm object at
0x7fea007336d0>>) from=internal, task_id=2b0c7a67-0f31-4f62-af70-a29e0b84dafb
(api:48)
2022-10-25 16:39:51,787+0800 INFO (periodic/3) [vdsm.api] FINISH sendExtendMsg
return=None from=internal, task_id=2b0c7a67-0f31-4f62-af70-a29e0b84dafb (api:54)
2022-10-25 16:39:52,635+0800 INFO (mailbox-hsm) [storage.MailBox.HsmMailMonitor]
HSM_MailMonitor sending mail to SPM - ['/usr/bin/dd',
'of=/rhev/data-center/81f9e45a-5879-11ea-a3bd-00163e27dfef/mastersd/dom_md/inbox',
'iflag=fullblock', 'oflag=direct', 'conv=notrunc',
'bs=4096', 'count=1', 'seek=1'] (mailbox:380)
2022-10-25 16:39:52,710+0800 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call
Host.ping2 succeeded in 0.00 seconds (__init__:312)
2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [api.virt] START getStats()
from=::1,39184, vmId=6579a9c4-d55e-4d61-9c3a-c745cdcfa430 (api:48)
2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [api] FINISH getStats error=Virtual
machine does not exist: {'vmId': u'6579a9c4-d55e-4d61-9c3a-c745cdcfa430'}
(api:129)
2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [api.virt] FINISH getStats
return={'status': {'message': "Virtual machine does not exist:
{'vmId': u'6579a9c4-d55e-4d61-9c3a-c745cdcfa430'}", 'code':
1}} from=::1,39184, vmId=6579a9c4-d55e-4d61-9c3a-c745cdcfa430 (api:54)
2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2022-10-25 16:39:53,786+0800 INFO (periodic/5) [virt.vm]
(vmId='a8d6fcaf-a673-46df-b00a-925a8b9791ef') Requesting extension for volume
581f65d2-3db5-4fde-960f-4c4ffe27341e on domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19
(apparent: 91268055040, capacity: 322122547200, allocated: 90872676352, physical:
91268055040 threshold_state: exceeded) (vm:1289)
2022-10-25 16:39:53,787+0800 INFO (periodic/5) [vdsm.api] START
sendExtendMsg(spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef',
volDict={'newSize': 92341796864, 'domainID':
'107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda',
'poolID': '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock':
<Clock(total=0.00*, extend-volume=0.00*)>, 'internal': False,
'volumeID': '581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID':
'2d1375ba-ce52-4da1-9873-d9181d9e775b'}, newSize=92341796864,
callbackFunc=<bound method Vm.__afterVolumeExtension of <vdsm.virt.vm.Vm object at
0x7fea007336d0>>) from=internal, task_id=197234ba-5cab-4098-86c8-7f2c979171d2
(api:48)
2022-10-25 16:39:53,787+0800 INFO (periodic/5) [vdsm.api] FINISH sendExtendMsg
return=None from=internal, task_id=197234ba-5cab-4098-86c8-7f2c979171d2 (api:54)
2022-10-25 16:39:54,167+0800 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call
Host.ping2 succeeded in 0.00 seconds (__init__:312)
2022-10-25 16:39:54,658+0800 INFO (mailbox-hsm) [storage.MailBox.HsmMailMonitor]
HSM_MailMonitor sending mail to SPM - ['/usr/bin/dd',
'of=/rhev/data-center/81f9e45a-5879-11ea-a3bd-00163e27dfef/mastersd/dom_md/inbox',
'iflag=fullblock', 'oflag=direct', 'conv=notrunc',
'bs=4096', 'count=1', 'seek=1'] (mailbox:380)
2022-10-25 16:39:54,661+0800 INFO (mailbox-hsm/3) [storage.ThreadPool.WorkerThread]
START task b81cf2c1-c795-4f97-9c5a-6b06a6a3173c (cmd=<function runTask at
0x7fea630c8500>, args=(<bound method Vm.__afterVolumeExtension of
<vdsm.virt.vm.Vm object at 0x7fea007336d0>>, {'newSize': 92341796864,
'domainID': '107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name':
'vda', 'poolID': '81f9e45a-5879-11ea-a3bd-00163e27dfef',
'clock': <Clock(total=2.88*, extend-volume=2.88, refresh-volume=0.00*)>,
'internal': False, 'volumeID':
'581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID':
'2d1375ba-ce52-4da1-9873-d9181d9e775b'})) (threadPool:208)
2022-10-25 16:39:54,663+0800 INFO (mailbox-hsm/3) [vdsm.api] START
refreshVolume(sdUUID='107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',
spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef',
imgUUID='2d1375ba-ce52-4da1-9873-d9181d9e775b',
volUUID='581f65d2-3db5-4fde-960f-4c4ffe27341e') from=internal,
task_id=918b7e3f-d730-4226-a941-fc0e60ee36b0 (api:48)
2022-10-25 16:39:54,809+0800 WARN (mailbox-hsm/3) [storage.LVM] Command
['/sbin/lvm', 'vgs', '--config', 'devices {
preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
write_cache_state=0 disable_after_error_count=3
filter=["a|^/dev/mapper/3600c0ff00051441d1e145f5e01000000$|^/dev/mapper/3600c0ff00051441de4f6555e01000000$|^/dev/mapper/364cd98f0bb99140025dd822d083dd183$|^/dev/mapper/364cd98f0bb99140025dd82520a7280ae$|",
"r|.*|"] } global { locking_type=4 prioritise_write_locks=1 wait_for_locks=1
use_lvmetad=0 } backup { retain_min=50 retain_days=0 }', '--noheadings',
'--units', 'b', '--nosuffix', '--separator', '|',
'--ignoreskippedcluster', '-o',
'uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name',
'--select', 'vg_name = 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19'] succeeded
with warnings: [' /dev/mapper/3600c0ff00051441d1e145f5e01000000: Checksum error at
offset 30099960109568', " Couldn't read volume
group metadata from /dev/mapper/3600c0ff00051441d1e145f5e01000000.", ' Metadata
location on /dev/mapper/3600c0ff00051441d1e145f5e01000000 at 30099960109568 has invalid
summary for VG.', ' Failed to read metadata summary from
/dev/mapper/3600c0ff00051441d1e145f5e01000000', ' Failed to scan VG from
/dev/mapper/3600c0ff00051441d1e145f5e01000000'] (lvm:466)
2022-10-25 16:39:54,824+0800 INFO (mailbox-hsm/3) [vdsm.api] FINISH refreshVolume
error=Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)
from=internal, task_id=918b7e3f-d730-4226-a941-fc0e60ee36b0 (api:52)
2022-10-25 16:39:54,824+0800 ERROR (mailbox-hsm/3) [storage.TaskManager.Task]
(Task='918b7e3f-d730-4226-a941-fc0e60ee36b0') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run
return fn(*args, **kargs)
File "<string>", line 2, in refreshVolume
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3033, in
refreshVolume
sdUUID=sdUUID).produceVolume(imgUUID=imgUUID,
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in
produce
domain.getRealDomain()
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in
getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in
_realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in
_findDomain
return findMethod(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1843, in
findDomain
return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1701, in
findDomainPath
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)
2022-10-25 16:39:54,825+0800 INFO (mailbox-hsm/3) [storage.TaskManager.Task]
(Task='918b7e3f-d730-4226-a941-fc0e60ee36b0') aborting: Task is aborted:
"Storage domain does not exist:
('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)" - code 358 (task:1181)
2022-10-25 16:39:54,825+0800 ERROR (mailbox-hsm/3) [storage.Dispatcher] FINISH
refreshVolume error=Storage domain does not exist:
('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) (dispatcher:83)
2022-10-25 16:39:54,826+0800 INFO (mailbox-hsm/3) [vdsm.api] START
getVolumeSize(sdUUID='107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',
spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef',
imgUUID='2d1375ba-ce52-4da1-9873-d9181d9e775b',
volUUID='581f65d2-3db5-4fde-960f-4c4ffe27341e', options=None) from=internal,
task_id=c5a7364d-3b39-4e12-b75b-22d5c4487cad (api:48)
2022-10-25 16:39:54,970+0800 WARN (mailbox-hsm/3) [storage.LVM] Command
['/sbin/lvm', 'vgs', '--config', 'devices {
preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
write_cache_state=0 disable_after_error_count=3
filter=["a|^/dev/mapper/3600c0ff00051441d1e145f5e01000000$|^/dev/mapper/3600c0ff00051441de4f6555e01000000$|^/dev/mapper/364cd98f0bb99140025dd822d083dd183$|^/dev/mapper/364cd98f0bb99140025dd82520a7280ae$|",
"r|.*|"] } global { locking_type=4 prioritise_write_locks=1 wait_for_locks=1
use_lvmetad=0 } backup { retain_min=50 retain_days=0 }', '--noheadings',
'--units', 'b', '--nosuffix', '--separator', '|',
'--ignoreskippedcluster', '-o',
'uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name',
'--select', 'vg_name = 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19'] succeeded
with warnings: [' /dev/mapper/3600c0ff00051441d1e145f5e01000000: Checksum error at
offset 30099960109568', " Couldn't read volume
group metadata from /dev/mapper/3600c0ff00051441d1e145f5e01000000.", ' Metadata
location on /dev/mapper/3600c0ff00051441d1e145f5e01000000 at 30099960109568 has invalid
summary for VG.', ' Failed to read metadata summary from
/dev/mapper/3600c0ff00051441d1e145f5e01000000', ' Failed to scan VG from
/dev/mapper/3600c0ff00051441d1e145f5e01000000'] (lvm:466)
2022-10-25 16:39:54,983+0800 INFO (mailbox-hsm/3) [vdsm.api] FINISH getVolumeSize
error=Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)
from=internal, task_id=c5a7364d-3b39-4e12-b75b-22d5c4487cad (api:52)
2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.TaskManager.Task]
(Task='c5a7364d-3b39-4e12-b75b-22d5c4487cad') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run
return fn(*args, **kargs)
File "<string>", line 2, in getVolumeSize
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3071, in
getVolumeSize
dom = sdCache.produce(sdUUID=sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in
produce
domain.getRealDomain()
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in
getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in
_realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in
_findDomain
return findMethod(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1843, in
findDomain
return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1701, in
findDomainPath
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)
2022-10-25 16:39:54,983+0800 INFO (mailbox-hsm/3) [storage.TaskManager.Task]
(Task='c5a7364d-3b39-4e12-b75b-22d5c4487cad') aborting: Task is aborted:
"Storage domain does not exist:
('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)" - code 358 (task:1181)
2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.Dispatcher] FINISH
getVolumeSize error=Storage domain does not exist:
('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) (dispatcher:83)
2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.TaskManager.Task]
(Task='033cec9a-5b65-4413-97c3-a273aafa6114') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run
return fn(*args, **kargs)
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1407, in
__afterVolumeExtension
volSize = self.__verifyVolumeExtension(volInfo)
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1325, in
__verifyVolumeExtension
volInfo['imageID'], volInfo['volumeID'])
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 6147, in
_getVolumeSize
(domainID, volumeID))
StorageUnavailableError: Unable to get volume size for domain
107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume 581f65d2-3db5-4fde-960f-4c4ffe27341e
2022-10-25 16:39:54,983+0800 INFO (mailbox-hsm/3) [storage.TaskManager.Task]
(Task='033cec9a-5b65-4413-97c3-a273aafa6114') aborting: Task is aborted:
u'Unable to get volume size for domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume
581f65d2-3db5-4fde-960f-4c4ffe27341e' - code 100 (task:1181)
2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.ThreadPool.WorkerThread]
FINISH task b81cf2c1-c795-4f97-9c5a-6b06a6a3173c failed (callback=None, cmd=<function
runTask at 0x7fea630c8500>, args=(<bound method Vm.__afterVolumeExtension of
<vdsm.virt.vm.Vm object at 0x7fea007336d0>>, {'newSize': 92341796864,
'domainID': '107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name':
'vda', 'poolID': '81f9e45a-5879-11ea-a3bd-00163e27dfef',
'clock': <Clock(total=3.20*, extend-volume=2.88, refresh-volume=0.16)>,
'internal': False, 'volumeID':
'581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID':
'2d1375ba-ce52-4da1-9873-d9181d9e775b'})) (threadPool:222)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/threadPool.py", line 209,
in _processNextTask
cmd(args)
File "/usr/lib/python2.7/site-packages/vdsm/storage/mailbox.py", line 85, in
runTask
ctask.prepare(cmd, *args)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in
wrapper
return m(self, *a, **kw)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in
prepare
raise self.error
StorageUnavailableError: Unable to get volume size for domain
107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume 581f65d2-3db5-4fde-960f-4c4ffe27341e
^C
This is what I get from vdsm when I ran tail -f.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/QRSLEBLJUE5...