3:40 a.m.
Hi Tomas,
[root@host1 vdsm]# tail -f vdsm.log
StorageUnavailableError: Unable to get volume size for domain
107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume 581f65d2-3db5-4fde-960f-4c4ffe27341e
2022-10-25 16:39:51,785+0800 INFO (periodic/3) [virt.vm]
(vmId='a8d6fcaf-a673-46df-b00a-925a8b9791ef') Requesting extension for volume
581f65d2-3db5-4fde-960f-4c4ffe27341e on domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19
(apparent: 91268055040, capacity: 322122547200, allocated: 90872676352, physical:
91268055040 threshold_state: exceeded) (vm:1289)
2022-10-25 16:39:51,787+0800 INFO (periodic/3) [vdsm.api] START
sendExtendMsg(spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef',
volDict={'newSize': 92341796864, 'domainID':
'107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda',
'poolID': '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock':
<Clock(total=0.00*, extend-volume=0.00*)>, 'internal': False,
'volumeID': '581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID':
'2d1375ba-ce52-4da1-9873-d9181d9e775b'}, newSize=92341796864,
callbackFunc=<bound method Vm.__afterVolumeExtension of <vdsm.virt.vm.Vm object at
0x7fea007336d0>>) from=internal, task_id=2b0c7a67-0f31-4f62-af70-a29e0b84dafb
(api:48)
2022-10-25 16:39:51,787+0800 INFO (periodic/3) [vdsm.api] FINISH sendExtendMsg
return=None from=internal, task_id=2b0c7a67-0f31-4f62-af70-a29e0b84dafb (api:54)
2022-10-25 16:39:52,635+0800 INFO (mailbox-hsm) [storage.MailBox.HsmMailMonitor]
HSM_MailMonitor sending mail to SPM - ['/usr/bin/dd',
'of=/rhev/data-center/81f9e45a-5879-11ea-a3bd-00163e27dfef/mastersd/dom_md/inbox',
'iflag=fullblock', 'oflag=direct', 'conv=notrunc',
'bs=4096', 'count=1', 'seek=1'] (mailbox:380)
2022-10-25 16:39:52,710+0800 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.ping2
succeeded in 0.00 seconds (__init__:312)
2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [api.virt] START getStats() from=::1,39184,
vmId=6579a9c4-d55e-4d61-9c3a-c745cdcfa430 (api:48)
2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'6579a9c4-d55e-4d61-9c3a-c745cdcfa430'}
(api:129)
2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [api.virt] FINISH getStats
return={'status': {'message': "Virtual machine does not exist:
{'vmId': u'6579a9c4-d55e-4d61-9c3a-c745cdcfa430'}", 'code':
1}} from=::1,39184, vmId=6579a9c4-d55e-4d61-9c3a-c745cdcfa430 (api:54)
2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2022-10-25 16:39:53,786+0800 INFO (periodic/5) [virt.vm]
(vmId='a8d6fcaf-a673-46df-b00a-925a8b9791ef') Requesting extension for volume
581f65d2-3db5-4fde-960f-4c4ffe27341e on domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19
(apparent: 91268055040, capacity: 322122547200, allocated: 90872676352, physical:
91268055040 threshold_state: exceeded) (vm:1289)
2022-10-25 16:39:53,787+0800 INFO (periodic/5) [vdsm.api] START
sendExtendMsg(spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef',
volDict={'newSize': 92341796864, 'domainID':
'107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda',
'poolID': '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock':
<Clock(total=0.00*, extend-volume=0.00*)>, 'internal': False,
'volumeID': '581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID':
'2d1375ba-ce52-4da1-9873-d9181d9e775b'}, newSize=92341796864,
callbackFunc=<bound method Vm.__afterVolumeExtension of <vdsm.virt.vm.Vm object at
0x7fea007336d0>>) from=internal, task_id=197234ba-5cab-4098-86c8-7f2c979171d2
(api:48)
2022-10-25 16:39:53,787+0800 INFO (periodic/5) [vdsm.api] FINISH sendExtendMsg
return=None from=internal, task_id=197234ba-5cab-4098-86c8-7f2c979171d2 (api:54)
2022-10-25 16:39:54,167+0800 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.ping2
succeeded in 0.00 seconds (__init__:312)
2022-10-25 16:39:54,658+0800 INFO (mailbox-hsm) [storage.MailBox.HsmMailMonitor]
HSM_MailMonitor sending mail to SPM - ['/usr/bin/dd',
'of=/rhev/data-center/81f9e45a-5879-11ea-a3bd-00163e27dfef/mastersd/dom_md/inbox',
'iflag=fullblock', 'oflag=direct', 'conv=notrunc',
'bs=4096', 'count=1', 'seek=1'] (mailbox:380)
2022-10-25 16:39:54,661+0800 INFO (mailbox-hsm/3) [storage.ThreadPool.WorkerThread] START
task b81cf2c1-c795-4f97-9c5a-6b06a6a3173c (cmd=<function runTask at 0x7fea630c8500>,
args=(<bound method Vm.__afterVolumeExtension of <vdsm.virt.vm.Vm object at
0x7fea007336d0>>, {'newSize': 92341796864, 'domainID':
'107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda',
'poolID': '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock':
<Clock(total=2.88*, extend-volume=2.88, refresh-volume=0.00*)>, 'internal':
False, 'volumeID': '581f65d2-3db5-4fde-960f-4c4ffe27341e',
'imageID': '2d1375ba-ce52-4da1-9873-d9181d9e775b'})) (threadPool:208)
2022-10-25 16:39:54,663+0800 INFO (mailbox-hsm/3) [vdsm.api] START
refreshVolume(sdUUID='107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',
spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef',
imgUUID='2d1375ba-ce52-4da1-9873-d9181d9e775b',
volUUID='581f65d2-3db5-4fde-960f-4c4ffe27341e') from=internal,
task_id=918b7e3f-d730-4226-a941-fc0e60ee36b0 (api:48)
2022-10-25 16:39:54,809+0800 WARN (mailbox-hsm/3) [storage.LVM] Command
['/sbin/lvm', 'vgs', '--config', 'devices {
preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
write_cache_state=0 disable_after_error_count=3
filter=["a|^/dev/mapper/3600c0ff00051441d1e145f5e01000000$|^/dev/mapper/3600c0ff00051441de4f6555e01000000$|^/dev/mapper/364cd98f0bb99140025dd822d083dd183$|^/dev/mapper/364cd98f0bb99140025dd82520a7280ae$|",
"r|.*|"] } global { locking_type=4 prioritise_write_locks=1 wait_for_locks=1
use_lvmetad=0 } backup { retain_min=50 retain_days=0 }', '--noheadings',
'--units', 'b', '--nosuffix', '--separator', '|',
'--ignoreskippedcluster', '-o',
'uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name',
'--select', 'vg_name = 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19'] succeeded
with warnings: [' /dev/mapper/3600c0ff00051441d1e145f5e01000000: Checksum error at
offset 30099960109568', " Couldn't read volume
group metadata from /dev/mapper/3600c0ff00051441d1e145f5e01000000.", ' Metadata
location on /dev/mapper/3600c0ff00051441d1e145f5e01000000 at 30099960109568 has invalid
summary for VG.', ' Failed to read metadata summary from
/dev/mapper/3600c0ff00051441d1e145f5e01000000', ' Failed to scan VG from
/dev/mapper/3600c0ff00051441d1e145f5e01000000'] (lvm:466)
2022-10-25 16:39:54,824+0800 INFO (mailbox-hsm/3) [vdsm.api] FINISH refreshVolume
error=Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)
from=internal, task_id=918b7e3f-d730-4226-a941-fc0e60ee36b0 (api:52)
2022-10-25 16:39:54,824+0800 ERROR (mailbox-hsm/3) [storage.TaskManager.Task]
(Task='918b7e3f-d730-4226-a941-fc0e60ee36b0') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run
return fn(*args, **kargs)
File "<string>", line 2, in refreshVolume
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3033, in
refreshVolume
sdUUID=sdUUID).produceVolume(imgUUID=imgUUID,
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in
produce
domain.getRealDomain()
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in
getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in
_realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in
_findDomain
return findMethod(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1843, in
findDomain
return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1701, in
findDomainPath
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)
2022-10-25 16:39:54,825+0800 INFO (mailbox-hsm/3) [storage.TaskManager.Task]
(Task='918b7e3f-d730-4226-a941-fc0e60ee36b0') aborting: Task is aborted:
"Storage domain does not exist:
('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)" - code 358 (task:1181)
2022-10-25 16:39:54,825+0800 ERROR (mailbox-hsm/3) [storage.Dispatcher] FINISH
refreshVolume error=Storage domain does not exist:
('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) (dispatcher:83)
2022-10-25 16:39:54,826+0800 INFO (mailbox-hsm/3) [vdsm.api] START
getVolumeSize(sdUUID='107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',
spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef',
imgUUID='2d1375ba-ce52-4da1-9873-d9181d9e775b',
volUUID='581f65d2-3db5-4fde-960f-4c4ffe27341e', options=None) from=internal,
task_id=c5a7364d-3b39-4e12-b75b-22d5c4487cad (api:48)
2022-10-25 16:39:54,970+0800 WARN (mailbox-hsm/3) [storage.LVM] Command
['/sbin/lvm', 'vgs', '--config', 'devices {
preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
write_cache_state=0 disable_after_error_count=3
filter=["a|^/dev/mapper/3600c0ff00051441d1e145f5e01000000$|^/dev/mapper/3600c0ff00051441de4f6555e01000000$|^/dev/mapper/364cd98f0bb99140025dd822d083dd183$|^/dev/mapper/364cd98f0bb99140025dd82520a7280ae$|",
"r|.*|"] } global { locking_type=4 prioritise_write_locks=1 wait_for_locks=1
use_lvmetad=0 } backup { retain_min=50 retain_days=0 }', '--noheadings',
'--units', 'b', '--nosuffix', '--separator', '|',
'--ignoreskippedcluster', '-o',
'uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name',
'--select', 'vg_name = 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19'] succeeded
with warnings: [' /dev/mapper/3600c0ff00051441d1e145f5e01000000: Checksum error at
offset 30099960109568', " Couldn't read volume
group metadata from /dev/mapper/3600c0ff00051441d1e145f5e01000000.", ' Metadata
location on /dev/mapper/3600c0ff00051441d1e145f5e01000000 at 30099960109568 has invalid
summary for VG.', ' Failed to read metadata summary from
/dev/mapper/3600c0ff00051441d1e145f5e01000000', ' Failed to scan VG from
/dev/mapper/3600c0ff00051441d1e145f5e01000000'] (lvm:466)
2022-10-25 16:39:54,983+0800 INFO (mailbox-hsm/3) [vdsm.api] FINISH getVolumeSize
error=Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)
from=internal, task_id=c5a7364d-3b39-4e12-b75b-22d5c4487cad (api:52)
2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.TaskManager.Task]
(Task='c5a7364d-3b39-4e12-b75b-22d5c4487cad') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run
return fn(*args, **kargs)
File "<string>", line 2, in getVolumeSize
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3071, in
getVolumeSize
dom = sdCache.produce(sdUUID=sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in
produce
domain.getRealDomain()
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in
getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in
_realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in
_findDomain
return findMethod(sdUUID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1843, in
findDomain
return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1701, in
findDomainPath
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)
2022-10-25 16:39:54,983+0800 INFO (mailbox-hsm/3) [storage.TaskManager.Task]
(Task='c5a7364d-3b39-4e12-b75b-22d5c4487cad') aborting: Task is aborted:
"Storage domain does not exist:
('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)" - code 358 (task:1181)
2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.Dispatcher] FINISH
getVolumeSize error=Storage domain does not exist:
('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) (dispatcher:83)
2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.TaskManager.Task]
(Task='033cec9a-5b65-4413-97c3-a273aafa6114') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run
return fn(*args, **kargs)
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1407, in
__afterVolumeExtension
volSize = self.__verifyVolumeExtension(volInfo)
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1325, in
__verifyVolumeExtension
volInfo['imageID'], volInfo['volumeID'])
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 6147, in
_getVolumeSize
(domainID, volumeID))
StorageUnavailableError: Unable to get volume size for domain
107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume 581f65d2-3db5-4fde-960f-4c4ffe27341e
2022-10-25 16:39:54,983+0800 INFO (mailbox-hsm/3) [storage.TaskManager.Task]
(Task='033cec9a-5b65-4413-97c3-a273aafa6114') aborting: Task is aborted:
u'Unable to get volume size for domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume
581f65d2-3db5-4fde-960f-4c4ffe27341e' - code 100 (task:1181)
2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.ThreadPool.WorkerThread]
FINISH task b81cf2c1-c795-4f97-9c5a-6b06a6a3173c failed (callback=None, cmd=<function
runTask at 0x7fea630c8500>, args=(<bound method Vm.__afterVolumeExtension of
<vdsm.virt.vm.Vm object at 0x7fea007336d0>>, {'newSize': 92341796864,
'domainID': '107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name':
'vda', 'poolID': '81f9e45a-5879-11ea-a3bd-00163e27dfef',
'clock': <Clock(total=3.20*, extend-volume=2.88, refresh-volume=0.16)>,
'internal': False, 'volumeID':
'581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID':
'2d1375ba-ce52-4da1-9873-d9181d9e775b'})) (threadPool:222)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/threadPool.py", line 209,
in _processNextTask
cmd(args)
File "/usr/lib/python2.7/site-packages/vdsm/storage/mailbox.py", line 85, in
runTask
ctask.prepare(cmd, *args)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in
wrapper
return m(self, *a, **kw)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in
prepare
raise self.error
StorageUnavailableError: Unable to get volume size for domain
107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume 581f65d2-3db5-4fde-960f-4c4ffe27341e
^C
This is what I get from vdsm when I ran tail -f.