URGENT HELP NEEDED!! Hosts are in non-operational state.
Oct 25, 2022, 4:10:01 PM - Failed to connect Host <hostname> to Storage Pool <storage pool> Oct 25, 2022, 4:10:01 PM - Host <hostname> cannot access the Storage Domain(s) VM_Data attached to the <data center>. Setting Host state to Non-Operational. Oct 25, 2022, 4:10:01 PM - Host <hostname> reports about one of the Active Storage Domains as Problematic. Oct 25, 2022, 4:09:52 PM - Activation of host <hostname> initiated by <user>. Hi All, Today morning, noticed that my 1st and 2nd host is in non-operational status and saw that my storage pool isn't being mounted and whenever I try to press Activate under Management, the above error happens. I have multiple VM running and I can't bring it up if it goes down, and it is my production server for my company. Any advise on how to check on this or what logs to look & provide to resolve this? The person who deployed this setup is no longer in my company and I have no one else to seek help for this issue.
On Tue, Oct 25, 2022 at 08:16:56AM -0000, eugene@knorydev.com wrote:
Oct 25, 2022, 4:10:01 PM - Failed to connect Host <hostname> to Storage Pool <storage pool> Oct 25, 2022, 4:10:01 PM - Host <hostname> cannot access the Storage Domain(s) VM_Data attached to the <data center>. Setting Host state to Non-Operational. Oct 25, 2022, 4:10:01 PM - Host <hostname> reports about one of the Active Storage Domains as Problematic. Oct 25, 2022, 4:09:52 PM - Activation of host <hostname> initiated by <user>. Hi All,
Today morning, noticed that my 1st and 2nd host is in non-operational status and saw that my storage pool isn't being mounted and whenever I try to press Activate under Management, the above error happens. I have multiple VM running and I can't bring it up if it goes down, and it is my production server for my company.
Any advise on how to check on this or what logs to look & provide to resolve this?
Start by logging into (one of) the failing host and check logs in /var/log/vdsm and journal. This should give you more clues. Tomas
The person who deployed this setup is no longer in my company and I have no one else to seek help for this issue. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/PRNQDIWMISQ4PV...
Hi Tomas, [root@host1 vdsm]# tail -f vdsm.log StorageUnavailableError: Unable to get volume size for domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume 581f65d2-3db5-4fde-960f-4c4ffe27341e 2022-10-25 16:39:51,785+0800 INFO (periodic/3) [virt.vm] (vmId='a8d6fcaf-a673-46df-b00a-925a8b9791ef') Requesting extension for volume 581f65d2-3db5-4fde-960f-4c4ffe27341e on domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 (apparent: 91268055040, capacity: 322122547200, allocated: 90872676352, physical: 91268055040 threshold_state: exceeded) (vm:1289) 2022-10-25 16:39:51,787+0800 INFO (periodic/3) [vdsm.api] START sendExtendMsg(spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef', volDict={'newSize': 92341796864, 'domainID': '107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda', 'poolID': '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock': <Clock(total=0.00*, extend-volume=0.00*)>, 'internal': False, 'volumeID': '581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID': '2d1375ba-ce52-4da1-9873-d9181d9e775b'}, newSize=92341796864, callbackFunc=<bound method Vm.__afterVolumeExtension of <vdsm.virt.vm.Vm object at 0x7fea007336d0>>) from=internal, task_id=2b0c7a67-0f31-4f62-af70-a29e0b84dafb (api:48) 2022-10-25 16:39:51,787+0800 INFO (periodic/3) [vdsm.api] FINISH sendExtendMsg return=None from=internal, task_id=2b0c7a67-0f31-4f62-af70-a29e0b84dafb (api:54) 2022-10-25 16:39:52,635+0800 INFO (mailbox-hsm) [storage.MailBox.HsmMailMonitor] HSM_MailMonitor sending mail to SPM - ['/usr/bin/dd', 'of=/rhev/data-center/81f9e45a-5879-11ea-a3bd-00163e27dfef/mastersd/dom_md/inbox', 'iflag=fullblock', 'oflag=direct', 'conv=notrunc', 'bs=4096', 'count=1', 'seek=1'] (mailbox:380) 2022-10-25 16:39:52,710+0800 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:312) 2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [api.virt] START getStats() from=::1,39184, vmId=6579a9c4-d55e-4d61-9c3a-c745cdcfa430 (api:48) 2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'6579a9c4-d55e-4d61-9c3a-c745cdcfa430'} (api:129) 2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [api.virt] FINISH getStats return={'status': {'message': "Virtual machine does not exist: {'vmId': u'6579a9c4-d55e-4d61-9c3a-c745cdcfa430'}", 'code': 1}} from=::1,39184, vmId=6579a9c4-d55e-4d61-9c3a-c745cdcfa430 (api:54) 2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2022-10-25 16:39:53,786+0800 INFO (periodic/5) [virt.vm] (vmId='a8d6fcaf-a673-46df-b00a-925a8b9791ef') Requesting extension for volume 581f65d2-3db5-4fde-960f-4c4ffe27341e on domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 (apparent: 91268055040, capacity: 322122547200, allocated: 90872676352, physical: 91268055040 threshold_state: exceeded) (vm:1289) 2022-10-25 16:39:53,787+0800 INFO (periodic/5) [vdsm.api] START sendExtendMsg(spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef', volDict={'newSize': 92341796864, 'domainID': '107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda', 'poolID': '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock': <Clock(total=0.00*, extend-volume=0.00*)>, 'internal': False, 'volumeID': '581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID': '2d1375ba-ce52-4da1-9873-d9181d9e775b'}, newSize=92341796864, callbackFunc=<bound method Vm.__afterVolumeExtension of <vdsm.virt.vm.Vm object at 0x7fea007336d0>>) from=internal, task_id=197234ba-5cab-4098-86c8-7f2c979171d2 (api:48) 2022-10-25 16:39:53,787+0800 INFO (periodic/5) [vdsm.api] FINISH sendExtendMsg return=None from=internal, task_id=197234ba-5cab-4098-86c8-7f2c979171d2 (api:54) 2022-10-25 16:39:54,167+0800 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:312) 2022-10-25 16:39:54,658+0800 INFO (mailbox-hsm) [storage.MailBox.HsmMailMonitor] HSM_MailMonitor sending mail to SPM - ['/usr/bin/dd', 'of=/rhev/data-center/81f9e45a-5879-11ea-a3bd-00163e27dfef/mastersd/dom_md/inbox', 'iflag=fullblock', 'oflag=direct', 'conv=notrunc', 'bs=4096', 'count=1', 'seek=1'] (mailbox:380) 2022-10-25 16:39:54,661+0800 INFO (mailbox-hsm/3) [storage.ThreadPool.WorkerThread] START task b81cf2c1-c795-4f97-9c5a-6b06a6a3173c (cmd=<function runTask at 0x7fea630c8500>, args=(<bound method Vm.__afterVolumeExtension of <vdsm.virt.vm.Vm object at 0x7fea007336d0>>, {'newSize': 92341796864, 'domainID': '107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda', 'poolID': '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock': <Clock(total=2.88*, extend-volume=2.88, refresh-volume=0.00*)>, 'internal': False, 'volumeID': '581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID': '2d1375ba-ce52-4da1-9873-d9181d9e775b'})) (threadPool:208) 2022-10-25 16:39:54,663+0800 INFO (mailbox-hsm/3) [vdsm.api] START refreshVolume(sdUUID='107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef', imgUUID='2d1375ba-ce52-4da1-9873-d9181d9e775b', volUUID='581f65d2-3db5-4fde-960f-4c4ffe27341e') from=internal, task_id=918b7e3f-d730-4226-a941-fc0e60ee36b0 (api:48) 2022-10-25 16:39:54,809+0800 WARN (mailbox-hsm/3) [storage.LVM] Command ['/sbin/lvm', 'vgs', '--config', 'devices { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter=["a|^/dev/mapper/3600c0ff00051441d1e145f5e01000000$|^/dev/mapper/3600c0ff00051441de4f6555e01000000$|^/dev/mapper/364cd98f0bb99140025dd822d083dd183$|^/dev/mapper/364cd98f0bb99140025dd82520a7280ae$|", "r|.*|"] } global { locking_type=4 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min=50 retain_days=0 }', '--noheadings', '--units', 'b', '--nosuffix', '--separator', '|', '--ignoreskippedcluster', '-o', 'uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name', '--select', 'vg_name = 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19'] succeeded with warnings: [' /dev/mapper/3600c0ff00051441d1e145f5e01000000: Checksum error at offset 30099960109568', " Couldn't read volume group metadata from /dev/mapper/3600c0ff00051441d1e145f5e01000000.", ' Metadata location on /dev/mapper/3600c0ff00051441d1e145f5e01000000 at 30099960109568 has invalid summary for VG.', ' Failed to read metadata summary from /dev/mapper/3600c0ff00051441d1e145f5e01000000', ' Failed to scan VG from /dev/mapper/3600c0ff00051441d1e145f5e01000000'] (lvm:466) 2022-10-25 16:39:54,824+0800 INFO (mailbox-hsm/3) [vdsm.api] FINISH refreshVolume error=Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) from=internal, task_id=918b7e3f-d730-4226-a941-fc0e60ee36b0 (api:52) 2022-10-25 16:39:54,824+0800 ERROR (mailbox-hsm/3) [storage.TaskManager.Task] (Task='918b7e3f-d730-4226-a941-fc0e60ee36b0') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in refreshVolume File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3033, in refreshVolume sdUUID=sdUUID).produceVolume(imgUUID=imgUUID, File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1843, in findDomain return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID)) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1701, in findDomainPath raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) 2022-10-25 16:39:54,825+0800 INFO (mailbox-hsm/3) [storage.TaskManager.Task] (Task='918b7e3f-d730-4226-a941-fc0e60ee36b0') aborting: Task is aborted: "Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)" - code 358 (task:1181) 2022-10-25 16:39:54,825+0800 ERROR (mailbox-hsm/3) [storage.Dispatcher] FINISH refreshVolume error=Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) (dispatcher:83) 2022-10-25 16:39:54,826+0800 INFO (mailbox-hsm/3) [vdsm.api] START getVolumeSize(sdUUID='107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef', imgUUID='2d1375ba-ce52-4da1-9873-d9181d9e775b', volUUID='581f65d2-3db5-4fde-960f-4c4ffe27341e', options=None) from=internal, task_id=c5a7364d-3b39-4e12-b75b-22d5c4487cad (api:48) 2022-10-25 16:39:54,970+0800 WARN (mailbox-hsm/3) [storage.LVM] Command ['/sbin/lvm', 'vgs', '--config', 'devices { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter=["a|^/dev/mapper/3600c0ff00051441d1e145f5e01000000$|^/dev/mapper/3600c0ff00051441de4f6555e01000000$|^/dev/mapper/364cd98f0bb99140025dd822d083dd183$|^/dev/mapper/364cd98f0bb99140025dd82520a7280ae$|", "r|.*|"] } global { locking_type=4 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min=50 retain_days=0 }', '--noheadings', '--units', 'b', '--nosuffix', '--separator', '|', '--ignoreskippedcluster', '-o', 'uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name', '--select', 'vg_name = 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19'] succeeded with warnings: [' /dev/mapper/3600c0ff00051441d1e145f5e01000000: Checksum error at offset 30099960109568', " Couldn't read volume group metadata from /dev/mapper/3600c0ff00051441d1e145f5e01000000.", ' Metadata location on /dev/mapper/3600c0ff00051441d1e145f5e01000000 at 30099960109568 has invalid summary for VG.', ' Failed to read metadata summary from /dev/mapper/3600c0ff00051441d1e145f5e01000000', ' Failed to scan VG from /dev/mapper/3600c0ff00051441d1e145f5e01000000'] (lvm:466) 2022-10-25 16:39:54,983+0800 INFO (mailbox-hsm/3) [vdsm.api] FINISH getVolumeSize error=Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) from=internal, task_id=c5a7364d-3b39-4e12-b75b-22d5c4487cad (api:52) 2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.TaskManager.Task] (Task='c5a7364d-3b39-4e12-b75b-22d5c4487cad') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in getVolumeSize File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3071, in getVolumeSize dom = sdCache.produce(sdUUID=sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1843, in findDomain return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID)) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1701, in findDomainPath raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) 2022-10-25 16:39:54,983+0800 INFO (mailbox-hsm/3) [storage.TaskManager.Task] (Task='c5a7364d-3b39-4e12-b75b-22d5c4487cad') aborting: Task is aborted: "Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)" - code 358 (task:1181) 2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.Dispatcher] FINISH getVolumeSize error=Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) (dispatcher:83) 2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.TaskManager.Task] (Task='033cec9a-5b65-4413-97c3-a273aafa6114') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1407, in __afterVolumeExtension volSize = self.__verifyVolumeExtension(volInfo) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1325, in __verifyVolumeExtension volInfo['imageID'], volInfo['volumeID']) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 6147, in _getVolumeSize (domainID, volumeID)) StorageUnavailableError: Unable to get volume size for domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume 581f65d2-3db5-4fde-960f-4c4ffe27341e 2022-10-25 16:39:54,983+0800 INFO (mailbox-hsm/3) [storage.TaskManager.Task] (Task='033cec9a-5b65-4413-97c3-a273aafa6114') aborting: Task is aborted: u'Unable to get volume size for domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume 581f65d2-3db5-4fde-960f-4c4ffe27341e' - code 100 (task:1181) 2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.ThreadPool.WorkerThread] FINISH task b81cf2c1-c795-4f97-9c5a-6b06a6a3173c failed (callback=None, cmd=<function runTask at 0x7fea630c8500>, args=(<bound method Vm.__afterVolumeExtension of <vdsm.virt.vm.Vm object at 0x7fea007336d0>>, {'newSize': 92341796864, 'domainID': '107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda', 'poolID': '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock': <Clock(total=3.20*, extend-volume=2.88, refresh-volume=0.16)>, 'internal': False, 'volumeID': '581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID': '2d1375ba-ce52-4da1-9873-d9181d9e775b'})) (threadPool:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/threadPool.py", line 209, in _processNextTask cmd(args) File "/usr/lib/python2.7/site-packages/vdsm/storage/mailbox.py", line 85, in runTask ctask.prepare(cmd, *args) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error StorageUnavailableError: Unable to get volume size for domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume 581f65d2-3db5-4fde-960f-4c4ffe27341e ^C This is what I get from vdsm when I ran tail -f.
Both the initial messages and the additional detail in these vdsm logs show that it can’t mount a storage volume. In a multi node environment, it’s usually iscsi, gluster, etc, so start by looking at your network and the storage devices and figuring out why it can’t mount them. Comparison to any remaining running nodes may be helpful.
On Oct 25, 2022, at 3:40 AM, eugene@knorydev.com wrote:
Hi Tomas,
[root@host1 vdsm]# tail -f vdsm.log StorageUnavailableError: Unable to get volume size for domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume 581f65d2-3db5-4fde-960f-4c4ffe27341e 2022-10-25 16:39:51,785+0800 INFO (periodic/3) [virt.vm] (vmId='a8d6fcaf-a673-46df-b00a-925a8b9791ef') Requesting extension for volume 581f65d2-3db5-4fde-960f-4c4ffe27341e on domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 (apparent: 91268055040, capacity: 322122547200, allocated: 90872676352, physical: 91268055040 threshold_state: exceeded) (vm:1289) 2022-10-25 16:39:51,787+0800 INFO (periodic/3) [vdsm.api] START sendExtendMsg(spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef', volDict={'newSize': 92341796864, 'domainID': '107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda', 'poolID': '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock': <Clock(total=0.00*, extend-volume=0.00*)>, 'internal': False, 'volumeID': '581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID': '2d1375ba-ce52-4da1-9873-d9181d9e775b'}, newSize=92341796864, callbackFunc=<bound method Vm.__afterVolumeExtension of <vdsm.virt.vm.Vm object at 0x7fea007336d0>>) from=internal, task_id=2b0c7a67-0f31-4f62-af70-a29e0b84dafb (api:48) 2022-10-25 16:39:51,787+0800 INFO (periodic/3) [vdsm.api] FINISH sendExtendMsg return=None from=internal, task_id=2b0c7a67-0f31-4f62-af70-a29e0b84dafb (api:54) 2022-10-25 16:39:52,635+0800 INFO (mailbox-hsm) [storage.MailBox.HsmMailMonitor] HSM_MailMonitor sending mail to SPM - ['/usr/bin/dd', 'of=/rhev/data-center/81f9e45a-5879-11ea-a3bd-00163e27dfef/mastersd/dom_md/inbox', 'iflag=fullblock', 'oflag=direct', 'conv=notrunc', 'bs=4096', 'count=1', 'seek=1'] (mailbox:380) 2022-10-25 16:39:52,710+0800 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:312) 2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [api.virt] START getStats() from=::1,39184, vmId=6579a9c4-d55e-4d61-9c3a-c745cdcfa430 (api:48) 2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'6579a9c4-d55e-4d61-9c3a-c745cdcfa430'} (api:129) 2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [api.virt] FINISH getStats return={'status': {'message': "Virtual machine does not exist: {'vmId': u'6579a9c4-d55e-4d61-9c3a-c745cdcfa430'}", 'code': 1}} from=::1,39184, vmId=6579a9c4-d55e-4d61-9c3a-c745cdcfa430 (api:54) 2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2022-10-25 16:39:53,786+0800 INFO (periodic/5) [virt.vm] (vmId='a8d6fcaf-a673-46df-b00a-925a8b9791ef') Requesting extension for volume 581f65d2-3db5-4fde-960f-4c4ffe27341e on domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 (apparent: 91268055040, capacity: 322122547200, allocated: 90872676352, physical: 91268055040 threshold_state: exceeded) (vm:1289) 2022-10-25 16:39:53,787+0800 INFO (periodic/5) [vdsm.api] START sendExtendMsg(spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef', volDict={'newSize': 92341796864, 'domainID': '107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda', 'poolID': '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock': <Clock(total=0.00*, extend-volume=0.00*)>, 'internal': False, 'volumeID': '581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID': '2d1375ba-ce52-4da1-9873-d9181d9e775b'}, newSize=92341796864, callbackFunc=<bound method Vm.__afterVolumeExtension of <vdsm.virt.vm.Vm object at 0x7fea007336d0>>) from=internal, task_id=197234ba-5cab-4098-86c8-7f2c979171d2 (api:48) 2022-10-25 16:39:53,787+0800 INFO (periodic/5) [vdsm.api] FINISH sendExtendMsg return=None from=internal, task_id=197234ba-5cab-4098-86c8-7f2c979171d2 (api:54) 2022-10-25 16:39:54,167+0800 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:312) 2022-10-25 16:39:54,658+0800 INFO (mailbox-hsm) [storage.MailBox.HsmMailMonitor] HSM_MailMonitor sending mail to SPM - ['/usr/bin/dd', 'of=/rhev/data-center/81f9e45a-5879-11ea-a3bd-00163e27dfef/mastersd/dom_md/inbox', 'iflag=fullblock', 'oflag=direct', 'conv=notrunc', 'bs=4096', 'count=1', 'seek=1'] (mailbox:380) 2022-10-25 16:39:54,661+0800 INFO (mailbox-hsm/3) [storage.ThreadPool.WorkerThread] START task b81cf2c1-c795-4f97-9c5a-6b06a6a3173c (cmd=<function runTask at 0x7fea630c8500>, args=(<bound method Vm.__afterVolumeExtension of <vdsm.virt.vm.Vm object at 0x7fea007336d0>>, {'newSize': 92341796864, 'domainID': '107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda', 'poolID': '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock': <Clock(total=2.88*, extend-volume=2.88, refresh-volume=0.00*)>, 'internal': False, 'volumeID': '581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID': '2d1375ba-ce52-4da1-9873-d9181d9e775b'})) (threadPool:208) 2022-10-25 16:39:54,663+0800 INFO (mailbox-hsm/3) [vdsm.api] START refreshVolume(sdUUID='107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef', imgUUID='2d1375ba-ce52-4da1-9873-d9181d9e775b', volUUID='581f65d2-3db5-4fde-960f-4c4ffe27341e') from=internal, task_id=918b7e3f-d730-4226-a941-fc0e60ee36b0 (api:48) 2022-10-25 16:39:54,809+0800 WARN (mailbox-hsm/3) [storage.LVM] Command ['/sbin/lvm', 'vgs', '--config', 'devices { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter=["a|^/dev/mapper/3600c0ff00051441d1e145f5e01000000$|^/dev/mapper/3600c0ff00051441de4f6555e01000000$|^/dev/mapper/364cd98f0bb99140025dd822d083dd183$|^/dev/mapper/364cd98f0bb99140025dd82520a7280ae$|", "r|.*|"] } global { locking_type=4 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min=50 retain_days=0 }', '--noheadings', '--units', 'b', '--nosuffix', '--separator', '|', '--ignoreskippedcluster', '-o', 'uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name', '--select', 'vg_name = 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19'] succeeded with warnings: [' /dev/mapper/3600c0ff00051441d1e145f5e01000000: Checksum error at offset 30099960109568', " Couldn't read volume group metadata from /dev/mapper/3600c0ff00051441d1e145f5e01000000.", ' Metadata location on /dev/mapper/3600c0ff00051441d1e145f5e01000000 at 30099960109568 has invalid summary for VG.', ' Failed to read metadata summary from /dev/mapper/3600c0ff00051441d1e145f5e01000000', ' Failed to scan VG from /dev/mapper/3600c0ff00051441d1e145f5e01000000'] (lvm:466) 2022-10-25 16:39:54,824+0800 INFO (mailbox-hsm/3) [vdsm.api] FINISH refreshVolume error=Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) from=internal, task_id=918b7e3f-d730-4226-a941-fc0e60ee36b0 (api:52) 2022-10-25 16:39:54,824+0800 ERROR (mailbox-hsm/3) [storage.TaskManager.Task] (Task='918b7e3f-d730-4226-a941-fc0e60ee36b0') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in refreshVolume File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3033, in refreshVolume sdUUID=sdUUID).produceVolume(imgUUID=imgUUID, File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1843, in findDomain return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID)) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1701, in findDomainPath raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) 2022-10-25 16:39:54,825+0800 INFO (mailbox-hsm/3) [storage.TaskManager.Task] (Task='918b7e3f-d730-4226-a941-fc0e60ee36b0') aborting: Task is aborted: "Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)" - code 358 (task:1181) 2022-10-25 16:39:54,825+0800 ERROR (mailbox-hsm/3) [storage.Dispatcher] FINISH refreshVolume error=Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) (dispatcher:83) 2022-10-25 16:39:54,826+0800 INFO (mailbox-hsm/3) [vdsm.api] START getVolumeSize(sdUUID='107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef', imgUUID='2d1375ba-ce52-4da1-9873-d9181d9e775b', volUUID='581f65d2-3db5-4fde-960f-4c4ffe27341e', options=None) from=internal, task_id=c5a7364d-3b39-4e12-b75b-22d5c4487cad (api:48) 2022-10-25 16:39:54,970+0800 WARN (mailbox-hsm/3) [storage.LVM] Command ['/sbin/lvm', 'vgs', '--config', 'devices { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter=["a|^/dev/mapper/3600c0ff00051441d1e145f5e01000000$|^/dev/mapper/3600c0ff00051441de4f6555e01000000$|^/dev/mapper/364cd98f0bb99140025dd822d083dd183$|^/dev/mapper/364cd98f0bb99140025dd82520a7280ae$|", "r|.*|"] } global { locking_type=4 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min=50 retain_days=0 }', '--noheadings', '--units', 'b', '--nosuffix', '--separator', '|', '--ignoreskippedcluster', '-o', 'uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name', '--select', 'vg_name = 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19'] succeeded with warnings: [' /dev/mapper/3600c0ff00051441d1e145f5e01000000: Checksum error at offset 30099960109568', " Couldn't read volume group metadata from /dev/mapper/3600c0ff00051441d1e145f5e01000000.", ' Metadata location on /dev/mapper/3600c0ff00051441d1e145f5e01000000 at 30099960109568 has invalid summary for VG.', ' Failed to read metadata summary from /dev/mapper/3600c0ff00051441d1e145f5e01000000', ' Failed to scan VG from /dev/mapper/3600c0ff00051441d1e145f5e01000000'] (lvm:466) 2022-10-25 16:39:54,983+0800 INFO (mailbox-hsm/3) [vdsm.api] FINISH getVolumeSize error=Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) from=internal, task_id=c5a7364d-3b39-4e12-b75b-22d5c4487cad (api:52) 2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.TaskManager.Task] (Task='c5a7364d-3b39-4e12-b75b-22d5c4487cad') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in getVolumeSize File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3071, in getVolumeSize dom = sdCache.produce(sdUUID=sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1843, in findDomain return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID)) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1701, in findDomainPath raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) 2022-10-25 16:39:54,983+0800 INFO (mailbox-hsm/3) [storage.TaskManager.Task] (Task='c5a7364d-3b39-4e12-b75b-22d5c4487cad') aborting: Task is aborted: "Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)" - code 358 (task:1181) 2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.Dispatcher] FINISH getVolumeSize error=Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) (dispatcher:83) 2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.TaskManager.Task] (Task='033cec9a-5b65-4413-97c3-a273aafa6114') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1407, in __afterVolumeExtension volSize = self.__verifyVolumeExtension(volInfo) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1325, in __verifyVolumeExtension volInfo['imageID'], volInfo['volumeID']) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 6147, in _getVolumeSize (domainID, volumeID)) StorageUnavailableError: Unable to get volume size for domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume 581f65d2-3db5-4fde-960f-4c4ffe27341e 2022-10-25 16:39:54,983+0800 INFO (mailbox-hsm/3) [storage.TaskManager.Task] (Task='033cec9a-5b65-4413-97c3-a273aafa6114') aborting: Task is aborted: u'Unable to get volume size for domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume 581f65d2-3db5-4fde-960f-4c4ffe27341e' - code 100 (task:1181) 2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.ThreadPool.WorkerThread] FINISH task b81cf2c1-c795-4f97-9c5a-6b06a6a3173c failed (callback=None, cmd=<function runTask at 0x7fea630c8500>, args=(<bound method Vm.__afterVolumeExtension of <vdsm.virt.vm.Vm object at 0x7fea007336d0>>, {'newSize': 92341796864, 'domainID': '107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda', 'poolID': '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock': <Clock(total=3.20*, extend-volume=2.88, refresh-volume=0.16)>, 'internal': False, 'volumeID': '581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID': '2d1375ba-ce52-4da1-9873-d9181d9e775b'})) (threadPool:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/threadPool.py", line 209, in _processNextTask cmd(args) File "/usr/lib/python2.7/site-packages/vdsm/storage/mailbox.py", line 85, in runTask ctask.prepare(cmd, *args) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error StorageUnavailableError: Unable to get volume size for domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume 581f65d2-3db5-4fde-960f-4c4ffe27341e ^C
This is what I get from vdsm when I ran tail -f. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QRSLEBLJUE5NY5...
On Tue, Oct 25, 2022 at 7:38 PM Darrell Budic <budic@onholyground.com> wrote:
Both the initial messages and the additional detail in these vdsm logs show that it can’t mount a storage volume. In a multi node environment, it’s usually iscsi, gluster, etc, so start by looking at your network and the storage devices and figuring out why it can’t mount them. Comparison to any remaining running nodes may be helpful.
Indeed, but in this particular case, it also provides more details: [' /dev/mapper/3600c0ff00051441d1 e145f5e01000000: Checksum error at offset 30099960109568', " Couldn't read volume group metadata from /dev/mapper/3600c0ff00051441d1e145f5e01000000.", ' Metadata location on /dev/mapper/3600c0ff00051441d1e145f5e01000000 at 30099960109568 has invalid summary for VG.', ' Failed to read metadata summary from /dev/mapper/3600c0ff00051441d1e145f5e01000000', ' Failed to scan VG from /dev/mapper/3600c0ff00051441d1e145f5e01000000'] If this VG does work well from some other host, perhaps it's a local issue. Otherwise, it's probably corrupted - if you have good and recent backups, it might be best to restore them, otherwise you can try fixing the VG - I'd start by searching the net for relevant parts of this error message and see how people handled similar cases. Good luck and best regards,
On Oct 25, 2022, at 3:40 AM, eugene@knorydev.com wrote:
Hi Tomas,
[root@host1 vdsm]# tail -f vdsm.log StorageUnavailableError: Unable to get volume size for domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume 581f65d2-3db5-4fde-960f-4c4ffe27341e 2022-10-25 16:39:51,785+0800 INFO (periodic/3) [virt.vm] (vmId='a8d6fcaf-a673-46df-b00a-925a8b9791ef') Requesting extension for volume 581f65d2-3db5-4fde-960f-4c4ffe27341e on domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 (apparent: 91268055040, capacity: 322122547200, allocated: 90872676352, physical: 91268055040 threshold_state: exceeded) (vm:1289) 2022-10-25 16:39:51,787+0800 INFO (periodic/3) [vdsm.api] START sendExtendMsg(spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef', volDict={'newSize': 92341796864, 'domainID': '107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda', 'poolID': '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock': <Clock(total=0.00*, extend-volume=0.00*)>, 'internal': False, 'volumeID': '581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID': '2d1375ba-ce52-4da1-9873-d9181d9e775b'}, newSize=92341796864, callbackFunc=<bound method Vm.__afterVolumeExtension of <vdsm.virt.vm.Vm object at 0x7fea007336d0>>) from=internal, task_id=2b0c7a67-0f31-4f62-af70-a29e0b84dafb (api:48) 2022-10-25 16:39:51,787+0800 INFO (periodic/3) [vdsm.api] FINISH sendExtendMsg return=None from=internal, task_id=2b0c7a67-0f31-4f62-af70-a29e0b84dafb (api:54) 2022-10-25 16:39:52,635+0800 INFO (mailbox-hsm) [storage.MailBox.HsmMailMonitor] HSM_MailMonitor sending mail to SPM - ['/usr/bin/dd', 'of=/rhev/data-center/81f9e45a-5879-11ea-a3bd-00163e27dfef/mastersd/dom_md/inbox', 'iflag=fullblock', 'oflag=direct', 'conv=notrunc', 'bs=4096', 'count=1', 'seek=1'] (mailbox:380) 2022-10-25 16:39:52,710+0800 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:312) 2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [api.virt] START getStats() from=::1,39184, vmId=6579a9c4-d55e-4d61-9c3a-c745cdcfa430 (api:48) 2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'6579a9c4-d55e-4d61-9c3a-c745cdcfa430'} (api:129) 2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [api.virt] FINISH getStats return={'status': {'message': "Virtual machine does not exist: {'vmId': u'6579a9c4-d55e-4d61-9c3a-c745cdcfa430'}", 'code': 1}} from=::1,39184, vmId=6579a9c4-d55e-4d61-9c3a-c745cdcfa430 (api:54) 2022-10-25 16:39:52,715+0800 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2022-10-25 16:39:53,786+0800 INFO (periodic/5) [virt.vm] (vmId='a8d6fcaf-a673-46df-b00a-925a8b9791ef') Requesting extension for volume 581f65d2-3db5-4fde-960f-4c4ffe27341e on domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 (apparent: 91268055040, capacity: 322122547200, allocated: 90872676352, physical: 91268055040 threshold_state: exceeded) (vm:1289) 2022-10-25 16:39:53,787+0800 INFO (periodic/5) [vdsm.api] START sendExtendMsg(spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef', volDict={'newSize': 92341796864, 'domainID': '107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda', 'poolID': '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock': <Clock(total=0.00*, extend-volume=0.00*)>, 'internal': False, 'volumeID': '581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID': '2d1375ba-ce52-4da1-9873-d9181d9e775b'}, newSize=92341796864, callbackFunc=<bound method Vm.__afterVolumeExtension of <vdsm.virt.vm.Vm object at 0x7fea007336d0>>) from=internal, task_id=197234ba-5cab-4098-86c8-7f2c979171d2 (api:48) 2022-10-25 16:39:53,787+0800 INFO (periodic/5) [vdsm.api] FINISH sendExtendMsg return=None from=internal, task_id=197234ba-5cab-4098-86c8-7f2c979171d2 (api:54) 2022-10-25 16:39:54,167+0800 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:312) 2022-10-25 16:39:54,658+0800 INFO (mailbox-hsm) [storage.MailBox.HsmMailMonitor] HSM_MailMonitor sending mail to SPM - ['/usr/bin/dd', 'of=/rhev/data-center/81f9e45a-5879-11ea-a3bd-00163e27dfef/mastersd/dom_md/inbox', 'iflag=fullblock', 'oflag=direct', 'conv=notrunc', 'bs=4096', 'count=1', 'seek=1'] (mailbox:380) 2022-10-25 16:39:54,661+0800 INFO (mailbox-hsm/3) [storage.ThreadPool.WorkerThread] START task b81cf2c1-c795-4f97-9c5a-6b06a6a3173c (cmd=<function runTask at 0x7fea630c8500>, args=(<bound method Vm.__afterVolumeExtension of <vdsm.virt.vm.Vm object at 0x7fea007336d0>>, {'newSize': 92341796864, 'domainID': '107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda', 'poolID': '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock': <Clock(total=2.88*, extend-volume=2.88, refresh-volume=0.00*)>, 'internal': False, 'volumeID': '581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID': '2d1375ba-ce52-4da1-9873-d9181d9e775b'})) (threadPool:208) 2022-10-25 16:39:54,663+0800 INFO (mailbox-hsm/3) [vdsm.api] START refreshVolume(sdUUID='107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef', imgUUID='2d1375ba-ce52-4da1-9873-d9181d9e775b', volUUID='581f65d2-3db5-4fde-960f-4c4ffe27341e') from=internal, task_id=918b7e3f-d730-4226-a941-fc0e60ee36b0 (api:48) 2022-10-25 16:39:54,809+0800 WARN (mailbox-hsm/3) [storage.LVM] Command ['/sbin/lvm', 'vgs', '--config', 'devices { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter=["a|^/dev/mapper/3600c0ff00051441d1e145f5e01000000$|^/dev/mapper/3600c0ff00051441de4f6555e01000000$|^/dev/mapper/364cd98f0bb99140025dd822d083dd183$|^/dev/mapper/364cd98f0bb99140025dd82520a7280ae$|", "r|.*|"] } global { locking_type=4 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min=50 retain_days=0 }', '--noheadings', '--units', 'b', '--nosuffix', '--separator', '|', '--ignoreskippedcluster', '-o', 'uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name', '--select', 'vg_name = 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19'] succeeded with warnings: [' /dev/mapper/3600c0ff00051441d1e145f5e01000000: Checksum error at offset 30099960109568', " Couldn't read volume group metadata from /dev/mapper/3600c0ff00051441d1e145f5e01000000.", ' Metadata location on /dev/mapper/3600c0ff00051441d1e145f5e01000000 at 30099960109568 has invalid summary for VG.', ' Failed to read metadata summary from /dev/mapper/3600c0ff00051441d1e145f5e01000000', ' Failed to scan VG from /dev/mapper/3600c0ff00051441d1e145f5e01000000'] (lvm:466) 2022-10-25 16:39:54,824+0800 INFO (mailbox-hsm/3) [vdsm.api] FINISH refreshVolume error=Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) from=internal, task_id=918b7e3f-d730-4226-a941-fc0e60ee36b0 (api:52) 2022-10-25 16:39:54,824+0800 ERROR (mailbox-hsm/3) [storage.TaskManager.Task] (Task='918b7e3f-d730-4226-a941-fc0e60ee36b0') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in refreshVolume File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3033, in refreshVolume sdUUID=sdUUID).produceVolume(imgUUID=imgUUID, File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1843, in findDomain return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID)) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1701, in findDomainPath raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) 2022-10-25 16:39:54,825+0800 INFO (mailbox-hsm/3) [storage.TaskManager.Task] (Task='918b7e3f-d730-4226-a941-fc0e60ee36b0') aborting: Task is aborted: "Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)" - code 358 (task:1181) 2022-10-25 16:39:54,825+0800 ERROR (mailbox-hsm/3) [storage.Dispatcher] FINISH refreshVolume error=Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) (dispatcher:83) 2022-10-25 16:39:54,826+0800 INFO (mailbox-hsm/3) [vdsm.api] START getVolumeSize(sdUUID='107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', spUUID='81f9e45a-5879-11ea-a3bd-00163e27dfef', imgUUID='2d1375ba-ce52-4da1-9873-d9181d9e775b', volUUID='581f65d2-3db5-4fde-960f-4c4ffe27341e', options=None) from=internal, task_id=c5a7364d-3b39-4e12-b75b-22d5c4487cad (api:48) 2022-10-25 16:39:54,970+0800 WARN (mailbox-hsm/3) [storage.LVM] Command ['/sbin/lvm', 'vgs', '--config', 'devices { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter=["a|^/dev/mapper/3600c0ff00051441d1e145f5e01000000$|^/dev/mapper/3600c0ff00051441de4f6555e01000000$|^/dev/mapper/364cd98f0bb99140025dd822d083dd183$|^/dev/mapper/364cd98f0bb99140025dd82520a7280ae$|", "r|.*|"] } global { locking_type=4 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min=50 retain_days=0 }', '--noheadings', '--units', 'b', '--nosuffix', '--separator', '|', '--ignoreskippedcluster', '-o', 'uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name', '--select', 'vg_name = 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19'] succeeded with warnings: [' /dev/mapper/3600c0ff00051441d1e145f5e01000000: Checksum error at offset 30099960109568', " Couldn't read volume group metadata from /dev/mapper/3600c0ff00051441d1e145f5e01000000.", ' Metadata location on /dev/mapper/3600c0ff00051441d1e145f5e01000000 at 30099960109568 has invalid summary for VG.', ' Failed to read metadata summary from /dev/mapper/3600c0ff00051441d1e145f5e01000000', ' Failed to scan VG from /dev/mapper/3600c0ff00051441d1e145f5e01000000'] (lvm:466) 2022-10-25 16:39:54,983+0800 INFO (mailbox-hsm/3) [vdsm.api] FINISH getVolumeSize error=Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) from=internal, task_id=c5a7364d-3b39-4e12-b75b-22d5c4487cad (api:52) 2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.TaskManager.Task] (Task='c5a7364d-3b39-4e12-b75b-22d5c4487cad') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in getVolumeSize File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3071, in getVolumeSize dom = sdCache.produce(sdUUID=sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1843, in findDomain return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID)) File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1701, in findDomainPath raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) 2022-10-25 16:39:54,983+0800 INFO (mailbox-hsm/3) [storage.TaskManager.Task] (Task='c5a7364d-3b39-4e12-b75b-22d5c4487cad') aborting: Task is aborted: "Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',)" - code 358 (task:1181) 2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.Dispatcher] FINISH getVolumeSize error=Storage domain does not exist: ('107f73a2-8c3e-43a7-bf18-ce8ef4be2a19',) (dispatcher:83) 2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.TaskManager.Task] (Task='033cec9a-5b65-4413-97c3-a273aafa6114') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1407, in __afterVolumeExtension volSize = self.__verifyVolumeExtension(volInfo) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1325, in __verifyVolumeExtension volInfo['imageID'], volInfo['volumeID']) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 6147, in _getVolumeSize (domainID, volumeID)) StorageUnavailableError: Unable to get volume size for domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume 581f65d2-3db5-4fde-960f-4c4ffe27341e 2022-10-25 16:39:54,983+0800 INFO (mailbox-hsm/3) [storage.TaskManager.Task] (Task='033cec9a-5b65-4413-97c3-a273aafa6114') aborting: Task is aborted: u'Unable to get volume size for domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume 581f65d2-3db5-4fde-960f-4c4ffe27341e' - code 100 (task:1181) 2022-10-25 16:39:54,983+0800 ERROR (mailbox-hsm/3) [storage.ThreadPool.WorkerThread] FINISH task b81cf2c1-c795-4f97-9c5a-6b06a6a3173c failed (callback=None, cmd=<function runTask at 0x7fea630c8500>, args=(<bound method Vm.__afterVolumeExtension of <vdsm.virt.vm.Vm object at 0x7fea007336d0>>, {'newSize': 92341796864, 'domainID': '107f73a2-8c3e-43a7-bf18-ce8ef4be2a19', 'name': 'vda', 'poolID': '81f9e45a-5879-11ea-a3bd-00163e27dfef', 'clock': <Clock(total=3.20*, extend-volume=2.88, refresh-volume=0.16)>, 'internal': False, 'volumeID': '581f65d2-3db5-4fde-960f-4c4ffe27341e', 'imageID': '2d1375ba-ce52-4da1-9873-d9181d9e775b'})) (threadPool:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/threadPool.py", line 209, in _processNextTask cmd(args) File "/usr/lib/python2.7/site-packages/vdsm/storage/mailbox.py", line 85, in runTask ctask.prepare(cmd, *args) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper return m(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1189, in prepare raise self.error StorageUnavailableError: Unable to get volume size for domain 107f73a2-8c3e-43a7-bf18-ce8ef4be2a19 volume 581f65d2-3db5-4fde-960f-4c4ffe27341e ^C
This is what I get from vdsm when I ran tail -f. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QRSLEBLJUE5NY5...
Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/Q6U3KM7BCJIQA6...
-- Didi
participants (4)
-
Darrell Budic -
eugene＠knorydev.com -
Tomáš Golembiovský -
Yedidyah Bar David