Hi. I have a 2 node cluster (both installed from the Ovirt-node iso) , 1 server is 100% fine, but one server I tried to update when there were issues with the ovirtmgmt network (I wasn't aware of at the time) - as a result the server update failed, and was still on ovirt-node 4.4.4 after - I could see the LVM layer for 4.4.5 though. The network is fine now (hence the other server being ok) - I tried reinstalling the package ovirt-node-ng-image-update-4.4.5.1-1.el8.noarch.rpm but it failed due to LVM lv for 4.4.5 already existing ... So I tried removing the LV - i.e # lvremove /dev/onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1 however now I'm sure I have less 4.4.4 lvs .. i.e ------ [root@ng2-ovirt-kvm3 mcox]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert home onn Vwi-aotz-- 1.00g pool00 10.05 ovirt-node-ng-4.4.4.1-0.20210208.0+1 onn Vwi-aotz-- 191.00g pool00 2.27 pool00 onn twi-aotz-- 218.00g 2.48 1.92 swap onn -wi-ao---- 4.00g tmp onn Vwi-aotz-- 1.00g pool00 2.24 var onn Vwi-aotz-- 15.00g pool00 2.80 var_crash onn Vwi-aotz-- 10.00g pool00 0.21 var_log onn Vwi-aotz-- 8.00g pool00 5.97 var_log_audit onn Vwi-aotz-- 2.00g pool00 1.14 ---- Where as on the working server I have these Ovirt Node LVs ovirt-node-ng-4.4.4.1-0.20210208.0 onn Vwi---tz-k 191.00g pool00 root ovirt-node-ng-4.4.4.1-0.20210208.0+1 onn Vwi-a-tz-- 191.00g pool00 ovirt-node-ng-4.4.4.1-0.20210208.0 1.86 ovirt-node-ng-4.4.5.1-0.20210323.0 onn Vri---tz-k 191.00g pool00 ovirt-node-ng-4.4.5.1-0.20210323.0+1 onn Vwi-aotz-- 191.00g pool00 ovirt-node-ng-4.4.5.1-0.20210323.0 1.58 Anyway I tried to reinstall the ovirt-node-ng-image-update-4.4.5.1-1.el8.noarch.rpm package after removal of lv and files in /boot But install still fails .. Is there anything I can do to update this node - or should I bite the bullet and reinstall ? this is from /var/log/imgbased.log -> -------------------------------------- 2021-04-14 16:05:56,634 [INFO] (MainThread) Adding a new layer after <Base ovirt-node-ng-4.4.5.1-0.20210323.0 [] /> 2021-04-14 16:05:56,634 [INFO] (MainThread) Adding a new layer after <Base ovirt-node-ng-4.4.5.1-0.20210323.0 [] /> 2021-04-14 16:05:56,634 [DEBUG] (MainThread) Basing new layer on previous: <Base ovirt-node-ng-4.4.5.1-0.20210323.0 [] /> 2021-04-14 16:05:56,634 [DEBUG] (MainThread) Finding next layer based on <Base ovirt-node-ng-4.4.5.1-0.20210323.0 [] /> 2021-04-14 16:05:56,634 [DEBUG] (MainThread) Suggesting for layer for base ovirt-node-ng-4.4.5.1-0.20210323.0 2021-04-14 16:05:56,635 [DEBUG] (MainThread) ... without layers 2021-04-14 16:05:56,635 [INFO] (MainThread) New layer will be: <Layer ovirt-node-ng-4.4.5.1-0.20210323.0+1 /> 2021-04-14 16:05:56,635 [DEBUG] (MainThread) Calling: (['vgs', '--noheadings', '--ignoreskippedcluster', '--select', 'vg_tags = imgbased:vg', '-o', 'vg_name'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True} 2021-04-14 16:05:56,697 [DEBUG] (MainThread) Returned: b'onn' 2021-04-14 16:05:56,698 [DEBUG] (MainThread) Calling: (['vgs', '--noheadings', '--ignoreskippedcluster', '@imgbased:pool', '-o', 'lv_full_name'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True} 2021-04-14 16:05:56,757 [DEBUG] (MainThread) Returned: b'onn/pool00' 2021-04-14 16:05:56,758 [DEBUG] (MainThread) Calling: (['lvs', '--noheadings', '--ignoreskippedcluster', '--nosuffix', '--units', 'm', '-o', 'metadata_percent,lv_metadata_size', 'onn/pool00'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True} 2021-04-14 16:05:56,829 [DEBUG] (MainThread) Returned: b'2.05 1024.00' 2021-04-14 16:05:56,830 [DEBUG] (MainThread) Pool: onn/pool00, metadata size=1024.0M (2.05%) 2021-04-14 16:05:56,830 [DEBUG] (MainThread) Calling: (['lvchange', '--activate', 'y', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0', '--ignoreactivationskip'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True} 2021-04-14 16:05:56,953 [DEBUG] (MainThread) Calling: (['lvcreate', '--snapshot', '--name', 'ovirt-node-ng-4.4.5.1-0.20210323.0+1', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0'],) {'close_fds': True, 'stderr': -2} 2021-04-14 16:05:57,111 [DEBUG] (MainThread) Returned: b'WARNING: Sum of all thin volume sizes (610.00 GiB) exceeds the size of thin pool onn/pool00 and the size of whole volume group (<277.78 GiB).\n Logical volume "ovirt-node-ng-4.4.5.1-0.20210323.0+1" created.' 2021-04-14 16:05:57,112 [DEBUG] (MainThread) Calling: (['lvchange', '--activate', 'y', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1', '--ignoreactivationskip'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True} 2021-04-14 16:05:57,231 [DEBUG] (MainThread) Calling: (['lvchange', '--addtag', 'imgbased:layer', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True} 2021-04-14 16:05:57,310 [DEBUG] (MainThread) Returned: b'Logical volume onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1 changed.' 2021-04-14 16:05:57,311 [DEBUG] (MainThread) Calling: (['lvs', '--noheadings', '--ignoreskippedcluster', '-olv_path', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True} 2021-04-14 16:05:57,383 [DEBUG] (MainThread) Returned: b'/dev/onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1' 2021-04-14 16:05:57,400 [DEBUG] (MainThread) Calling: (['mktemp', '-d', '--tmpdir', 'mnt.XXXXX'],) {'close_fds': True, 'stderr': -2} 2021-04-14 16:05:57,404 [DEBUG] (MainThread) Returned: b'/tmp/mnt.aGJ3A' 2021-04-14 16:05:57,404 [DEBUG] (MainThread) Calling: (['mount', '-onouuid', '/dev/onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1', '/tmp/mnt.aGJ3A'],) {'close_fds': True, 'stderr': -2} 2021-04-14 16:05:57,537 [DEBUG] (MainThread) Calling: (['umount', '-l', '/tmp/mnt.aGJ3A'],) {'close_fds': True, 'stderr': -2} 2021-04-14 16:05:57,561 [DEBUG] (MainThread) Calling: (['rmdir', '/tmp/mnt.aGJ3A'],) {'close_fds': True, 'stderr': -2} 2021-04-14 16:05:57,565 [DEBUG] (MainThread) Running: ['xfs_admin', '-U', 'generate', '/dev/onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'] 2021-04-14 16:05:57,566 [DEBUG] (MainThread) Calling: (['xfs_admin', '-U', 'generate', '/dev/onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'],) {'stderr': -2, 'close_fds': True} 2021-04-14 16:05:57,957 [DEBUG] (MainThread) Calling: (['lvchange', '--setactivationskip', 'n', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True} 2021-04-14 16:05:58,040 [DEBUG] (MainThread) Returned: b'Logical volume onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1 changed.' 2021-04-14 16:05:58,041 [DEBUG] (MainThread) Calling: (['lvchange', '--setactivationskip', 'y', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True} 2021-04-14 16:05:58,116 [DEBUG] (MainThread) Returned: b'Logical volume onn/ovirt-node-ng-4.4.5.1-0.20210323.0 changed.' 2021-04-14 16:05:58,117 [DEBUG] (MainThread) Got: <LV 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1' /> and <LV 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0' /> 2021-04-14 16:05:58,117 [DEBUG] (MainThread) Calling: (['mount', '--make-rprivate', '/var'],) {'close_fds': True, 'stderr': -2} 2021-04-14 16:05:58,122 [DEBUG] (MainThread) Calling: (['systemctl', 'is-active', 'vdsmd'],) {'close_fds': True, 'stderr': -2} 2021-04-14 16:05:58,141 [DEBUG] (MainThread) Returned: b'active' 2021-04-14 16:05:58,141 [DEBUG] (MainThread) Calling: (['systemctl', 'stop', 'vdsmd'],) {'close_fds': True, 'stderr': -2} 2021-04-14 16:06:08,940 [DEBUG] (MainThread) Calling: (['systemctl', 'stop', 'supervdsmd'],) {'close_fds': True, 'stderr': -2} 2021-04-14 16:06:09,072 [DEBUG] (MainThread) Calling: (['systemctl', 'stop', 'vdsm-network'],) {'close_fds': True, 'stderr': -2} 2021-04-14 16:06:09,090 [DEBUG] (MainThread) Calling: (['vgchange', '-ay', '--select', 'vg_tags = imgbased:vg'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True} 2021-04-14 16:06:09,167 [DEBUG] (MainThread) Returned: b'11 logical volume(s) in volume group "onn" now active' 2021-04-14 16:06:09,168 [DEBUG] (MainThread) Calling: (['lvs', '--noheadings', '--ignoreskippedcluster', '-o', 'lv_full_name', '--select', 'lv_tags = imgbased:base || lv_tags = imgbased:layer'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True} 2021-04-14 16:06:09,229 [DEBUG] (MainThread) Returned: b'onn/ovirt-node-ng-4.4.4.1-0.20210208.0+1\n onn/ovirt-node-ng-4.4.5.1-0.20210323.0 \n onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1' 2021-04-14 16:06:09,230 [DEBUG] (MainThread) All LV names: ['onn/ovirt-node-ng-4.4.4.1-0.20210208.0+1', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'] 2021-04-14 16:06:09,230 [DEBUG] (MainThread) All LVS: [<LV 'onn/ovirt-node-ng-4.4.4.1-0.20210208.0+1' />, <LV 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0' />, <LV 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1' />] 2021-04-14 16:06:09,230 [DEBUG] (MainThread) Calling: (['lvs', '--noheadings', '--ignoreskippedcluster', '-olv_tags', 'onn/ovirt-node-ng-4.4.4.1-0.20210208.0+1'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True} 2021-04-14 16:06:09,293 [DEBUG] (MainThread) Returned: b'imgbased:layer' 2021-04-14 16:06:09,294 [DEBUG] (MainThread) Calling: (['lvs', '--noheadings', '--ignoreskippedcluster', '-olv_tags', 'onn/ovirt-node-ng-4.4.4.1-0.20210208.0+1'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True} 2021-04-14 16:06:09,357 [DEBUG] (MainThread) Returned: b'imgbased:layer' 2021-04-14 16:06:09,358 [DEBUG] (MainThread) Calling: (['lvs', '--noheadings', '--ignoreskippedcluster', '-olv_tags', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True} 2021-04-14 16:06:09,417 [DEBUG] (MainThread) Returned: b'imgbased:base' 2021-04-14 16:06:09,418 [DEBUG] (MainThread) Calling: (['lvs', '--noheadings', '--ignoreskippedcluster', '-olv_tags', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True} 2021-04-14 16:06:09,490 [DEBUG] (MainThread) Returned: b'imgbased:layer' 2021-04-14 16:06:09,491 [DEBUG] (MainThread) Calling: (['lvs', '--noheadings', '--ignoreskippedcluster', '-olv_tags', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True} 2021-04-14 16:06:09,557 [DEBUG] (MainThread) Returned: b'imgbased:layer' 2021-04-14 16:06:09,558 [DEBUG] (MainThread) Our LVS: [<LV 'onn/ovirt-node-ng-4.4.4.1-0.20210208.0+1' />, <LV 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0' />, <LV 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1' />] 2021-04-14 16:06:09,558 [DEBUG] (MainThread) Names: ['ovirt-node-ng-4.4.4.1-0.20210208.0+1', 'ovirt-node-ng-4.4.5.1-0.20210323.0', 'ovirt-node-ng-4.4.5.1-0.20210323.0+1'] 2021-04-14 16:06:09,558 [DEBUG] (MainThread) Images: [<Layer ovirt-node-ng-4.4.4.1-0.20210208.0+1 />, <Base ovirt-node-ng-4.4.5.1-0.20210323.0 [] />, <Layer ovirt-node-ng-4.4.5.1-0.20210323.0+1 />] 2021-04-14 16:06:09,558 [DEBUG] (MainThread) Calling: (['umount', '-l', '/tmp/mnt.o5bFp'],) {'close_fds': True, 'stderr': -2} 2021-04-14 16:06:09,579 [DEBUG] (MainThread) Calling: (['rmdir', '/tmp/mnt.o5bFp'],) {'close_fds': True, 'stderr': -2} 2021-04-14 16:06:09,583 [DEBUG] (MainThread) Calling: (['umount', '-l', '/tmp/mnt.CcHrE'],) {'close_fds': True, 'stderr': -2} 2021-04-14 16:06:09,601 [DEBUG] (MainThread) Calling: (['rmdir', '/tmp/mnt.CcHrE'],) {'close_fds': True, 'stderr': -2} 2021-04-14 16:06:09,605 [ERROR] (MainThread) Update failed, resetting registered LVs 2021-04-14 16:06:09,606 [DEBUG] (MainThread) Calling: (['sync'],) {'close_fds': True, 'stderr': -2} 2021-04-14 16:06:09,615 [DEBUG] (MainThread) Calling: (['lvs', '--noheadings', '--ignoreskippedcluster', '-olv_dm_path', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True} 2021-04-14 16:06:09,683 [DEBUG] (MainThread) Returned: b'/dev/mapper/onn-ovirt--node--ng--4.4.5.1--0.20210323.0' 2021-04-14 16:06:09,684 [DEBUG] (MainThread) Calling: (['lvremove', '-ff', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True} 2021-04-14 16:06:09,820 [DEBUG] (MainThread) Returned: b'Logical volume "ovirt-node-ng-4.4.5.1-0.20210323.0" successfully removed' 2021-04-14 16:06:09,821 [DEBUG] (MainThread) Calling: (['lvs', '--noheadings', '--ignoreskippedcluster', '-olv_dm_path', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True} 2021-04-14 16:06:09,886 [DEBUG] (MainThread) Returned: b'/dev/mapper/onn-ovirt--node--ng--4.4.5.1--0.20210323.0+1' 2021-04-14 16:06:09,887 [DEBUG] (MainThread) Calling: (['lvremove', '-ff', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True} 2021-04-14 16:06:10,207 [DEBUG] (MainThread) Returned: b'Logical volume "ovirt-node-ng-4.4.5.1-0.20210323.0+1" successfully removed' Traceback (most recent call last): File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/__main__.py", line 53, in <module> CliApplication() File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/__init__.py", line 82, in CliApplication app.hooks.emit("post-arg-parse", args) File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/hooks.py", line 120, in emit cb(self.context, *args) File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/plugins/update.py", line 75, in post_argparse six.reraise(*exc_info) File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/six.py", line 703, in reraise raise value File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/plugins/update.py", line 66, in post_argparse base, _ = LiveimgExtractor(app.imgbase).extract(args.FILENAME) File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/plugins/update.py", line 148, in extract "%s" % size, nvr) File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/plugins/update.py", line 128, in add_base_with_tree new_layer_lv = self.imgbase.add_layer(new_base) File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/imgbase.py", line 213, in add_layer self.hooks.emit("new-layer-added", prev_lv, new_lv) File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/hooks.py", line 120, in emit cb(self.context, *args) File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/plugins/osupdater.py", line 88, in on_new_layer previous_layer_lv = get_prev_layer_lv(imgbase, new_lv) File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/plugins/osupdater.py", line 143, in get_prev_layer_lv layer_before = imgbase.naming.layer_before(new_layer) File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/naming.py", line 74, in layer_before layers = self.layers() File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/naming.py", line 58, in layers for b in self.tree(): File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/naming.py", line 226, in tree bases[img.base.nvr].layers.append(img) KeyError: <NVR ovirt-node-ng-4.4.4.1-0.20210208.0 /> --------------------