Hi.
I have a 2 node cluster (both installed from the Ovirt-node iso) , 1 server is 100% fine, but one server I tried to update when there were issues with the ovirtmgmt network (I wasn't aware of at the time) - as a result the server update failed, and was still on ovirt-node 4.4.4 after - I could see the LVM layer for 4.4.5 though.
The network is fine now (hence the other server being ok) - I tried reinstalling the package ovirt-node-ng-image-update-4.4.5.1-1.el8.noarch.rpm but it failed due to LVM lv for 4.4.5 already existing ...
So I tried removing the LV - i.e
# lvremove /dev/onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1
however now I'm sure I have less 4.4.4 lvs ..
i.e
------
[root@ng2-ovirt-kvm3 mcox]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
home onn Vwi-aotz-- 1.00g pool00 10.05
ovirt-node-ng-4.4.4.1-0.20210208.0+1 onn Vwi-aotz-- 191.00g pool00 2.27
pool00 onn twi-aotz-- 218.00g 2.48 1.92
swap onn -wi-ao---- 4.00g
tmp onn Vwi-aotz-- 1.00g pool00 2.24
var onn Vwi-aotz-- 15.00g pool00 2.80
var_crash onn Vwi-aotz-- 10.00g pool00 0.21
var_log onn Vwi-aotz-- 8.00g pool00 5.97
var_log_audit onn Vwi-aotz-- 2.00g pool00 1.14
----
Where as on the working server I have these Ovirt Node LVs
ovirt-node-ng-4.4.4.1-0.20210208.0 onn Vwi---tz-k 191.00g pool00 root
ovirt-node-ng-4.4.4.1-0.20210208.0+1 onn Vwi-a-tz-- 191.00g pool00 ovirt-node-ng-4.4.4.1-0.20210208.0 1.86
ovirt-node-ng-4.4.5.1-0.20210323.0 onn Vri---tz-k 191.00g pool00
ovirt-node-ng-4.4.5.1-0.20210323.0+1 onn Vwi-aotz-- 191.00g pool00 ovirt-node-ng-4.4.5.1-0.20210323.0 1.58
Anyway I tried to reinstall the ovirt-node-ng-image-update-4.4.5.1-1.el8.noarch.rpm package after removal of lv and files in /boot
But install still fails .. Is there anything I can do to update this node - or should I bite the bullet and reinstall ?
this is from /var/log/imgbased.log ->
--------------------------------------
2021-04-14 16:05:56,634 [INFO] (MainThread) Adding a new layer after <Base ovirt-node-ng-4.4.5.1-0.20210323.0 [] />
2021-04-14 16:05:56,634 [INFO] (MainThread) Adding a new layer after <Base ovirt-node-ng-4.4.5.1-0.20210323.0 [] />
2021-04-14 16:05:56,634 [DEBUG] (MainThread) Basing new layer on previous: <Base ovirt-node-ng-4.4.5.1-0.20210323.0 [] />
2021-04-14 16:05:56,634 [DEBUG] (MainThread) Finding next layer based on <Base ovirt-node-ng-4.4.5.1-0.20210323.0 [] />
2021-04-14 16:05:56,634 [DEBUG] (MainThread) Suggesting for layer for base ovirt-node-ng-4.4.5.1-0.20210323.0
2021-04-14 16:05:56,635 [DEBUG] (MainThread) ... without layers
2021-04-14 16:05:56,635 [INFO] (MainThread) New layer will be: <Layer ovirt-node-ng-4.4.5.1-0.20210323.0+1 />
2021-04-14 16:05:56,635 [DEBUG] (MainThread) Calling: (['vgs', '--noheadings', '--ignoreskippedcluster', '--select', 'vg_tags = imgbased:vg', '-o', 'vg_name'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:05:56,697 [DEBUG] (MainThread) Returned: b'onn'
2021-04-14 16:05:56,698 [DEBUG] (MainThread) Calling: (['vgs', '--noheadings', '--ignoreskippedcluster', '@imgbased:pool', '-o', 'lv_full_name'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:05:56,757 [DEBUG] (MainThread) Returned: b'onn/pool00'
2021-04-14 16:05:56,758 [DEBUG] (MainThread) Calling: (['lvs', '--noheadings', '--ignoreskippedcluster', '--nosuffix', '--units', 'm', '-o', 'metadata_percent,lv_metadata_size', 'onn/pool00'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:05:56,829 [DEBUG] (MainThread) Returned: b'2.05 1024.00'
2021-04-14 16:05:56,830 [DEBUG] (MainThread) Pool: onn/pool00, metadata size=1024.0M (2.05%)
2021-04-14 16:05:56,830 [DEBUG] (MainThread) Calling: (['lvchange', '--activate', 'y', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0', '--ignoreactivationskip'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:05:56,953 [DEBUG] (MainThread) Calling: (['lvcreate', '--snapshot', '--name', 'ovirt-node-ng-4.4.5.1-0.20210323.0+1', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0'],) {'close_fds': True, 'stderr': -2}
2021-04-14 16:05:57,111 [DEBUG] (MainThread) Returned: b'WARNING: Sum of all thin volume sizes (610.00 GiB) exceeds the size of thin pool onn/pool00 and the size of whole volume group (<277.78 GiB).\n Logical volume "ovirt-node-ng-4.4.5.1-0.20210323.0+1" created.'
2021-04-14 16:05:57,112 [DEBUG] (MainThread) Calling: (['lvchange', '--activate', 'y', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1', '--ignoreactivationskip'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:05:57,231 [DEBUG] (MainThread) Calling: (['lvchange', '--addtag', 'imgbased:layer', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:05:57,310 [DEBUG] (MainThread) Returned: b'Logical volume onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1 changed.'
2021-04-14 16:05:57,311 [DEBUG] (MainThread) Calling: (['lvs', '--noheadings', '--ignoreskippedcluster', '-olv_path', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:05:57,383 [DEBUG] (MainThread) Returned: b'/dev/onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'
2021-04-14 16:05:57,400 [DEBUG] (MainThread) Calling: (['mktemp', '-d', '--tmpdir', 'mnt.XXXXX'],) {'close_fds': True, 'stderr': -2}
2021-04-14 16:05:57,404 [DEBUG] (MainThread) Returned: b'/tmp/mnt.aGJ3A'
2021-04-14 16:05:57,404 [DEBUG] (MainThread) Calling: (['mount', '-onouuid', '/dev/onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1', '/tmp/mnt.aGJ3A'],) {'close_fds': True, 'stderr': -2}
2021-04-14 16:05:57,537 [DEBUG] (MainThread) Calling: (['umount', '-l', '/tmp/mnt.aGJ3A'],) {'close_fds': True, 'stderr': -2}
2021-04-14 16:05:57,561 [DEBUG] (MainThread) Calling: (['rmdir', '/tmp/mnt.aGJ3A'],) {'close_fds': True, 'stderr': -2}
2021-04-14 16:05:57,565 [DEBUG] (MainThread) Running: ['xfs_admin', '-U', 'generate', '/dev/onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1']
2021-04-14 16:05:57,566 [DEBUG] (MainThread) Calling: (['xfs_admin', '-U', 'generate', '/dev/onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'],) {'stderr': -2, 'close_fds': True}
2021-04-14 16:05:57,957 [DEBUG] (MainThread) Calling: (['lvchange', '--setactivationskip', 'n', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:05:58,040 [DEBUG] (MainThread) Returned: b'Logical volume onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1 changed.'
2021-04-14 16:05:58,041 [DEBUG] (MainThread) Calling: (['lvchange', '--setactivationskip', 'y', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:05:58,116 [DEBUG] (MainThread) Returned: b'Logical volume onn/ovirt-node-ng-4.4.5.1-0.20210323.0 changed.'
2021-04-14 16:05:58,117 [DEBUG] (MainThread) Got: <LV 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1' /> and <LV 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0' />
2021-04-14 16:05:58,117 [DEBUG] (MainThread) Calling: (['mount', '--make-rprivate', '/var'],) {'close_fds': True, 'stderr': -2}
2021-04-14 16:05:58,122 [DEBUG] (MainThread) Calling: (['systemctl', 'is-active', 'vdsmd'],) {'close_fds': True, 'stderr': -2}
2021-04-14 16:05:58,141 [DEBUG] (MainThread) Returned: b'active'
2021-04-14 16:05:58,141 [DEBUG] (MainThread) Calling: (['systemctl', 'stop', 'vdsmd'],) {'close_fds': True, 'stderr': -2}
2021-04-14 16:06:08,940 [DEBUG] (MainThread) Calling: (['systemctl', 'stop', 'supervdsmd'],) {'close_fds': True, 'stderr': -2}
2021-04-14 16:06:09,072 [DEBUG] (MainThread) Calling: (['systemctl', 'stop', 'vdsm-network'],) {'close_fds': True, 'stderr': -2}
2021-04-14 16:06:09,090 [DEBUG] (MainThread) Calling: (['vgchange', '-ay', '--select', 'vg_tags = imgbased:vg'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:06:09,167 [DEBUG] (MainThread) Returned: b'11 logical volume(s) in volume group "onn" now active'
2021-04-14 16:06:09,168 [DEBUG] (MainThread) Calling: (['lvs', '--noheadings', '--ignoreskippedcluster', '-o', 'lv_full_name', '--select', 'lv_tags = imgbased:base || lv_tags = imgbased:layer'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:06:09,229 [DEBUG] (MainThread) Returned: b'onn/ovirt-node-ng-4.4.4.1-0.20210208.0+1\n onn/ovirt-node-ng-4.4.5.1-0.20210323.0 \n onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'
2021-04-14 16:06:09,230 [DEBUG] (MainThread) All LV names: ['onn/ovirt-node-ng-4.4.4.1-0.20210208.0+1', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1']
2021-04-14 16:06:09,230 [DEBUG] (MainThread) All LVS: [<LV 'onn/ovirt-node-ng-4.4.4.1-0.20210208.0+1' />, <LV 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0' />, <LV 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1' />]
2021-04-14 16:06:09,230 [DEBUG] (MainThread) Calling: (['lvs', '--noheadings', '--ignoreskippedcluster', '-olv_tags', 'onn/ovirt-node-ng-4.4.4.1-0.20210208.0+1'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:06:09,293 [DEBUG] (MainThread) Returned: b'imgbased:layer'
2021-04-14 16:06:09,294 [DEBUG] (MainThread) Calling: (['lvs', '--noheadings', '--ignoreskippedcluster', '-olv_tags', 'onn/ovirt-node-ng-4.4.4.1-0.20210208.0+1'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:06:09,357 [DEBUG] (MainThread) Returned: b'imgbased:layer'
2021-04-14 16:06:09,358 [DEBUG] (MainThread) Calling: (['lvs', '--noheadings', '--ignoreskippedcluster', '-olv_tags', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:06:09,417 [DEBUG] (MainThread) Returned: b'imgbased:base'
2021-04-14 16:06:09,418 [DEBUG] (MainThread) Calling: (['lvs', '--noheadings', '--ignoreskippedcluster', '-olv_tags', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:06:09,490 [DEBUG] (MainThread) Returned: b'imgbased:layer'
2021-04-14 16:06:09,491 [DEBUG] (MainThread) Calling: (['lvs', '--noheadings', '--ignoreskippedcluster', '-olv_tags', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:06:09,557 [DEBUG] (MainThread) Returned: b'imgbased:layer'
2021-04-14 16:06:09,558 [DEBUG] (MainThread) Our LVS: [<LV 'onn/ovirt-node-ng-4.4.4.1-0.20210208.0+1' />, <LV 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0' />, <LV 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1' />]
2021-04-14 16:06:09,558 [DEBUG] (MainThread) Names: ['ovirt-node-ng-4.4.4.1-0.20210208.0+1', 'ovirt-node-ng-4.4.5.1-0.20210323.0', 'ovirt-node-ng-4.4.5.1-0.20210323.0+1']
2021-04-14 16:06:09,558 [DEBUG] (MainThread) Images: [<Layer ovirt-node-ng-4.4.4.1-0.20210208.0+1 />, <Base ovirt-node-ng-4.4.5.1-0.20210323.0 [] />, <Layer ovirt-node-ng-4.4.5.1-0.20210323.0+1 />]
2021-04-14 16:06:09,558 [DEBUG] (MainThread) Calling: (['umount', '-l', '/tmp/mnt.o5bFp'],) {'close_fds': True, 'stderr': -2}
2021-04-14 16:06:09,579 [DEBUG] (MainThread) Calling: (['rmdir', '/tmp/mnt.o5bFp'],) {'close_fds': True, 'stderr': -2}
2021-04-14 16:06:09,583 [DEBUG] (MainThread) Calling: (['umount', '-l', '/tmp/mnt.CcHrE'],) {'close_fds': True, 'stderr': -2}
2021-04-14 16:06:09,601 [DEBUG] (MainThread) Calling: (['rmdir', '/tmp/mnt.CcHrE'],) {'close_fds': True, 'stderr': -2}
2021-04-14 16:06:09,605 [ERROR] (MainThread) Update failed, resetting registered LVs
2021-04-14 16:06:09,606 [DEBUG] (MainThread) Calling: (['sync'],) {'close_fds': True, 'stderr': -2}
2021-04-14 16:06:09,615 [DEBUG] (MainThread) Calling: (['lvs', '--noheadings', '--ignoreskippedcluster', '-olv_dm_path', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:06:09,683 [DEBUG] (MainThread) Returned: b'/dev/mapper/onn-ovirt--node--ng--4.4.5.1--0.20210323.0'
2021-04-14 16:06:09,684 [DEBUG] (MainThread) Calling: (['lvremove', '-ff', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:06:09,820 [DEBUG] (MainThread) Returned: b'Logical volume "ovirt-node-ng-4.4.5.1-0.20210323.0" successfully removed'
2021-04-14 16:06:09,821 [DEBUG] (MainThread) Calling: (['lvs', '--noheadings', '--ignoreskippedcluster', '-olv_dm_path', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:06:09,886 [DEBUG] (MainThread) Returned: b'/dev/mapper/onn-ovirt--node--ng--4.4.5.1--0.20210323.0+1'
2021-04-14 16:06:09,887 [DEBUG] (MainThread) Calling: (['lvremove', '-ff', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'],) {'stderr': <_io.TextIOWrapper name='/dev/null' mode='w' encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:06:10,207 [DEBUG] (MainThread) Returned: b'Logical volume "ovirt-node-ng-4.4.5.1-0.20210323.0+1" successfully removed'
Traceback (most recent call last):
File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/__main__.py", line 53, in <module>
CliApplication()
File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/__init__.py", line 82, in CliApplication
app.hooks.emit("post-arg-parse", args)
File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/hooks.py", line 120, in emit
cb(self.context, *args)
File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/plugins/update.py", line 75, in post_argparse
six.reraise(*exc_info)
File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
raise value
File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/plugins/update.py", line 66, in post_argparse
base, _ = LiveimgExtractor(app.imgbase).extract(args.FILENAME)
File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/plugins/update.py", line 148, in extract
"%s" % size, nvr)
File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/plugins/update.py", line 128, in add_base_with_tree
new_layer_lv = self.imgbase.add_layer(new_base)
File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/imgbase.py", line 213, in add_layer
self.hooks.emit("new-layer-added", prev_lv, new_lv)
File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/hooks.py", line 120, in emit
cb(self.context, *args)
File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/plugins/osupdater.py", line 88, in on_new_layer
previous_layer_lv = get_prev_layer_lv(imgbase, new_lv)
File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/plugins/osupdater.py", line 143, in get_prev_layer_lv
layer_before = imgbase.naming.layer_before(new_layer)
File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/naming.py", line 74, in layer_before
layers = self.layers()
File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/naming.py", line 58, in layers
for b in self.tree():
File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/naming.py", line 226, in tree
bases[img.base.nvr].layers.append(img)
KeyError: <NVR ovirt-node-ng-4.4.4.1-0.20210208.0 />
--------------------
