10:53 a.m.
Hi.
I have a 2 node cluster (both installed from the Ovirt-node iso) , 1 server is 100% fine,
but one server I tried to update when there were issues with the ovirtmgmt network (I
wasn't aware of at the time) - as a result the server update failed, and was still on
ovirt-node 4.4.4 after - I could see the LVM layer for 4.4.5 though.
The network is fine now (hence the other server being ok) - I tried reinstalling the
package ovirt-node-ng-image-update-4.4.5.1-1.el8.noarch.rpm but it failed due to LVM lv
for 4.4.5 already existing ...
So I tried removing the LV - i.e
# lvremove /dev/onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1
however now I'm sure I have less 4.4.4 lvs ..
i.e
------
[root@ng2-ovirt-kvm3 mcox]# lvs
LV VG Attr LSize Pool Origin Data% Meta%
Move Log Cpy%Sync Convert
home onn Vwi-aotz-- 1.00g pool00 10.05
ovirt-node-ng-4.4.4.1-0.20210208.0+1 onn Vwi-aotz-- 191.00g pool00 2.27
pool00 onn twi-aotz-- 218.00g 2.48 1.92
swap onn -wi-ao---- 4.00g
tmp onn Vwi-aotz-- 1.00g pool00 2.24
var onn Vwi-aotz-- 15.00g pool00 2.80
var_crash onn Vwi-aotz-- 10.00g pool00 0.21
var_log onn Vwi-aotz-- 8.00g pool00 5.97
var_log_audit onn Vwi-aotz-- 2.00g pool00 1.14
----
Where as on the working server I have these Ovirt Node LVs
ovirt-node-ng-4.4.4.1-0.20210208.0 onn Vwi---tz-k 191.00g pool00 root
ovirt-node-ng-4.4.4.1-0.20210208.0+1 onn Vwi-a-tz-- 191.00g pool00
ovirt-node-ng-4.4.4.1-0.20210208.0 1.86
ovirt-node-ng-4.4.5.1-0.20210323.0 onn Vri---tz-k 191.00g pool00
ovirt-node-ng-4.4.5.1-0.20210323.0+1 onn Vwi-aotz-- 191.00g pool00
ovirt-node-ng-4.4.5.1-0.20210323.0 1.58
Anyway I tried to reinstall the ovirt-node-ng-image-update-4.4.5.1-1.el8.noarch.rpm
package after removal of lv and files in /boot
But install still fails .. Is there anything I can do to update this node - or should I
bite the bullet and reinstall ?
this is from /var/log/imgbased.log ->
--------------------------------------
2021-04-14 16:05:56,634 [INFO] (MainThread) Adding a new layer after <Base
ovirt-node-ng-4.4.5.1-0.20210323.0 [] />
2021-04-14 16:05:56,634 [INFO] (MainThread) Adding a new layer after <Base
ovirt-node-ng-4.4.5.1-0.20210323.0 [] />
2021-04-14 16:05:56,634 [DEBUG] (MainThread) Basing new layer on previous: <Base
ovirt-node-ng-4.4.5.1-0.20210323.0 [] />
2021-04-14 16:05:56,634 [DEBUG] (MainThread) Finding next layer based on <Base
ovirt-node-ng-4.4.5.1-0.20210323.0 [] />
2021-04-14 16:05:56,634 [DEBUG] (MainThread) Suggesting for layer for base
ovirt-node-ng-4.4.5.1-0.20210323.0
2021-04-14 16:05:56,635 [DEBUG] (MainThread) ... without layers
2021-04-14 16:05:56,635 [INFO] (MainThread) New layer will be: <Layer
ovirt-node-ng-4.4.5.1-0.20210323.0+1 />
2021-04-14 16:05:56,635 [DEBUG] (MainThread) Calling: (['vgs',
'--noheadings', '--ignoreskippedcluster', '--select', 'vg_tags
= imgbased:vg', '-o', 'vg_name'],) {'stderr':
<_io.TextIOWrapper name='/dev/null' mode='w'
encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:05:56,697 [DEBUG] (MainThread) Returned: b'onn'
2021-04-14 16:05:56,698 [DEBUG] (MainThread) Calling: (['vgs',
'--noheadings', '--ignoreskippedcluster', '@imgbased:pool',
'-o', 'lv_full_name'],) {'stderr': <_io.TextIOWrapper
name='/dev/null' mode='w' encoding='UTF-8'>,
'close_fds': True}
2021-04-14 16:05:56,757 [DEBUG] (MainThread) Returned: b'onn/pool00'
2021-04-14 16:05:56,758 [DEBUG] (MainThread) Calling: (['lvs',
'--noheadings', '--ignoreskippedcluster', '--nosuffix',
'--units', 'm', '-o', 'metadata_percent,lv_metadata_size',
'onn/pool00'],) {'stderr': <_io.TextIOWrapper name='/dev/null'
mode='w' encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:05:56,829 [DEBUG] (MainThread) Returned: b'2.05 1024.00'
2021-04-14 16:05:56,830 [DEBUG] (MainThread) Pool: onn/pool00, metadata size=1024.0M
(2.05%)
2021-04-14 16:05:56,830 [DEBUG] (MainThread) Calling: (['lvchange',
'--activate', 'y', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0',
'--ignoreactivationskip'],) {'stderr': <_io.TextIOWrapper
name='/dev/null' mode='w' encoding='UTF-8'>,
'close_fds': True}
2021-04-14 16:05:56,953 [DEBUG] (MainThread) Calling: (['lvcreate',
'--snapshot', '--name', 'ovirt-node-ng-4.4.5.1-0.20210323.0+1',
'onn/ovirt-node-ng-4.4.5.1-0.20210323.0'],) {'close_fds': True,
'stderr': -2}
2021-04-14 16:05:57,111 [DEBUG] (MainThread) Returned: b'WARNING: Sum of all thin
volume sizes (610.00 GiB) exceeds the size of thin pool onn/pool00 and the size of whole
volume group (<277.78 GiB).\n Logical volume
"ovirt-node-ng-4.4.5.1-0.20210323.0+1" created.'
2021-04-14 16:05:57,112 [DEBUG] (MainThread) Calling: (['lvchange',
'--activate', 'y', 'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1',
'--ignoreactivationskip'],) {'stderr': <_io.TextIOWrapper
name='/dev/null' mode='w' encoding='UTF-8'>,
'close_fds': True}
2021-04-14 16:05:57,231 [DEBUG] (MainThread) Calling: (['lvchange',
'--addtag', 'imgbased:layer',
'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'],) {'stderr':
<_io.TextIOWrapper name='/dev/null' mode='w'
encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:05:57,310 [DEBUG] (MainThread) Returned: b'Logical volume
onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1 changed.'
2021-04-14 16:05:57,311 [DEBUG] (MainThread) Calling: (['lvs',
'--noheadings', '--ignoreskippedcluster', '-olv_path',
'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'],) {'stderr':
<_io.TextIOWrapper name='/dev/null' mode='w'
encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:05:57,383 [DEBUG] (MainThread) Returned:
b'/dev/onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'
2021-04-14 16:05:57,400 [DEBUG] (MainThread) Calling: (['mktemp', '-d',
'--tmpdir', 'mnt.XXXXX'],) {'close_fds': True, 'stderr':
-2}
2021-04-14 16:05:57,404 [DEBUG] (MainThread) Returned: b'/tmp/mnt.aGJ3A'
2021-04-14 16:05:57,404 [DEBUG] (MainThread) Calling: (['mount',
'-onouuid', '/dev/onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1',
'/tmp/mnt.aGJ3A'],) {'close_fds': True, 'stderr': -2}
2021-04-14 16:05:57,537 [DEBUG] (MainThread) Calling: (['umount', '-l',
'/tmp/mnt.aGJ3A'],) {'close_fds': True, 'stderr': -2}
2021-04-14 16:05:57,561 [DEBUG] (MainThread) Calling: (['rmdir',
'/tmp/mnt.aGJ3A'],) {'close_fds': True, 'stderr': -2}
2021-04-14 16:05:57,565 [DEBUG] (MainThread) Running: ['xfs_admin', '-U',
'generate', '/dev/onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1']
2021-04-14 16:05:57,566 [DEBUG] (MainThread) Calling: (['xfs_admin', '-U',
'generate', '/dev/onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'],)
{'stderr': -2, 'close_fds': True}
2021-04-14 16:05:57,957 [DEBUG] (MainThread) Calling: (['lvchange',
'--setactivationskip', 'n',
'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'],) {'stderr':
<_io.TextIOWrapper name='/dev/null' mode='w'
encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:05:58,040 [DEBUG] (MainThread) Returned: b'Logical volume
onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1 changed.'
2021-04-14 16:05:58,041 [DEBUG] (MainThread) Calling: (['lvchange',
'--setactivationskip', 'y',
'onn/ovirt-node-ng-4.4.5.1-0.20210323.0'],) {'stderr':
<_io.TextIOWrapper name='/dev/null' mode='w'
encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:05:58,116 [DEBUG] (MainThread) Returned: b'Logical volume
onn/ovirt-node-ng-4.4.5.1-0.20210323.0 changed.'
2021-04-14 16:05:58,117 [DEBUG] (MainThread) Got: <LV
'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1' /> and <LV
'onn/ovirt-node-ng-4.4.5.1-0.20210323.0' />
2021-04-14 16:05:58,117 [DEBUG] (MainThread) Calling: (['mount',
'--make-rprivate', '/var'],) {'close_fds': True, 'stderr':
-2}
2021-04-14 16:05:58,122 [DEBUG] (MainThread) Calling: (['systemctl',
'is-active', 'vdsmd'],) {'close_fds': True, 'stderr': -2}
2021-04-14 16:05:58,141 [DEBUG] (MainThread) Returned: b'active'
2021-04-14 16:05:58,141 [DEBUG] (MainThread) Calling: (['systemctl',
'stop', 'vdsmd'],) {'close_fds': True, 'stderr': -2}
2021-04-14 16:06:08,940 [DEBUG] (MainThread) Calling: (['systemctl',
'stop', 'supervdsmd'],) {'close_fds': True, 'stderr': -2}
2021-04-14 16:06:09,072 [DEBUG] (MainThread) Calling: (['systemctl',
'stop', 'vdsm-network'],) {'close_fds': True, 'stderr':
-2}
2021-04-14 16:06:09,090 [DEBUG] (MainThread) Calling: (['vgchange', '-ay',
'--select', 'vg_tags = imgbased:vg'],) {'stderr':
<_io.TextIOWrapper name='/dev/null' mode='w'
encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:06:09,167 [DEBUG] (MainThread) Returned: b'11 logical volume(s) in
volume group "onn" now active'
2021-04-14 16:06:09,168 [DEBUG] (MainThread) Calling: (['lvs',
'--noheadings', '--ignoreskippedcluster', '-o',
'lv_full_name', '--select', 'lv_tags = imgbased:base || lv_tags =
imgbased:layer'],) {'stderr': <_io.TextIOWrapper name='/dev/null'
mode='w' encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:06:09,229 [DEBUG] (MainThread) Returned:
b'onn/ovirt-node-ng-4.4.4.1-0.20210208.0+1\n onn/ovirt-node-ng-4.4.5.1-0.20210323.0
\n onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'
2021-04-14 16:06:09,230 [DEBUG] (MainThread) All LV names:
['onn/ovirt-node-ng-4.4.4.1-0.20210208.0+1',
'onn/ovirt-node-ng-4.4.5.1-0.20210323.0',
'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1']
2021-04-14 16:06:09,230 [DEBUG] (MainThread) All LVS: [<LV
'onn/ovirt-node-ng-4.4.4.1-0.20210208.0+1' />, <LV
'onn/ovirt-node-ng-4.4.5.1-0.20210323.0' />, <LV
'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1' />]
2021-04-14 16:06:09,230 [DEBUG] (MainThread) Calling: (['lvs',
'--noheadings', '--ignoreskippedcluster', '-olv_tags',
'onn/ovirt-node-ng-4.4.4.1-0.20210208.0+1'],) {'stderr':
<_io.TextIOWrapper name='/dev/null' mode='w'
encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:06:09,293 [DEBUG] (MainThread) Returned: b'imgbased:layer'
2021-04-14 16:06:09,294 [DEBUG] (MainThread) Calling: (['lvs',
'--noheadings', '--ignoreskippedcluster', '-olv_tags',
'onn/ovirt-node-ng-4.4.4.1-0.20210208.0+1'],) {'stderr':
<_io.TextIOWrapper name='/dev/null' mode='w'
encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:06:09,357 [DEBUG] (MainThread) Returned: b'imgbased:layer'
2021-04-14 16:06:09,358 [DEBUG] (MainThread) Calling: (['lvs',
'--noheadings', '--ignoreskippedcluster', '-olv_tags',
'onn/ovirt-node-ng-4.4.5.1-0.20210323.0'],) {'stderr':
<_io.TextIOWrapper name='/dev/null' mode='w'
encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:06:09,417 [DEBUG] (MainThread) Returned: b'imgbased:base'
2021-04-14 16:06:09,418 [DEBUG] (MainThread) Calling: (['lvs',
'--noheadings', '--ignoreskippedcluster', '-olv_tags',
'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'],) {'stderr':
<_io.TextIOWrapper name='/dev/null' mode='w'
encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:06:09,490 [DEBUG] (MainThread) Returned: b'imgbased:layer'
2021-04-14 16:06:09,491 [DEBUG] (MainThread) Calling: (['lvs',
'--noheadings', '--ignoreskippedcluster', '-olv_tags',
'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'],) {'stderr':
<_io.TextIOWrapper name='/dev/null' mode='w'
encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:06:09,557 [DEBUG] (MainThread) Returned: b'imgbased:layer'
2021-04-14 16:06:09,558 [DEBUG] (MainThread) Our LVS: [<LV
'onn/ovirt-node-ng-4.4.4.1-0.20210208.0+1' />, <LV
'onn/ovirt-node-ng-4.4.5.1-0.20210323.0' />, <LV
'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1' />]
2021-04-14 16:06:09,558 [DEBUG] (MainThread) Names:
['ovirt-node-ng-4.4.4.1-0.20210208.0+1',
'ovirt-node-ng-4.4.5.1-0.20210323.0',
'ovirt-node-ng-4.4.5.1-0.20210323.0+1']
2021-04-14 16:06:09,558 [DEBUG] (MainThread) Images: [<Layer
ovirt-node-ng-4.4.4.1-0.20210208.0+1 />, <Base ovirt-node-ng-4.4.5.1-0.20210323.0 []
/>, <Layer ovirt-node-ng-4.4.5.1-0.20210323.0+1 />]
2021-04-14 16:06:09,558 [DEBUG] (MainThread) Calling: (['umount', '-l',
'/tmp/mnt.o5bFp'],) {'close_fds': True, 'stderr': -2}
2021-04-14 16:06:09,579 [DEBUG] (MainThread) Calling: (['rmdir',
'/tmp/mnt.o5bFp'],) {'close_fds': True, 'stderr': -2}
2021-04-14 16:06:09,583 [DEBUG] (MainThread) Calling: (['umount', '-l',
'/tmp/mnt.CcHrE'],) {'close_fds': True, 'stderr': -2}
2021-04-14 16:06:09,601 [DEBUG] (MainThread) Calling: (['rmdir',
'/tmp/mnt.CcHrE'],) {'close_fds': True, 'stderr': -2}
2021-04-14 16:06:09,605 [ERROR] (MainThread) Update failed, resetting registered LVs
2021-04-14 16:06:09,606 [DEBUG] (MainThread) Calling: (['sync'],)
{'close_fds': True, 'stderr': -2}
2021-04-14 16:06:09,615 [DEBUG] (MainThread) Calling: (['lvs',
'--noheadings', '--ignoreskippedcluster', '-olv_dm_path',
'onn/ovirt-node-ng-4.4.5.1-0.20210323.0'],) {'stderr':
<_io.TextIOWrapper name='/dev/null' mode='w'
encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:06:09,683 [DEBUG] (MainThread) Returned:
b'/dev/mapper/onn-ovirt--node--ng--4.4.5.1--0.20210323.0'
2021-04-14 16:06:09,684 [DEBUG] (MainThread) Calling: (['lvremove', '-ff',
'onn/ovirt-node-ng-4.4.5.1-0.20210323.0'],) {'stderr':
<_io.TextIOWrapper name='/dev/null' mode='w'
encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:06:09,820 [DEBUG] (MainThread) Returned: b'Logical volume
"ovirt-node-ng-4.4.5.1-0.20210323.0" successfully removed'
2021-04-14 16:06:09,821 [DEBUG] (MainThread) Calling: (['lvs',
'--noheadings', '--ignoreskippedcluster', '-olv_dm_path',
'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'],) {'stderr':
<_io.TextIOWrapper name='/dev/null' mode='w'
encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:06:09,886 [DEBUG] (MainThread) Returned:
b'/dev/mapper/onn-ovirt--node--ng--4.4.5.1--0.20210323.0+1'
2021-04-14 16:06:09,887 [DEBUG] (MainThread) Calling: (['lvremove', '-ff',
'onn/ovirt-node-ng-4.4.5.1-0.20210323.0+1'],) {'stderr':
<_io.TextIOWrapper name='/dev/null' mode='w'
encoding='UTF-8'>, 'close_fds': True}
2021-04-14 16:06:10,207 [DEBUG] (MainThread) Returned: b'Logical volume
"ovirt-node-ng-4.4.5.1-0.20210323.0+1" successfully removed'
Traceback (most recent call last):
File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File
"/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/__main__.py", line
53, in <module>
CliApplication()
File
"/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/__init__.py", line
82, in CliApplication
app.hooks.emit("post-arg-parse", args)
File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/hooks.py",
line 120, in emit
cb(self.context, *args)
File
"/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/plugins/update.py",
line 75, in post_argparse
six.reraise(*exc_info)
File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/six.py", line 703,
in reraise
raise value
File
"/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/plugins/update.py",
line 66, in post_argparse
base, _ = LiveimgExtractor(app.imgbase).extract(args.FILENAME)
File
"/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/plugins/update.py",
line 148, in extract
"%s" % size, nvr)
File
"/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/plugins/update.py",
line 128, in add_base_with_tree
new_layer_lv = self.imgbase.add_layer(new_base)
File
"/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/imgbase.py", line
213, in add_layer
self.hooks.emit("new-layer-added", prev_lv, new_lv)
File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/hooks.py",
line 120, in emit
cb(self.context, *args)
File
"/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/plugins/osupdater.py",
line 88, in on_new_layer
previous_layer_lv = get_prev_layer_lv(imgbase, new_lv)
File
"/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/plugins/osupdater.py",
line 143, in get_prev_layer_lv
layer_before = imgbase.naming.layer_before(new_layer)
File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/naming.py",
line 74, in layer_before
layers = self.layers()
File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/naming.py",
line 58, in layers
for b in self.tree():
File "/tmp/tmp.cyXrOTxRgb/usr/lib/python3.6/site-packages/imgbased/naming.py",
line 226, in tree
bases[img.base.nvr].layers.append(img)
KeyError: <NVR ovirt-node-ng-4.4.4.1-0.20210208.0 />
--------------------