Hi Yuval,

yes, it was a manual lvremove.

[root@ovn-monster tmp]# lvm
lvm> lvs onn_ovn-monster
  LV                                   VG              Attr       LSize    Pool   Origin                             Data%  Meta%  Move Log Cpy%Sync Convert
  home                                 onn_ovn-monster Vwi-aotz--    1,00g pool00                                    4,79                                   
  ovirt-node-ng-4.2.1.1-0.20180223.0   onn_ovn-monster Vwi---tz-k <252,38g pool00 root                                                                      
  ovirt-node-ng-4.2.1.1-0.20180223.0+1 onn_ovn-monster Vwi-a-tz-- <252,38g pool00 ovirt-node-ng-4.2.1.1-0.20180223.0 2,53                                   
  ovirt-node-ng-4.2.3-0.20180524.0     onn_ovn-monster Vri---tz-k <252,38g pool00                                                                           
  ovirt-node-ng-4.2.3-0.20180524.0+1   onn_ovn-monster Vwi-aotz-- <252,38g pool00 ovirt-node-ng-4.2.3-0.20180524.0   2,63                                   
  ovirt-node-ng-4.2.3.1-0.20180530.0   onn_ovn-monster Vri---tz-k <252,38g pool00                                                                           
  ovirt-node-ng-4.2.3.1-0.20180530.0+1 onn_ovn-monster Vwi-a-tz-- <252,38g pool00 ovirt-node-ng-4.2.3.1-0.20180530.0 0,86                                   
  ovirt-node-ng-4.2.4-0.20180626.0     onn_ovn-monster Vri---tz-k <252,38g pool00                                                                           
  ovirt-node-ng-4.2.4-0.20180626.0+1   onn_ovn-monster Vwi-a-tz-- <252,38g pool00 ovirt-node-ng-4.2.4-0.20180626.0   0,87                                   
  pool00                               onn_ovn-monster twi-aotz-- <279,38g                                           8,19   1,27                            
  root                                 onn_ovn-monster Vwi-a-tz-- <252,38g pool00                                    1,24                                   
  swap                                 onn_ovn-monster -wi-ao----    4,00g                                                                                  
  tmp                                  onn_ovn-monster Vwi-aotz--    1,00g pool00                                    5,00                                   
  var                                  onn_ovn-monster Vwi-aotz--   15,00g pool00                                    3,55                                   
  var_crash                            onn_ovn-monster Vwi-aotz--   10,00g pool00                                    2,86                                   
  var_log                              onn_ovn-monster Vwi-aotz--    8,00g pool00                                    38,62                                  
  var_log_audit                        onn_ovn-monster Vwi-aotz--    2,00g pool00                                    6,75                                   
lvm> lvremove      onn_ovn-monster ovirt-node-ng-4.2.4-0.20180626.0 ovirt-node-ng-4.2.4-0.20180626.0+1
  Logical volume onn_ovn-monster/swap in use.
Removing pool "pool00" will remove 15 dependent volume(s). Proceed? [y/n]: n
  Logical volume "pool00" not removed.
  Logical volume onn_ovn-monster/var_log_audit contains a filesystem in use.
  Logical volume onn_ovn-monster/var_log contains a filesystem in use.
  Logical volume onn_ovn-monster/var contains a filesystem in use.
  Logical volume onn_ovn-monster/tmp contains a filesystem in use.
  Logical volume onn_ovn-monster/home contains a filesystem in use.
Do you really want to remove active logical volume onn_ovn-monster/root? [y/n]: n
  Logical volume onn_ovn-monster/root not removed.


  Logical volume "ovirt-node-ng-4.2.1.1-0.20180223.0" successfully removed
Do you really want to remove active logical volume onn_ovn-monster/ovirt-node-ng-4.2.1.1-0.20180223.0+1? [y/n]: n  ###### my mistake here !



  Logical volume onn_ovn-monster/ovirt-node-ng-4.2.1.1-0.20180223.0+1 not removed.
  Logical volume onn_ovn-monster/var_crash contains a filesystem in use.
  Logical volume "ovirt-node-ng-4.2.3-0.20180524.0" successfully removed
  Logical volume onn_ovn-monster/ovirt-node-ng-4.2.3-0.20180524.0+1 contains a filesystem in use.
  Logical volume "ovirt-node-ng-4.2.3.1-0.20180530.0" successfully removed
Do you really want to remove active logical volume onn_ovn-monster/ovirt-node-ng-4.2.3.1-0.20180530.0+1? [y/n]: n
  Logical volume onn_ovn-monster/ovirt-node-ng-4.2.3.1-0.20180530.0+1 not removed.
  Logical volume "ovirt-node-ng-4.2.4-0.20180626.0" successfully removed
Do you really want to remove active logical volume onn_ovn-monster/ovirt-node-ng-4.2.4-0.20180626.0+1? [y/n]: y
  Logical volume "ovirt-node-ng-4.2.4-0.20180626.0+1" successfully removed
  Volume group "ovirt-node-ng-4.2.4-0.20180626.0" not found
  Cannot process volume group ovirt-node-ng-4.2.4-0.20180626.0
  Volume group "ovirt-node-ng-4.2.4-0.20180626.0+1" not found
  Cannot process volume group ovirt-node-ng-4.2.4-0.20180626.0+1
lvm> 
lvm> lvs
  LV                                   VG              Attr       LSize    Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home                                 onn_ovn-monster Vwi-aotz--    1,00g pool00        4,79                                   
  ovirt-node-ng-4.2.1.1-0.20180223.0+1 onn_ovn-monster Vwi-a-tz-- <252,38g pool00        2,53                                   
  ovirt-node-ng-4.2.3-0.20180524.0+1   onn_ovn-monster Vwi-aotz-- <252,38g pool00        2,63                                   
  ovirt-node-ng-4.2.3.1-0.20180530.0+1 onn_ovn-monster Vwi-a-tz-- <252,38g pool00        0,86                                   
  pool00                               onn_ovn-monster twi-aotz-- <279,38g               7,34   1,11                            
  root                                 onn_ovn-monster Vwi-a-tz-- <252,38g pool00        1,24                                   
  swap                                 onn_ovn-monster -wi-ao----    4,00g                                                      
  tmp                                  onn_ovn-monster Vwi-aotz--    1,00g pool00        5,00                                   
  var                                  onn_ovn-monster Vwi-aotz--   15,00g pool00        3,55                                   
  var_crash                            onn_ovn-monster Vwi-aotz--   10,00g pool00        2,86                                   
  var_log                              onn_ovn-monster Vwi-aotz--    8,00g pool00        38,62                                  
  var_log_audit                        onn_ovn-monster Vwi-aotz--    2,00g pool00        6,75                                   
  turbo                                ovirt           -wi-ao----  894,25g  

##### Correct mistake

lvm> lvremove onn_ovn-monster/ovirt-node-ng-4.2.1.1-0.20180223.0+1 
Do you really want to remove active logical volume onn_ovn-monster/ovirt-node-ng-4.2.1.1-0.20180223.0+1? [y/n]: y
  Logical volume "ovirt-node-ng-4.2.1.1-0.20180223.0+1" successfully removed
lvm> quit
  Exiting.

try re-reinstall ...
Am 03.07.2018 um 21:57 schrieb Yuval Turgeman <yturgema@redhat.com>:

Hi Oliver,

The KeyError happens because there are no bases for the layers.  For each LV that ends with a +1, there should be a base read-only LV without +1.  So for 3 ovirt-node-ng images, you're supposed to have 6 layers.  This is the reason nodectl info fails, and the upgrade will fail also.  In your original email it looks OK - I have never seen this happen, was this a manual lvremove ? I need to reproduce this and check what can be done.

You can find me on #ovirt (irc.oftc.net) also :)


On Tue, Jul 3, 2018 at 10:41 PM, Oliver Riesener <Oliver.Riesener@hs-bremen.de> wrote:
Yuval, here comes the lvs output.

The IO Errors are because Node is in maintenance.
The LV root is from previous installed centos 7.5.
The i have installed node-ng 4.2.1 and got this MIX.
The LV turbo is a SSD in it’s own VG named ovirt.

I have removed LV ovirt-node-ng-4.2.1-0.20180223.0 and (+1) removed
because nodectl info error:

KeyError: <NVR ovirt-node-ng-4.2.1.1-0.20180223.0

Now i get the error @4.2.3:
[root@ovn-monster ~]# nodectl info
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/site-packages/nodectl/__main__.py", line 42, in <module>
    CliApplication()
  File "/usr/lib/python2.7/site-packages/nodectl/__init__.py", line 200, in CliApplication
    return cmdmap.command(args)
  File "/usr/lib/python2.7/site-packages/nodectl/__init__.py", line 118, in command
    return self.commands[command](**kwargs)
  File "/usr/lib/python2.7/site-packages/nodectl/__init__.py", line 76, in info
    Info(self.imgbased, self.machine).write()
  File "/usr/lib/python2.7/site-packages/nodectl/info.py", line 45, in __init__
    self._fetch_information()
  File "/usr/lib/python2.7/site-packages/nodectl/info.py", line 49, in _fetch_information
    self._get_layout()
  File "/usr/lib/python2.7/site-packages/nodectl/info.py", line 66, in _get_layout
    layout = LayoutParser(self.app.imgbase.layout()).parse()
  File "/usr/lib/python2.7/site-packages/imgbased/imgbase.py", line 155, in layout
    return self.naming.layout()
  File "/usr/lib/python2.7/site-packages/imgbased/naming.py", line 109, in layout
    tree = self.tree(lvs)
  File "/usr/lib/python2.7/site-packages/imgbased/naming.py", line 224, in tree
    bases[img.base.nvr].layers.append(img)
KeyError: <NVR ovirt-node-ng-4.2.3-0.20180524.0 />

lvs -a

[root@ovn-monster ~]# lvs -a
  /dev/mapper/36090a02860ee716bee5e05b11dc52616: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
  /dev/mapper/36090a02860ee716bee5e05b11dc52616: read failed after 0 of 4096 at 5497568559104: Eingabe-/Ausgabefehler
  /dev/mapper/36090a02860ee716bee5e05b11dc52616: read failed after 0 of 4096 at 5497568616448: Eingabe-/Ausgabefehler
  /dev/mapper/36090a02860ee716bee5e05b11dc52616: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
  /dev/mapper/36090a02860ee9137c5ae35cd4bc5f6b8: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
  /dev/mapper/36090a02860ee9137c5ae35cd4bc5f6b8: read failed after 0 of 4096 at 1099526242304: Eingabe-/Ausgabefehler
  /dev/mapper/36090a02860ee9137c5ae35cd4bc5f6b8: read failed after 0 of 4096 at 1099526299648: Eingabe-/Ausgabefehler
  /dev/mapper/36090a02860ee9137c5ae35cd4bc5f6b8: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
  /dev/mapper/36090a02860eea13dc5aed55e4cc57698: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
  /dev/mapper/36090a02860eea13dc5aed55e4cc57698: read failed after 0 of 4096 at 1099526242304: Eingabe-/Ausgabefehler
  /dev/mapper/36090a02860eea13dc5aed55e4cc57698: read failed after 0 of 4096 at 1099526299648: Eingabe-/Ausgabefehler
  /dev/mapper/36090a02860eea13dc5aed55e4cc57698: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/metadata: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/metadata: read failed after 0 of 4096 at 536805376: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/metadata: read failed after 0 of 4096 at 536862720: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/metadata: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/ids: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/ids: read failed after 0 of 4096 at 134152192: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/ids: read failed after 0 of 4096 at 134209536: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/ids: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/leases: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/leases: read failed after 0 of 4096 at 2147418112: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/leases: read failed after 0 of 4096 at 2147475456: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/leases: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/outbox: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/outbox: read failed after 0 of 4096 at 134152192: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/outbox: read failed after 0 of 4096 at 134209536: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/outbox: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/xleases: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/xleases: read failed after 0 of 4096 at 1073676288: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/xleases: read failed after 0 of 4096 at 1073733632: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/xleases: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/inbox: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/inbox: read failed after 0 of 4096 at 134152192: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/inbox: read failed after 0 of 4096 at 134209536: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/inbox: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/master: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/master: read failed after 0 of 4096 at 1073676288: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/master: read failed after 0 of 4096 at 1073733632: Eingabe-/Ausgabefehler
  /dev/675cb45d-3746-4f3b-b9ee-516612da50e5/master: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/metadata: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/metadata: read failed after 0 of 4096 at 536805376: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/metadata: read failed after 0 of 4096 at 536862720: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/metadata: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/metadata: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/metadata: read failed after 0 of 4096 at 536805376: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/metadata: read failed after 0 of 4096 at 536862720: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/metadata: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/ids: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/ids: read failed after 0 of 4096 at 134152192: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/ids: read failed after 0 of 4096 at 134209536: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/ids: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/ids: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/ids: read failed after 0 of 4096 at 134152192: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/ids: read failed after 0 of 4096 at 134209536: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/ids: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/leases: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/leases: read failed after 0 of 4096 at 2147418112: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/leases: read failed after 0 of 4096 at 2147475456: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/leases: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/leases: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/leases: read failed after 0 of 4096 at 2147418112: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/leases: read failed after 0 of 4096 at 2147475456: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/leases: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/outbox: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/outbox: read failed after 0 of 4096 at 134152192: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/outbox: read failed after 0 of 4096 at 134209536: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/outbox: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/xleases: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/xleases: read failed after 0 of 4096 at 1073676288: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/xleases: read failed after 0 of 4096 at 1073733632: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/xleases: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/inbox: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/inbox: read failed after 0 of 4096 at 134152192: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/inbox: read failed after 0 of 4096 at 134209536: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/inbox: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/master: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/master: read failed after 0 of 4096 at 1073676288: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/master: read failed after 0 of 4096 at 1073733632: Eingabe-/Ausgabefehler
  /dev/bcdbb66e-6196-4366-be25-a3e9ab948839/master: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/outbox: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/outbox: read failed after 0 of 4096 at 134152192: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/outbox: read failed after 0 of 4096 at 134209536: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/outbox: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/inbox: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/inbox: read failed after 0 of 4096 at 134152192: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/inbox: read failed after 0 of 4096 at 134209536: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/inbox: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/master: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/master: read failed after 0 of 4096 at 1073676288: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/master: read failed after 0 of 4096 at 1073733632: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/master: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/xleases: read failed after 0 of 4096 at 0: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/xleases: read failed after 0 of 4096 at 1073676288: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/xleases: read failed after 0 of 4096 at 1073733632: Eingabe-/Ausgabefehler
  /dev/c91974bf-fd64-4630-8005-e785b73acbef/xleases: read failed after 0 of 4096 at 4096: Eingabe-/Ausgabefehler
  LV                                   VG              Attr       LSize    Pool   Origin                           Data%  Meta%  Move Log Cpy%Sync Convert
  home                                 onn_ovn-monster Vwi-aotz--    1,00g pool00                                  4,79                                   
  [lvol0_pmspare]                      onn_ovn-monster ewi-------  144,00m                                                                                
  ovirt-node-ng-4.2.3-0.20180524.0+1   onn_ovn-monster Vwi-aotz-- <252,38g pool00                                  2,88                                   
  ovirt-node-ng-4.2.3.1-0.20180530.0+1 onn_ovn-monster Vwi-a-tz-- <252,38g pool00                                  0,86                                   
  ovirt-node-ng-4.2.4-0.20180626.0     onn_ovn-monster Vri-a-tz-k <252,38g pool00                                  0,85                                   
  ovirt-node-ng-4.2.4-0.20180626.0+1   onn_ovn-monster Vwi-a-tz-- <252,38g pool00 ovirt-node-ng-4.2.4-0.20180626.0 0,85                                   
  pool00                               onn_ovn-monster twi-aotz-- <279,38g                                         6,76   1,01                            
  [pool00_tdata]                       onn_ovn-monster Twi-ao---- <279,38g                                                                                
  [pool00_tmeta]                       onn_ovn-monster ewi-ao----    1,00g                                                                                
  root                                 onn_ovn-monster Vwi-a-tz-- <252,38g pool00                                  1,24                                   
  swap                                 onn_ovn-monster -wi-ao----    4,00g                                                                                
  tmp                                  onn_ovn-monster Vwi-aotz--    1,00g pool00                                  5,01                                   
  var                                  onn_ovn-monster Vwi-aotz--   15,00g pool00                                  3,56                                   
  var_crash                            onn_ovn-monster Vwi-aotz--   10,00g pool00                                  2,86                                   
  var_log                              onn_ovn-monster Vwi-aotz--    8,00g pool00                                  38,48                                  
  var_log_audit                        onn_ovn-monster Vwi-aotz--    2,00g pool00                                  6,77                                   
  turbo                                ovirt           -wi-ao----  894,25g                                                                                

Am 03.07.2018 um 12:58 schrieb Yuval Turgeman <yturgema@redhat.com>:

Oliver, can you share the output from lvs ?

On Tue, Jul 3, 2018 at 12:06 AM, Oliver Riesener <Oliver.Riesener@hs-bremen.de> wrote:
Hi Yuval,

* reinstallation failed, because LV already exists.
  ovirt-node-ng-4.2.4-0.20180626.0     onn_ovn-monster Vri-a-tz-k <252,38g pool00                                  0,85
  ovirt-node-ng-4.2.4-0.20180626.0+1   onn_ovn-monster Vwi-a-tz-- <252,38g pool00 ovirt-node-ng-4.2.4-0.20180626.0 0,85 
See attachment imgbased.reinstall.log

* I removed them and re-reinstall without luck.

I got KeyError: <NVR ovirt-node-ng-4.2.1.1-0.20180223.0 />

See attachment imgbased.rereinstall.log

Also a new problem with nodectl info
[root@ovn-monster tmp]# nodectl info
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/site-packages/nodectl/__main__.py", line 42, in <module>
    CliApplication()
  File "/usr/lib/python2.7/site-packages/nodectl/__init__.py", line 200, in CliApplication
    return cmdmap.command(args)
  File "/usr/lib/python2.7/site-packages/nodectl/__init__.py", line 118, in command
    return self.commands[command](**kwargs)
  File "/usr/lib/python2.7/site-packages/nodectl/__init__.py", line 76, in info
    Info(self.imgbased, self.machine).write()
  File "/usr/lib/python2.7/site-packages/nodectl/info.py", line 45, in __init__
    self._fetch_information()
  File "/usr/lib/python2.7/site-packages/nodectl/info.py", line 49, in _fetch_information
    self._get_layout()
  File "/usr/lib/python2.7/site-packages/nodectl/info.py", line 66, in _get_layout
    layout = LayoutParser(self.app.imgbase.layout()).parse()
  File "/usr/lib/python2.7/site-packages/imgbased/imgbase.py", line 155, in layout
    return self.naming.layout()
  File "/usr/lib/python2.7/site-packages/imgbased/naming.py", line 109, in layout
    tree = self.tree(lvs)
  File "/usr/lib/python2.7/site-packages/imgbased/naming.py", line 224, in tree
    bases[img.base.nvr].layers.append(img)
KeyError: <NVR ovirt-node-ng-4.2.3-0.20180524.0 />






Am 02.07.2018 um 22:22 schrieb Oliver Riesener <Oliver.Riesener@hs-bremen.de>:

Hi Yuval,

yes you are right, there was a unused and deactivated var_crash LV.

* I activated and mount it to /var/crash via /etc/fstab.
* /var/crash was empty, and LV has already ext4 fs.
  var_crash                            onn_ovn-monster Vwi-aotz--   10,00g pool00                                    2,86                                   

* Now i will try to upgrade again.
  * yum reinstall ovirt-node-ng-image-update.noarch

BTW, no more imgbased.log files found.

Am 02.07.2018 um 20:57 schrieb Yuval Turgeman <yturgema@redhat.com>:

From your log: 

AssertionError: Path is already a volume: /var/crash

Basically, it means that you already have an LV for /var/crash but it's not mounted for some reason, so either mount it (if the data good) or remove it and then reinstall the image-update rpm.  Before that, check that you dont have any other LVs in that same state - or you can post the output for lvs... btw, do you have any more imgbased.log files laying around ?

You can find more details about this here:


On Mon, Jul 2, 2018 at 8:12 PM, Oliver Riesener <Oliver.Riesener@hs-bremen.de> wrote:
Hi, 

i attached my /tmp/imgbased.log

Sheers

Oliver



Am 02.07.2018 um 13:58 schrieb Yuval Turgeman <yuvalt@redhat.com>:

Looks like the upgrade script failed - can you please attach /var/log/imgbased.log or /tmp/imgbased.log ?

Thanks,
Yuval.

On Mon, Jul 2, 2018 at 2:54 PM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Yuval, can you please have a look?

2018-06-30 7:48 GMT+02:00 Oliver Riesener <Oliver.Riesener@hs-bremen.de>:
Yes, here is the same.

It seams the bootloader isn’t configured right ?
 
I did the Upgrade and reboot to 4.2.4 from UI and got:

[root@ovn-monster ~]# nodectl info
layers: 
  ovirt-node-ng-4.2.4-0.20180626.0: 
    ovirt-node-ng-4.2.4-0.20180626.0+1
  ovirt-node-ng-4.2.3.1-0.20180530.0: 
    ovirt-node-ng-4.2.3.1-0.20180530.0+1
  ovirt-node-ng-4.2.3-0.20180524.0: 
    ovirt-node-ng-4.2.3-0.20180524.0+1
  ovirt-node-ng-4.2.1.1-0.20180223.0: 
    ovirt-node-ng-4.2.1.1-0.20180223.0+1
bootloader: 
  default: ovirt-node-ng-4.2.3-0.20180524.0+1
  entries: 
    ovirt-node-ng-4.2.3-0.20180524.0+1: 
      index: 0
      title: ovirt-node-ng-4.2.3-0.20180524.0
      kernel: /boot/ovirt-node-ng-4.2.3-0.20180524.0+1/vmlinuz-3.10.0-862.3.2.el7.x86_64
      args: "ro crashkernel=auto rd.lvm.lv=onn_ovn-monster/ovirt-node-ng-4.2.3-0.20180524.0+1 rd.lvm.lv=onn_ovn-monster/swap rd.md.uuid=c6c3013b:027a9346:67dfd181:89635587 rhgb quiet LANG=de_DE.UTF-8 img.bootid=ovirt-node-ng-4.2.3-0.20180524.0+1"
      initrd: /boot/ovirt-node-ng-4.2.3-0.20180524.0+1/initramfs-3.10.0-862.3.2.el7.x86_64.img
      root: /dev/onn_ovn-monster/ovirt-node-ng-4.2.3-0.20180524.0+1
    ovirt-node-ng-4.2.1.1-0.20180223.0+1: 
      index: 1
      title: ovirt-node-ng-4.2.1.1-0.20180223.0
      kernel: /boot/ovirt-node-ng-4.2.1.1-0.20180223.0+1/vmlinuz-3.10.0-693.17.1.el7.x86_64
      args: "ro crashkernel=auto rd.lvm.lv=onn_ovn-monster/ovirt-node-ng-4.2.1.1-0.20180223.0+1 rd.lvm.lv=onn_ovn-monster/swap rd.md.uuid=c6c3013b:027a9346:67dfd181:89635587 rhgb quiet LANG=de_DE.UTF-8 img.bootid=ovirt-node-ng-4.2.1.1-0.20180223.0+1"
      initrd: /boot/ovirt-node-ng-4.2.1.1-0.20180223.0+1/initramfs-3.10.0-693.17.1.el7.x86_64.img
      root: /dev/onn_ovn-monster/ovirt-node-ng-4.2.1.1-0.20180223.0+1
current_layer: ovirt-node-ng-4.2.3-0.20180524.0+1
[root@ovn-monster ~]# uptime
 07:35:27 up 2 days, 15:42,  1 user,  load average: 1,07, 1,00, 0,95

Am 29.06.2018 um 23:53 schrieb Matt Simonsen <matt@khoza.com>:

Hello,

I did yum updates on 2 of my oVirt 4.2.3 nodes running the prebuilt node platform and it doesn't appear the updates worked.


[root@node6-g8-h4 ~]# yum update
Loaded plugins: enabled_repos_upload, fastestmirror, imgbased-persist,
              : package_upload, product-id, search-disabled-repos, subscription-
              : manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
Loading mirror speeds from cached hostfile
 * ovirt-4.2-epel: linux.mirrors.es.net
Resolving Dependencies
--> Running transaction check
---> Package ovirt-node-ng-image-update.noarch 0:4.2.3.1-1.el7 will be updated
---> Package ovirt-node-ng-image-update.noarch 0:4.2.4-1.el7 will be obsoleting
---> Package ovirt-node-ng-image-update-placeholder.noarch 0:4.2.3.1-1.el7 will be obsoleted
--> Finished Dependency Resolution

Dependencies Resolved

=========================================================================================================================
 Package                                  Arch Version                     Repository               Size
=========================================================================================================================
Installing:
 ovirt-node-ng-image-update               noarch 4.2.4-1.el7                 ovirt-4.2               647 M
     replacing  ovirt-node-ng-image-update-placeholder.noarch 4.2.3.1-1.el7

Transaction Summary
=========================================================================================================================
Install  1 Package

Total download size: 647 M
Is this ok [y/d/N]: y
Downloading packages:
warning: /var/cache/yum/x86_64/7/ovirt-4.2/packages/ovirt-node-ng-image-update-4.2.4-1.el7.noarch.rpm: Header V4 RSA/SHA1 Signature, key ID fe590cb7: NOKEY
Public key for ovirt-node-ng-image-update-4.2.4-1.el7.noarch.rpm is not installed
ovirt-node-ng-image-update-4.2.4-1.el7.noarch.rpm | 647 MB  00:02:07
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-ovirt-4.2
Importing GPG key 0xFE590CB7:
 Userid     : "oVirt <infra@ovirt.org>"
 Fingerprint: 31a5 d783 7fad 7cb2 86cd 3469 ab8c 4f9d fe59 0cb7
 Package    : ovirt-release42-4.2.3.1-1.el7.noarch (installed)
 From       : /etc/pki/rpm-gpg/RPM-GPG-ovirt-4.2
Is this ok [y/N]: y
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : ovirt-node-ng-image-update-4.2.4-1.el7.noarch 1/3
warning: %post(ovirt-node-ng-image-update-4.2.4-1.el7.noarch) scriptlet failed, exit status 1
Non-fatal POSTIN scriptlet failure in rpm package ovirt-node-ng-image-update-4.2.4-1.el7.noarch
  Erasing    : ovirt-node-ng-image-update-placeholder-4.2.3.1-1.el7.noarch 2/3
  Cleanup    : ovirt-node-ng-image-update-4.2.3.1-1.el7.noarch 3/3
warning: file /usr/share/ovirt-node-ng/image/ovirt-node-ng-4.2.0-0.20180530.0.el7.squashfs.img: remove failed: No such file or directory
Uploading Package Profile
Unable to upload Package Profile
  Verifying  : ovirt-node-ng-image-update-4.2.4-1.el7.noarch 1/3
  Verifying  : ovirt-node-ng-image-update-4.2.3.1-1.el7.noarch 2/3
  Verifying  : ovirt-node-ng-image-update-placeholder-4.2.3.1-1.el7.noarch 3/3

Installed:
  ovirt-node-ng-image-update.noarch 0:4.2.4-1.el7

Replaced:
  ovirt-node-ng-image-update-placeholder.noarch 0:4.2.3.1-1.el7

Complete!
Uploading Enabled Repositories Report
Loaded plugins: fastestmirror, product-id, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
Cannot upload enabled repos report, is this client registered?


My engine shows the nodes as having no updates, however the major components including the kernel version and port 9090 admin GUI show 4.2.3

Is there anything I can provide to help diagnose the issue?


[root@node6-g8-h4 ~]# rpm -qa | grep ovirt

ovirt-imageio-common-1.3.1.2-0.el7.centos.noarch
ovirt-host-deploy-1.7.3-1.el7.centos.noarch
ovirt-vmconsole-host-1.0.5-4.el7.centos.noarch
ovirt-provider-ovn-driver-1.2.10-1.el7.centos.noarch
ovirt-engine-sdk-python-3.6.9.1-1.el7.noarch
ovirt-setup-lib-1.1.4-1.el7.centos.noarch
ovirt-release42-4.2.3.1-1.el7.noarch
ovirt-imageio-daemon-1.3.1.2-0.el7.centos.noarch
ovirt-hosted-engine-setup-2.2.20-1.el7.centos.noarch
ovirt-host-dependencies-4.2.2-2.el7.centos.x86_64
ovirt-hosted-engine-ha-2.2.11-1.el7.centos.noarch
ovirt-host-4.2.2-2.el7.centos.x86_64
ovirt-node-ng-image-update-4.2.4-1.el7.noarch
ovirt-vmconsole-1.0.5-4.el7.centos.noarch
ovirt-release-host-node-4.2.3.1-1.el7.noarch
cockpit-ovirt-dashboard-0.11.24-1.el7.centos.noarch
ovirt-node-ng-nodectl-4.2.0-0.20180524.0.el7.noarch
python-ovirt-engine-sdk4-4.2.6-2.el7.centos.x86_64

[root@node6-g8-h4 ~]# yum update
Loaded plugins: enabled_repos_upload, fastestmirror, imgbased-persist, package_upload, product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
Loading mirror speeds from cached hostfile
 * ovirt-4.2-epel: linux.mirrors.es.net
No packages marked for update
Uploading Enabled Repositories Report
Loaded plugins: fastestmirror, product-id, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
Cannot upload enabled repos report, is this client registered?
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/UHQMGULUHL4GBBHUBNGOAICJEM6W3RVW/


_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RTOXFNAXQ3NJBWX7RXOYK5H5RZBHX2OK/




-- 
SANDRO BONAZZOLA

MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV

sbonazzo@redhat.com   


_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ACKMTWOUNXB7NWOCRZIOWJI7U5HIQAUV/



_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/E5MFPG7WW5BIQ5BOC7LG5C23YZ7B7UJ2/

_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/IPKGIHH7SFTW7K7YOKL4QXBYXR5KMKNB/



_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/6HKLX746IND2NQVRRGKACMSQCA3GK6GA/