[ovirt-users] node-ng update failed from ovirt-node-ng-4.1.0-0 to ovirt-node-ng-image-4.1.0-1, and NM + iscsi boo issue

Tue Feb 7 08:50:40 UTC 2017

There is no new lv, I can remove rpm and reinstall manually to reproduce with any debug options, if you want.
I left host in maintenance mode. 
should I also remove ovirt-node-ng-image ? I found something strange:
# rpm -q ovirt-node-ng-image  
ovirt-node-ng-image-4.1.0-1.el7.centos.noarch

# rpm -ql ovirt-node-ng-image
/usr/share/ovirt-node-ng/image
/usr/share/ovirt-node-ng/image/ovirt-node-ng-4.1.0-0.20170201.0.el7.squashfs.img
/usr/share/ovirt-node-ng/image/product.img

version is 4.1.0-1, but squashfs file is 4.1.0-0

About iscsi... I've disabled NM in 4.0.6, everything was ok, it was re-enabled only after update, I don't
think iscsi root affects NM status.
The problem is only with reboot, 1 time out of ~4 reboots it's falling to RO.
Also I have centos hosts and NM stays disabled there, maybe I should try "unmanaged-devices="
in NM config, but I think it's not ideal, NM is useless for this setup)

-- 

 Tuesday, February 7, 2017, 07:14:55:

> Hey Sergey -

> If you check "lvs" and ensure that there's not actually a new LV
> from the update, you can cleanly 'rpm -e
> ovirt-node-ng-image-update', and be ok without redeploying.

> Unfortunately, it's hard to tell from the logs (and '--justdb'
> hanging) what's happening here, but I'll try to reproduce.

> NetworkManager disablement should "stick" across upgrades, but it's
> possible that iscsi roots are doing something here. I'll check for a dracut flag, also...

> On Mon, Feb 6, 2017 at 1:14 PM, Sandro Bonazzola <sbonazzo at redhat.com> wrote:

> Adding Douglas and Ryan

> Il 06/Feb/2017 13:32, "Sergey Kulikov" <serg_k at msm.ru> ha scritto:

>  1) I've updated from 4.0.6 to 4.1.0 (on Feb 01 node-ng was at version 4.1.0-0)
>  After some time engine alerted, that this node have updates to ovirt-node-ng-image-4.1.0-1,
>  but update from engine timed out, there were hanging processes in ps on this node:
>  
>  root     36309  0.0  0.0 113120  1564 ?        Ss   19:04   0:00
> bash -c umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t
> ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null
> 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar
> --warning=no-timestamp -C "${MYTMP}" -x && 
> "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True
>  root     36339  0.2  0.0 496700 94208 ?        S    19:04   0:21
> /bin/python /tmp/ovirt-GCmVusccfe/pythonlib/otopi/__main__.py
> "BASE/pluginPath=str:/tmp/ovirt-GCmVusccfe/otopi-plugins" 
> APPEND:BASE/pluginGroups=str:ovirt-host-common:ovirt-host-mgmt
> DIALOG/dialect=str:machine DIALOG/customization=bool:True
>  root     37498  0.0  0.0 113124  1452 ?        S    19:09   0:00 /bin/sh /var/tmp/rpm-tmp.4UqJ4e 1
>  root     37560  0.0  0.0      0     0 ?        S<   21:42   0:00 [kworker/21:2H]
>  root     37626  0.0  0.0 174516  5996 ?        S    19:09   0:00
> rpm -Uvh --quiet --justdb
> /usr/share/imgbased/ovirt-node-ng-image-update-4.1.0-1.el7.centos.noarch.rpm
>  
>  they were hanging forever, I ended up with rebooting the node, no
> errors in log, it was just hanging at:
>  
>  2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:SEND       ***CONFIRM GPG_KEY Confirm
> use of GPG Key userid=oVirt <infra at ovirt.org> hexkeyid=FE590CB7
>  2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:SEND       ###
>  2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:SEND       ### Please confirm
> 'GPG_KEY' Confirm use of GPG Key userid=oVirt <infra at ovirt.org> hexkeyid=FE590CB7
>  2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:SEND       ### Response is CONFIRM GPG_KEY=yes|no or ABORT GPG_KEY
>  2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:RECEIVE    CONFIRM GPG_KEY=yes
>  2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
> yumpackager.info:80 Yum Status: Running Test Transaction
>  Running Transaction Check
>  2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
> yumpackager.info:80 Yum Status: Running Transaction
>  2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
> yumpackager.info:80 Yum install: 1/2:
> ovirt-node-ng-image-4.1.0-1.el7.centos.noarch
>  2017-02-03 19:09:20 DEBUG
> otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum
> Done: ovirt-node-ng-image-4.1.0-1.el7.centos.noarch
>  2017-02-03 19:09:20 INFO otopi.plugins.otopi.packagers.yumpackager
> yumpackager.info:80 Yum install: 2/2:
> ovirt-node-ng-image-update-4.1.0-1.el7.centos.noarch
>  
>  now my node have this layout:
>  # imgbase layout
>  ovirt-node-ng-4.1.0-0.20170201.0
>   +- ovirt-node-ng-4.1.0-0.20170201.0+1
>  (so update failed)
>  but 4.1.0-1 rpms are marked as "installed" and yum can't find any
> updates, can I rollback to base layout without installed  4.1.0-1 rms ?
>  imgbase rollback needs at least 2 layers over base.
>  
>  Or maybe the only way is to reinstall this node?
>  
>  2) And another question, how can I disable NetworkManger
> permanently, or exclude some interfaces permanently?
>  I've tried to disable NetworkManger by systemctl, but after update
> from 4.0 to 4.1 it was re-enabled(so it's not persistent between updates).
>  I've an issue with iscsi root and enabled NetworkManger, because
> NM tries to bring down\up my iscsi interfaces on boot, and sometimes FS remounting RO
>  because of IO errors, I can't put NM_CONTROLLED=no in ifcfg,
> because ifcfg is generated by dracut at every boot.
>  
>  
>  -
>  
>  _______________________________________________
>  Users mailing list
>  Users at ovirt.org
>  http://lists.ovirt.org/mailman/listinfo/users
>  
>  