node-ng update failed from ovirt-node-ng-4.1.0-0 to ovirt-node-ng-image-4.1.0-1, and NM + iscsi boo issue
1) I've updated from 4.0.6 to 4.1.0 (on Feb 01 node-ng was at version 4.1.0-0) After some time engine alerted, that this node have updates to ovirt-node-ng-image-4.1.0-1, but update from engine timed out, there were hanging processes in ps on this node: root 36309 0.0 0.0 113120 1564 ? Ss 19:04 0:00 bash -c umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True root 36339 0.2 0.0 496700 94208 ? S 19:04 0:21 /bin/python /tmp/ovirt-GCmVusccfe/pythonlib/otopi/__main__.py "BASE/pluginPath=str:/tmp/ovirt-GCmVusccfe/otopi-plugins" APPEND:BASE/pluginGroups=str:ovirt-host-common:ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True root 37498 0.0 0.0 113124 1452 ? S 19:09 0:00 /bin/sh /var/tmp/rpm-tmp.4UqJ4e 1 root 37560 0.0 0.0 0 0 ? S< 21:42 0:00 [kworker/21:2H] root 37626 0.0 0.0 174516 5996 ? S 19:09 0:00 rpm -Uvh --quiet --justdb /usr/share/imgbased/ovirt-node-ng-image-update-4.1.0-1.el7.centos.noarch.rpm they were hanging forever, I ended up with rebooting the node, no errors in log, it was just hanging at: 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ***CONFIRM GPG_KEY Confirm use of GPG Key userid=oVirt <infra@ovirt.org> hexkeyid=FE590CB7 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### Please confirm 'GPG_KEY' Confirm use of GPG Key userid=oVirt <infra@ovirt.org> hexkeyid=FE590CB7 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### Response is CONFIRM GPG_KEY=yes|no or ABORT GPG_KEY 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:RECEIVE CONFIRM GPG_KEY=yes 2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum Status: Running Test Transaction Running Transaction Check 2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum Status: Running Transaction 2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum install: 1/2: ovirt-node-ng-image-4.1.0-1.el7.centos.noarch 2017-02-03 19:09:20 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Done: ovirt-node-ng-image-4.1.0-1.el7.centos.noarch 2017-02-03 19:09:20 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum install: 2/2: ovirt-node-ng-image-update-4.1.0-1.el7.centos.noarch now my node have this layout: # imgbase layout ovirt-node-ng-4.1.0-0.20170201.0 +- ovirt-node-ng-4.1.0-0.20170201.0+1 (so update failed) but 4.1.0-1 rpms are marked as "installed" and yum can't find any updates, can I rollback to base layout without installed 4.1.0-1 rms ? imgbase rollback needs at least 2 layers over base. Or maybe the only way is to reinstall this node? 2) And another question, how can I disable NetworkManger permanently, or exclude some interfaces permanently? I've tried to disable NetworkManger by systemctl, but after update from 4.0 to 4.1 it was re-enabled(so it's not persistent between updates). I've an issue with iscsi root and enabled NetworkManger, because NM tries to bring down\up my iscsi interfaces on boot, and sometimes FS remounting RO because of IO errors, I can't put NM_CONTROLLED=no in ifcfg, because ifcfg is generated by dracut at every boot. -
Adding Douglas and Ryan Il 06/Feb/2017 13:32, "Sergey Kulikov" <serg_k@msm.ru> ha scritto:
1) I've updated from 4.0.6 to 4.1.0 (on Feb 01 node-ng was at version 4.1.0-0) After some time engine alerted, that this node have updates to ovirt-node-ng-image-4.1.0-1, but update from engine timed out, there were hanging processes in ps on this node:
root 36309 0.0 0.0 113120 1564 ? Ss 19:04 0:00 bash -c umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True root 36339 0.2 0.0 496700 94208 ? S 19:04 0:21 /bin/python /tmp/ovirt-GCmVusccfe/pythonlib/otopi/__main__.py "BASE/pluginPath=str:/tmp/ovirt-GCmVusccfe/otopi-plugins" APPEND:BASE/pluginGroups=str:ovirt-host-common:ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True root 37498 0.0 0.0 113124 1452 ? S 19:09 0:00 /bin/sh /var/tmp/rpm-tmp.4UqJ4e 1 root 37560 0.0 0.0 0 0 ? S< 21:42 0:00 [kworker/21:2H] root 37626 0.0 0.0 174516 5996 ? S 19:09 0:00 rpm -Uvh --quiet --justdb /usr/share/imgbased/ovirt-node-ng-image-update-4.1.0-1. el7.centos.noarch.rpm
they were hanging forever, I ended up with rebooting the node, no errors in log, it was just hanging at:
2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ***CONFIRM GPG_KEY Confirm use of GPG Key userid=oVirt <infra@ovirt.org> hexkeyid=FE590CB7 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### Please confirm 'GPG_KEY' Confirm use of GPG Key userid=oVirt <infra@ovirt.org> hexkeyid=FE590CB7 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### Response is CONFIRM GPG_KEY=yes|no or ABORT GPG_KEY 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:RECEIVE CONFIRM GPG_KEY=yes 2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum Status: Running Test Transaction Running Transaction Check 2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum Status: Running Transaction 2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum install: 1/2: ovirt-node-ng-image-4.1.0-1. el7.centos.noarch 2017-02-03 19:09:20 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Done: ovirt-node-ng-image-4.1.0-1. el7.centos.noarch 2017-02-03 19:09:20 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum install: 2/2: ovirt-node-ng-image-update-4. 1.0-1.el7.centos.noarch
now my node have this layout: # imgbase layout ovirt-node-ng-4.1.0-0.20170201.0 +- ovirt-node-ng-4.1.0-0.20170201.0+1 (so update failed) but 4.1.0-1 rpms are marked as "installed" and yum can't find any updates, can I rollback to base layout without installed 4.1.0-1 rms ? imgbase rollback needs at least 2 layers over base.
Or maybe the only way is to reinstall this node?
2) And another question, how can I disable NetworkManger permanently, or exclude some interfaces permanently? I've tried to disable NetworkManger by systemctl, but after update from 4.0 to 4.1 it was re-enabled(so it's not persistent between updates). I've an issue with iscsi root and enabled NetworkManger, because NM tries to bring down\up my iscsi interfaces on boot, and sometimes FS remounting RO because of IO errors, I can't put NM_CONTROLLED=no in ifcfg, because ifcfg is generated by dracut at every boot.
-
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hey Sergey - If you check "lvs" and ensure that there's not actually a new LV from the update, you can cleanly 'rpm -e ovirt-node-ng-image-update', and be ok without redeploying. Unfortunately, it's hard to tell from the logs (and '--justdb' hanging) what's happening here, but I'll try to reproduce. NetworkManager disablement should "stick" across upgrades, but it's possible that iscsi roots are doing something here. I'll check for a dracut flag, also... On Mon, Feb 6, 2017 at 1:14 PM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Adding Douglas and Ryan
Il 06/Feb/2017 13:32, "Sergey Kulikov" <serg_k@msm.ru> ha scritto:
1) I've updated from 4.0.6 to 4.1.0 (on Feb 01 node-ng was at version 4.1.0-0) After some time engine alerted, that this node have updates to ovirt-node-ng-image-4.1.0-1, but update from engine timed out, there were hanging processes in ps on this node:
root 36309 0.0 0.0 113120 1564 ? Ss 19:04 0:00 bash -c umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True root 36339 0.2 0.0 496700 94208 ? S 19:04 0:21 /bin/python /tmp/ovirt-GCmVusccfe/pythonlib/otopi/__main__.py "BASE/pluginPath=str:/tmp/ovirt-GCmVusccfe/otopi-plugins" APPEND:BASE/pluginGroups=str:ovirt-host-common:ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True root 37498 0.0 0.0 113124 1452 ? S 19:09 0:00 /bin/sh /var/tmp/rpm-tmp.4UqJ4e 1 root 37560 0.0 0.0 0 0 ? S< 21:42 0:00 [kworker/21:2H] root 37626 0.0 0.0 174516 5996 ? S 19:09 0:00 rpm -Uvh --quiet --justdb /usr/share/imgbased/ovirt-node -ng-image-update-4.1.0-1.el7.centos.noarch.rpm
they were hanging forever, I ended up with rebooting the node, no errors in log, it was just hanging at:
2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ***CONFIRM GPG_KEY Confirm use of GPG Key userid=oVirt <infra@ovirt.org> hexkeyid=FE590CB7 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### Please confirm 'GPG_KEY' Confirm use of GPG Key userid=oVirt <infra@ovirt.org> hexkeyid=FE590CB7 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### Response is CONFIRM GPG_KEY=yes|no or ABORT GPG_KEY 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:RECEIVE CONFIRM GPG_KEY=yes 2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum Status: Running Test Transaction Running Transaction Check 2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum Status: Running Transaction 2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum install: 1/2: ovirt-node-ng-image-4.1.0-1.el 7.centos.noarch 2017-02-03 19:09:20 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Done: ovirt-node-ng-image-4.1.0-1.el 7.centos.noarch 2017-02-03 19:09:20 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum install: 2/2: ovirt-node-ng-image-update-4.1 .0-1.el7.centos.noarch
now my node have this layout: # imgbase layout ovirt-node-ng-4.1.0-0.20170201.0 +- ovirt-node-ng-4.1.0-0.20170201.0+1 (so update failed) but 4.1.0-1 rpms are marked as "installed" and yum can't find any updates, can I rollback to base layout without installed 4.1.0-1 rms ? imgbase rollback needs at least 2 layers over base.
Or maybe the only way is to reinstall this node?
2) And another question, how can I disable NetworkManger permanently, or exclude some interfaces permanently? I've tried to disable NetworkManger by systemctl, but after update from 4.0 to 4.1 it was re-enabled(so it's not persistent between updates). I've an issue with iscsi root and enabled NetworkManger, because NM tries to bring down\up my iscsi interfaces on boot, and sometimes FS remounting RO because of IO errors, I can't put NM_CONTROLLED=no in ifcfg, because ifcfg is generated by dracut at every boot.
-
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
There is no new lv, I can remove rpm and reinstall manually to reproduce with any debug options, if you want. I left host in maintenance mode. should I also remove ovirt-node-ng-image ? I found something strange: # rpm -q ovirt-node-ng-image ovirt-node-ng-image-4.1.0-1.el7.centos.noarch # rpm -ql ovirt-node-ng-image /usr/share/ovirt-node-ng/image /usr/share/ovirt-node-ng/image/ovirt-node-ng-4.1.0-0.20170201.0.el7.squashfs.img /usr/share/ovirt-node-ng/image/product.img version is 4.1.0-1, but squashfs file is 4.1.0-0 About iscsi... I've disabled NM in 4.0.6, everything was ok, it was re-enabled only after update, I don't think iscsi root affects NM status. The problem is only with reboot, 1 time out of ~4 reboots it's falling to RO. Also I have centos hosts and NM stays disabled there, maybe I should try "unmanaged-devices=" in NM config, but I think it's not ideal, NM is useless for this setup) -- Tuesday, February 7, 2017, 07:14:55:
Hey Sergey -
If you check "lvs" and ensure that there's not actually a new LV from the update, you can cleanly 'rpm -e ovirt-node-ng-image-update', and be ok without redeploying.
Unfortunately, it's hard to tell from the logs (and '--justdb' hanging) what's happening here, but I'll try to reproduce.
NetworkManager disablement should "stick" across upgrades, but it's possible that iscsi roots are doing something here. I'll check for a dracut flag, also...
On Mon, Feb 6, 2017 at 1:14 PM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Adding Douglas and Ryan
Il 06/Feb/2017 13:32, "Sergey Kulikov" <serg_k@msm.ru> ha scritto:
1) I've updated from 4.0.6 to 4.1.0 (on Feb 01 node-ng was at version 4.1.0-0) After some time engine alerted, that this node have updates to ovirt-node-ng-image-4.1.0-1, but update from engine timed out, there were hanging processes in ps on this node:
root 36309 0.0 0.0 113120 1564 ? Ss 19:04 0:00 bash -c umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True root 36339 0.2 0.0 496700 94208 ? S 19:04 0:21 /bin/python /tmp/ovirt-GCmVusccfe/pythonlib/otopi/__main__.py "BASE/pluginPath=str:/tmp/ovirt-GCmVusccfe/otopi-plugins" APPEND:BASE/pluginGroups=str:ovirt-host-common:ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True root 37498 0.0 0.0 113124 1452 ? S 19:09 0:00 /bin/sh /var/tmp/rpm-tmp.4UqJ4e 1 root 37560 0.0 0.0 0 0 ? S< 21:42 0:00 [kworker/21:2H] root 37626 0.0 0.0 174516 5996 ? S 19:09 0:00 rpm -Uvh --quiet --justdb /usr/share/imgbased/ovirt-node-ng-image-update-4.1.0-1.el7.centos.noarch.rpm
they were hanging forever, I ended up with rebooting the node, no errors in log, it was just hanging at:
2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ***CONFIRM GPG_KEY Confirm use of GPG Key userid=oVirt <infra@ovirt.org> hexkeyid=FE590CB7 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### Please confirm 'GPG_KEY' Confirm use of GPG Key userid=oVirt <infra@ovirt.org> hexkeyid=FE590CB7 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### Response is CONFIRM GPG_KEY=yes|no or ABORT GPG_KEY 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:RECEIVE CONFIRM GPG_KEY=yes 2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum Status: Running Test Transaction Running Transaction Check 2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum Status: Running Transaction 2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum install: 1/2: ovirt-node-ng-image-4.1.0-1.el7.centos.noarch 2017-02-03 19:09:20 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum Done: ovirt-node-ng-image-4.1.0-1.el7.centos.noarch 2017-02-03 19:09:20 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum install: 2/2: ovirt-node-ng-image-update-4.1.0-1.el7.centos.noarch
now my node have this layout: # imgbase layout ovirt-node-ng-4.1.0-0.20170201.0 +- ovirt-node-ng-4.1.0-0.20170201.0+1 (so update failed) but 4.1.0-1 rpms are marked as "installed" and yum can't find any updates, can I rollback to base layout without installed 4.1.0-1 rms ? imgbase rollback needs at least 2 layers over base.
Or maybe the only way is to reinstall this node?
2) And another question, how can I disable NetworkManger permanently, or exclude some interfaces permanently? I've tried to disable NetworkManger by systemctl, but after update from 4.0 to 4.1 it was re-enabled(so it's not persistent between updates). I've an issue with iscsi root and enabled NetworkManger, because NM tries to bring down\up my iscsi interfaces on boot, and sometimes FS remounting RO because of IO errors, I can't put NM_CONTROLLED=no in ifcfg, because ifcfg is generated by dracut at every boot.
-
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (3)
-
Ryan Barry -
Sandro Bonazzola -
Sergey Kulikov