node-ng update failed from ovirt-node-ng-4.1.0-0 to ovirt-node-ng-image-4.1.0-1, and NM + iscsi boo issue
1:32 p.m.
1) I've updated from 4.0.6 to 4.1.0 (on Feb 01 node-ng was at version 4.1.0-0)
After some time engine alerted, that this node have updates to
ovirt-node-ng-image-4.1.0-1,
but update from engine timed out, there were hanging processes in ps on this node:
root 36309 0.0 0.0 113120 1564 ? Ss 19:04 0:00 bash -c umask 0077;
MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)";
trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr
\"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C
"${MYTMP}" -x && "${MYTMP}"/ovirt-host-mgmt
DIALOG/dialect=str:machine DIALOG/customization=bool:True
root 36339 0.2 0.0 496700 94208 ? S 19:04 0:21 /bin/python
/tmp/ovirt-GCmVusccfe/pythonlib/otopi/__main__.py
"BASE/pluginPath=str:/tmp/ovirt-GCmVusccfe/otopi-plugins"
APPEND:BASE/pluginGroups=str:ovirt-host-common:ovirt-host-mgmt DIALOG/dialect=str:machine
DIALOG/customization=bool:True
root 37498 0.0 0.0 113124 1452 ? S 19:09 0:00 /bin/sh
/var/tmp/rpm-tmp.4UqJ4e 1
root 37560 0.0 0.0 0 0 ? S< 21:42 0:00 [kworker/21:2H]
root 37626 0.0 0.0 174516 5996 ? S 19:09 0:00 rpm -Uvh --quiet --justdb
/usr/share/imgbased/ovirt-node-ng-image-update-4.1.0-1.el7.centos.noarch.rpm
they were hanging forever, I ended up with rebooting the node, no errors in log, it was
just hanging at:
2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204
DIALOG:SEND ***CONFIRM GPG_KEY Confirm use of GPG Key userid=oVirt
<infra(a)ovirt.org> hexkeyid=FE590CB7
2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204
DIALOG:SEND ###
2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204
DIALOG:SEND ### Please confirm 'GPG_KEY' Confirm use of GPG Key userid=oVirt
<infra(a)ovirt.org> hexkeyid=FE590CB7
2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204
DIALOG:SEND ### Response is CONFIRM GPG_KEY=yes|no or ABORT GPG_KEY
2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204
DIALOG:RECEIVE CONFIRM GPG_KEY=yes
2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum
Status: Running Test Transaction
Running Transaction Check
2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum
Status: Running Transaction
2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum
install: 1/2: ovirt-node-ng-image-4.1.0-1.el7.centos.noarch
2017-02-03 19:09:20 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76
Yum Done: ovirt-node-ng-image-4.1.0-1.el7.centos.noarch
2017-02-03 19:09:20 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum
install: 2/2: ovirt-node-ng-image-update-4.1.0-1.el7.centos.noarch
now my node have this layout:
# imgbase layout
ovirt-node-ng-4.1.0-0.20170201.0
+- ovirt-node-ng-4.1.0-0.20170201.0+1
(so update failed)
but 4.1.0-1 rpms are marked as "installed" and yum can't find any updates,
can I rollback to base layout without installed 4.1.0-1 rms ?
imgbase rollback needs at least 2 layers over base.
Or maybe the only way is to reinstall this node?
2) And another question, how can I disable NetworkManger permanently, or exclude some
interfaces permanently?
I've tried to disable NetworkManger by systemctl, but after update from 4.0 to 4.1 it
was re-enabled(so it's not persistent between updates).
I've an issue with iscsi root and enabled NetworkManger, because NM tries to bring
down\up my iscsi interfaces on boot, and sometimes FS remounting RO
because of IO errors, I can't put NM_CONTROLLED=no in ifcfg, because ifcfg is
generated by dracut at every boot.
-
5:14 a.m.
Hey Sergey -
If you check "lvs" and ensure that there's not actually a new LV from the
update, you can cleanly 'rpm -e ovirt-node-ng-image-update', and be ok
without redeploying.
Unfortunately, it's hard to tell from the logs (and '--justdb' hanging)
what's happening here, but I'll try to reproduce.
NetworkManager disablement should "stick" across upgrades, but it's
possible that iscsi roots are doing something here. I'll check for a dracut
flag, also...
On Mon, Feb 6, 2017 at 1:14 PM, Sandro Bonazzola <sbonazzo(a)redhat.com>
wrote:
9:50 a.m.
There is no new lv, I can remove rpm and reinstall manually to reproduce with any debug
options, if you want.
I left host in maintenance mode.
should I also remove ovirt-node-ng-image ? I found something strange:
# rpm -q ovirt-node-ng-image
ovirt-node-ng-image-4.1.0-1.el7.centos.noarch
# rpm -ql ovirt-node-ng-image
/usr/share/ovirt-node-ng/image
/usr/share/ovirt-node-ng/image/ovirt-node-ng-4.1.0-0.20170201.0.el7.squashfs.img
/usr/share/ovirt-node-ng/image/product.img
version is 4.1.0-1, but squashfs file is 4.1.0-0
About iscsi... I've disabled NM in 4.0.6, everything was ok, it was re-enabled only
after update, I don't
think iscsi root affects NM status.
The problem is only with reboot, 1 time out of ~4 reboots it's falling to RO.
Also I have centos hosts and NM stays disabled there, maybe I should try
"unmanaged-devices="
in NM config, but I think it's not ideal, NM is useless for this setup)
--
Tuesday, February 7, 2017, 07:14:55:
Hey Sergey -
Unfortunately, it's hard to tell from the logs (and
'--justdb'
hanging) what's happening here, but I'll try to reproduce.
On Mon, Feb 6, 2017 at 1:14 PM, Sandro Bonazzola
<sbonazzo(a)redhat.com> wrote:
Adding Douglas and Ryan
Il 06/Feb/2017 13:32, "Sergey Kulikov"
<serg_k(a)msm.ru> ha scritto:
2982
days inactive
2983
days old
3 comments
3 participants
participants (3)
-
Ryan Barry -
Sandro Bonazzola -
Sergey Kulikov