node-ng update failed from ovirt-node-ng-4.1.0-0 to ovirt-node-ng-image-4.1.0-1, and NM + iscsi boo issue
6:32 a.m.
1) I've updated from 4.0.6 to 4.1.0 (on Feb 01 node-ng was at version 4.1.0-0)
After some time engine alerted, that this node have updates to
ovirt-node-ng-image-4.1.0-1,
but update from engine timed out, there were hanging processes in ps on this node:
root 36309 0.0 0.0 113120 1564 ? Ss 19:04 0:00 bash -c umask 0077;
MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)";
trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr
\"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C
"${MYTMP}" -x && "${MYTMP}"/ovirt-host-mgmt
DIALOG/dialect=str:machine DIALOG/customization=bool:True
root 36339 0.2 0.0 496700 94208 ? S 19:04 0:21 /bin/python
/tmp/ovirt-GCmVusccfe/pythonlib/otopi/__main__.py
"BASE/pluginPath=str:/tmp/ovirt-GCmVusccfe/otopi-plugins"
APPEND:BASE/pluginGroups=str:ovirt-host-common:ovirt-host-mgmt DIALOG/dialect=str:machine
DIALOG/customization=bool:True
root 37498 0.0 0.0 113124 1452 ? S 19:09 0:00 /bin/sh
/var/tmp/rpm-tmp.4UqJ4e 1
root 37560 0.0 0.0 0 0 ? S< 21:42 0:00 [kworker/21:2H]
root 37626 0.0 0.0 174516 5996 ? S 19:09 0:00 rpm -Uvh --quiet --justdb
/usr/share/imgbased/ovirt-node-ng-image-update-4.1.0-1.el7.centos.noarch.rpm
they were hanging forever, I ended up with rebooting the node, no errors in log, it was
just hanging at:
2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204
DIALOG:SEND ***CONFIRM GPG_KEY Confirm use of GPG Key userid=oVirt
<infra(a)ovirt.org> hexkeyid=FE590CB7
2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204
DIALOG:SEND ###
2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204
DIALOG:SEND ### Please confirm 'GPG_KEY' Confirm use of GPG Key userid=oVirt
<infra(a)ovirt.org> hexkeyid=FE590CB7
2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204
DIALOG:SEND ### Response is CONFIRM GPG_KEY=yes|no or ABORT GPG_KEY
2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204
DIALOG:RECEIVE CONFIRM GPG_KEY=yes
2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum
Status: Running Test Transaction
Running Transaction Check
2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum
Status: Running Transaction
2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum
install: 1/2: ovirt-node-ng-image-4.1.0-1.el7.centos.noarch
2017-02-03 19:09:20 DEBUG otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76
Yum Done: ovirt-node-ng-image-4.1.0-1.el7.centos.noarch
2017-02-03 19:09:20 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum
install: 2/2: ovirt-node-ng-image-update-4.1.0-1.el7.centos.noarch
now my node have this layout:
# imgbase layout
ovirt-node-ng-4.1.0-0.20170201.0
+- ovirt-node-ng-4.1.0-0.20170201.0+1
(so update failed)
but 4.1.0-1 rpms are marked as "installed" and yum can't find any updates,
can I rollback to base layout without installed 4.1.0-1 rms ?
imgbase rollback needs at least 2 layers over base.
Or maybe the only way is to reinstall this node?
2) And another question, how can I disable NetworkManger permanently, or exclude some
interfaces permanently?
I've tried to disable NetworkManger by systemctl, but after update from 4.0 to 4.1 it
was re-enabled(so it's not persistent between updates).
I've an issue with iscsi root and enabled NetworkManger, because NM tries to bring
down\up my iscsi interfaces on boot, and sometimes FS remounting RO
because of IO errors, I can't put NM_CONTROLLED=no in ifcfg, because ifcfg is
generated by dracut at every boot.
-
2:14 p.m.
Adding Douglas and Ryan
Il 06/Feb/2017 13:32, "Sergey Kulikov" <serg_k(a)msm.ru> ha scritto:
1) I've updated from 4.0.6 to 4.1.0 (on Feb 01 node-ng was at version
4.1.0-0)
After some time engine alerted, that this node have updates to
ovirt-node-ng-image-4.1.0-1,
but update from engine timed out, there were hanging processes in ps on
this node:
root 36309 0.0 0.0 113120 1564 ? Ss 19:04 0:00 bash -c
umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t
ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null
2>&1; rm
-fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp
-C
"${MYTMP}" -x && "${MYTMP}"/ovirt-host-mgmt
DIALOG/dialect=str:machine
DIALOG/customization=bool:True
root 36339 0.2 0.0 496700 94208 ? S 19:04 0:21
/bin/python /tmp/ovirt-GCmVusccfe/pythonlib/otopi/__main__.py
"BASE/pluginPath=str:/tmp/ovirt-GCmVusccfe/otopi-plugins"
APPEND:BASE/pluginGroups=str:ovirt-host-common:ovirt-host-mgmt
DIALOG/dialect=str:machine DIALOG/customization=bool:True
root 37498 0.0 0.0 113124 1452 ? S 19:09 0:00 /bin/sh
/var/tmp/rpm-tmp.4UqJ4e 1
root 37560 0.0 0.0 0 0 ? S< 21:42 0:00
[kworker/21:2H]
root 37626 0.0 0.0 174516 5996 ? S 19:09 0:00 rpm -Uvh
--quiet --justdb /usr/share/imgbased/ovirt-node-ng-image-update-4.1.0-1.
el7.centos.noarch.rpm
they were hanging forever, I ended up with rebooting the node, no errors
in log, it was just hanging at:
2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
dialog.__logString:204 DIALOG:SEND ***CONFIRM GPG_KEY Confirm use of
GPG Key userid=oVirt <infra(a)ovirt.org> hexkeyid=FE590CB7
2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
dialog.__logString:204 DIALOG:SEND ###
2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
dialog.__logString:204 DIALOG:SEND ### Please confirm 'GPG_KEY'
Confirm use of GPG Key userid=oVirt <infra(a)ovirt.org> hexkeyid=FE590CB7
2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
dialog.__logString:204 DIALOG:SEND ### Response is CONFIRM
GPG_KEY=yes|no or ABORT GPG_KEY
2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
dialog.__logString:204 DIALOG:RECEIVE CONFIRM GPG_KEY=yes
2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
yumpackager.info:80 Yum Status: Running Test Transaction
Running Transaction Check
2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
yumpackager.info:80 Yum Status: Running Transaction
2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
yumpackager.info:80 Yum install: 1/2: ovirt-node-ng-image-4.1.0-1.
el7.centos.noarch
2017-02-03 19:09:20 DEBUG otopi.plugins.otopi.packagers.yumpackager
yumpackager.verbose:76 Yum Done: ovirt-node-ng-image-4.1.0-1.
el7.centos.noarch
2017-02-03 19:09:20 INFO otopi.plugins.otopi.packagers.yumpackager
yumpackager.info:80 Yum install: 2/2: ovirt-node-ng-image-update-4.
1.0-1.el7.centos.noarch
now my node have this layout:
# imgbase layout
ovirt-node-ng-4.1.0-0.20170201.0
+- ovirt-node-ng-4.1.0-0.20170201.0+1
(so update failed)
but 4.1.0-1 rpms are marked as "installed" and yum can't find any updates,
can I rollback to base layout without installed 4.1.0-1 rms ?
imgbase rollback needs at least 2 layers over base.
Or maybe the only way is to reinstall this node?
2) And another question, how can I disable NetworkManger permanently, or
exclude some interfaces permanently?
I've tried to disable NetworkManger by systemctl, but after update from
4.0 to 4.1 it was re-enabled(so it's not persistent between updates).
I've an issue with iscsi root and enabled NetworkManger, because NM tries
to bring down\up my iscsi interfaces on boot, and sometimes FS remounting RO
because of IO errors, I can't put NM_CONTROLLED=no in ifcfg, because ifcfg
is generated by dracut at every boot.
-
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
10:14 p.m.
Hey Sergey -
If you check "lvs" and ensure that there's not actually a new LV from the
update, you can cleanly 'rpm -e ovirt-node-ng-image-update', and be ok
without redeploying.
Unfortunately, it's hard to tell from the logs (and '--justdb' hanging)
what's happening here, but I'll try to reproduce.
NetworkManager disablement should "stick" across upgrades, but it's
possible that iscsi roots are doing something here. I'll check for a dracut
flag, also...
On Mon, Feb 6, 2017 at 1:14 PM, Sandro Bonazzola <sbonazzo(a)redhat.com>
wrote:
Adding Douglas and Ryan
Il 06/Feb/2017 13:32, "Sergey Kulikov" <serg_k(a)msm.ru> ha scritto:
>
> 1) I've updated from 4.0.6 to 4.1.0 (on Feb 01 node-ng was at version
> 4.1.0-0)
> After some time engine alerted, that this node have updates to
> ovirt-node-ng-image-4.1.0-1,
> but update from engine timed out, there were hanging processes in ps on
> this node:
>
> root 36309 0.0 0.0 113120 1564 ? Ss 19:04 0:00 bash -c
> umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t
> ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" >
/dev/null 2>&1; rm
> -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar
--warning=no-timestamp -C
> "${MYTMP}" -x && "${MYTMP}"/ovirt-host-mgmt
DIALOG/dialect=str:machine
> DIALOG/customization=bool:True
> root 36339 0.2 0.0 496700 94208 ? S 19:04 0:21
> /bin/python /tmp/ovirt-GCmVusccfe/pythonlib/otopi/__main__.py
> "BASE/pluginPath=str:/tmp/ovirt-GCmVusccfe/otopi-plugins"
> APPEND:BASE/pluginGroups=str:ovirt-host-common:ovirt-host-mgmt
> DIALOG/dialect=str:machine DIALOG/customization=bool:True
> root 37498 0.0 0.0 113124 1452 ? S 19:09 0:00 /bin/sh
> /var/tmp/rpm-tmp.4UqJ4e 1
> root 37560 0.0 0.0 0 0 ? S< 21:42 0:00
> [kworker/21:2H]
> root 37626 0.0 0.0 174516 5996 ? S 19:09 0:00 rpm -Uvh
> --quiet --justdb /usr/share/imgbased/ovirt-node
> -ng-image-update-4.1.0-1.el7.centos.noarch.rpm
>
> they were hanging forever, I ended up with rebooting the node, no errors
> in log, it was just hanging at:
>
> 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:SEND ***CONFIRM GPG_KEY Confirm use of
> GPG Key userid=oVirt <infra(a)ovirt.org> hexkeyid=FE590CB7
> 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:SEND ###
> 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:SEND ### Please confirm 'GPG_KEY'
> Confirm use of GPG Key userid=oVirt <infra(a)ovirt.org> hexkeyid=FE590CB7
> 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:SEND ### Response is CONFIRM
> GPG_KEY=yes|no or ABORT GPG_KEY
> 2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
> dialog.__logString:204 DIALOG:RECEIVE CONFIRM GPG_KEY=yes
> 2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
> yumpackager.info:80 Yum Status: Running Test Transaction
> Running Transaction Check
> 2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
> yumpackager.info:80 Yum Status: Running Transaction
> 2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
> yumpackager.info:80 Yum install: 1/2: ovirt-node-ng-image-4.1.0-1.el
> 7.centos.noarch
> 2017-02-03 19:09:20 DEBUG otopi.plugins.otopi.packagers.yumpackager
> yumpackager.verbose:76 Yum Done: ovirt-node-ng-image-4.1.0-1.el
> 7.centos.noarch
> 2017-02-03 19:09:20 INFO otopi.plugins.otopi.packagers.yumpackager
> yumpackager.info:80 Yum install: 2/2: ovirt-node-ng-image-update-4.1
> .0-1.el7.centos.noarch
>
> now my node have this layout:
> # imgbase layout
> ovirt-node-ng-4.1.0-0.20170201.0
> +- ovirt-node-ng-4.1.0-0.20170201.0+1
> (so update failed)
> but 4.1.0-1 rpms are marked as "installed" and yum can't find any
> updates, can I rollback to base layout without installed 4.1.0-1 rms ?
> imgbase rollback needs at least 2 layers over base.
>
> Or maybe the only way is to reinstall this node?
>
> 2) And another question, how can I disable NetworkManger permanently, or
> exclude some interfaces permanently?
> I've tried to disable NetworkManger by systemctl, but after update from
> 4.0 to 4.1 it was re-enabled(so it's not persistent between updates).
> I've an issue with iscsi root and enabled NetworkManger, because NM tries
> to bring down\up my iscsi interfaces on boot, and sometimes FS remounting RO
> because of IO errors, I can't put NM_CONTROLLED=no in ifcfg, because
> ifcfg is generated by dracut at every boot.
>
>
> -
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
2:50 a.m.
There is no new lv, I can remove rpm and reinstall manually to reproduce with any debug
options, if you want.
I left host in maintenance mode.
should I also remove ovirt-node-ng-image ? I found something strange:
# rpm -q ovirt-node-ng-image
ovirt-node-ng-image-4.1.0-1.el7.centos.noarch
# rpm -ql ovirt-node-ng-image
/usr/share/ovirt-node-ng/image
/usr/share/ovirt-node-ng/image/ovirt-node-ng-4.1.0-0.20170201.0.el7.squashfs.img
/usr/share/ovirt-node-ng/image/product.img
version is 4.1.0-1, but squashfs file is 4.1.0-0
About iscsi... I've disabled NM in 4.0.6, everything was ok, it was re-enabled only
after update, I don't
think iscsi root affects NM status.
The problem is only with reboot, 1 time out of ~4 reboots it's falling to RO.
Also I have centos hosts and NM stays disabled there, maybe I should try
"unmanaged-devices="
in NM config, but I think it's not ideal, NM is useless for this setup)
--
Tuesday, February 7, 2017, 07:14:55:
Hey Sergey -
If you check "lvs" and ensure that there's not actually
a new LV
from the update, you can cleanly 'rpm -e
ovirt-node-ng-image-update', and be ok without redeploying.
Unfortunately, it's hard to tell from the logs (and
'--justdb'
hanging) what's happening here, but I'll try to reproduce.
NetworkManager disablement should "stick" across upgrades,
but it's
possible that iscsi roots are doing something here. I'll check for a dracut flag,
also...
On Mon, Feb 6, 2017 at 1:14 PM, Sandro Bonazzola
<sbonazzo(a)redhat.com> wrote:
Adding Douglas and Ryan
Il 06/Feb/2017 13:32, "Sergey Kulikov"
<serg_k(a)msm.ru> ha scritto:
1) I've updated from 4.0.6 to 4.1.0 (on Feb 01 node-ng was at
version 4.1.0-0)
After some time engine alerted, that this node have updates to
ovirt-node-ng-image-4.1.0-1,
but update from engine timed out, there were hanging processes in ps on this node:
root 36309 0.0 0.0 113120 1564 ? Ss 19:04 0:00
bash -c umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t
ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null
2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar
--warning=no-timestamp -C "${MYTMP}" -x &&
"${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine
DIALOG/customization=bool:True
root 36339 0.2 0.0 496700 94208 ? S 19:04 0:21
/bin/python /tmp/ovirt-GCmVusccfe/pythonlib/otopi/__main__.py
"BASE/pluginPath=str:/tmp/ovirt-GCmVusccfe/otopi-plugins"
APPEND:BASE/pluginGroups=str:ovirt-host-common:ovirt-host-mgmt
DIALOG/dialect=str:machine DIALOG/customization=bool:True
root 37498 0.0 0.0 113124 1452 ? S 19:09 0:00 /bin/sh
/var/tmp/rpm-tmp.4UqJ4e 1
root 37560 0.0 0.0 0 0 ? S< 21:42 0:00 [kworker/21:2H]
root 37626 0.0 0.0 174516 5996 ? S 19:09 0:00
rpm -Uvh --quiet --justdb
/usr/share/imgbased/ovirt-node-ng-image-update-4.1.0-1.el7.centos.noarch.rpm
they were hanging forever, I ended up with rebooting the node, no
errors in log, it was just hanging at:
2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
dialog.__logString:204 DIALOG:SEND ***CONFIRM GPG_KEY Confirm
use of GPG Key userid=oVirt <infra(a)ovirt.org> hexkeyid=FE590CB7
2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
dialog.__logString:204 DIALOG:SEND ###
2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
dialog.__logString:204 DIALOG:SEND ### Please confirm
'GPG_KEY' Confirm use of GPG Key userid=oVirt <infra(a)ovirt.org>
hexkeyid=FE590CB7
2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
dialog.__logString:204 DIALOG:SEND ### Response is CONFIRM GPG_KEY=yes|no or ABORT
GPG_KEY
2017-02-03 19:09:16 DEBUG otopi.plugins.otopi.dialog.machine
dialog.__logString:204 DIALOG:RECEIVE CONFIRM GPG_KEY=yes
2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
yumpackager.info:80 Yum Status: Running Test Transaction
Running Transaction Check
2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
yumpackager.info:80 Yum Status: Running Transaction
2017-02-03 19:09:16 INFO otopi.plugins.otopi.packagers.yumpackager
yumpackager.info:80 Yum install: 1/2:
ovirt-node-ng-image-4.1.0-1.el7.centos.noarch
2017-02-03 19:09:20 DEBUG
otopi.plugins.otopi.packagers.yumpackager yumpackager.verbose:76 Yum
Done: ovirt-node-ng-image-4.1.0-1.el7.centos.noarch
2017-02-03 19:09:20 INFO otopi.plugins.otopi.packagers.yumpackager
yumpackager.info:80 Yum install: 2/2:
ovirt-node-ng-image-update-4.1.0-1.el7.centos.noarch
now my node have this layout:
# imgbase layout
ovirt-node-ng-4.1.0-0.20170201.0
+- ovirt-node-ng-4.1.0-0.20170201.0+1
(so update failed)
but 4.1.0-1 rpms are marked as "installed" and yum can't find any
updates, can I rollback to base layout without installed 4.1.0-1 rms ?
imgbase rollback needs at least 2 layers over base.
Or maybe the only way is to reinstall this node?
2) And another question, how can I disable NetworkManger
permanently, or exclude some interfaces permanently?
I've tried to disable NetworkManger by systemctl, but after update
from 4.0 to 4.1 it was re-enabled(so it's not persistent between updates).
I've an issue with iscsi root and enabled NetworkManger, because
NM tries to bring down\up my iscsi interfaces on boot, and sometimes FS remounting RO
because of IO errors, I can't put NM_CONTROLLED=no in ifcfg,
because ifcfg is generated by dracut at every boot.
-
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
2643
days inactive
2644
days old
3 comments
3 participants
participants (3)
-
Ryan Barry -
Sandro Bonazzola -
Sergey Kulikov