[25-1-18] [ OST Failure Report] [oVirt Master (vdsm)] [post-002_bootstrap]

25 Jan 2018

      Hi,

We failed to setup host in OST upgrade from 4.1 to master suite.
Please note that the upgrade suite installs 4.1 engine, then upgrades it to
master and then tries to set up a host.

*Links:*

   1. Link to failed job
   <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5093/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-002_bootstrap.py/>
   2. Suspected patch: Gerrit 86474/33
   <https://gerrit.ovirt.org/#/c/86474/33>

*Error snippet from engine.log (engine):*

2018-01-24 15:13:20,257-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [34609a2f] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511),
An error has occurred during installation of Host
lago-upgrade-from-release-suite-master-host0: Failed to execute stage
'Closing up': Failed to start service 'vdsmd'.
2018-01-24 15:13:20,301-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [34609a2f] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509),
Installing Host lago-upgrade-from-release-suite-master-host0. Stage:
Clean up.
2018-01-24 15:13:20,304-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [34609a2f] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509),
Installing Host lago-upgrade-from-release-suite-master-host0. Stage:
Pre-termination.
2018-01-24 15:13:20,332-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [34609a2f] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509),
Installing Host lago-upgrade-from-release-suite-master-host0.
Retrieving installation logs to:
'/var/log/ovirt-engine/host-deploy/ovirt-host-deploy-20180124151320-lago-upgrade-from-release-suite-master-host0-34609a2f.log'.
2018-01-24 15:13:29,227-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [34609a2f] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509),
Installing Host lago-upgrade-from-release-suite-master-host0. Stage:
Termination.
2018-01-24 15:13:29,321-05 ERROR
[org.ovirt.engine.core.uutils.ssh.SSHDialog]
(EE-ManagedThreadFactory-engine-Thread-1) [34609a2f] SSH error running
command root@lago-upgrade-from-release-suite-master-host0:'umask 0077;
MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)";
trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr
\"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C
"${MYTMP}" -x &&  "${MYTMP}"/ovirt-host-deploy
DIALOG/dialect=str:machine DIALOG/customization=bool:True':
IOException: Command returned failure code 1 during SSH session
'root@lago-upgrade-from-release-suite-master-host0'
2018-01-24 15:13:29,322-05 ERROR
[org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase]
(EE-ManagedThreadFactory-engine-Thread-1) [34609a2f] Error during host
lago-upgrade-from-release-suite-master-host0 install
2018-01-24 15:13:29,324-05 ERROR
[org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [34609a2f] Host installation
failed for host '4d681c3b-e8db-4a71-b5e3-0db096e3ae9c',
'lago-upgrade-from-release-suite-master-host0': Command returned
failure code 1 during SSH session
'root@lago-upgrade-from-release-suite-master-host0'
2018-01-24 15:13:29,330-05 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [34609a2f] START,
SetVdsStatusVDSCommand(HostName =
lago-upgrade-from-release-suite-master-host0,
SetVdsStatusVDSCommandParameters:{hostId='4d681c3b-e8db-4a71-b5e3-0db096e3ae9c',
status='InstallFailed', nonOperationalReason='NONE',
stopSpmFailureLogged='false', maintenanceReason='null'}), log id:
5e6c4a3e
2018-01-24 15:13:29,339-05 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [34609a2f] FINISH,
SetVdsStatusVDSCommand, log id: 5e6c4a3e
2018-01-24 15:13:29,346-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-1) [34609a2f] EVENT_ID:
VDS_INSTALL_FAILED(505), Host
lago-upgrade-from-release-suite-master-host0 installation failed.
Command returned failure code 1 during SSH session
'root@lago-upgrade-from-release-suite-master-host0'.

*Error snippet from /var/log/messages (host0):*

15:13:19 host0 NetworkManager[580]: <info>  (bondscan-Ncw7DP): new
Bond device (carrier: OFF, driver: 'bonding', ifindex: 4)
15:13:19 host0 vdsm-tool: Traceback (most recent call last):
15:13:19 host0 vdsm-tool: File "/usr/bin/vdsm-tool", line 219, in main
15:13:19 host0 vdsm-tool: return tool_command[cmd]["command"](*args)
15:13:19 host0 vdsm-tool: File
"/usr/lib/python2.7/site-packages/vdsm/tool/network.py", line 97, in
dump_bonding_options
15:13:19 host0 vdsm-tool: sysfs_options_mapper.dump_bonding_options()
15:13:19 host0 vdsm-tool: File
"/usr/lib/python2.7/site-packages/vdsm/network/link/bond/sysfs_options_mapper.py",
line 48, in dump_bonding_options
15:13:19 host0 vdsm-tool: jdump(_get_default_bonding_options(), f)
15:13:19 host0 vdsm-tool: File
"/usr/lib/python2.7/site-packages/vdsm/network/link/bond/sysfs_options_mapper.py",
line 60, in _get_default_bonding_options
15:13:19 host0 vdsm-tool: with _bond_device(bond_name):
15:13:19 host0 vdsm-tool: File "/usr/lib64/python2.7/contextlib.py",
line 17, in __enter__
15:13:19 host0 vdsm-tool: return self.gen.next()
15:13:19 host0 vdsm-tool: File
"/usr/lib/python2.7/site-packages/vdsm/network/link/bond/sysfs_options_mapper.py",
line 102, in _bond_device
15:13:19 host0 vdsm-tool: _unmanage_nm_device(bond_name)
15:13:19 host0 vdsm-tool: File
"/usr/lib/python2.7/site-packages/vdsm/network/link/bond/sysfs_options_mapper.py",
line 116, in _unmanage_nm_device
15:13:19 host0 vdsm-tool: dev.managed = False
15:13:19 host0 vdsm-tool: File
"/usr/lib/python2.7/site-packages/vdsm/network/nm/networkmanager.py",
line 90, in managed
15:13:19 host0 vdsm-tool: self._device.managed = value
15:13:19 host0 vdsm-tool: File
"/usr/lib/python2.7/site-packages/vdsm/network/nm/nmdbus/device.py",
line 81, in managed
15:13:19 host0 vdsm-tool: return self._set_property('Managed', value)
15:13:19 host0 vdsm-tool: File
"/usr/lib/python2.7/site-packages/vdsm/network/nm/nmdbus/device.py",
line 88, in _set_property
15:13:19 host0 vdsm-tool: self.IF_NAME, property_name, property_value)
15:13:19 host0 vdsm-tool: File
"/usr/lib64/python2.7/site-packages/dbus/proxies.py", line 70, in
__call__
15:13:19 host0 vdsm-tool: return self._proxy_method(*args, **keywords)
15:13:19 host0 vdsm-tool: File
"/usr/lib64/python2.7/site-packages/dbus/proxies.py", line 145, in
__call__
15:13:19 host0 vdsm-tool: **keywords)
15:13:19 host0 vdsm-tool: File
"/usr/lib64/python2.7/site-packages/dbus/connection.py", line 651, in
call_blocking
15:13:19 host0 vdsm-tool: message, timeout)
15:13:19 host0 vdsm-tool: DBusException:
org.freedesktop.DBus.Error.AccessDenied: Property "Managed" of
interface "org.freedesktop.NetworkManager.Device" is not settable

Thanks,
-- 

DANIEL BELENKY

RHV DEVOPS

Daniel Belenky

Eyal Edri

Edward Haas

Eyal Edri

Edward Haas

Daniel Belenky

Edward Haas

tags

participants (3)