[ovirt-devel] [25-1-18] [ OST Failure Report] [oVirt Master (vdsm)] [post-002_bootstrap]

Thu Jan 25 11:26:02 UTC 2018

The revert was merged.

On Thu, Jan 25, 2018 at 12:32 PM, Daniel Belenky <dbelenky at redhat.com>
wrote:

> Have you tried running OST with rpms from the suspected patch to reproduce?
>

I meant reproducing it manually, not though OST.

> On Thu, Jan 25, 2018 at 12:24 PM Edward Haas <ehaas at redhat.com> wrote:
>
>> We have two options, a revert or a fix:
>> Revert: https://gerrit.ovirt.org/#/c/86789/
>> Fix: https://gerrit.ovirt.org/#/c/86785/
>>
>> We are not sure about the fix because we cannot reproduce the problem
>> manually.
>>
>>
>> On Thu, Jan 25, 2018 at 10:45 AM, Eyal Edri <eedri at redhat.com> wrote:
>>
>>> Once you have RPMs, you can run the upgrade suite from the manual job.
>>>
>>> On Thu, Jan 25, 2018 at 10:43 AM, Edward Haas <ehaas at redhat.com> wrote:
>>>
>>>> Can we test if this one fixes this problem?
>>>> https://gerrit.ovirt.org/#/c/86781
>>>>
>>>> On Thu, Jan 25, 2018 at 10:00 AM, Eyal Edri <eedri at redhat.com> wrote:
>>>>
>>>>> Indeed, the patch looks relevant,
>>>>> Dan, can we revert it or send a fix ASAP to avoid building up a large
>>>>> queue?
>>>>>
>>>>> On Thu, Jan 25, 2018 at 9:29 AM, Daniel Belenky <dbelenky at redhat.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> We failed to setup host in OST upgrade from 4.1 to master suite.
>>>>>> Please note that the upgrade suite installs 4.1 engine, then upgrades
>>>>>> it to master and then tries to set up a host.
>>>>>>
>>>>>> *Links:*
>>>>>>
>>>>>>    1. Link to failed job
>>>>>>    <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/5093/artifact/exported-artifacts/upgrade-from-release-suit-master-el7/test_logs/upgrade-from-release-suite-master/post-002_bootstrap.py/>
>>>>>>    2. Suspected patch: Gerrit 86474/33
>>>>>>    <https://gerrit.ovirt.org/#/c/86474/33>
>>>>>>
>>>>>> *Error snippet from engine.log (engine):*
>>>>>>
>>>>>> 2018-01-24 15:13:20,257-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [34609a2f] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host lago-upgrade-from-release-suite-master-host0: Failed to execute stage 'Closing up': Failed to start service 'vdsmd'.
>>>>>> 2018-01-24 15:13:20,301-05 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [34609a2f] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host0. Stage: Clean up.
>>>>>> 2018-01-24 15:13:20,304-05 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [34609a2f] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host0. Stage: Pre-termination.
>>>>>> 2018-01-24 15:13:20,332-05 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [34609a2f] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host0. Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-host-deploy-20180124151320-lago-upgrade-from-release-suite-master-host0-34609a2f.log'.
>>>>>> 2018-01-24 15:13:29,227-05 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [34609a2f] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host0. Stage: Termination.
>>>>>> 2018-01-24 15:13:29,321-05 ERROR [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [34609a2f] SSH error running command root at lago-upgrade-from-release-suite-master-host0:'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x &&  "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True': IOException: Command returned failure code 1 during SSH session 'root at lago-upgrade-from-release-suite-master-host0'
>>>>>> 2018-01-24 15:13:29,322-05 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [34609a2f] Error during host lago-upgrade-from-release-suite-master-host0 install
>>>>>> 2018-01-24 15:13:29,324-05 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [34609a2f] Host installation failed for host '4d681c3b-e8db-4a71-b5e3-0db096e3ae9c', 'lago-upgrade-from-release-suite-master-host0': Command returned failure code 1 during SSH session 'root at lago-upgrade-from-release-suite-master-host0'
>>>>>> 2018-01-24 15:13:29,330-05 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [34609a2f] START, SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host0, SetVdsStatusVDSCommandParameters:{hostId='4d681c3b-e8db-4a71-b5e3-0db096e3ae9c', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5e6c4a3e
>>>>>> 2018-01-24 15:13:29,339-05 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [34609a2f] FINISH, SetVdsStatusVDSCommand, log id: 5e6c4a3e
>>>>>> 2018-01-24 15:13:29,346-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [34609a2f] EVENT_ID: VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host0 installation failed. Command returned failure code 1 during SSH session 'root at lago-upgrade-from-release-suite-master-host0'.
>>>>>>
>>>>>> *Error snippet from /var/log/messages (host0):*
>>>>>>
>>>>>> 15:13:19 host0 NetworkManager[580]: <info>  (bondscan-Ncw7DP): new Bond device (carrier: OFF, driver: 'bonding', ifindex: 4)
>>>>>> 15:13:19 host0 vdsm-tool: Traceback (most recent call last):
>>>>>> 15:13:19 host0 vdsm-tool: File "/usr/bin/vdsm-tool", line 219, in main
>>>>>> 15:13:19 host0 vdsm-tool: return tool_command[cmd]["command"](*args)
>>>>>> 15:13:19 host0 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/tool/network.py", line 97, in dump_bonding_options
>>>>>> 15:13:19 host0 vdsm-tool: sysfs_options_mapper.dump_bonding_options()
>>>>>> 15:13:19 host0 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/link/bond/sysfs_options_mapper.py", line 48, in dump_bonding_options
>>>>>> 15:13:19 host0 vdsm-tool: jdump(_get_default_bonding_options(), f)
>>>>>> 15:13:19 host0 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/link/bond/sysfs_options_mapper.py", line 60, in _get_default_bonding_options
>>>>>> 15:13:19 host0 vdsm-tool: with _bond_device(bond_name):
>>>>>> 15:13:19 host0 vdsm-tool: File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
>>>>>> 15:13:19 host0 vdsm-tool: return self.gen.next()
>>>>>> 15:13:19 host0 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/link/bond/sysfs_options_mapper.py", line 102, in _bond_device
>>>>>> 15:13:19 host0 vdsm-tool: _unmanage_nm_device(bond_name)
>>>>>> 15:13:19 host0 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/link/bond/sysfs_options_mapper.py", line 116, in _unmanage_nm_device
>>>>>> 15:13:19 host0 vdsm-tool: dev.managed = False
>>>>>> 15:13:19 host0 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/nm/networkmanager.py", line 90, in managed
>>>>>> 15:13:19 host0 vdsm-tool: self._device.managed = value
>>>>>> 15:13:19 host0 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/nm/nmdbus/device.py", line 81, in managed
>>>>>> 15:13:19 host0 vdsm-tool: return self._set_property('Managed', value)
>>>>>> 15:13:19 host0 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/network/nm/nmdbus/device.py", line 88, in _set_property
>>>>>> 15:13:19 host0 vdsm-tool: self.IF_NAME, property_name, property_value)
>>>>>> 15:13:19 host0 vdsm-tool: File "/usr/lib64/python2.7/site-packages/dbus/proxies.py", line 70, in __call__
>>>>>> 15:13:19 host0 vdsm-tool: return self._proxy_method(*args, **keywords)
>>>>>> 15:13:19 host0 vdsm-tool: File "/usr/lib64/python2.7/site-packages/dbus/proxies.py", line 145, in __call__
>>>>>> 15:13:19 host0 vdsm-tool: **keywords)
>>>>>> 15:13:19 host0 vdsm-tool: File "/usr/lib64/python2.7/site-packages/dbus/connection.py", line 651, in call_blocking
>>>>>> 15:13:19 host0 vdsm-tool: message, timeout)
>>>>>> 15:13:19 host0 vdsm-tool: DBusException: org.freedesktop.DBus.Error.AccessDenied: Property "Managed" of interface "org.freedesktop.NetworkManager.Device" is not settable
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> --
>>>>>>
>>>>>> DANIEL BELENKY
>>>>>>
>>>>>> RHV DEVOPS
>>>>>>
>>>>>> _______________________________________________
>>>>>> Devel mailing list
>>>>>> Devel at ovirt.org
>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Eyal edri
>>>>>
>>>>>
>>>>> MANAGER
>>>>>
>>>>> RHV DevOps
>>>>>
>>>>> EMEA VIRTUALIZATION R&D
>>>>>
>>>>>
>>>>> Red Hat EMEA <https://www.redhat.com/>
>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED.
>>>>> <https://redhat.com/trusted>
>>>>> phone: +972-9-7692018 <+972%209-769-2018>
>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Eyal edri
>>>
>>>
>>> MANAGER
>>>
>>> RHV DevOps
>>>
>>> EMEA VIRTUALIZATION R&D
>>>
>>>
>>> Red Hat EMEA <https://www.redhat.com/>
>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED.
>>> <https://redhat.com/trusted>
>>> phone: +972-9-7692018 <+972%209-769-2018>
>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>
>>
>> --
> Daniel Belenky
> RHV DevOps
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20180125/c3e9e220/attachment-0001.html>