[ovirt-devel] [ ovirt-devel ] [OST Failure Report ] [ oVirt Master ] [ 002_bootstrap ] [ 20/08/17 ]

Tue Aug 22 09:57:29 UTC 2017

On Tue, Aug 22, 2017 at 12:52 PM, Anton Marchukov <amarchuk at redhat.com> wrote:
> Hello All.
>
> Any news on this?  I see the latest failures for vdsm is the same [1] and
> the job is still not working for it.
>
> [1]
> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1901/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/lago-basic-suite-master-engine/_var_log/ovirt-engine/host-deploy/ovirt-host-deploy-20170822035135-lago-basic-suite-master-host0-1f46d892.log

This log has:

2017-08-22 03:51:28,272-0400 DEBUG otopi.context
context._executeMethod:128 Stage closeup METHOD
otopi.plugins.ovirt_host_deploy.vdsm.packages.Plugin._reconfigure
2017-08-22 03:51:28,272-0400 DEBUG
otopi.plugins.ovirt_host_deploy.vdsm.packages plugin.executeRaw:813
execute: ('/bin/vdsm-tool', 'configure', '--force'),
executable='None', cwd='None', env=None
2017-08-22 03:51:30,687-0400 DEBUG
otopi.plugins.ovirt_host_deploy.vdsm.packages plugin.executeRaw:863
execute-result: ('/bin/vdsm-tool', 'configure', '--force'), rc=1
2017-08-22 03:51:30,688-0400 DEBUG
otopi.plugins.ovirt_host_deploy.vdsm.packages plugin.execute:921
execute-output: ('/bin/vdsm-tool', 'configure', '--force') stdout:

Checking configuration status...

abrt is not configured for vdsm
WARNING: LVM local configuration: /etc/lvm/lvmlocal.conf is not based
on vdsm configuration
lvm requires configuration
libvirt is not configured for vdsm yet
FAILED: conflicting vdsm and libvirt-qemu tls configuration.
vdsm.conf with ssl=True requires the following changes:
libvirtd.conf: listen_tcp=0, auth_tcp="sasl", listen_tls=1
qemu.conf: spice_tls=1.
multipath requires configuration

Running configure...
Reconfiguration of abrt is done.
Reconfiguration of passwd is done.
WARNING: LVM local configuration: /etc/lvm/lvmlocal.conf is not based
on vdsm configuration
Backing up /etc/lvm/lvmlocal.conf to /etc/lvm/lvmlocal.conf.201708220351
Installing /usr/share/vdsm/lvmlocal.conf at /etc/lvm/lvmlocal.conf
Units need configuration: {'lvm2-lvmetad.service': {'LoadState':
'loaded', 'ActiveState': 'active'}, 'lvm2-lvmetad.socket':
{'LoadState': 'loaded', 'ActiveState': 'active'}}
Reconfiguration of lvm is done.
Reconfiguration of sebool is done.

2017-08-22 03:51:30,688-0400 DEBUG
otopi.plugins.ovirt_host_deploy.vdsm.packages plugin.execute:926
execute-output: ('/bin/vdsm-tool', 'configure', '--force') stderr:
Error:  ServiceNotExistError: Tried all alternatives but failed:
ServiceNotExistError: dev-hugepages1G.mount is not native systemctl service
ServiceNotExistError: dev-hugepages1G.mount is not a SysV service

2017-08-22 03:51:30,689-0400 WARNING
otopi.plugins.ovirt_host_deploy.vdsm.packages
packages._reconfigure:155 Cannot configure vdsm

Nir, any idea?

>
>
>
> On Sun, Aug 20, 2017 at 12:39 PM, Nir Soffer <nsoffer at redhat.com> wrote:
>>
>> On Sun, Aug 20, 2017 at 11:08 AM Dan Kenigsberg <danken at redhat.com> wrote:
>>>
>>> On Sun, Aug 20, 2017 at 10:39 AM, Yaniv Kaul <ykaul at redhat.com> wrote:
>>> >
>>> >
>>> > On Sun, Aug 20, 2017 at 8:48 AM, Daniel Belenky <dbelenky at redhat.com>
>>> > wrote:
>>> >>
>>> >> Failed test: basic_suite_master/002_bootstrap
>>> >> Version: oVirt Master
>>> >> Link to failed job: ovirt-master_change-queue-tester/1860/
>>> >> Link to logs (Jenkins): test logs
>>> >> Suspected patch: https://gerrit.ovirt.org/#/c/80749/3
>>> >>
>>> >> From what I was able to find, It seems that for some reason VDSM
>>> >> failed to
>>> >> start on host 1. The VDSM log is empty, and the only error I could
>>> >> find in
>>> >> supervdsm.log is that start of LLDP failed (Not sure if it's related)
>>> >
>>> >
>>> > Can you check the networking on the hosts? Something's very strange
>>> > there.
>>> > For example:
>>> > Aug 19 16:38:42 lago-basic-suite-master-host0 NetworkManager[685]:
>>> > <info>
>>> > [1503175122.2682] manager: (e7NZWeNDXwIjQia): new Bond device
>>> > (/org/freedesktop/NetworkManager/Devices/17)
>>> > Aug 19 16:38:42 lago-basic-suite-master-host0 kernel: e7NZWeNDXwIjQia:
>>> > Setting xmit hash policy to layer2+3 (2)
>>> > Aug 19 16:38:42 lago-basic-suite-master-host0 kernel: e7NZWeNDXwIjQia:
>>> > Setting xmit hash policy to encap2+3 (3)
>>> > Aug 19 16:38:42 lago-basic-suite-master-host0 kernel: e7NZWeNDXwIjQia:
>>> > Setting xmit hash policy to encap3+4 (4)
>>> > Aug 19 16:38:42 lago-basic-suite-master-host0 kernel: e7NZWeNDXwIjQia:
>>> > option xmit_hash_policy: invalid value (5)
>>> > Aug 19 16:38:42 lago-basic-suite-master-host0 kernel: e7NZWeNDXwIjQia:
>>> > Setting primary_reselect to always (0)
>>> > Aug 19 16:38:42 lago-basic-suite-master-host0 kernel: e7NZWeNDXwIjQia:
>>> > Setting primary_reselect to better (1)
>>> > Aug 19 16:38:42 lago-basic-suite-master-host0 kernel: e7NZWeNDXwIjQia:
>>> > Setting primary_reselect to failure (2)
>>> > Aug 19 16:38:42 lago-basic-suite-master-host0 kernel: e7NZWeNDXwIjQia:
>>> > option primary_reselect: invalid value (3)
>>> > Aug 19 16:38:42 lago-basic-suite-master-host0 kernel: e7NZWeNDXwIjQia:
>>> > Setting arp_all_targets to any (0)
>>> > Aug 19 16:38:42 lago-basic-suite-master-host0 kernel: e7NZWeNDXwIjQia:
>>> > Setting arp_all_targets to all (1)
>>> > Aug 19 16:38:42 lago-basic-suite-master-host0 kernel: e7NZWeNDXwIjQia:
>>> > option arp_all_targets: invalid value (2)
>>> > Aug 19 16:38:42 lago-basic-suite-master-host0 kernel: bonding:
>>> > e7NZWeNDXwIjQia is being deleted...
>>> > Aug 19 16:38:42 lago-basic-suite-master-host0 lldpad: recvfrom(Event
>>> > interface): No buffer space available
>>> >
>>> > Y.
>>>
>>>
>>>
>>> The post-boot noise with funny-looking bonds is due to our calling of
>>> `vdsm-tool dump-bonding-options` every boot, in order to find the
>>> bonding defaults for the current kernel.
>>>
>>> >
>>> >>
>>> >> From host-deploy log:
>>> >>
>>> >> 2017-08-19 16:38:41,476-0400 DEBUG
>>> >> otopi.plugins.otopi.services.systemd
>>> >> systemd.state:130 starting service vdsmd
>>> >> 2017-08-19 16:38:41,476-0400 DEBUG
>>> >> otopi.plugins.otopi.services.systemd
>>> >> plugin.executeRaw:813 execute: ('/bin/systemctl', 'start',
>>> >> 'vdsmd.service'),
>>> >> executable='None', cwd='None', env=None
>>> >> 2017-08-19 16:38:44,628-0400 DEBUG
>>> >> otopi.plugins.otopi.services.systemd
>>> >> plugin.executeRaw:863 execute-result: ('/bin/systemctl', 'start',
>>> >> 'vdsmd.service'), rc=1
>>> >> 2017-08-19 16:38:44,630-0400 DEBUG
>>> >> otopi.plugins.otopi.services.systemd
>>> >> plugin.execute:921 execute-output: ('/bin/systemctl', 'start',
>>> >> 'vdsmd.service') stdout:
>>> >>
>>> >>
>>> >> 2017-08-19 16:38:44,630-0400 DEBUG
>>> >> otopi.plugins.otopi.services.systemd
>>> >> plugin.execute:926 execute-output: ('/bin/systemctl', 'start',
>>> >> 'vdsmd.service') stderr:
>>> >> Job for vdsmd.service failed because the control process exited with
>>> >> error
>>> >> code. See "systemctl status vdsmd.service" and "journalctl -xe" for
>>> >> details.
>>> >>
>>> >> 2017-08-19 16:38:44,631-0400 DEBUG otopi.context
>>> >> context._executeMethod:142 method exception
>>> >> Traceback (most recent call last):
>>> >>   File "/tmp/ovirt-dunwHj8Njn/pythonlib/otopi/context.py", line 132,
>>> >> in
>>> >> _executeMethod
>>> >>     method['method']()
>>> >>   File
>>> >>
>>> >> "/tmp/ovirt-dunwHj8Njn/otopi-plugins/ovirt-host-deploy/vdsm/packages.py",
>>> >> line 224, in _start
>>> >>     self.services.state('vdsmd', True)
>>> >>   File
>>> >> "/tmp/ovirt-dunwHj8Njn/otopi-plugins/otopi/services/systemd.py",
>>> >> line 141, in state
>>> >>     service=name,
>>> >> RuntimeError: Failed to start service 'vdsmd'
>>> >>
>>> >>
>>> >> From /var/log/messages:
>>> >>
>>> >> Aug 19 16:38:44 lago-basic-suite-master-host0 vdsmd_init_common.sh:
>>> >> Error:
>>> >> Aug 19 16:38:44 lago-basic-suite-master-host0 vdsmd_init_common.sh:
>>> >> One of
>>> >> the modules is not configured to work with VDSM.
>>> >> Aug 19 16:38:44 lago-basic-suite-master-host0 vdsmd_init_common.sh: To
>>> >> configure the module use the following:
>>> >> Aug 19 16:38:44 lago-basic-suite-master-host0 vdsmd_init_common.sh:
>>> >> 'vdsm-tool configure [--module module-name]'.
>>> >> Aug 19 16:38:44 lago-basic-suite-master-host0 vdsmd_init_common.sh: If
>>> >> all
>>> >> modules are not configured try to use:
>>> >> Aug 19 16:38:44 lago-basic-suite-master-host0 vdsmd_init_common.sh:
>>> >> 'vdsm-tool configure --force'
>>> >> Aug 19 16:38:44 lago-basic-suite-master-host0 vdsmd_init_common.sh:
>>> >> (The
>>> >> force flag will stop the module's service and start it
>>> >> Aug 19 16:38:44 lago-basic-suite-master-host0 vdsmd_init_common.sh:
>>> >> afterwards automatically to load the new configuration.)
>>> >> Aug 19 16:38:44 lago-basic-suite-master-host0 vdsmd_init_common.sh:
>>> >> abrt
>>> >> is already configured for vdsm
>>> >> Aug 19 16:38:44 lago-basic-suite-master-host0 vdsmd_init_common.sh:
>>> >> lvm is
>>> >> configured for vdsm
>>> >> Aug 19 16:38:44 lago-basic-suite-master-host0 vdsmd_init_common.sh:
>>> >> libvirt is already configured for vdsm
>>> >> Aug 19 16:38:44 lago-basic-suite-master-host0 vdsmd_init_common.sh:
>>> >> multipath requires configuration
>>> >> Aug 19 16:38:44 lago-basic-suite-master-host0 vdsmd_init_common.sh:
>>> >> Modules sanlock, multipath are not configured
>>
>>
>> This means the host was not deployed correctly. When deploying vdsm
>> host deploy must run "vdsm-tool configure --force", which configures
>> multipath and sanlock.
>>
>> We did not change anything in multipath and sanlock configurators lately.
>>
>> Didi, can you check this?
>>
>> _______________________________________________
>> Devel mailing list
>> Devel at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>
>
>
>
> --
> Anton Marchukov
> Team Lead - Release Management - RHV DevOps - Red Hat
>

-- 
Didi