[ovirt-devel] [ OST Failure Report ] [ oVirt master ] [ 26.02.2017 ] [test-repo_ovirt_experimental_master]

[Piotr Kliczewski]

Looking at the logs I see that the issue occurred when we were in
setupNetworks and in between we called getStats.

We called:

2017-02-26 05:43:44,655-05 DEBUG
[org.ovirt.vdsm.jsonrpc.client.reactors.stomp.StompCommonClient]
(org.ovirt.thread.pool-7-thread-4) [] Message sent: SEND
destination:jms.topic.vdsm_requests
content-length:370
reply-to:jms.topic.vdsm_responses

<JsonRpcRequest id: "8eea12f1-f41a-4174-8032-03f34fa2e806", method:
Host.setupNetworks, params: {networks={VLAN200_Network={vlan=200,
netmask=255.255.255.0, ipv6autoconf=false, nic=eth0, bridged=false,
ipaddr=192.0.3.1, dhcpv6=false, mtu=1500, switch=legacy}}, bondings={},
options={connectivityTimeout=120, connectivityCheck=true}}>

in between we see:

2017-02-26 05:43:51,741-05 DEBUG
[org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message]
(DefaultQuartzScheduler5) [63e30973] SEND
destination:jms.topic.vdsm_requests
reply-to:jms.topic.vdsm_responses
content-length:98

{"jsonrpc":"2.0","method":"Host.getStats","params":{},"id":"27ffe862-ddda-46a1-a59f-5c5e3478a436"}\00

and setupNetworks response arrived at:

2017-02-26 05:43:51,762-05 DEBUG
[org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp
Reactor) [40fe71c] MESSAGE
content-length:106
destination:jms.topic.vdsm_responses
content-type:application/json
subscription:c1dfc84f-c309-4ea0-8f1c-92462aacbd8e

{"jsonrpc": "2.0", "id": "8eea12f1-f41a-4174-8032-03f34fa2e806", "result":
{"message": "Done", "code": 0}}\00

Response to Host.getStats arrived only partially. I thought that we hold
host level lock to stop host monitoring to run in parallel with
setupNetworks.

On Sun, Feb 26, 2017 at 2:23 PM, Nir Soffer <nsoffer at redhat.com> wrote:

> On Sun, Feb 26, 2017 at 3:18 PM, Yaniv Kaul <ykaul at redhat.com> wrote:
>
>> On Sun, Feb 26, 2017 at 3:04 PM Shlomo Ben David <sbendavi at redhat.com>
>> wrote:
>>
>> Hi,
>>
>>
>> Test failed: [ test-repo_ovirt_experimental_master ]
>>
>> Link to Job: [1]
>>
>> Link to all logs: [2]
>>
>> Link to error log: [3]
>>
>>
>> [1] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/5538
>>
>> [2] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ma
>> ster/5538/artifact/exported-artifacts/basic-suit-master-el7/
>> test_logs/basic-suite-master/post-006_migrations.py/
>>
>> [3] http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ma
>> ster/5538/artifact/exported-artifacts/basic-suit-master-el7/
>> nosetests-006_migrations.py.xml
>>
>> Error snippet from the log:
>>
>>
>> The below is not the issue. The issue is (engine log):
>> 2017-02-26 05:43:52,178-05 ERROR [org.ovirt.engine.core.bll.network.host.HostValidator]
>> (default task-11) [d3b3a59d-6cc0-4896-b3f2-8483f9b77fe2] Unable to setup
>> network: operation can only be done when Host status is one of:
>> Maintenance, Up, NonOperational; current status is Connecting
>>
>> And it comes from the host disconnecting a bit from engine, with various
>> errors such as (again, engine log):
>>
>>
>> 2017-02-26 05:43:26,763-05 ERROR [org.ovirt.engine.core.dal.dbb
>> roker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5)
>> [63e30973] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID:
>> null, Call Stack: null, Custom Event ID: -1, Message: VDSM
>> lago-basic-suite-master-host1 *command FullListVDS failed: Unrecognized
>> message received*
>>
>
> The error Shlomo posted may not be the error failing the test, but is is
> very important error that must not be in vdsm logs.
>
> Nir
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20170226/39dc9608/attachment.html>