<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jan 11, 2017 at 12:49 PM, Milan Zamazal <span dir="ltr"><<a href="mailto:mzamazal@redhat.com" target="_blank">mzamazal@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I just ran ovirt-system-tests on two very different machines. It passed<br>
on one of them, while it failed on the other one, at a different place:<br>
<br>
@ Run test: 005_network_by_label.py:<br>
nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']<br>
# assign_hosts_network_label:<br>
Error while running thread<br>
Traceback (most recent call last):<br>
File "/usr/lib/python2.7/site-<wbr>packages/lago/utils.py", line 55, in _ret_via_queue<br>
queue.put({'return': func()})<br>
File "/var/local/lago/ovirt-system-<wbr>tests/basic-suite-master/test-<wbr>scenarios/005_network_by_<wbr>label.py", line 56, in _assign_host_network_label<br>
host_nic=nic<br>
File "/usr/lib/python2.7/site-<wbr>packages/ovirtsdk/<wbr>infrastructure/brokers.py", line 16231, in add<br>
headers={"Correlation-Id":<wbr>correlation_id, "Expect":expect}<br>
File "/usr/lib/python2.7/site-<wbr>packages/ovirtsdk/<wbr>infrastructure/proxy.py", line 79, in add<br>
return self.request('POST', url, body, headers, cls=cls)<br>
File "/usr/lib/python2.7/site-<wbr>packages/ovirtsdk/<wbr>infrastructure/proxy.py", line 122, in request<br>
persistent_auth=self.__<wbr>persistent_auth<br>
File "/usr/lib/python2.7/site-<wbr>packages/ovirtsdk/<wbr>infrastructure/<wbr>connectionspool.py", line 79, in do_request<br>
persistent_auth)<br>
File "/usr/lib/python2.7/site-<wbr>packages/ovirtsdk/<wbr>infrastructure/<wbr>connectionspool.py", line 162, in __do_request<br>
raise errors.RequestError(response_<wbr>code, response_reason, response_body)<br>
RequestError:<br>
status: 409<br>
reason: Conflict<br>
detail: Cannot add Label. Operation can be performed only when Host status is Maintenance, Up, NonOperational.<br></blockquote><div><br></div><div>This is an issue we've seen from time to time and have not figured it out yet. Do you have engine logs for it?</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I can also see occasional errors like the following in vdsm.log:<br>
<br>
ERROR (JsonRpc (StompReactor)) [vds.dispatcher] SSL error receiving from <yajsonrpc.betterAsyncore.<wbr>Dispatcher connected ('::ffff:192.168.201.3', 47434, 0, 0) at 0x271fd88>: (104, 'Connection reset by peer') (betterAsyncore:119)<br></blockquote><div><br></div><div>This is the core issue of today's - but probably unrelated to the issue you've just described, that we have seen happening from time to time in the past (I'd say that I've seen it happening last time ~2 weeks ago or so, but it's not reproducible easily to me).</div><div>Y.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
So we are probably dealing with an error that occurs "randomly" and is<br>
not related to a particular test.<br>
<br>
Daniel Belenky <<a href="mailto:dbelenky@redhat.com">dbelenky@redhat.com</a>> writes:<br>
<br>
> Link to Jenkins<br>
> <<a href="http://jenkins.ovirt.org/view/experimental%20jobs/job/test-repo_ovirt_experimental_master/4648/artifact/exported-artifacts/basic_suite_master.sh-el7/exported-artifacts/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/<wbr>view/experimental%20jobs/job/<wbr>test-repo_ovirt_experimental_<wbr>master/4648/artifact/exported-<wbr>artifacts/basic_suite_master.<wbr>sh-el7/exported-artifacts/</a>><br>
<div class="HOEnZb"><div class="h5">><br>
> On Wed, Jan 11, 2017 at 10:26 AM, Francesco Romani <<a href="mailto:fromani@redhat.com">fromani@redhat.com</a>><br>
> wrote:<br>
><br>
>> Hi all<br>
>><br>
>> On 01/11/2017 08:52 AM, Eyal Edri wrote:<br>
>><br>
>> Adding Tomas from Virt.<br>
>><br>
>> On Tue, Jan 10, 2017 at 10:54 AM, Piotr Kliczewski <<br>
>> <a href="mailto:piotr.kliczewski@gmail.com">piotr.kliczewski@gmail.com</a>> wrote:<br>
>><br>
>>> On Tue, Jan 10, 2017 at 9:29 AM, Daniel Belenky <<a href="mailto:dbelenky@redhat.com">dbelenky@redhat.com</a>><br>
>>> wrote:<br>
>>> > Hi all,<br>
>>> ><br>
>>> > test-repo_ovirt_experimental_<wbr>master (link to Jenkins) job failed on<br>
>>> > basic_sanity scenario.<br>
>>> > The job was triggered by <a href="https://gerrit.ovirt.org/#/c/69845/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>69845/</a><br>
>>> ><br>
>>> > From looking at the logs, it seems that the reason is VDSM.<br>
>>> ><br>
>>> > In the VDSM log, i see the following error:<br>
>>> ><br>
>>> > 2017-01-09 16:47:41,331 ERROR (JsonRpc (StompReactor)) [vds.dispatcher]<br>
>>> SSL<br>
>>> > error receiving from <yajsonrpc.betterAsyncore.<wbr>Dispatcher connected<br>
>>> ('::1',<br>
>>> > 34942, 0, 0) at 0x36b95f0>: unexpected eof (betterAsyncore:119)<br>
>>><br>
>><br>
>> Daniel, could you please remind me the jenkins link? I see something<br>
>> suspicious on the Vdsm log.<br>
>> Most notably, Vdsm received SIGTERM. Is this expected and part of the test?<br>
>><br>
>> ><br>
>>><br>
>>> This issue means that the client closed connection while vdsm was<br>
>>> replying. It can happen at any time<br>
>>> when the client is not nice with the connection. As you can see the<br>
>>> client connected locally '::1'.<br>
>>><br>
>>> ><br>
>>> > Also, when looking at the MOM logs, I see the the following:<br>
>>> ><br>
>>> > 2017-01-09 16:43:39,508 - mom.vdsmInterface - ERROR - Cannot connect to<br>
>>> > VDSM! [Errno 111] Connection refused<br>
>>> ><br>
>>><br>
>>> Looking at the log at this time vdsm had no open socket.<br>
>><br>
>><br>
>><br>
>> Correct, but IIRC we have a race on startup - that's the reason why MOM<br>
>> retries to connect. After the new try, MOM seems to behave<br>
>> correctly:<br>
>><br>
>> 2017-01-09 16:44:05,672 - mom.RPCServer - INFO - ping()<br>
>> 2017-01-09 16:44:05,673 - mom.RPCServer - INFO - getStatistics()<br>
>><br>
>> --<br>
>> Francesco Romani<br>
>> Red Hat Engineering Virtualization R & D<br>
>> IRC: fromani<br>
>><br>
>><br>
</div></div><div class="HOEnZb"><div class="h5">______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br>
</div></div></blockquote></div><br></div></div>