Subject: [ OST Failure Report ] [ master ] [20-09-2017 ] [ post-002_bootstrap.py ]
This is a multi-part message in MIME format. --------------04175AE2D1C2AC0FFF61E4CE Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, We have a failure in test post-002_bootstrap.py from the logs I can see that lago has an unhandled exception from ovirt sdk which makes sense since the patch reported on CQ is a change to the sdk. ** *Link to suspected patches: https://gerrit.ovirt.org/#/c/82022/* * Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734 Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734/artifact/... (Relevant) error snippet from the log: <error> 2017-09-20 09:17:15,676::testlib.py::assert_equals_within::227::ovirtlago.testlib::ERROR:: * Unhandled exception in <function _host_is_up_4 at 0x3dd25f0> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 338, in _host_is_up_4 host_status = host_service.get().status File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 31529, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 202, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 53, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 199, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 121, in _check_fault body = self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 307, in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 800, in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 840, in _check_content_type raise Error(msg) Error: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML 2017-09-20 09:17:15,683::utils.py::wrapper::480::lago.utils::DEBUG::Looking for a workdir </error> Thanks, * Dafna* * --------------04175AE2D1C2AC0FFF61E4CE Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <p>Hi, <br> </p> <p>We have a failure in test post-002_bootstrap.py</p> <p>from the logs I can see that lago has an unhandled exception from ovirt sdk which makes sense since the patch reported on CQ is a change to the sdk. <br> </p> <p><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-9ef6-91f4-38de-b7313f855cc2"> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/82022/">https://gerrit.ovirt.org/#/c/82022/</a></span></p> <br> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734</a></span></p> <br> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/</a></span></p> <br> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: </span></p> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error></span></p> <br> 2017-09-20 09:17:15,676::testlib.py::assert_equals_within::227::ovirtlago.testlib::ERROR:: * Unhandled exception in <function _host_is_up_4 at 0x3dd25f0><br> Traceback (most recent call last):<br> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within<br> res = func()<br> File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 338, in _host_is_up_4<br> host_status = host_service.get().status<br> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 31529, in get<br> return self._internal_get(headers, query, wait)<br> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 202, in _internal_get<br> return future.wait() if wait else future<br> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 53, in wait<br> return self._code(response)<br> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 199, in callback<br> self._check_fault(response)<br> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 121, in _check_fault<br> body = self._internal_read_body(response)<br> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 307, in _internal_read_body<br> self._connection.check_xml_content_type(response)<br> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 800, in check_xml_content_type<br> response.headers<br> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 840, in _check_content_type<br> raise Error(msg)<br> Error: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML<br> 2017-09-20 09:17:15,683::utils.py::wrapper::480::lago.utils::DEBUG::Looking for a workdir<br> <br> <br> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error></span></p> <br> Thanks, <br> </b></p> <p><span style="font-weight:normal;">Dafna</span><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-9ef6-91f4-38de-b7313f855cc2"><br> </b></p> </body> </html> --------------04175AE2D1C2AC0FFF61E4CE--
Ondro, could you please take a look? On Wed, Sep 20, 2017 at 1:07 PM, Dafna Ron <dron@redhat.com> wrote:
Hi,
We have a failure in test post-002_bootstrap.py
from the logs I can see that lago has an unhandled exception from ovirt sdk which makes sense since the patch reported on CQ is a change to the sdk.
*Link to suspected patches: https://gerrit.ovirt.org/#/c/82022/ <https://gerrit.ovirt.org/#/c/82022/>*
* Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734 <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734> Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734/artifact/... <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/> (Relevant) error snippet from the log: <error> 2017-09-20 09:17:15,676::testlib.py::assert_equals_within::227::ovirtlago.testlib::ERROR:: * Unhandled exception in <function _host_is_up_4 at 0x3dd25f0> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 338, in _host_is_up_4 host_status = host_service.get().status File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 31529, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 202, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 53, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 199, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 121, in _check_fault body = self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 307, in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 800, in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 840, in _check_content_type raise Error(msg) Error: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML 2017-09-20 09:17:15,683::utils.py::wrapper::480::lago.utils::DEBUG::Looking for a workdir </error> Thanks, *
Dafna
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
That SDK patch is for the *Ruby* SDK, which isn't used in OST, so I think it isn't related. On 09/20/2017 02:11 PM, Martin Perina wrote:
Ondro, could you please take a look?
On Wed, Sep 20, 2017 at 1:07 PM, Dafna Ron <dron@redhat.com> wrote:
Hi,
We have a failure in test post-002_bootstrap.py
from the logs I can see that lago has an unhandled exception from ovirt sdk which makes sense since the patch reported on CQ is a change to the sdk.
*Link to suspected patches: https://gerrit.ovirt.org/#/c/82022/ <https://gerrit.ovirt.org/#/c/82022/>*
* Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734 <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734> Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734/artifact/... <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/> (Relevant) error snippet from the log: <error> 2017-09-20 09:17:15,676::testlib.py::assert_equals_within::227::ovirtlago.testlib::ERROR:: * Unhandled exception in <function _host_is_up_4 at 0x3dd25f0> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 338, in _host_is_up_4 host_status = host_service.get().status File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 31529, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 202, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 53, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 199, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 121, in _check_fault body = self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 307, in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 800, in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 840, in _check_content_type raise Error(msg) Error: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML 2017-09-20 09:17:15,683::utils.py::wrapper::480::lago.utils::DEBUG::Looking for a workdir </error> Thanks, *
Dafna
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
The http returned 503: 192.168.201.1 - - [20/Sep/2017:05:17:10 -0400] "GET /ovirt-engine/api/hosts/06b9cdf3-5bcb-4be8-9e11-8ad5123749f0 HTTP/1.1" 200 5249 192.168.201.1 - - [20/Sep/2017:05:17:13 -0400] "GET /ovirt-engine/api/hosts/06b9cdf3-5bcb-4be8-9e11-8ad5123749f0 HTTP/1.1" 503 299 192.168.201.4 - - [20/Sep/2017:05:17:11 -0400] "POST /ovirt-engine/sso/oauth/token-info HTTP/1.1" 500 527 At that time postgres was killed by oom: Sep 20 05:17:15 lago-basic-suite-master-engine kernel: postgres invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 So from Python SDK perpepective all looks OK. On Wed, Sep 20, 2017 at 2:11 PM, Martin Perina <mperina@redhat.com> wrote:
Ondro, could you please take a look?
On Wed, Sep 20, 2017 at 1:07 PM, Dafna Ron <dron@redhat.com> wrote:
Hi,
We have a failure in test post-002_bootstrap.py
from the logs I can see that lago has an unhandled exception from ovirt sdk which makes sense since the patch reported on CQ is a change to the sdk.
Link to suspected patches: https://gerrit.ovirt.org/#/c/82022/
Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734
Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734/artifact/...
(Relevant) error snippet from the log:
<error>
2017-09-20 09:17:15,676::testlib.py::assert_equals_within::227::ovirtlago.testlib::ERROR:: * Unhandled exception in <function _host_is_up_4 at 0x3dd25f0> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 338, in _host_is_up_4 host_status = host_service.get().status File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 31529, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 202, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 53, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 199, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 121, in _check_fault body = self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 307, in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 800, in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 840, in _check_content_type raise Error(msg) Error: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML 2017-09-20 09:17:15,683::utils.py::wrapper::480::lago.utils::DEBUG::Looking for a workdir
</error>
Thanks,
Dafna
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
This is a multi-part message in MIME format. --------------76F54851C80F784CD6DAB3CE Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit We just had a second failure which CQ is reporting to be from the same patch: failed job: http://jenkins.ovirt.org/job/ovirt-4.1_change-queue-tester/1034/console if you look at the console output you can see this: *12:27:15* ovirt-engine-sdk-ruby_4.1_build-artifacts-fc24-ppc64le (122) failed building This is the job that failed on build: http://jenkins.ovirt.org/job/ovirt-engine-sdk-ruby_4.1_build-artifacts-fc24-... I re-ran the sdk build and it succeeded this time: http://jenkins.ovirt.org/job/ovirt-engine-sdk-ruby_4.1_build-artifacts-fc24-... please try to re-submit the patch? maybe we have a coincident but if it fails again it would be 3ed time... Thanks, Dafna On 09/20/2017 02:20 PM, Ondra Machacek wrote:
The http returned 503:
192.168.201.1 - - [20/Sep/2017:05:17:10 -0400] "GET /ovirt-engine/api/hosts/06b9cdf3-5bcb-4be8-9e11-8ad5123749f0 HTTP/1.1" 200 5249 192.168.201.1 - - [20/Sep/2017:05:17:13 -0400] "GET /ovirt-engine/api/hosts/06b9cdf3-5bcb-4be8-9e11-8ad5123749f0 HTTP/1.1" 503 299 192.168.201.4 - - [20/Sep/2017:05:17:11 -0400] "POST /ovirt-engine/sso/oauth/token-info HTTP/1.1" 500 527
At that time postgres was killed by oom:
Sep 20 05:17:15 lago-basic-suite-master-engine kernel: postgres invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
So from Python SDK perpepective all looks OK.
On Wed, Sep 20, 2017 at 2:11 PM, Martin Perina <mperina@redhat.com> wrote:
Ondro, could you please take a look?
On Wed, Sep 20, 2017 at 1:07 PM, Dafna Ron <dron@redhat.com> wrote:
Hi,
We have a failure in test post-002_bootstrap.py
from the logs I can see that lago has an unhandled exception from ovirt sdk which makes sense since the patch reported on CQ is a change to the sdk.
Link to suspected patches: https://gerrit.ovirt.org/#/c/82022/
Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734
Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734/artifact/...
(Relevant) error snippet from the log:
<error>
2017-09-20 09:17:15,676::testlib.py::assert_equals_within::227::ovirtlago.testlib::ERROR:: * Unhandled exception in <function _host_is_up_4 at 0x3dd25f0> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 338, in _host_is_up_4 host_status = host_service.get().status File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 31529, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 202, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 53, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 199, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 121, in _check_fault body = self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 307, in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 800, in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 840, in _check_content_type raise Error(msg) Error: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML 2017-09-20 09:17:15,683::utils.py::wrapper::480::lago.utils::DEBUG::Looking for a workdir
</error>
Thanks,
Dafna
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
--------------76F54851C80F784CD6DAB3CE Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <div class="moz-cite-prefix">We just had a second failure which CQ is reporting to be from the same patch: <br> <br> failed job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-4.1_change-queue-tester/1034/console">http://jenkins.ovirt.org/job/ovirt-4.1_change-queue-tester/1034/console</a><br> <br> if you look at the console output you can see this: <br> <br> <pre class="console-output" style="box-sizing: border-box; white-space: pre-wrap; word-wrap: break-word; margin: 0px; color: rgb(51, 51, 51); font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;"><span class="timestamp" style="box-sizing: border-box;"><b style="box-sizing: border-box;">12:27:15</b> </span>ovirt-engine-sdk-ruby_4.1_build-artifacts-fc24-ppc64le (122) failed building This is the job that failed on build: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-engine-sdk-ruby_4.1_build-artifacts-fc24-ppc64le/122/">http://jenkins.ovirt.org/job/ovirt-engine-sdk-ruby_4.1_build-artifacts-fc24-ppc64le/122/</a> I re-ran the sdk build and it succeeded this time: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-engine-sdk-ruby_4.1_build-artifacts-fc24-ppc64le/123/">http://jenkins.ovirt.org/job/ovirt-engine-sdk-ruby_4.1_build-artifacts-fc24-ppc64le/123/</a> please try to re-submit the patch? maybe we have a coincident but if it fails again it would be 3ed time... Thanks, Dafna </pre> <br> <br> On 09/20/2017 02:20 PM, Ondra Machacek wrote:<br> </div> <blockquote type="cite" cite="mid:CAJCqMsV9tTemu+5s-BdY9PLu_E3HW8pO8=--RVH0TyFCjopXzg@mail.gmail.com"> <pre wrap="">The http returned 503: 192.168.201.1 - - [20/Sep/2017:05:17:10 -0400] "GET /ovirt-engine/api/hosts/06b9cdf3-5bcb-4be8-9e11-8ad5123749f0 HTTP/1.1" 200 5249 192.168.201.1 - - [20/Sep/2017:05:17:13 -0400] "GET /ovirt-engine/api/hosts/06b9cdf3-5bcb-4be8-9e11-8ad5123749f0 HTTP/1.1" 503 299 192.168.201.4 - - [20/Sep/2017:05:17:11 -0400] "POST /ovirt-engine/sso/oauth/token-info HTTP/1.1" 500 527 At that time postgres was killed by oom: Sep 20 05:17:15 lago-basic-suite-master-engine kernel: postgres invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 So from Python SDK perpepective all looks OK. On Wed, Sep 20, 2017 at 2:11 PM, Martin Perina <a class="moz-txt-link-rfc2396E" href="mailto:mperina@redhat.com"><mperina@redhat.com></a> wrote: </pre> <blockquote type="cite"> <pre wrap="">Ondro, could you please take a look? On Wed, Sep 20, 2017 at 1:07 PM, Dafna Ron <a class="moz-txt-link-rfc2396E" href="mailto:dron@redhat.com"><dron@redhat.com></a> wrote: </pre> <blockquote type="cite"> <pre wrap=""> Hi, We have a failure in test post-002_bootstrap.py from the logs I can see that lago has an unhandled exception from ovirt sdk which makes sense since the patch reported on CQ is a change to the sdk. Link to suspected patches: <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/82022/">https://gerrit.ovirt.org/#/c/82022/</a> Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734</a> Link to all logs: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/</a> (Relevant) error snippet from the log: <error> 2017-09-20 09:17:15,676::testlib.py::assert_equals_within::227::ovirtlago.testlib::ERROR:: * Unhandled exception in <function _host_is_up_4 at 0x3dd25f0> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 338, in _host_is_up_4 host_status = host_service.get().status File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 31529, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 202, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 53, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 199, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 121, in _check_fault body = self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 307, in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 800, in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 840, in _check_content_type raise Error(msg) Error: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML 2017-09-20 09:17:15,683::utils.py::wrapper::480::lago.utils::DEBUG::Looking for a workdir </error> Thanks, Dafna _______________________________________________ Devel mailing list <a class="moz-txt-link-abbreviated" href="mailto:Devel@ovirt.org">Devel@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/devel">http://lists.ovirt.org/mailman/listinfo/devel</a> </pre> </blockquote> <pre wrap=""> </pre> </blockquote> </blockquote> <p><br> </p> </body> </html> --------------76F54851C80F784CD6DAB3CE--
I am aware of that build failure. The tests of the *Ruby* SDK hang apparently random in ppc64le environments. I am trying to find out why, but not making much progress, to be honest. Meanwhile I wonder why this is a failure to run OST tests, as the Ruby SDK isn't used at all in those tests, as far as I know. On 09/20/2017 04:09 PM, Dafna Ron wrote:
We just had a second failure which CQ is reporting to be from the same patch:
failed job: http://jenkins.ovirt.org/job/ovirt-4.1_change-queue-tester/1034/console
if you look at the console output you can see this:
*12:27:15* ovirt-engine-sdk-ruby_4.1_build-artifacts-fc24-ppc64le (122) failed building
This is the job that failed on build: http://jenkins.ovirt.org/job/ovirt-engine-sdk-ruby_4.1_build-artifacts-fc24-...
I re-ran the sdk build and it succeeded this time: http://jenkins.ovirt.org/job/ovirt-engine-sdk-ruby_4.1_build-artifacts-fc24-...
please try to re-submit the patch? maybe we have a coincident but if it fails again it would be 3ed time...
Thanks, Dafna
On 09/20/2017 02:20 PM, Ondra Machacek wrote:
The http returned 503:
192.168.201.1 - - [20/Sep/2017:05:17:10 -0400] "GET /ovirt-engine/api/hosts/06b9cdf3-5bcb-4be8-9e11-8ad5123749f0 HTTP/1.1" 200 5249 192.168.201.1 - - [20/Sep/2017:05:17:13 -0400] "GET /ovirt-engine/api/hosts/06b9cdf3-5bcb-4be8-9e11-8ad5123749f0 HTTP/1.1" 503 299 192.168.201.4 - - [20/Sep/2017:05:17:11 -0400] "POST /ovirt-engine/sso/oauth/token-info HTTP/1.1" 500 527
At that time postgres was killed by oom:
Sep 20 05:17:15 lago-basic-suite-master-engine kernel: postgres invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
So from Python SDK perpepective all looks OK.
On Wed, Sep 20, 2017 at 2:11 PM, Martin Perina <mperina@redhat.com> wrote:
Ondro, could you please take a look?
On Wed, Sep 20, 2017 at 1:07 PM, Dafna Ron <dron@redhat.com> wrote:
Hi,
We have a failure in test post-002_bootstrap.py
from the logs I can see that lago has an unhandled exception from ovirt sdk which makes sense since the patch reported on CQ is a change to the sdk.
Link to suspected patches: https://gerrit.ovirt.org/#/c/82022/
Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734
Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734/artifact/...
(Relevant) error snippet from the log:
<error>
2017-09-20 09:17:15,676::testlib.py::assert_equals_within::227::ovirtlago.testlib::ERROR:: * Unhandled exception in <function _host_is_up_4 at 0x3dd25f0> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 338, in _host_is_up_4 host_status = host_service.get().status File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 31529, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 202, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 53, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 199, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 121, in _check_fault body = self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 307, in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 800, in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 840, in _check_content_type raise Error(msg) Error: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML 2017-09-20 09:17:15,683::utils.py::wrapper::480::lago.utils::DEBUG::Looking for a workdir
</error>
Thanks,
Dafna
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
בתאריך 20 בספט׳ 2017 05:44 PM, "Juan Hernández" <jhernand@redhat.com> כתב: I am aware of that build failure. The tests of the *Ruby* SDK hang apparently random in ppc64le environments. I am trying to find out why, but not making much progress, to be honest. Meanwhile I wonder why this is a failure to run OST tests, as the Ruby SDK isn't used at all in those tests, as far as I know. This is not actually failing OST, its making CQ not run OST at all, since it verifies builds pass before trying. Barak Korren bkorren@redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/
participants (5)
-
Barak Korren -
Dafna Ron -
Juan Hernández -
Martin Perina -
Ondra Machacek