From dron at redhat.com Wed Sep 20 11:07:28 2017 Content-Type: multipart/mixed; boundary="===============1942547012476988326==" MIME-Version: 1.0 From: Dafna Ron To: devel at ovirt.org Subject: [ovirt-devel] Subject: [ OST Failure Report ] [ master ] [20-09-2017 ] [ post-002_bootstrap.py ] Date: Wed, 20 Sep 2017 12:07:23 +0100 Message-ID: <99870607-0361-fab3-d24e-3423b74e12c2@redhat.com> --===============1942547012476988326== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------04175AE2D1C2AC0FFF61E4CE Content-Type: text/plain; charset=3Dutf-8 Content-Transfer-Encoding: 7bit Hi, We have a failure in test post-002_bootstrap.py from the logs I can see that lago has an unhandled exception from ovirt sdk which makes sense since the patch reported on CQ is a change to the sdk. ** *Link to suspected patches: https://gerrit.ovirt.org/#/c/82022/* * Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734 Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734/artifact= /exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post= -002_bootstrap.py/ (Relevant) error snippet from the log: 2017-09-20 09:17:15,676::testlib.py::assert_equals_within::227::ovirtlago.testlib::ERR= OR:: = * Unhandled exception in Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within res =3D func() File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-test= s/basic-suite-master/test-scenarios/002_bootstrap.py", line 338, in _host_is_up_4 host_status =3D host_service.get().status File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 31529, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 202, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 53, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 199, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 121, in _check_fault body =3D self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 307, in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 800, in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 840, in _check_content_type raise Error(msg) Error: The response content type 'text/html; charset=3Diso-8859-1' isn't the expected XML 2017-09-20 09:17:15,683::utils.py::wrapper::480::lago.utils::DEBUG::Looking for a workdir Thanks, * Dafna* * --------------04175AE2D1C2AC0FFF61E4CE Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: 8bit

Hi,

We have a failure in test post-002_bootstrap.py

from the logs I can see that lago has an unhandled exception from ovirt sdk which makes sense since the patch reported on CQ is a change to the sdk.

Link to suspecte= d patches: https://gerrit.ovirt.org/#/c/82022/


Link to Job: http://jenkins.ovirt.org/job/ovirt-master_= change-queue-tester/2734


Link to all logs= : http://jen= kins.ovirt.org/job/ovirt-master_change-queue-tester/2734/artifact/exported-= artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_boots= trap.py/


(Relevant) error= snippet from the log:

<error>


2017-09-20 09:17:15,676::testlib.py::assert_equals_within::227::ovirtlago.testlib::ERR= OR::=C2=A0=C2=A0=C2=A0 * Unhandled exception in <function _host_is_up_4 at 0x3dd25f0>
Traceback (most recent call last):
=C2=A0 File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_equals_within
=C2=A0=C2=A0=C2=A0 res =3D func()
=C2=A0 File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-test= s/basic-suite-master/test-scenarios/002_bootstrap.py", line 338, in _host_is_up_4
=C2=A0=C2=A0=C2=A0 host_status =3D host_service.get().status
=C2=A0 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 31529, in get
=C2=A0=C2=A0=C2=A0 return self._internal_get(headers, query, wait)<= br> =C2=A0 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 202, in _internal_get
=C2=A0=C2=A0=C2=A0 return future.wait() if wait else future
=C2=A0 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 53, in wait
=C2=A0=C2=A0=C2=A0 return self._code(response)
=C2=A0 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 199, in callback
=C2=A0=C2=A0=C2=A0 self._check_fault(response)
=C2=A0 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 121, in _check_fault
=C2=A0=C2=A0=C2=A0 body =3D self._internal_read_body(response)
=C2=A0 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 307, in _internal_read_body
=C2=A0=C2=A0=C2=A0 self._connection.check_xml_content_type(response= )
=C2=A0 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 800, in check_xml_content_type
=C2=A0=C2=A0=C2=A0 response.headers
=C2=A0 File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 840, in _check_content_type
=C2=A0=C2=A0=C2=A0 raise Error(msg)
Error: The response content type 'text/html; charset=3Diso-8859-1' isn't the expected XML
2017-09-20 09:17:15,683::utils.py::wrapper::480::lago.utils::DEBUG::Looking for a workdir


</error>


Thanks,

Dafna

--------------04175AE2D1C2AC0FFF61E4CE-- --===============1942547012476988326== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS0wNDE3NUFFMkQxQzJBQzBGRkY2MUU0Q0UKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXV0Zi04CkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDdiaXQKCkhpLAoKV2UgaGF2ZSBh IGZhaWx1cmUgaW4gdGVzdCBwb3N0LTAwMl9ib290c3RyYXAucHkKCmZyb20gdGhlIGxvZ3MgSSBj YW4gc2VlIHRoYXQgbGFnbyBoYXMgYW4gdW5oYW5kbGVkIGV4Y2VwdGlvbiBmcm9tIG92aXJ0CnNk ayB3aGljaCBtYWtlcyBzZW5zZSBzaW5jZSB0aGUgcGF0Y2ggcmVwb3J0ZWQgb24gQ1EgaXMgYSBj aGFuZ2UgdG8gdGhlCnNkay4KCioqCgoqTGluayB0byBzdXNwZWN0ZWQgcGF0Y2hlczogaHR0cHM6 Ly9nZXJyaXQub3ZpcnQub3JnLyMvYy84MjAyMi8qCgoqCgpMaW5rIHRvIEpvYjoKaHR0cDovL2pl bmtpbnMub3ZpcnQub3JnL2pvYi9vdmlydC1tYXN0ZXJfY2hhbmdlLXF1ZXVlLXRlc3Rlci8yNzM0 CgoKTGluayB0byBhbGwgbG9nczoKaHR0cDovL2plbmtpbnMub3ZpcnQub3JnL2pvYi9vdmlydC1t YXN0ZXJfY2hhbmdlLXF1ZXVlLXRlc3Rlci8yNzM0L2FydGlmYWN0L2V4cG9ydGVkLWFydGlmYWN0 cy9iYXNpYy1zdWl0LW1hc3Rlci1lbDcvdGVzdF9sb2dzL2Jhc2ljLXN1aXRlLW1hc3Rlci9wb3N0 LTAwMl9ib290c3RyYXAucHkvCgoKKFJlbGV2YW50KSBlcnJvciBzbmlwcGV0IGZyb20gdGhlIGxv ZzoKCjxlcnJvcj4KCgoyMDE3LTA5LTIwCjA5OjE3OjE1LDY3Njo6dGVzdGxpYi5weTo6YXNzZXJ0 X2VxdWFsc193aXRoaW46OjIyNzo6b3ZpcnRsYWdvLnRlc3RsaWI6OkVSUk9SOjogICAKKiBVbmhh bmRsZWQgZXhjZXB0aW9uIGluIDxmdW5jdGlvbiBfaG9zdF9pc191cF80IGF0IDB4M2RkMjVmMD4K VHJhY2ViYWNrIChtb3N0IHJlY2VudCBjYWxsIGxhc3QpOgogIEZpbGUgIi91c3IvbGliL3B5dGhv bjIuNy9zaXRlLXBhY2thZ2VzL292aXJ0bGFnby90ZXN0bGliLnB5IiwgbGluZQoyMTksIGluIGFz c2VydF9lcXVhbHNfd2l0aGluCiAgICByZXMgPSBmdW5jKCkKICBGaWxlCiIvaG9tZS9qZW5raW5z L3dvcmtzcGFjZS9vdmlydC1tYXN0ZXJfY2hhbmdlLXF1ZXVlLXRlc3Rlci9vdmlydC1zeXN0ZW0t dGVzdHMvYmFzaWMtc3VpdGUtbWFzdGVyL3Rlc3Qtc2NlbmFyaW9zLzAwMl9ib290c3RyYXAucHki LApsaW5lIDMzOCwgaW4gX2hvc3RfaXNfdXBfNAogICAgaG9zdF9zdGF0dXMgPSBob3N0X3NlcnZp Y2UuZ2V0KCkuc3RhdHVzCiAgRmlsZSAiL3Vzci9saWI2NC9weXRob24yLjcvc2l0ZS1wYWNrYWdl cy9vdmlydHNkazQvc2VydmljZXMucHkiLCBsaW5lCjMxNTI5LCBpbiBnZXQKICAgIHJldHVybiBz ZWxmLl9pbnRlcm5hbF9nZXQoaGVhZGVycywgcXVlcnksIHdhaXQpCiAgRmlsZSAiL3Vzci9saWI2 NC9weXRob24yLjcvc2l0ZS1wYWNrYWdlcy9vdmlydHNkazQvc2VydmljZS5weSIsIGxpbmUKMjAy LCBpbiBfaW50ZXJuYWxfZ2V0CiAgICByZXR1cm4gZnV0dXJlLndhaXQoKSBpZiB3YWl0IGVsc2Ug ZnV0dXJlCiAgRmlsZSAiL3Vzci9saWI2NC9weXRob24yLjcvc2l0ZS1wYWNrYWdlcy9vdmlydHNk azQvc2VydmljZS5weSIsIGxpbmUKNTMsIGluIHdhaXQKICAgIHJldHVybiBzZWxmLl9jb2RlKHJl c3BvbnNlKQogIEZpbGUgIi91c3IvbGliNjQvcHl0aG9uMi43L3NpdGUtcGFja2FnZXMvb3ZpcnRz ZGs0L3NlcnZpY2UucHkiLCBsaW5lCjE5OSwgaW4gY2FsbGJhY2sKICAgIHNlbGYuX2NoZWNrX2Zh dWx0KHJlc3BvbnNlKQogIEZpbGUgIi91c3IvbGliNjQvcHl0aG9uMi43L3NpdGUtcGFja2FnZXMv b3ZpcnRzZGs0L3NlcnZpY2UucHkiLCBsaW5lCjEyMSwgaW4gX2NoZWNrX2ZhdWx0CiAgICBib2R5 ID0gc2VsZi5faW50ZXJuYWxfcmVhZF9ib2R5KHJlc3BvbnNlKQogIEZpbGUgIi91c3IvbGliNjQv cHl0aG9uMi43L3NpdGUtcGFja2FnZXMvb3ZpcnRzZGs0L3NlcnZpY2UucHkiLCBsaW5lCjMwNywg aW4gX2ludGVybmFsX3JlYWRfYm9keQogICAgc2VsZi5fY29ubmVjdGlvbi5jaGVja194bWxfY29u dGVudF90eXBlKHJlc3BvbnNlKQogIEZpbGUgIi91c3IvbGliNjQvcHl0aG9uMi43L3NpdGUtcGFj a2FnZXMvb3ZpcnRzZGs0L19faW5pdF9fLnB5IiwgbGluZQo4MDAsIGluIGNoZWNrX3htbF9jb250 ZW50X3R5cGUKICAgIHJlc3BvbnNlLmhlYWRlcnMKICBGaWxlICIvdXNyL2xpYjY0L3B5dGhvbjIu Ny9zaXRlLXBhY2thZ2VzL292aXJ0c2RrNC9fX2luaXRfXy5weSIsIGxpbmUKODQwLCBpbiBfY2hl Y2tfY29udGVudF90eXBlCiAgICByYWlzZSBFcnJvcihtc2cpCkVycm9yOiBUaGUgcmVzcG9uc2Ug Y29udGVudCB0eXBlICd0ZXh0L2h0bWw7IGNoYXJzZXQ9aXNvLTg4NTktMScgaXNuJ3QKdGhlIGV4 cGVjdGVkIFhNTAoyMDE3LTA5LTIwCjA5OjE3OjE1LDY4Mzo6dXRpbHMucHk6OndyYXBwZXI6OjQ4 MDo6bGFnby51dGlsczo6REVCVUc6Okxvb2tpbmcgZm9yIGEKd29ya2RpcgoKCjwvZXJyb3I+CgoK VGhhbmtzLAoqCgpEYWZuYSoKKgoKCi0tLS0tLS0tLS0tLS0tMDQxNzVBRTJEMUMyQUMwRkZGNjFF NENFCkNvbnRlbnQtVHlwZTogdGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04CkNvbnRlbnQtVHJhbnNm ZXItRW5jb2Rpbmc6IDhiaXQKCjxodG1sPgogIDxoZWFkPgoKICAgIDxtZXRhIGh0dHAtZXF1aXY9 ImNvbnRlbnQtdHlwZSIgY29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4KICA8L2hl YWQ+CiAgPGJvZHkgdGV4dD0iIzAwMDAwMCIgYmdjb2xvcj0iI0ZGRkZGRiI+CiAgICA8cD5IaSwg PGJyPgogICAgPC9wPgogICAgPHA+V2UgaGF2ZSBhIGZhaWx1cmUgaW4gdGVzdCBwb3N0LTAwMl9i b290c3RyYXAucHk8L3A+CiAgICA8cD5mcm9tIHRoZSBsb2dzIEkgY2FuIHNlZSB0aGF0IGxhZ28g aGFzIGFuIHVuaGFuZGxlZCBleGNlcHRpb24gZnJvbQogICAgICBvdmlydCBzZGsgd2hpY2ggbWFr ZXMgc2Vuc2Ugc2luY2UgdGhlIHBhdGNoIHJlcG9ydGVkIG9uIENRIGlzIGEKICAgICAgY2hhbmdl IHRvIHRoZSBzZGsuIDxicj4KICAgIDwvcD4KICAgIDxwPjxiIHN0eWxlPSJmb250LXdlaWdodDpu b3JtYWw7IgogICAgICAgIGlkPSJkb2NzLWludGVybmFsLWd1aWQtNTg1OWI3YTEtOWVmNi05MWY0 LTM4ZGUtYjczMTNmODU1Y2MyIj4KICAgICAgICA8cCBkaXI9Imx0ciIKICAgICAgICAgIHN0eWxl PSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0b206MHB0OyI+PHNw YW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7 YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7Zm9udC1zdHlsZTpu b3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2ZXJ0aWNhbC1h bGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcDsiPkxpbmsgdG8gc3VzcGVjdGVkIHBh dGNoZXM6IDxhIGNsYXNzPSJtb3otdHh0LWxpbmstZnJlZXRleHQiIGhyZWY9Imh0dHBzOi8vZ2Vy cml0Lm92aXJ0Lm9yZy8jL2MvODIwMjIvIj5odHRwczovL2dlcnJpdC5vdmlydC5vcmcvIy9jLzgy MDIyLzwvYT48L3NwYW4+PC9wPgogICAgICAgIDxicj4KICAgICAgICA8cCBkaXI9Imx0ciIKICAg ICAgICAgIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0 b206MHB0OyI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2Nv bG9yOiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7 Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9u ZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcDsiPkxpbmsgdG8g Sm9iOiA8YSBjbGFzcz0ibW96LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRwOi8vamVua2lu cy5vdmlydC5vcmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVyLzI3MzQiPmh0 dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2Ivb3ZpcnQtbWFzdGVyX2NoYW5nZS1xdWV1ZS10ZXN0 ZXIvMjczNDwvYT48L3NwYW4+PC9wPgogICAgICAgIDxicj4KICAgICAgICA8cCBkaXI9Imx0ciIK ICAgICAgICAgIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1i b3R0b206MHB0OyI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFs O2NvbG9yOiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0 MDA7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246 bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcDsiPkxpbmsg dG8gYWxsIGxvZ3M6IDxhIGNsYXNzPSJtb3otdHh0LWxpbmstZnJlZXRleHQiIGhyZWY9Imh0dHA6 Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2Ivb3ZpcnQtbWFzdGVyX2NoYW5nZS1xdWV1ZS10ZXN0ZXIv MjczNC9hcnRpZmFjdC9leHBvcnRlZC1hcnRpZmFjdHMvYmFzaWMtc3VpdC1tYXN0ZXItZWw3L3Rl c3RfbG9ncy9iYXNpYy1zdWl0ZS1tYXN0ZXIvcG9zdC0wMDJfYm9vdHN0cmFwLnB5LyI+aHR0cDov L2plbmtpbnMub3ZpcnQub3JnL2pvYi9vdmlydC1tYXN0ZXJfY2hhbmdlLXF1ZXVlLXRlc3Rlci8y NzM0L2FydGlmYWN0L2V4cG9ydGVkLWFydGlmYWN0cy9iYXNpYy1zdWl0LW1hc3Rlci1lbDcvdGVz dF9sb2dzL2Jhc2ljLXN1aXRlLW1hc3Rlci9wb3N0LTAwMl9ib290c3RyYXAucHkvPC9hPjwvc3Bh bj48L3A+CiAgICAgICAgPGJyPgogICAgICAgIDxwIGRpcj0ibHRyIgogICAgICAgICAgc3R5bGU9 ImxpbmUtaGVpZ2h0OjEuMzg7bWFyZ2luLXRvcDowcHQ7bWFyZ2luLWJvdHRvbTowcHQ7Ij48c3Bh biBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7Y29sb3I6IzAwMDAwMDti YWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0OjQwMDtmb250LXN0eWxlOm5v cm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3JhdGlvbjpub25lO3ZlcnRpY2FsLWFs aWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFwOyI+KFJlbGV2YW50KSBlcnJvciBzbmlw cGV0IGZyb20gdGhlIGxvZzogPC9zcGFuPjwvcD4KICAgICAgICA8cCBkaXI9Imx0ciIKICAgICAg ICAgIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdpbi1ib3R0b206 MHB0OyI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9y OiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7Zm9u dC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2 ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcDsiPiZsdDtlcnJvciZn dDs8L3NwYW4+PC9wPgogICAgICAgIDxicj4KICAgICAgICAyMDE3LTA5LTIwCjA5OjE3OjE1LDY3 Njo6dGVzdGxpYi5weTo6YXNzZXJ0X2VxdWFsc193aXRoaW46OjIyNzo6b3ZpcnRsYWdvLnRlc3Rs aWI6OkVSUk9SOjrCoMKgwqAKICAgICAgICAqIFVuaGFuZGxlZCBleGNlcHRpb24gaW4gJmx0O2Z1 bmN0aW9uIF9ob3N0X2lzX3VwXzQgYXQKICAgICAgICAweDNkZDI1ZjAmZ3Q7PGJyPgogICAgICAg IFRyYWNlYmFjayAobW9zdCByZWNlbnQgY2FsbCBsYXN0KTo8YnI+CiAgICAgICAgwqAgRmlsZSAi L3Vzci9saWIvcHl0aG9uMi43L3NpdGUtcGFja2FnZXMvb3ZpcnRsYWdvL3Rlc3RsaWIucHkiLAog ICAgICAgIGxpbmUgMjE5LCBpbiBhc3NlcnRfZXF1YWxzX3dpdGhpbjxicj4KICAgICAgICDCoMKg wqAgcmVzID0gZnVuYygpPGJyPgogICAgICAgIMKgIEZpbGUKIi9ob21lL2plbmtpbnMvd29ya3Nw YWNlL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVyL292aXJ0LXN5c3RlbS10ZXN0cy9i YXNpYy1zdWl0ZS1tYXN0ZXIvdGVzdC1zY2VuYXJpb3MvMDAyX2Jvb3RzdHJhcC5weSIsCiAgICAg ICAgbGluZSAzMzgsIGluIF9ob3N0X2lzX3VwXzQ8YnI+CiAgICAgICAgwqDCoMKgIGhvc3Rfc3Rh dHVzID0gaG9zdF9zZXJ2aWNlLmdldCgpLnN0YXR1czxicj4KICAgICAgICDCoCBGaWxlCiAgICAg ICAgIi91c3IvbGliNjQvcHl0aG9uMi43L3NpdGUtcGFja2FnZXMvb3ZpcnRzZGs0L3NlcnZpY2Vz LnB5IiwgbGluZQogICAgICAgIDMxNTI5LCBpbiBnZXQ8YnI+CiAgICAgICAgwqDCoMKgIHJldHVy biBzZWxmLl9pbnRlcm5hbF9nZXQoaGVhZGVycywgcXVlcnksIHdhaXQpPGJyPgogICAgICAgIMKg IEZpbGUKICAgICAgICAiL3Vzci9saWI2NC9weXRob24yLjcvc2l0ZS1wYWNrYWdlcy9vdmlydHNk azQvc2VydmljZS5weSIsIGxpbmUKICAgICAgICAyMDIsIGluIF9pbnRlcm5hbF9nZXQ8YnI+CiAg ICAgICAgwqDCoMKgIHJldHVybiBmdXR1cmUud2FpdCgpIGlmIHdhaXQgZWxzZSBmdXR1cmU8YnI+ CiAgICAgICAgwqAgRmlsZQogICAgICAgICIvdXNyL2xpYjY0L3B5dGhvbjIuNy9zaXRlLXBhY2th Z2VzL292aXJ0c2RrNC9zZXJ2aWNlLnB5IiwgbGluZQogICAgICAgIDUzLCBpbiB3YWl0PGJyPgog ICAgICAgIMKgwqDCoCByZXR1cm4gc2VsZi5fY29kZShyZXNwb25zZSk8YnI+CiAgICAgICAgwqAg RmlsZQogICAgICAgICIvdXNyL2xpYjY0L3B5dGhvbjIuNy9zaXRlLXBhY2thZ2VzL292aXJ0c2Rr NC9zZXJ2aWNlLnB5IiwgbGluZQogICAgICAgIDE5OSwgaW4gY2FsbGJhY2s8YnI+CiAgICAgICAg wqDCoMKgIHNlbGYuX2NoZWNrX2ZhdWx0KHJlc3BvbnNlKTxicj4KICAgICAgICDCoCBGaWxlCiAg ICAgICAgIi91c3IvbGliNjQvcHl0aG9uMi43L3NpdGUtcGFja2FnZXMvb3ZpcnRzZGs0L3NlcnZp Y2UucHkiLCBsaW5lCiAgICAgICAgMTIxLCBpbiBfY2hlY2tfZmF1bHQ8YnI+CiAgICAgICAgwqDC oMKgIGJvZHkgPSBzZWxmLl9pbnRlcm5hbF9yZWFkX2JvZHkocmVzcG9uc2UpPGJyPgogICAgICAg IMKgIEZpbGUKICAgICAgICAiL3Vzci9saWI2NC9weXRob24yLjcvc2l0ZS1wYWNrYWdlcy9vdmly dHNkazQvc2VydmljZS5weSIsIGxpbmUKICAgICAgICAzMDcsIGluIF9pbnRlcm5hbF9yZWFkX2Jv ZHk8YnI+CiAgICAgICAgwqDCoMKgIHNlbGYuX2Nvbm5lY3Rpb24uY2hlY2tfeG1sX2NvbnRlbnRf dHlwZShyZXNwb25zZSk8YnI+CiAgICAgICAgwqAgRmlsZQogICAgICAgICIvdXNyL2xpYjY0L3B5 dGhvbjIuNy9zaXRlLXBhY2thZ2VzL292aXJ0c2RrNC9fX2luaXRfXy5weSIsIGxpbmUKICAgICAg ICA4MDAsIGluIGNoZWNrX3htbF9jb250ZW50X3R5cGU8YnI+CiAgICAgICAgwqDCoMKgIHJlc3Bv bnNlLmhlYWRlcnM8YnI+CiAgICAgICAgwqAgRmlsZQogICAgICAgICIvdXNyL2xpYjY0L3B5dGhv bjIuNy9zaXRlLXBhY2thZ2VzL292aXJ0c2RrNC9fX2luaXRfXy5weSIsIGxpbmUKICAgICAgICA4 NDAsIGluIF9jaGVja19jb250ZW50X3R5cGU8YnI+CiAgICAgICAgwqDCoMKgIHJhaXNlIEVycm9y KG1zZyk8YnI+CiAgICAgICAgRXJyb3I6IFRoZSByZXNwb25zZSBjb250ZW50IHR5cGUgJ3RleHQv aHRtbDsgY2hhcnNldD1pc28tODg1OS0xJwogICAgICAgIGlzbid0IHRoZSBleHBlY3RlZCBYTUw8 YnI+CiAgICAgICAgMjAxNy0wOS0yMAogICAgICAgIDA5OjE3OjE1LDY4Mzo6dXRpbHMucHk6Ondy YXBwZXI6OjQ4MDo6bGFnby51dGlsczo6REVCVUc6Okxvb2tpbmcKICAgICAgICBmb3IgYSB3b3Jr ZGlyPGJyPgogICAgICAgIDxicj4KICAgICAgICA8YnI+CiAgICAgICAgPHAgZGlyPSJsdHIiCiAg ICAgICAgICBzdHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90 dG9tOjBwdDsiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTFwdDtmb250LWZhbWlseTpBcmlhbDtj b2xvcjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6dHJhbnNwYXJlbnQ7Zm9udC13ZWlnaHQ6NDAw O2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7dGV4dC1kZWNvcmF0aW9uOm5v bmU7dmVydGljYWwtYWxpZ246YmFzZWxpbmU7d2hpdGUtc3BhY2U6cHJlLXdyYXA7Ij4mbHQ7L2Vy cm9yJmd0Ozwvc3Bhbj48L3A+CiAgICAgICAgPGJyPgogICAgICAgIFRoYW5rcywgPGJyPgogICAg ICA8L2I+PC9wPgogICAgPHA+PHNwYW4gc3R5bGU9ImZvbnQtd2VpZ2h0Om5vcm1hbDsiPkRhZm5h PC9zcGFuPjxiCiAgICAgICAgc3R5bGU9ImZvbnQtd2VpZ2h0Om5vcm1hbDsiCiAgICAgICAgaWQ9 ImRvY3MtaW50ZXJuYWwtZ3VpZC01ODU5YjdhMS05ZWY2LTkxZjQtMzhkZS1iNzMxM2Y4NTVjYzIi Pjxicj4KICAgICAgPC9iPjwvcD4KICA8L2JvZHk+CjwvaHRtbD4KCi0tLS0tLS0tLS0tLS0tMDQx NzVBRTJEMUMyQUMwRkZGNjFFNENFLS0K --===============1942547012476988326==-- From mperina at redhat.com Wed Sep 20 12:11:04 2017 Content-Type: multipart/mixed; boundary="===============8654375161725098321==" MIME-Version: 1.0 From: Martin Perina To: devel at ovirt.org Subject: Re: [ovirt-devel] Subject: [ OST Failure Report ] [ master ] [20-09-2017 ] [ post-002_bootstrap.py ] Date: Wed, 20 Sep 2017 14:11:02 +0200 Message-ID: In-Reply-To: 99870607-0361-fab3-d24e-3423b74e12c2@redhat.com --===============8654375161725098321== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Ondro, could you please take a look? On Wed, Sep 20, 2017 at 1:07 PM, Dafna Ron wrote: > Hi, > > We have a failure in test post-002_bootstrap.py > > from the logs I can see that lago has an unhandled exception from ovirt > sdk which makes sense since the patch reported on CQ is a change to the > sdk. > > *Link to suspected patches: https://gerrit.ovirt.org/#/c/82022/ > * > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * Link to Job: > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734 > Link > to all logs: > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734/artifa= ct/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/po= st-002_bootstrap.py/ > > (Relevant) error snippet from the log: 2017-09-20 > 09:17:15,676::testlib.py::assert_equals_within::227::ovirtlago.testlib::E= RROR:: > * Unhandled exception in Traceback > (most recent call last): File > "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in > assert_equals_within res =3D func() File > "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-te= sts/basic-suite-master/test-scenarios/002_bootstrap.py", > line 338, in _host_is_up_4 host_status =3D host_service.get().status > File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line > 31529, in get return self._internal_get(headers, query, wait) File > "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 202, in > _internal_get return future.wait() if wait else future File > "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 53, in wa= it > return self._code(response) File > "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 199, in > callback self._check_fault(response) File > "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 121, in > _check_fault body =3D self._internal_read_body(response) File > "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 307, in > _internal_read_body self._connection.check_xml_content_type(response) > File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 800, > in check_xml_content_type response.headers File > "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 840, in > _check_content_type raise Error(msg) Error: The response content type > 'text/html; charset=3Diso-8859-1' isn't the expected XML 2017-09-20 > 09:17:15,683::utils.py::wrapper::480::lago.utils::DEBUG::Looking for a > workdir Thanks, * > > Dafna > > _______________________________________________ > Devel mailing list > Devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel > --===============8654375161725098321== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGRpdiBkaXI9Imx0ciI+PGRpdiBjbGFzcz0iZ21haWxfZGVmYXVsdCIgc3R5bGU9ImZvbnQtZmFt aWx5OmFyaWFsLGhlbHZldGljYSxzYW5zLXNlcmlmIj5PbmRybywgY291bGQgeW91IHBsZWFzZSB0 YWtlIGEgbG9vaz88YnI+PC9kaXY+PC9kaXY+PGRpdiBjbGFzcz0iZ21haWxfZXh0cmEiPjxicj48 ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+T24gV2VkLCBTZXAgMjAsIDIwMTcgYXQgMTowNyBQTSwg RGFmbmEgUm9uIDxzcGFuIGRpcj0ibHRyIj4mbHQ7PGEgaHJlZj0ibWFpbHRvOmRyb25AcmVkaGF0 LmNvbSIgdGFyZ2V0PSJfYmxhbmsiPmRyb25AcmVkaGF0LmNvbTwvYT4mZ3Q7PC9zcGFuPiB3cm90 ZTo8YnI+PGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjAgMCAw IC44ZXg7Ym9yZGVyLWxlZnQ6MXB4ICNjY2Mgc29saWQ7cGFkZGluZy1sZWZ0OjFleCI+CiAgCgog ICAgCiAgCiAgPGRpdiB0ZXh0PSIjMDAwMDAwIiBiZ2NvbG9yPSIjRkZGRkZGIj4KICAgIDxwPkhp LCA8YnI+CiAgICA8L3A+CiAgICA8cD5XZSBoYXZlIGEgZmFpbHVyZSBpbiB0ZXN0IHBvc3QtMDAy X2Jvb3RzdHJhcC5weTwvcD4KICAgIDxwPmZyb20gdGhlIGxvZ3MgSSBjYW4gc2VlIHRoYXQgbGFn byBoYXMgYW4gdW5oYW5kbGVkIGV4Y2VwdGlvbiBmcm9tCiAgICAgIG92aXJ0IHNkayB3aGljaCBt YWtlcyBzZW5zZSBzaW5jZSB0aGUgcGF0Y2ggcmVwb3J0ZWQgb24gQ1EgaXMgYQogICAgICBjaGFu Z2UgdG8gdGhlIHNkay4gPGJyPgogICAgPC9wPgogICAgPHA+PGIgc3R5bGU9ImZvbnQtd2VpZ2h0 Om5vcm1hbCIgaWQ9Im1fLTQ3NzUyNDUyMzAyNDE5MDY2NjZkb2NzLWludGVybmFsLWd1aWQtNTg1 OWI3YTEtOWVmNi05MWY0LTM4ZGUtYjczMTNmODU1Y2MyIj4KICAgICAgICA8L2I+PC9wPjxwIGRp cj0ibHRyIiBzdHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90 dG9tOjBwdCI+PGIgc3R5bGU9ImZvbnQtd2VpZ2h0Om5vcm1hbCIgaWQ9Im1fLTQ3NzUyNDUyMzAy NDE5MDY2NjZkb2NzLWludGVybmFsLWd1aWQtNTg1OWI3YTEtOWVmNi05MWY0LTM4ZGUtYjczMTNm ODU1Y2MyIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJpYWw7Y29s b3I6IzAwMDAwMDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0OjQwMDtm b250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3JhdGlvbjpub25l O3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFwIj5MaW5rIHRvIHN1 c3BlY3RlZCBwYXRjaGVzOiA8YSBjbGFzcz0ibV8tNDc3NTI0NTIzMDI0MTkwNjY2Nm1vei10eHQt bGluay1mcmVldGV4dCIgaHJlZj0iaHR0cHM6Ly9nZXJyaXQub3ZpcnQub3JnLyMvYy84MjAyMi8i IHRhcmdldD0iX2JsYW5rIj5odHRwczovL2dlcnJpdC5vdmlydC5vcmcvIy9jLzx3YnI+ODIwMjIv PC9hPjwvc3Bhbj48L2I+PC9wPjxiIHN0eWxlPSJmb250LXdlaWdodDpub3JtYWwiIGlkPSJtXy00 Nzc1MjQ1MjMwMjQxOTA2NjY2ZG9jcy1pbnRlcm5hbC1ndWlkLTU4NTliN2ExLTllZjYtOTFmNC0z OGRlLWI3MzEzZjg1NWNjMiI+CiAgICAgICAgPGJyPgogICAgICAgIDxwIGRpcj0ibHRyIiBzdHls ZT0ibGluZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBwdCI+PHNw YW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7 YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7Zm9udC1zdHlsZTpu b3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2ZXJ0aWNhbC1h bGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+TGluayB0byBKb2I6IDxhIGNsYXNz PSJtXy00Nzc1MjQ1MjMwMjQxOTA2NjY2bW96LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRw Oi8vamVua2lucy5vdmlydC5vcmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVy LzI3MzQiIHRhcmdldD0iX2JsYW5rIj5odHRwOi8vamVua2lucy5vdmlydC5vcmcvam9iLzx3YnI+ b3ZpcnQtbWFzdGVyX2NoYW5nZS1xdWV1ZS08d2JyPnRlc3Rlci8yNzM0PC9hPjwvc3Bhbj48L3A+ CiAgICAgICAgPGJyPgogICAgICAgIDxwIGRpcj0ibHRyIiBzdHlsZT0ibGluZS1oZWlnaHQ6MS4z ODttYXJnaW4tdG9wOjBwdDttYXJnaW4tYm90dG9tOjBwdCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6 ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFsO2NvbG9yOiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0 cmFuc3BhcmVudDtmb250LXdlaWdodDo0MDA7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50 Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0 ZS1zcGFjZTpwcmUtd3JhcCI+TGluayB0byBhbGwgbG9nczogPGEgY2xhc3M9Im1fLTQ3NzUyNDUy MzAyNDE5MDY2NjZtb3otdHh0LWxpbmstZnJlZXRleHQiIGhyZWY9Imh0dHA6Ly9qZW5raW5zLm92 aXJ0Lm9yZy9qb2Ivb3ZpcnQtbWFzdGVyX2NoYW5nZS1xdWV1ZS10ZXN0ZXIvMjczNC9hcnRpZmFj dC9leHBvcnRlZC1hcnRpZmFjdHMvYmFzaWMtc3VpdC1tYXN0ZXItZWw3L3Rlc3RfbG9ncy9iYXNp Yy1zdWl0ZS1tYXN0ZXIvcG9zdC0wMDJfYm9vdHN0cmFwLnB5LyIgdGFyZ2V0PSJfYmxhbmsiPmh0 dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2IvPHdicj5vdmlydC1tYXN0ZXJfY2hhbmdlLXF1ZXVl LTx3YnI+dGVzdGVyLzI3MzQvYXJ0aWZhY3QvZXhwb3J0ZWQtPHdicj5hcnRpZmFjdHMvYmFzaWMt c3VpdC1tYXN0ZXItPHdicj5lbDcvdGVzdF9sb2dzL2Jhc2ljLXN1aXRlLTx3YnI+bWFzdGVyL3Bv c3QtMDAyX2Jvb3RzdHJhcC5weS88L2E+PC9zcGFuPjwvcD4KICAgICAgICA8YnI+CiAgICAgICAg PHAgZGlyPSJsdHIiIHN0eWxlPSJsaW5lLWhlaWdodDoxLjM4O21hcmdpbi10b3A6MHB0O21hcmdp bi1ib3R0b206MHB0Ij48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExcHQ7Zm9udC1mYW1pbHk6QXJp YWw7Y29sb3I6IzAwMDAwMDtiYWNrZ3JvdW5kLWNvbG9yOnRyYW5zcGFyZW50O2ZvbnQtd2VpZ2h0 OjQwMDtmb250LXN0eWxlOm5vcm1hbDtmb250LXZhcmlhbnQ6bm9ybWFsO3RleHQtZGVjb3JhdGlv bjpub25lO3ZlcnRpY2FsLWFsaWduOmJhc2VsaW5lO3doaXRlLXNwYWNlOnByZS13cmFwIj4oUmVs ZXZhbnQpIGVycm9yIHNuaXBwZXQgZnJvbSB0aGUgbG9nOiA8L3NwYW4+PC9wPgogICAgICAgIDxw IGRpcj0ibHRyIiBzdHlsZT0ibGluZS1oZWlnaHQ6MS4zODttYXJnaW4tdG9wOjBwdDttYXJnaW4t Ym90dG9tOjBwdCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMXB0O2ZvbnQtZmFtaWx5OkFyaWFs O2NvbG9yOiMwMDAwMDA7YmFja2dyb3VuZC1jb2xvcjp0cmFuc3BhcmVudDtmb250LXdlaWdodDo0 MDA7Zm9udC1zdHlsZTpub3JtYWw7Zm9udC12YXJpYW50Om5vcm1hbDt0ZXh0LWRlY29yYXRpb246 bm9uZTt2ZXJ0aWNhbC1hbGlnbjpiYXNlbGluZTt3aGl0ZS1zcGFjZTpwcmUtd3JhcCI+Jmx0O2Vy cm9yJmd0Ozwvc3Bhbj48L3A+CiAgICAgICAgPGJyPgogICAgICAgIDIwMTctMDktMjAKMDk6MTc6 MTUsNjc2Ojp0ZXN0bGliLnB5Ojo8d2JyPmFzc2VydF9lcXVhbHNfd2l0aGluOjoyMjc6Ojx3YnI+ b3ZpcnRsYWdvLnRlc3RsaWI6OkVSUk9SOjrCoMKgwqAKICAgICAgICAqIFVuaGFuZGxlZCBleGNl cHRpb24gaW4gJmx0O2Z1bmN0aW9uIF9ob3N0X2lzX3VwXzQgYXQKICAgICAgICAweDNkZDI1ZjAm Z3Q7PGJyPgogICAgICAgIFRyYWNlYmFjayAobW9zdCByZWNlbnQgY2FsbCBsYXN0KTo8YnI+CiAg ICAgICAgwqAgRmlsZSAmcXVvdDsvdXNyL2xpYi9weXRob24yLjcvc2l0ZS08d2JyPnBhY2thZ2Vz L292aXJ0bGFnby90ZXN0bGliLnB5JnF1b3Q7PHdicj4sCiAgICAgICAgbGluZSAyMTksIGluIGFz c2VydF9lcXVhbHNfd2l0aGluPGJyPgogICAgICAgIMKgwqDCoCByZXMgPSBmdW5jKCk8YnI+CiAg ICAgICAgwqAgRmlsZQomcXVvdDsvaG9tZS9qZW5raW5zL3dvcmtzcGFjZS88d2JyPm92aXJ0LW1h c3Rlcl9jaGFuZ2UtcXVldWUtPHdicj50ZXN0ZXIvb3ZpcnQtc3lzdGVtLXRlc3RzLzx3YnI+YmFz aWMtc3VpdGUtbWFzdGVyL3Rlc3QtPHdicj5zY2VuYXJpb3MvMDAyX2Jvb3RzdHJhcC5weSZxdW90 OywKICAgICAgICBsaW5lIDMzOCwgaW4gX2hvc3RfaXNfdXBfNDxicj4KICAgICAgICDCoMKgwqAg aG9zdF9zdGF0dXMgPSBob3N0X3NlcnZpY2UuZ2V0KCkuc3RhdHVzPGJyPgogICAgICAgIMKgIEZp bGUKICAgICAgICAmcXVvdDsvdXNyL2xpYjY0L3B5dGhvbjIuNy9zaXRlLTx3YnI+cGFja2FnZXMv b3ZpcnRzZGs0L3NlcnZpY2VzLjx3YnI+cHkmcXVvdDssIGxpbmUKICAgICAgICAzMTUyOSwgaW4g Z2V0PGJyPgogICAgICAgIMKgwqDCoCByZXR1cm4gc2VsZi5faW50ZXJuYWxfZ2V0KGhlYWRlcnMs IHF1ZXJ5LCB3YWl0KTxicj4KICAgICAgICDCoCBGaWxlCiAgICAgICAgJnF1b3Q7L3Vzci9saWI2 NC9weXRob24yLjcvc2l0ZS08d2JyPnBhY2thZ2VzL292aXJ0c2RrNC9zZXJ2aWNlLnB5JnF1b3Q7 PHdicj4sIGxpbmUKICAgICAgICAyMDIsIGluIF9pbnRlcm5hbF9nZXQ8YnI+CiAgICAgICAgwqDC oMKgIHJldHVybiBmdXR1cmUud2FpdCgpIGlmIHdhaXQgZWxzZSBmdXR1cmU8YnI+CiAgICAgICAg wqAgRmlsZQogICAgICAgICZxdW90Oy91c3IvbGliNjQvcHl0aG9uMi43L3NpdGUtPHdicj5wYWNr YWdlcy9vdmlydHNkazQvc2VydmljZS5weSZxdW90Ozx3YnI+LCBsaW5lCiAgICAgICAgNTMsIGlu IHdhaXQ8YnI+CiAgICAgICAgwqDCoMKgIHJldHVybiBzZWxmLl9jb2RlKHJlc3BvbnNlKTxicj4K ICAgICAgICDCoCBGaWxlCiAgICAgICAgJnF1b3Q7L3Vzci9saWI2NC9weXRob24yLjcvc2l0ZS08 d2JyPnBhY2thZ2VzL292aXJ0c2RrNC9zZXJ2aWNlLnB5JnF1b3Q7PHdicj4sIGxpbmUKICAgICAg ICAxOTksIGluIGNhbGxiYWNrPGJyPgogICAgICAgIMKgwqDCoCBzZWxmLl9jaGVja19mYXVsdChy ZXNwb25zZSk8YnI+CiAgICAgICAgwqAgRmlsZQogICAgICAgICZxdW90Oy91c3IvbGliNjQvcHl0 aG9uMi43L3NpdGUtPHdicj5wYWNrYWdlcy9vdmlydHNkazQvc2VydmljZS5weSZxdW90Ozx3YnI+ LCBsaW5lCiAgICAgICAgMTIxLCBpbiBfY2hlY2tfZmF1bHQ8YnI+CiAgICAgICAgwqDCoMKgIGJv ZHkgPSBzZWxmLl9pbnRlcm5hbF9yZWFkX2JvZHkoPHdicj5yZXNwb25zZSk8YnI+CiAgICAgICAg wqAgRmlsZQogICAgICAgICZxdW90Oy91c3IvbGliNjQvcHl0aG9uMi43L3NpdGUtPHdicj5wYWNr YWdlcy9vdmlydHNkazQvc2VydmljZS5weSZxdW90Ozx3YnI+LCBsaW5lCiAgICAgICAgMzA3LCBp biBfaW50ZXJuYWxfcmVhZF9ib2R5PGJyPgogICAgICAgIMKgwqDCoCBzZWxmLl9jb25uZWN0aW9u LmNoZWNrX3htbF88d2JyPmNvbnRlbnRfdHlwZShyZXNwb25zZSk8YnI+CiAgICAgICAgwqAgRmls ZQogICAgICAgICZxdW90Oy91c3IvbGliNjQvcHl0aG9uMi43L3NpdGUtPHdicj5wYWNrYWdlcy9v dmlydHNkazQvX19pbml0X18uPHdicj5weSZxdW90OywgbGluZQogICAgICAgIDgwMCwgaW4gY2hl Y2tfeG1sX2NvbnRlbnRfdHlwZTxicj4KICAgICAgICDCoMKgwqAgcmVzcG9uc2UuaGVhZGVyczxi cj4KICAgICAgICDCoCBGaWxlCiAgICAgICAgJnF1b3Q7L3Vzci9saWI2NC9weXRob24yLjcvc2l0 ZS08d2JyPnBhY2thZ2VzL292aXJ0c2RrNC9fX2luaXRfXy48d2JyPnB5JnF1b3Q7LCBsaW5lCiAg ICAgICAgODQwLCBpbiBfY2hlY2tfY29udGVudF90eXBlPGJyPgogICAgICAgIMKgwqDCoCByYWlz ZSBFcnJvcihtc2cpPGJyPgogICAgICAgIEVycm9yOiBUaGUgcmVzcG9uc2UgY29udGVudCB0eXBl ICYjMzk7dGV4dC9odG1sOyBjaGFyc2V0PWlzby04ODU5LTEmIzM5OwogICAgICAgIGlzbiYjMzk7 dCB0aGUgZXhwZWN0ZWQgWE1MPGJyPgogICAgICAgIDIwMTctMDktMjAKICAgICAgICAwOToxNzox NSw2ODM6OnV0aWxzLnB5Ojo8d2JyPndyYXBwZXI6OjQ4MDo6bGFnby51dGlsczo6PHdicj5ERUJV Rzo6TG9va2luZwogICAgICAgIGZvciBhIHdvcmtkaXI8YnI+CiAgICAgICAgPGJyPgogICAgICAg IDxicj4KICAgICAgICA8cCBkaXI9Imx0ciIgc3R5bGU9ImxpbmUtaGVpZ2h0OjEuMzg7bWFyZ2lu LXRvcDowcHQ7bWFyZ2luLWJvdHRvbTowcHQiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTFwdDtm b250LWZhbWlseTpBcmlhbDtjb2xvcjojMDAwMDAwO2JhY2tncm91bmQtY29sb3I6dHJhbnNwYXJl bnQ7Zm9udC13ZWlnaHQ6NDAwO2ZvbnQtc3R5bGU6bm9ybWFsO2ZvbnQtdmFyaWFudDpub3JtYWw7 dGV4dC1kZWNvcmF0aW9uOm5vbmU7dmVydGljYWwtYWxpZ246YmFzZWxpbmU7d2hpdGUtc3BhY2U6 cHJlLXdyYXAiPiZsdDsvZXJyb3ImZ3Q7PC9zcGFuPjwvcD4KICAgICAgICA8YnI+CiAgICAgICAg VGhhbmtzLCA8YnI+CiAgICAgIDwvYj48cD48L3A+CiAgICA8cD48c3BhbiBzdHlsZT0iZm9udC13 ZWlnaHQ6bm9ybWFsIj5EYWZuYTwvc3Bhbj48YiBzdHlsZT0iZm9udC13ZWlnaHQ6bm9ybWFsIiBp ZD0ibV8tNDc3NTI0NTIzMDI0MTkwNjY2NmRvY3MtaW50ZXJuYWwtZ3VpZC01ODU5YjdhMS05ZWY2 LTkxZjQtMzhkZS1iNzMxM2Y4NTVjYzIiPjxicj4KICAgICAgPC9iPjwvcD4KICA8L2Rpdj4KCjxi cj5fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188d2JyPl9fX19fX19fX19fX19fX19fPGJy PgpEZXZlbCBtYWlsaW5nIGxpc3Q8YnI+CjxhIGhyZWY9Im1haWx0bzpEZXZlbEBvdmlydC5vcmci PkRldmVsQG92aXJ0Lm9yZzwvYT48YnI+CjxhIGhyZWY9Imh0dHA6Ly9saXN0cy5vdmlydC5vcmcv bWFpbG1hbi9saXN0aW5mby9kZXZlbCIgcmVsPSJub3JlZmVycmVyIiB0YXJnZXQ9Il9ibGFuayI+ aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy88d2JyPm1haWxtYW4vbGlzdGluZm8vZGV2ZWw8L2E+PGJy PjwvYmxvY2txdW90ZT48L2Rpdj48YnI+PC9kaXY+Cg== --===============8654375161725098321==-- From jhernand at redhat.com Wed Sep 20 12:32:42 2017 Content-Type: multipart/mixed; boundary="===============0962990200359484105==" MIME-Version: 1.0 From: =?utf-8?q?Juan_Hern=C3=A1ndez_=3Cjhernand_at_redhat=2Ecom=3E?= To: devel at ovirt.org Subject: Re: [ovirt-devel] Subject: [ OST Failure Report ] [ master ] [20-09-2017 ] [ post-002_bootstrap.py ] Date: Wed, 20 Sep 2017 14:32:39 +0200 Message-ID: <17ce3a58-3f19-d043-0235-198d9e7ffb97@redhat.com> In-Reply-To: CAP5iht46UryqLZ7hZW9SRV37j4cRJr8hZnyZ7v7EU8N=Fj2HDQ@mail.gmail.com --===============0962990200359484105== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable That SDK patch is for the *Ruby* SDK, which isn't used in OST, so I = think it isn't related. On 09/20/2017 02:11 PM, Martin Perina wrote: > Ondro, could you please take a look? > = > On Wed, Sep 20, 2017 at 1:07 PM, Dafna Ron wrote: > = >> Hi, >> >> We have a failure in test post-002_bootstrap.py >> >> from the logs I can see that lago has an unhandled exception from ovirt >> sdk which makes sense since the patch reported on CQ is a change to the >> sdk. >> >> *Link to suspected patches: https://gerrit.ovirt.org/#/c/82022/ >> * >> >> * Link to Job: >> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734 >> Link >> to all logs: >> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734/artif= act/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/p= ost-002_bootstrap.py/ >> >> (Relevant) error snippet from the log: 2017-09-20 >> 09:17:15,676::testlib.py::assert_equals_within::227::ovirtlago.testlib::= ERROR:: >> * Unhandled exception in Traceback >> (most recent call last): File >> "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in >> assert_equals_within res =3D func() File >> "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-t= ests/basic-suite-master/test-scenarios/002_bootstrap.py", >> line 338, in _host_is_up_4 host_status =3D host_service.get().status >> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line >> 31529, in get return self._internal_get(headers, query, wait) File >> "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 202, in >> _internal_get return future.wait() if wait else future File >> "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 53, in w= ait >> return self._code(response) File >> "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 199, in >> callback self._check_fault(response) File >> "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 121, in >> _check_fault body =3D self._internal_read_body(response) File >> "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 307, in >> _internal_read_body self._connection.check_xml_content_type(response) >> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 80= 0, >> in check_xml_content_type response.headers File >> "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 840, in >> _check_content_type raise Error(msg) Error: The response content type >> 'text/html; charset=3Diso-8859-1' isn't the expected XML 2017-09-20 >> 09:17:15,683::utils.py::wrapper::480::lago.utils::DEBUG::Looking for a >> workdir Thanks, * >> >> Dafna >> >> _______________________________________________ >> Devel mailing list >> Devel(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/devel >> > = --===============0962990200359484105==-- From omachace at redhat.com Wed Sep 20 13:21:00 2017 Content-Type: multipart/mixed; boundary="===============1591833353537850806==" MIME-Version: 1.0 From: Ondra Machacek To: devel at ovirt.org Subject: Re: [ovirt-devel] Subject: [ OST Failure Report ] [ master ] [20-09-2017 ] [ post-002_bootstrap.py ] Date: Wed, 20 Sep 2017 15:20:59 +0200 Message-ID: In-Reply-To: CAP5iht46UryqLZ7hZW9SRV37j4cRJr8hZnyZ7v7EU8N=Fj2HDQ@mail.gmail.com --===============1591833353537850806== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable The http returned 503: 192.168.201.1 - - [20/Sep/2017:05:17:10 -0400] "GET /ovirt-engine/api/hosts/06b9cdf3-5bcb-4be8-9e11-8ad5123749f0 HTTP/1.1" 200 5249 192.168.201.1 - - [20/Sep/2017:05:17:13 -0400] "GET /ovirt-engine/api/hosts/06b9cdf3-5bcb-4be8-9e11-8ad5123749f0 HTTP/1.1" 503 299 192.168.201.4 - - [20/Sep/2017:05:17:11 -0400] "POST /ovirt-engine/sso/oauth/token-info HTTP/1.1" 500 527 At that time postgres was killed by oom: Sep 20 05:17:15 lago-basic-suite-master-engine kernel: postgres invoked oom-killer: gfp_mask=3D0x280da, order=3D0, oom_score_adj=3D0 So from Python SDK perpepective all looks OK. On Wed, Sep 20, 2017 at 2:11 PM, Martin Perina wrote: > Ondro, could you please take a look? > > On Wed, Sep 20, 2017 at 1:07 PM, Dafna Ron wrote: >> >> Hi, >> >> We have a failure in test post-002_bootstrap.py >> >> from the logs I can see that lago has an unhandled exception from ovirt >> sdk which makes sense since the patch reported on CQ is a change to the = sdk. >> >> Link to suspected patches: https://gerrit.ovirt.org/#/c/82022/ >> >> >> Link to Job: >> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734 >> >> >> Link to all logs: >> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734/artif= act/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/p= ost-002_bootstrap.py/ >> >> >> (Relevant) error snippet from the log: >> >> >> >> >> 2017-09-20 >> 09:17:15,676::testlib.py::assert_equals_within::227::ovirtlago.testlib::= ERROR:: >> * Unhandled exception in >> Traceback (most recent call last): >> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, >> in assert_equals_within >> res =3D func() >> File >> "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-t= ests/basic-suite-master/test-scenarios/002_bootstrap.py", >> line 338, in _host_is_up_4 >> host_status =3D host_service.get().status >> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line >> 31529, in get >> return self._internal_get(headers, query, wait) >> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line >> 202, in _internal_get >> return future.wait() if wait else future >> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 5= 3, >> in wait >> return self._code(response) >> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line >> 199, in callback >> self._check_fault(response) >> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line >> 121, in _check_fault >> body =3D self._internal_read_body(response) >> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line >> 307, in _internal_read_body >> self._connection.check_xml_content_type(response) >> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line >> 800, in check_xml_content_type >> response.headers >> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line >> 840, in _check_content_type >> raise Error(msg) >> Error: The response content type 'text/html; charset=3Diso-8859-1' isn't= the >> expected XML >> 2017-09-20 >> 09:17:15,683::utils.py::wrapper::480::lago.utils::DEBUG::Looking for a >> workdir >> >> >> >> >> >> Thanks, >> >> Dafna >> >> >> _______________________________________________ >> Devel mailing list >> Devel(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/devel > > --===============1591833353537850806==-- From dron at redhat.com Wed Sep 20 14:09:33 2017 Content-Type: multipart/mixed; boundary="===============7339011374300912530==" MIME-Version: 1.0 From: Dafna Ron To: devel at ovirt.org Subject: Re: [ovirt-devel] Subject: [ OST Failure Report ] [ master ] [20-09-2017 ] [ post-002_bootstrap.py ] Date: Wed, 20 Sep 2017 15:09:28 +0100 Message-ID: In-Reply-To: CAJCqMsV9tTemu+5s-BdY9PLu_E3HW8pO8=--RVH0TyFCjopXzg@mail.gmail.com --===============7339011374300912530== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------76F54851C80F784CD6DAB3CE Content-Type: text/plain; charset=3Dutf-8 Content-Transfer-Encoding: 7bit We just had a second failure which CQ is reporting to be from the same patch: failed job: http://jenkins.ovirt.org/job/ovirt-4.1_change-queue-tester/1034/console if you look at the console output you can see this: *12:27:15* ovirt-engine-sdk-ruby_4.1_build-artifacts-fc24-ppc64le (122) fai= led building This is the job that failed on build: http://jenkins.ovirt.org/job/ovirt-en= gine-sdk-ruby_4.1_build-artifacts-fc24-ppc64le/122/ I re-ran the sdk build and it succeeded this time: http://jenkins.ovirt.org= /job/ovirt-engine-sdk-ruby_4.1_build-artifacts-fc24-ppc64le/123/ please try to re-submit the patch? maybe we have a coincident but if it fai= ls again it would be 3ed time... = Thanks, = Dafna On 09/20/2017 02:20 PM, Ondra Machacek wrote: > The http returned 503: > > 192.168.201.1 - - [20/Sep/2017:05:17:10 -0400] "GET > /ovirt-engine/api/hosts/06b9cdf3-5bcb-4be8-9e11-8ad5123749f0 HTTP/1.1" > 200 5249 > 192.168.201.1 - - [20/Sep/2017:05:17:13 -0400] "GET > /ovirt-engine/api/hosts/06b9cdf3-5bcb-4be8-9e11-8ad5123749f0 HTTP/1.1" > 503 299 > 192.168.201.4 - - [20/Sep/2017:05:17:11 -0400] "POST > /ovirt-engine/sso/oauth/token-info HTTP/1.1" 500 527 > > At that time postgres was killed by oom: > > Sep 20 05:17:15 lago-basic-suite-master-engine kernel: postgres > invoked oom-killer: gfp_mask=3D0x280da, order=3D0, oom_score_adj=3D0 > > > So from Python SDK perpepective all looks OK. > > > On Wed, Sep 20, 2017 at 2:11 PM, Martin Perina wro= te: >> Ondro, could you please take a look? >> >> On Wed, Sep 20, 2017 at 1:07 PM, Dafna Ron wrote: >>> Hi, >>> >>> We have a failure in test post-002_bootstrap.py >>> >>> from the logs I can see that lago has an unhandled exception from ovirt >>> sdk which makes sense since the patch reported on CQ is a change to the= sdk. >>> >>> Link to suspected patches: https://gerrit.ovirt.org/#/c/82022/ >>> >>> >>> Link to Job: >>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734 >>> >>> >>> Link to all logs: >>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734/arti= fact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/= post-002_bootstrap.py/ >>> >>> >>> (Relevant) error snippet from the log: >>> >>> >>> >>> >>> 2017-09-20 >>> 09:17:15,676::testlib.py::assert_equals_within::227::ovirtlago.testlib:= :ERROR:: >>> * Unhandled exception in >>> Traceback (most recent call last): >>> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 21= 9, >>> in assert_equals_within >>> res =3D func() >>> File >>> "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-= tests/basic-suite-master/test-scenarios/002_bootstrap.py", >>> line 338, in _host_is_up_4 >>> host_status =3D host_service.get().status >>> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line >>> 31529, in get >>> return self._internal_get(headers, query, wait) >>> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line >>> 202, in _internal_get >>> return future.wait() if wait else future >>> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line = 53, >>> in wait >>> return self._code(response) >>> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line >>> 199, in callback >>> self._check_fault(response) >>> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line >>> 121, in _check_fault >>> body =3D self._internal_read_body(response) >>> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line >>> 307, in _internal_read_body >>> self._connection.check_xml_content_type(response) >>> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line >>> 800, in check_xml_content_type >>> response.headers >>> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line >>> 840, in _check_content_type >>> raise Error(msg) >>> Error: The response content type 'text/html; charset=3Diso-8859-1' isn'= t the >>> expected XML >>> 2017-09-20 >>> 09:17:15,683::utils.py::wrapper::480::lago.utils::DEBUG::Looking for a >>> workdir >>> >>> >>> >>> >>> >>> Thanks, >>> >>> Dafna >>> >>> >>> _______________________________________________ >>> Devel mailing list >>> Devel(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/devel >> --------------76F54851C80F784CD6DAB3CE Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: 7bit
We just had a second failure which CQ is reporting to be from the same patch:

failed job: http://jenkins.ovirt.org/job= /ovirt-4.1_change-queue-tester/1034/console

if you look at the console output you can see this: 

12:27:15=
 ovirt-engine-sdk-ruby_4.1_build-artifacts-fc24-ppc64le (122) failed=
 building

This is the job that failed on build: http://jenkins.ovirt.org/job/ovirt-engine-sdk-ruby_4.1=
_build-artifacts-fc24-ppc64le/122/

I re-ran the sdk build and it succeeded this time: http://jenkins.ovirt.org/job/ovirt-engine=
-sdk-ruby_4.1_build-artifacts-fc24-ppc64le/123/

please try to re-submit the patch? maybe we have a coincident but if it fai=
ls again it would be 3ed time... =


Thanks, =

Dafna


On 09/20/2017 02:20 PM, Ondra Machacek wrote:
The http returned 503:

192.168.201.1 - - [20/Sep/2017:05:17:10 -0400] "GET
/ovirt-engine/api/hosts/06b9cdf3-5bcb-4be8-9e11-8ad5123749f0 HTTP/1.1"
200 5249
192.168.201.1 - - [20/Sep/2017:05:17:13 -0400] "GET
/ovirt-engine/api/hosts/06b9cdf3-5bcb-4be8-9e11-8ad5123749f0 HTTP/1.1"
503 299
192.168.201.4 - - [20/Sep/2017:05:17:11 -0400] "POST
/ovirt-engine/sso/oauth/token-info HTTP/1.1" 500 527

At that time postgres was killed by oom:

Sep 20 05:17:15 lago-basic-suite-master-engine kernel: postgres
invoked oom-killer: gfp_mask=3D0x280da, order=3D0, oom_score_adj=3D0


So from Python SDK perpepective all looks OK.


On Wed, Sep 20, 2017 at 2:11 PM, Martin Perina <mperina(a)redhat.com> wrote:

Ondro, could you please take a look?

On Wed, Sep 20, 2017 at 1:07 PM, Dafna Ron <dron(a)redhat.com> wrote:

Hi,

We have a failure in test post-002_bootstrap.py

from the logs I can see that lago has an unhandled exception from ovirt
sdk which makes sense since the patch reported on CQ is a change to the sdk.

Link to suspected patches: https://gerrit.ovirt.org/#/c/82022/


Link to Job:
http://jenkins.ovirt.org/job/ovirt-mast=
er_change-queue-tester/2734


Link to all logs:
http://jenki=
ns.ovirt.org/job/ovirt-master_change-queue-tester/2734/artifact/exported-ar=
tifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstr=
ap.py/


(Relevant) error snippet from the log:

<error>


2017-09-20
09:17:15,676::testlib.py::assert_equals_within::227::ovirtlago.testlib::ERR=
OR::
* Unhandled exception in <function _host_is_up_4 at 0x3dd25f0>
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219,
in assert_equals_within
    res =3D func()
  File
"/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-test=
s/basic-suite-master/test-scenarios/002_bootstrap.py",
line 338, in _host_is_up_4
    host_status =3D host_service.get().status
  File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line
31529, in get
    return self._internal_get(headers, query, wait)
  File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line
202, in _internal_get
    return future.wait() if wait else future
  File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 53,
in wait
    return self._code(response)
  File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line
199, in callback
    self._check_fault(response)
  File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line
121, in _check_fault
    body =3D self._internal_read_body(response)
  File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line
307, in _internal_read_body
    self._connection.check_xml_content_type(response)
  File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line
800, in check_xml_content_type
    response.headers
  File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line
840, in _check_content_type
    raise Error(msg)
Error: The response content type 'text/html; charset=3Diso-8859-1' isn't the
expected XML
2017-09-20
09:17:15,683::utils.py::wrapper::480::lago.utils::DEBUG::Looking for a
workdir


</error>


Thanks,

Dafna


_______________________________________________
Devel mailing list
Dev=
el(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel


--------------76F54851C80F784CD6DAB3CE-- --===============7339011374300912530== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS03NkY1NDg1MUM4MEY3ODRDRDZEQUIzQ0UKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXV0Zi04CkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDdiaXQKCldlIGp1c3QgaGFkIGEg c2Vjb25kIGZhaWx1cmUgd2hpY2ggQ1EgaXMgcmVwb3J0aW5nIHRvIGJlIGZyb20gdGhlIHNhbWUK cGF0Y2g6CgpmYWlsZWQgam9iOgpodHRwOi8vamVua2lucy5vdmlydC5vcmcvam9iL292aXJ0LTQu MV9jaGFuZ2UtcXVldWUtdGVzdGVyLzEwMzQvY29uc29sZQoKaWYgeW91IGxvb2sgYXQgdGhlIGNv bnNvbGUgb3V0cHV0IHlvdSBjYW4gc2VlIHRoaXM6CgoqMTI6Mjc6MTUqIG92aXJ0LWVuZ2luZS1z ZGstcnVieV80LjFfYnVpbGQtYXJ0aWZhY3RzLWZjMjQtcHBjNjRsZSAoMTIyKSBmYWlsZWQgYnVp bGRpbmcKClRoaXMgaXMgdGhlIGpvYiB0aGF0IGZhaWxlZCBvbiBidWlsZDogaHR0cDovL2plbmtp bnMub3ZpcnQub3JnL2pvYi9vdmlydC1lbmdpbmUtc2RrLXJ1YnlfNC4xX2J1aWxkLWFydGlmYWN0 cy1mYzI0LXBwYzY0bGUvMTIyLwoKSSByZS1yYW4gdGhlIHNkayBidWlsZCBhbmQgaXQgc3VjY2Vl ZGVkIHRoaXMgdGltZTogaHR0cDovL2plbmtpbnMub3ZpcnQub3JnL2pvYi9vdmlydC1lbmdpbmUt c2RrLXJ1YnlfNC4xX2J1aWxkLWFydGlmYWN0cy1mYzI0LXBwYzY0bGUvMTIzLwoKcGxlYXNlIHRy eSB0byByZS1zdWJtaXQgdGhlIHBhdGNoPyBtYXliZSB3ZSBoYXZlIGEgY29pbmNpZGVudCBidXQg aWYgaXQgZmFpbHMgYWdhaW4gaXQgd291bGQgYmUgM2VkIHRpbWUuLi4gCgpUaGFua3MsIApEYWZu YQoKCgpPbiAwOS8yMC8yMDE3IDAyOjIwIFBNLCBPbmRyYSBNYWNoYWNlayB3cm90ZToKPiBUaGUg aHR0cCByZXR1cm5lZCA1MDM6Cj4KPiAxOTIuMTY4LjIwMS4xIC0gLSBbMjAvU2VwLzIwMTc6MDU6 MTc6MTAgLTA0MDBdICJHRVQKPiAvb3ZpcnQtZW5naW5lL2FwaS9ob3N0cy8wNmI5Y2RmMy01YmNi LTRiZTgtOWUxMS04YWQ1MTIzNzQ5ZjAgSFRUUC8xLjEiCj4gMjAwIDUyNDkKPiAxOTIuMTY4LjIw MS4xIC0gLSBbMjAvU2VwLzIwMTc6MDU6MTc6MTMgLTA0MDBdICJHRVQKPiAvb3ZpcnQtZW5naW5l L2FwaS9ob3N0cy8wNmI5Y2RmMy01YmNiLTRiZTgtOWUxMS04YWQ1MTIzNzQ5ZjAgSFRUUC8xLjEi Cj4gNTAzIDI5OQo+IDE5Mi4xNjguMjAxLjQgLSAtIFsyMC9TZXAvMjAxNzowNToxNzoxMSAtMDQw MF0gIlBPU1QKPiAvb3ZpcnQtZW5naW5lL3Nzby9vYXV0aC90b2tlbi1pbmZvIEhUVFAvMS4xIiA1 MDAgNTI3Cj4KPiBBdCB0aGF0IHRpbWUgcG9zdGdyZXMgd2FzIGtpbGxlZCBieSBvb206Cj4KPiBT ZXAgMjAgMDU6MTc6MTUgbGFnby1iYXNpYy1zdWl0ZS1tYXN0ZXItZW5naW5lIGtlcm5lbDogcG9z dGdyZXMKPiBpbnZva2VkIG9vbS1raWxsZXI6IGdmcF9tYXNrPTB4MjgwZGEsIG9yZGVyPTAsIG9v bV9zY29yZV9hZGo9MAo+Cj4KPiBTbyBmcm9tIFB5dGhvbiBTREsgcGVycGVwZWN0aXZlIGFsbCBs b29rcyBPSy4KPgo+Cj4gT24gV2VkLCBTZXAgMjAsIDIwMTcgYXQgMjoxMSBQTSwgTWFydGluIFBl cmluYSA8bXBlcmluYUByZWRoYXQuY29tPiB3cm90ZToKPj4gT25kcm8sIGNvdWxkIHlvdSBwbGVh c2UgdGFrZSBhIGxvb2s/Cj4+Cj4+IE9uIFdlZCwgU2VwIDIwLCAyMDE3IGF0IDE6MDcgUE0sIERh Zm5hIFJvbiA8ZHJvbkByZWRoYXQuY29tPiB3cm90ZToKPj4+IEhpLAo+Pj4KPj4+IFdlIGhhdmUg YSBmYWlsdXJlIGluIHRlc3QgcG9zdC0wMDJfYm9vdHN0cmFwLnB5Cj4+Pgo+Pj4gZnJvbSB0aGUg bG9ncyBJIGNhbiBzZWUgdGhhdCBsYWdvIGhhcyBhbiB1bmhhbmRsZWQgZXhjZXB0aW9uIGZyb20g b3ZpcnQKPj4+IHNkayB3aGljaCBtYWtlcyBzZW5zZSBzaW5jZSB0aGUgcGF0Y2ggcmVwb3J0ZWQg b24gQ1EgaXMgYSBjaGFuZ2UgdG8gdGhlIHNkay4KPj4+Cj4+PiBMaW5rIHRvIHN1c3BlY3RlZCBw YXRjaGVzOiBodHRwczovL2dlcnJpdC5vdmlydC5vcmcvIy9jLzgyMDIyLwo+Pj4KPj4+Cj4+PiBM aW5rIHRvIEpvYjoKPj4+IGh0dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2Ivb3ZpcnQtbWFzdGVy X2NoYW5nZS1xdWV1ZS10ZXN0ZXIvMjczNAo+Pj4KPj4+Cj4+PiBMaW5rIHRvIGFsbCBsb2dzOgo+ Pj4gaHR0cDovL2plbmtpbnMub3ZpcnQub3JnL2pvYi9vdmlydC1tYXN0ZXJfY2hhbmdlLXF1ZXVl LXRlc3Rlci8yNzM0L2FydGlmYWN0L2V4cG9ydGVkLWFydGlmYWN0cy9iYXNpYy1zdWl0LW1hc3Rl ci1lbDcvdGVzdF9sb2dzL2Jhc2ljLXN1aXRlLW1hc3Rlci9wb3N0LTAwMl9ib290c3RyYXAucHkv Cj4+Pgo+Pj4KPj4+IChSZWxldmFudCkgZXJyb3Igc25pcHBldCBmcm9tIHRoZSBsb2c6Cj4+Pgo+ Pj4gPGVycm9yPgo+Pj4KPj4+Cj4+PiAyMDE3LTA5LTIwCj4+PiAwOToxNzoxNSw2NzY6OnRlc3Rs aWIucHk6OmFzc2VydF9lcXVhbHNfd2l0aGluOjoyMjc6Om92aXJ0bGFnby50ZXN0bGliOjpFUlJP Ujo6Cj4+PiAqIFVuaGFuZGxlZCBleGNlcHRpb24gaW4gPGZ1bmN0aW9uIF9ob3N0X2lzX3VwXzQg YXQgMHgzZGQyNWYwPgo+Pj4gVHJhY2ViYWNrIChtb3N0IHJlY2VudCBjYWxsIGxhc3QpOgo+Pj4g ICBGaWxlICIvdXNyL2xpYi9weXRob24yLjcvc2l0ZS1wYWNrYWdlcy9vdmlydGxhZ28vdGVzdGxp Yi5weSIsIGxpbmUgMjE5LAo+Pj4gaW4gYXNzZXJ0X2VxdWFsc193aXRoaW4KPj4+ICAgICByZXMg PSBmdW5jKCkKPj4+ICAgRmlsZQo+Pj4gIi9ob21lL2plbmtpbnMvd29ya3NwYWNlL292aXJ0LW1h c3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVyL292aXJ0LXN5c3RlbS10ZXN0cy9iYXNpYy1zdWl0ZS1t YXN0ZXIvdGVzdC1zY2VuYXJpb3MvMDAyX2Jvb3RzdHJhcC5weSIsCj4+PiBsaW5lIDMzOCwgaW4g X2hvc3RfaXNfdXBfNAo+Pj4gICAgIGhvc3Rfc3RhdHVzID0gaG9zdF9zZXJ2aWNlLmdldCgpLnN0 YXR1cwo+Pj4gICBGaWxlICIvdXNyL2xpYjY0L3B5dGhvbjIuNy9zaXRlLXBhY2thZ2VzL292aXJ0 c2RrNC9zZXJ2aWNlcy5weSIsIGxpbmUKPj4+IDMxNTI5LCBpbiBnZXQKPj4+ICAgICByZXR1cm4g c2VsZi5faW50ZXJuYWxfZ2V0KGhlYWRlcnMsIHF1ZXJ5LCB3YWl0KQo+Pj4gICBGaWxlICIvdXNy L2xpYjY0L3B5dGhvbjIuNy9zaXRlLXBhY2thZ2VzL292aXJ0c2RrNC9zZXJ2aWNlLnB5IiwgbGlu ZQo+Pj4gMjAyLCBpbiBfaW50ZXJuYWxfZ2V0Cj4+PiAgICAgcmV0dXJuIGZ1dHVyZS53YWl0KCkg aWYgd2FpdCBlbHNlIGZ1dHVyZQo+Pj4gICBGaWxlICIvdXNyL2xpYjY0L3B5dGhvbjIuNy9zaXRl LXBhY2thZ2VzL292aXJ0c2RrNC9zZXJ2aWNlLnB5IiwgbGluZSA1MywKPj4+IGluIHdhaXQKPj4+ ICAgICByZXR1cm4gc2VsZi5fY29kZShyZXNwb25zZSkKPj4+ICAgRmlsZSAiL3Vzci9saWI2NC9w eXRob24yLjcvc2l0ZS1wYWNrYWdlcy9vdmlydHNkazQvc2VydmljZS5weSIsIGxpbmUKPj4+IDE5 OSwgaW4gY2FsbGJhY2sKPj4+ICAgICBzZWxmLl9jaGVja19mYXVsdChyZXNwb25zZSkKPj4+ICAg RmlsZSAiL3Vzci9saWI2NC9weXRob24yLjcvc2l0ZS1wYWNrYWdlcy9vdmlydHNkazQvc2Vydmlj ZS5weSIsIGxpbmUKPj4+IDEyMSwgaW4gX2NoZWNrX2ZhdWx0Cj4+PiAgICAgYm9keSA9IHNlbGYu X2ludGVybmFsX3JlYWRfYm9keShyZXNwb25zZSkKPj4+ICAgRmlsZSAiL3Vzci9saWI2NC9weXRo b24yLjcvc2l0ZS1wYWNrYWdlcy9vdmlydHNkazQvc2VydmljZS5weSIsIGxpbmUKPj4+IDMwNywg aW4gX2ludGVybmFsX3JlYWRfYm9keQo+Pj4gICAgIHNlbGYuX2Nvbm5lY3Rpb24uY2hlY2tfeG1s X2NvbnRlbnRfdHlwZShyZXNwb25zZSkKPj4+ICAgRmlsZSAiL3Vzci9saWI2NC9weXRob24yLjcv c2l0ZS1wYWNrYWdlcy9vdmlydHNkazQvX19pbml0X18ucHkiLCBsaW5lCj4+PiA4MDAsIGluIGNo ZWNrX3htbF9jb250ZW50X3R5cGUKPj4+ICAgICByZXNwb25zZS5oZWFkZXJzCj4+PiAgIEZpbGUg Ii91c3IvbGliNjQvcHl0aG9uMi43L3NpdGUtcGFja2FnZXMvb3ZpcnRzZGs0L19faW5pdF9fLnB5 IiwgbGluZQo+Pj4gODQwLCBpbiBfY2hlY2tfY29udGVudF90eXBlCj4+PiAgICAgcmFpc2UgRXJy b3IobXNnKQo+Pj4gRXJyb3I6IFRoZSByZXNwb25zZSBjb250ZW50IHR5cGUgJ3RleHQvaHRtbDsg Y2hhcnNldD1pc28tODg1OS0xJyBpc24ndCB0aGUKPj4+IGV4cGVjdGVkIFhNTAo+Pj4gMjAxNy0w OS0yMAo+Pj4gMDk6MTc6MTUsNjgzOjp1dGlscy5weTo6d3JhcHBlcjo6NDgwOjpsYWdvLnV0aWxz OjpERUJVRzo6TG9va2luZyBmb3IgYQo+Pj4gd29ya2Rpcgo+Pj4KPj4+Cj4+PiA8L2Vycm9yPgo+ Pj4KPj4+Cj4+PiBUaGFua3MsCj4+Pgo+Pj4gRGFmbmEKPj4+Cj4+Pgo+Pj4gX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KPj4+IERldmVsIG1haWxpbmcgbGlz dAo+Pj4gRGV2ZWxAb3ZpcnQub3JnCj4+PiBodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4v bGlzdGluZm8vZGV2ZWwKPj4KCgotLS0tLS0tLS0tLS0tLTc2RjU0ODUxQzgwRjc4NENENkRBQjND RQpDb250ZW50LVR5cGU6IHRleHQvaHRtbDsgY2hhcnNldD11dGYtOApDb250ZW50LVRyYW5zZmVy LUVuY29kaW5nOiA3Yml0Cgo8aHRtbD4KICA8aGVhZD4KICAgIDxtZXRhIGh0dHAtZXF1aXY9IkNv bnRlbnQtVHlwZSIgY29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4KICA8L2hlYWQ+ CiAgPGJvZHkgdGV4dD0iIzAwMDAwMCIgYmdjb2xvcj0iI0ZGRkZGRiI+CiAgICA8ZGl2IGNsYXNz PSJtb3otY2l0ZS1wcmVmaXgiPldlIGp1c3QgaGFkIGEgc2Vjb25kIGZhaWx1cmUgd2hpY2ggQ1EK ICAgICAgaXMgcmVwb3J0aW5nIHRvIGJlIGZyb20gdGhlIHNhbWUgcGF0Y2g6IDxicj4KICAgICAg PGJyPgogICAgICBmYWlsZWQgam9iOgogICAgICA8YSBjbGFzcz0ibW96LXR4dC1saW5rLWZyZWV0 ZXh0IiBocmVmPSJodHRwOi8vamVua2lucy5vdmlydC5vcmcvam9iL292aXJ0LTQuMV9jaGFuZ2Ut cXVldWUtdGVzdGVyLzEwMzQvY29uc29sZSI+aHR0cDovL2plbmtpbnMub3ZpcnQub3JnL2pvYi9v dmlydC00LjFfY2hhbmdlLXF1ZXVlLXRlc3Rlci8xMDM0L2NvbnNvbGU8L2E+PGJyPgogICAgICA8 YnI+CiAgICAgIGlmIHlvdSBsb29rIGF0IHRoZSBjb25zb2xlIG91dHB1dCB5b3UgY2FuIHNlZSB0 aGlzOiA8YnI+CiAgICAgIDxicj4KICAgICAgPHByZSBjbGFzcz0iY29uc29sZS1vdXRwdXQiIHN0 eWxlPSJib3gtc2l6aW5nOiBib3JkZXItYm94OyB3aGl0ZS1zcGFjZTogcHJlLXdyYXA7IHdvcmQt d3JhcDogYnJlYWstd29yZDsgbWFyZ2luOiAwcHg7IGNvbG9yOiByZ2IoNTEsIDUxLCA1MSk7IGZv bnQtc2l6ZTogMTRweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQtbGlnYXR1cmVz OiBub3JtYWw7IGZvbnQtdmFyaWFudC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiBub3JtYWw7 IGxldHRlci1zcGFjaW5nOiBub3JtYWw7IG9ycGhhbnM6IDI7IHRleHQtYWxpZ246IHN0YXJ0OyB0 ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2lkb3dzOiAyOyB3b3JkLXNw YWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyB0ZXh0LWRlY29yYXRp b24tc3R5bGU6IGluaXRpYWw7IHRleHQtZGVjb3JhdGlvbi1jb2xvcjogaW5pdGlhbDsiPjxzcGFu IGNsYXNzPSJ0aW1lc3RhbXAiIHN0eWxlPSJib3gtc2l6aW5nOiBib3JkZXItYm94OyI+PGIgc3R5 bGU9ImJveC1zaXppbmc6IGJvcmRlci1ib3g7Ij4xMjoyNzoxNTwvYj4gPC9zcGFuPm92aXJ0LWVu Z2luZS1zZGstcnVieV80LjFfYnVpbGQtYXJ0aWZhY3RzLWZjMjQtcHBjNjRsZSAoMTIyKSBmYWls ZWQgYnVpbGRpbmcKClRoaXMgaXMgdGhlIGpvYiB0aGF0IGZhaWxlZCBvbiBidWlsZDogPGEgY2xh c3M9Im1vei10eHQtbGluay1mcmVldGV4dCIgaHJlZj0iaHR0cDovL2plbmtpbnMub3ZpcnQub3Jn L2pvYi9vdmlydC1lbmdpbmUtc2RrLXJ1YnlfNC4xX2J1aWxkLWFydGlmYWN0cy1mYzI0LXBwYzY0 bGUvMTIyLyI+aHR0cDovL2plbmtpbnMub3ZpcnQub3JnL2pvYi9vdmlydC1lbmdpbmUtc2RrLXJ1 YnlfNC4xX2J1aWxkLWFydGlmYWN0cy1mYzI0LXBwYzY0bGUvMTIyLzwvYT4KCkkgcmUtcmFuIHRo ZSBzZGsgYnVpbGQgYW5kIGl0IHN1Y2NlZWRlZCB0aGlzIHRpbWU6IDxhIGNsYXNzPSJtb3otdHh0 LWxpbmstZnJlZXRleHQiIGhyZWY9Imh0dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2Ivb3ZpcnQt ZW5naW5lLXNkay1ydWJ5XzQuMV9idWlsZC1hcnRpZmFjdHMtZmMyNC1wcGM2NGxlLzEyMy8iPmh0 dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2Ivb3ZpcnQtZW5naW5lLXNkay1ydWJ5XzQuMV9idWls ZC1hcnRpZmFjdHMtZmMyNC1wcGM2NGxlLzEyMy88L2E+CgpwbGVhc2UgdHJ5IHRvIHJlLXN1Ym1p dCB0aGUgcGF0Y2g/IG1heWJlIHdlIGhhdmUgYSBjb2luY2lkZW50IGJ1dCBpZiBpdCBmYWlscyBh Z2FpbiBpdCB3b3VsZCBiZSAzZWQgdGltZS4uLiAKClRoYW5rcywgCkRhZm5hCgo8L3ByZT4KICAg ICAgPGJyPgogICAgICA8YnI+CiAgICAgIE9uIDA5LzIwLzIwMTcgMDI6MjAgUE0sIE9uZHJhIE1h Y2hhY2VrIHdyb3RlOjxicj4KICAgIDwvZGl2PgogICAgPGJsb2NrcXVvdGUgdHlwZT0iY2l0ZSIK Y2l0ZT0ibWlkOkNBSkNxTXNWOXRUZW11KzVzLUJkWTlQTHVfRTNIVzhwTzg9LS1SVkgwVHlGQ2pv cFh6Z0BtYWlsLmdtYWlsLmNvbSI+CiAgICAgIDxwcmUgd3JhcD0iIj5UaGUgaHR0cCByZXR1cm5l ZCA1MDM6CgoxOTIuMTY4LjIwMS4xIC0gLSBbMjAvU2VwLzIwMTc6MDU6MTc6MTAgLTA0MDBdICJH RVQKL292aXJ0LWVuZ2luZS9hcGkvaG9zdHMvMDZiOWNkZjMtNWJjYi00YmU4LTllMTEtOGFkNTEy Mzc0OWYwIEhUVFAvMS4xIgoyMDAgNTI0OQoxOTIuMTY4LjIwMS4xIC0gLSBbMjAvU2VwLzIwMTc6 MDU6MTc6MTMgLTA0MDBdICJHRVQKL292aXJ0LWVuZ2luZS9hcGkvaG9zdHMvMDZiOWNkZjMtNWJj Yi00YmU4LTllMTEtOGFkNTEyMzc0OWYwIEhUVFAvMS4xIgo1MDMgMjk5CjE5Mi4xNjguMjAxLjQg LSAtIFsyMC9TZXAvMjAxNzowNToxNzoxMSAtMDQwMF0gIlBPU1QKL292aXJ0LWVuZ2luZS9zc28v b2F1dGgvdG9rZW4taW5mbyBIVFRQLzEuMSIgNTAwIDUyNwoKQXQgdGhhdCB0aW1lIHBvc3RncmVz IHdhcyBraWxsZWQgYnkgb29tOgoKU2VwIDIwIDA1OjE3OjE1IGxhZ28tYmFzaWMtc3VpdGUtbWFz dGVyLWVuZ2luZSBrZXJuZWw6IHBvc3RncmVzCmludm9rZWQgb29tLWtpbGxlcjogZ2ZwX21hc2s9 MHgyODBkYSwgb3JkZXI9MCwgb29tX3Njb3JlX2Fkaj0wCgoKU28gZnJvbSBQeXRob24gU0RLIHBl cnBlcGVjdGl2ZSBhbGwgbG9va3MgT0suCgoKT24gV2VkLCBTZXAgMjAsIDIwMTcgYXQgMjoxMSBQ TSwgTWFydGluIFBlcmluYSA8YSBjbGFzcz0ibW96LXR4dC1saW5rLXJmYzIzOTZFIiBocmVmPSJt YWlsdG86bXBlcmluYUByZWRoYXQuY29tIj4mbHQ7bXBlcmluYUByZWRoYXQuY29tJmd0OzwvYT4g d3JvdGU6CjwvcHJlPgogICAgICA8YmxvY2txdW90ZSB0eXBlPSJjaXRlIj4KICAgICAgICA8cHJl IHdyYXA9IiI+T25kcm8sIGNvdWxkIHlvdSBwbGVhc2UgdGFrZSBhIGxvb2s/CgpPbiBXZWQsIFNl cCAyMCwgMjAxNyBhdCAxOjA3IFBNLCBEYWZuYSBSb24gPGEgY2xhc3M9Im1vei10eHQtbGluay1y ZmMyMzk2RSIgaHJlZj0ibWFpbHRvOmRyb25AcmVkaGF0LmNvbSI+Jmx0O2Ryb25AcmVkaGF0LmNv bSZndDs8L2E+IHdyb3RlOgo8L3ByZT4KICAgICAgICA8YmxvY2txdW90ZSB0eXBlPSJjaXRlIj4K ICAgICAgICAgIDxwcmUgd3JhcD0iIj4KSGksCgpXZSBoYXZlIGEgZmFpbHVyZSBpbiB0ZXN0IHBv c3QtMDAyX2Jvb3RzdHJhcC5weQoKZnJvbSB0aGUgbG9ncyBJIGNhbiBzZWUgdGhhdCBsYWdvIGhh cyBhbiB1bmhhbmRsZWQgZXhjZXB0aW9uIGZyb20gb3ZpcnQKc2RrIHdoaWNoIG1ha2VzIHNlbnNl IHNpbmNlIHRoZSBwYXRjaCByZXBvcnRlZCBvbiBDUSBpcyBhIGNoYW5nZSB0byB0aGUgc2RrLgoK TGluayB0byBzdXNwZWN0ZWQgcGF0Y2hlczogPGEgY2xhc3M9Im1vei10eHQtbGluay1mcmVldGV4 dCIgaHJlZj0iaHR0cHM6Ly9nZXJyaXQub3ZpcnQub3JnLyMvYy84MjAyMi8iPmh0dHBzOi8vZ2Vy cml0Lm92aXJ0Lm9yZy8jL2MvODIwMjIvPC9hPgoKCkxpbmsgdG8gSm9iOgo8YSBjbGFzcz0ibW96 LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRwOi8vamVua2lucy5vdmlydC5vcmcvam9iL292 aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVyLzI3MzQiPmh0dHA6Ly9qZW5raW5zLm92aXJ0 Lm9yZy9qb2Ivb3ZpcnQtbWFzdGVyX2NoYW5nZS1xdWV1ZS10ZXN0ZXIvMjczNDwvYT4KCgpMaW5r IHRvIGFsbCBsb2dzOgo8YSBjbGFzcz0ibW96LXR4dC1saW5rLWZyZWV0ZXh0IiBocmVmPSJodHRw Oi8vamVua2lucy5vdmlydC5vcmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVy LzI3MzQvYXJ0aWZhY3QvZXhwb3J0ZWQtYXJ0aWZhY3RzL2Jhc2ljLXN1aXQtbWFzdGVyLWVsNy90 ZXN0X2xvZ3MvYmFzaWMtc3VpdGUtbWFzdGVyL3Bvc3QtMDAyX2Jvb3RzdHJhcC5weS8iPmh0dHA6 Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2Ivb3ZpcnQtbWFzdGVyX2NoYW5nZS1xdWV1ZS10ZXN0ZXIv MjczNC9hcnRpZmFjdC9leHBvcnRlZC1hcnRpZmFjdHMvYmFzaWMtc3VpdC1tYXN0ZXItZWw3L3Rl c3RfbG9ncy9iYXNpYy1zdWl0ZS1tYXN0ZXIvcG9zdC0wMDJfYm9vdHN0cmFwLnB5LzwvYT4KCgoo UmVsZXZhbnQpIGVycm9yIHNuaXBwZXQgZnJvbSB0aGUgbG9nOgoKJmx0O2Vycm9yJmd0OwoKCjIw MTctMDktMjAKMDk6MTc6MTUsNjc2Ojp0ZXN0bGliLnB5Ojphc3NlcnRfZXF1YWxzX3dpdGhpbjo6 MjI3OjpvdmlydGxhZ28udGVzdGxpYjo6RVJST1I6OgoqIFVuaGFuZGxlZCBleGNlcHRpb24gaW4g Jmx0O2Z1bmN0aW9uIF9ob3N0X2lzX3VwXzQgYXQgMHgzZGQyNWYwJmd0OwpUcmFjZWJhY2sgKG1v c3QgcmVjZW50IGNhbGwgbGFzdCk6CiAgRmlsZSAiL3Vzci9saWIvcHl0aG9uMi43L3NpdGUtcGFj a2FnZXMvb3ZpcnRsYWdvL3Rlc3RsaWIucHkiLCBsaW5lIDIxOSwKaW4gYXNzZXJ0X2VxdWFsc193 aXRoaW4KICAgIHJlcyA9IGZ1bmMoKQogIEZpbGUKIi9ob21lL2plbmtpbnMvd29ya3NwYWNlL292 aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVyL292aXJ0LXN5c3RlbS10ZXN0cy9iYXNpYy1z dWl0ZS1tYXN0ZXIvdGVzdC1zY2VuYXJpb3MvMDAyX2Jvb3RzdHJhcC5weSIsCmxpbmUgMzM4LCBp biBfaG9zdF9pc191cF80CiAgICBob3N0X3N0YXR1cyA9IGhvc3Rfc2VydmljZS5nZXQoKS5zdGF0 dXMKICBGaWxlICIvdXNyL2xpYjY0L3B5dGhvbjIuNy9zaXRlLXBhY2thZ2VzL292aXJ0c2RrNC9z ZXJ2aWNlcy5weSIsIGxpbmUKMzE1MjksIGluIGdldAogICAgcmV0dXJuIHNlbGYuX2ludGVybmFs X2dldChoZWFkZXJzLCBxdWVyeSwgd2FpdCkKICBGaWxlICIvdXNyL2xpYjY0L3B5dGhvbjIuNy9z aXRlLXBhY2thZ2VzL292aXJ0c2RrNC9zZXJ2aWNlLnB5IiwgbGluZQoyMDIsIGluIF9pbnRlcm5h bF9nZXQKICAgIHJldHVybiBmdXR1cmUud2FpdCgpIGlmIHdhaXQgZWxzZSBmdXR1cmUKICBGaWxl ICIvdXNyL2xpYjY0L3B5dGhvbjIuNy9zaXRlLXBhY2thZ2VzL292aXJ0c2RrNC9zZXJ2aWNlLnB5 IiwgbGluZSA1MywKaW4gd2FpdAogICAgcmV0dXJuIHNlbGYuX2NvZGUocmVzcG9uc2UpCiAgRmls ZSAiL3Vzci9saWI2NC9weXRob24yLjcvc2l0ZS1wYWNrYWdlcy9vdmlydHNkazQvc2VydmljZS5w eSIsIGxpbmUKMTk5LCBpbiBjYWxsYmFjawogICAgc2VsZi5fY2hlY2tfZmF1bHQocmVzcG9uc2Up CiAgRmlsZSAiL3Vzci9saWI2NC9weXRob24yLjcvc2l0ZS1wYWNrYWdlcy9vdmlydHNkazQvc2Vy dmljZS5weSIsIGxpbmUKMTIxLCBpbiBfY2hlY2tfZmF1bHQKICAgIGJvZHkgPSBzZWxmLl9pbnRl cm5hbF9yZWFkX2JvZHkocmVzcG9uc2UpCiAgRmlsZSAiL3Vzci9saWI2NC9weXRob24yLjcvc2l0 ZS1wYWNrYWdlcy9vdmlydHNkazQvc2VydmljZS5weSIsIGxpbmUKMzA3LCBpbiBfaW50ZXJuYWxf cmVhZF9ib2R5CiAgICBzZWxmLl9jb25uZWN0aW9uLmNoZWNrX3htbF9jb250ZW50X3R5cGUocmVz cG9uc2UpCiAgRmlsZSAiL3Vzci9saWI2NC9weXRob24yLjcvc2l0ZS1wYWNrYWdlcy9vdmlydHNk azQvX19pbml0X18ucHkiLCBsaW5lCjgwMCwgaW4gY2hlY2tfeG1sX2NvbnRlbnRfdHlwZQogICAg cmVzcG9uc2UuaGVhZGVycwogIEZpbGUgIi91c3IvbGliNjQvcHl0aG9uMi43L3NpdGUtcGFja2Fn ZXMvb3ZpcnRzZGs0L19faW5pdF9fLnB5IiwgbGluZQo4NDAsIGluIF9jaGVja19jb250ZW50X3R5 cGUKICAgIHJhaXNlIEVycm9yKG1zZykKRXJyb3I6IFRoZSByZXNwb25zZSBjb250ZW50IHR5cGUg J3RleHQvaHRtbDsgY2hhcnNldD1pc28tODg1OS0xJyBpc24ndCB0aGUKZXhwZWN0ZWQgWE1MCjIw MTctMDktMjAKMDk6MTc6MTUsNjgzOjp1dGlscy5weTo6d3JhcHBlcjo6NDgwOjpsYWdvLnV0aWxz OjpERUJVRzo6TG9va2luZyBmb3IgYQp3b3JrZGlyCgoKJmx0Oy9lcnJvciZndDsKCgpUaGFua3Ms CgpEYWZuYQoKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f CkRldmVsIG1haWxpbmcgbGlzdAo8YSBjbGFzcz0ibW96LXR4dC1saW5rLWFiYnJldmlhdGVkIiBo cmVmPSJtYWlsdG86RGV2ZWxAb3ZpcnQub3JnIj5EZXZlbEBvdmlydC5vcmc8L2E+CjxhIGNsYXNz PSJtb3otdHh0LWxpbmstZnJlZXRleHQiIGhyZWY9Imh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFp bG1hbi9saXN0aW5mby9kZXZlbCI+aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3Rp bmZvL2RldmVsPC9hPgo8L3ByZT4KICAgICAgICA8L2Jsb2NrcXVvdGU+CiAgICAgICAgPHByZSB3 cmFwPSIiPgoKPC9wcmU+CiAgICAgIDwvYmxvY2txdW90ZT4KICAgIDwvYmxvY2txdW90ZT4KICAg IDxwPjxicj4KICAgIDwvcD4KICA8L2JvZHk+CjwvaHRtbD4KCi0tLS0tLS0tLS0tLS0tNzZGNTQ4 NTFDODBGNzg0Q0Q2REFCM0NFLS0K --===============7339011374300912530==-- From jhernand at redhat.com Wed Sep 20 14:44:01 2017 Content-Type: multipart/mixed; boundary="===============4825524077343505997==" MIME-Version: 1.0 From: =?utf-8?q?Juan_Hern=C3=A1ndez_=3Cjhernand_at_redhat=2Ecom=3E?= To: devel at ovirt.org Subject: Re: [ovirt-devel] Subject: [ OST Failure Report ] [ master ] [20-09-2017 ] [ post-002_bootstrap.py ] Date: Wed, 20 Sep 2017 16:43:58 +0200 Message-ID: <6a94e41f-4d14-353a-26e2-ac471490b652@redhat.com> In-Reply-To: e11da0cc-7555-a2f3-35d5-640f841d9c87@redhat.com --===============4825524077343505997== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable I am aware of that build failure. The tests of the *Ruby* SDK hang = apparently random in ppc64le environments. I am trying to find out why, = but not making much progress, to be honest. Meanwhile I wonder why this is a failure to run OST tests, as the Ruby = SDK isn't used at all in those tests, as far as I know. On 09/20/2017 04:09 PM, Dafna Ron wrote: > We just had a second failure which CQ is reporting to be from the same > patch: > = > failed job: > http://jenkins.ovirt.org/job/ovirt-4.1_change-queue-tester/1034/console > = > if you look at the console output you can see this: > = > *12:27:15* ovirt-engine-sdk-ruby_4.1_build-artifacts-fc24-ppc64le (122) f= ailed building > = > This is the job that failed on build: http://jenkins.ovirt.org/job/ovirt-= engine-sdk-ruby_4.1_build-artifacts-fc24-ppc64le/122/ > = > I re-ran the sdk build and it succeeded this time: http://jenkins.ovirt.o= rg/job/ovirt-engine-sdk-ruby_4.1_build-artifacts-fc24-ppc64le/123/ > = > please try to re-submit the patch? maybe we have a coincident but if it f= ails again it would be 3ed time... > = > Thanks, > Dafna > = > = > = > On 09/20/2017 02:20 PM, Ondra Machacek wrote: >> The http returned 503: >> >> 192.168.201.1 - - [20/Sep/2017:05:17:10 -0400] "GET >> /ovirt-engine/api/hosts/06b9cdf3-5bcb-4be8-9e11-8ad5123749f0 HTTP/1.1" >> 200 5249 >> 192.168.201.1 - - [20/Sep/2017:05:17:13 -0400] "GET >> /ovirt-engine/api/hosts/06b9cdf3-5bcb-4be8-9e11-8ad5123749f0 HTTP/1.1" >> 503 299 >> 192.168.201.4 - - [20/Sep/2017:05:17:11 -0400] "POST >> /ovirt-engine/sso/oauth/token-info HTTP/1.1" 500 527 >> >> At that time postgres was killed by oom: >> >> Sep 20 05:17:15 lago-basic-suite-master-engine kernel: postgres >> invoked oom-killer: gfp_mask=3D0x280da, order=3D0, oom_score_adj=3D0 >> >> >> So from Python SDK perpepective all looks OK. >> >> >> On Wed, Sep 20, 2017 at 2:11 PM, Martin Perina wr= ote: >>> Ondro, could you please take a look? >>> >>> On Wed, Sep 20, 2017 at 1:07 PM, Dafna Ron wrote: >>>> Hi, >>>> >>>> We have a failure in test post-002_bootstrap.py >>>> >>>> from the logs I can see that lago has an unhandled exception from ovirt >>>> sdk which makes sense since the patch reported on CQ is a change to th= e sdk. >>>> >>>> Link to suspected patches: https://gerrit.ovirt.org/#/c/82022/ >>>> >>>> >>>> Link to Job: >>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734 >>>> >>>> >>>> Link to all logs: >>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2734/art= ifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master= /post-002_bootstrap.py/ >>>> >>>> >>>> (Relevant) error snippet from the log: >>>> >>>> >>>> >>>> >>>> 2017-09-20 >>>> 09:17:15,676::testlib.py::assert_equals_within::227::ovirtlago.testlib= ::ERROR:: >>>> * Unhandled exception in >>>> Traceback (most recent call last): >>>> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line = 219, >>>> in assert_equals_within >>>> res =3D func() >>>> File >>>> "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system= -tests/basic-suite-master/test-scenarios/002_bootstrap.py", >>>> line 338, in _host_is_up_4 >>>> host_status =3D host_service.get().status >>>> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", li= ne >>>> 31529, in get >>>> return self._internal_get(headers, query, wait) >>>> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line >>>> 202, in _internal_get >>>> return future.wait() if wait else future >>>> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", lin= e 53, >>>> in wait >>>> return self._code(response) >>>> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line >>>> 199, in callback >>>> self._check_fault(response) >>>> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line >>>> 121, in _check_fault >>>> body =3D self._internal_read_body(response) >>>> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line >>>> 307, in _internal_read_body >>>> self._connection.check_xml_content_type(response) >>>> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", li= ne >>>> 800, in check_xml_content_type >>>> response.headers >>>> File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", li= ne >>>> 840, in _check_content_type >>>> raise Error(msg) >>>> Error: The response content type 'text/html; charset=3Diso-8859-1' isn= 't the >>>> expected XML >>>> 2017-09-20 >>>> 09:17:15,683::utils.py::wrapper::480::lago.utils::DEBUG::Looking for a >>>> workdir >>>> >>>> >>>> >>>> >>>> >>>> Thanks, >>>> >>>> Dafna >>>> >>>> >>>> _______________________________________________ >>>> Devel mailing list >>>> Devel(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/devel >>> > = > = --===============4825524077343505997==-- From bkorren at redhat.com Thu Sep 21 06:26:15 2017 Content-Type: multipart/mixed; boundary="===============0761872571488581844==" MIME-Version: 1.0 From: Barak Korren To: devel at ovirt.org Subject: Re: [ovirt-devel] Subject: [ OST Failure Report ] [ master ] [20-09-2017 ] [ post-002_bootstrap.py ] Date: Thu, 21 Sep 2017 09:26:12 +0300 Message-ID: In-Reply-To: 6a94e41f-4d14-353a-26e2-ac471490b652@redhat.com --===============0761872571488581844== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable =D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A 20 =D7=91=D7=A1=D7=A4=D7=98=D7=B3 2017= 05:44 PM,=E2=80=8F "Juan Hern=C3=A1ndez" =D7=9B=D7= =AA=D7=91: I am aware of that build failure. The tests of the *Ruby* SDK hang apparently random in ppc64le environments. I am trying to find out why, but not making much progress, to be honest. Meanwhile I wonder why this is a failure to run OST tests, as the Ruby SDK isn't used at all in those tests, as far as I know. This is not actually failing OST, its making CQ not run OST at all, since it verifies builds pass before trying. Barak Korren bkorren(a)redhat.com RHCE, RHCi, RHV-DevOps Team https://ifireball.wordpress.com/ >> --===============0761872571488581844== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGRpdiBkaXI9ImF1dG8iPjxkaXY+PGJyPjxkaXYgZGF0YS1zbWFydG1haWw9ImdtYWlsX3NpZ25h dHVyZSI+PGJyPjwvZGl2PjxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj48YnI+PGRpdiBjbGFzcz0i Z21haWxfcXVvdGUiPteR16rXkNeo15nXmiAyMCDXkdeh16TXmNezIDIwMTcgMDU6NDQgUE0s4oCP ICZxdW90O0p1YW4gSGVybsOhbmRleiZxdW90OyAmbHQ7PGEgaHJlZj0ibWFpbHRvOmpoZXJuYW5k QHJlZGhhdC5jb20iPmpoZXJuYW5kQHJlZGhhdC5jb208L2E+Jmd0OyDXm9eq15E6PGJyIHR5cGU9 ImF0dHJpYnV0aW9uIj48YmxvY2txdW90ZSBjbGFzcz0icXVvdGUiIHN0eWxlPSJtYXJnaW46MCAw IDAgLjhleDtib3JkZXItbGVmdDoxcHggI2NjYyBzb2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij5JIGFt IGF3YXJlIG9mIHRoYXQgYnVpbGQgZmFpbHVyZS4gVGhlIHRlc3RzIG9mIHRoZSAqUnVieSogU0RL IGhhbmcgYXBwYXJlbnRseSByYW5kb20gaW4gcHBjNjRsZSBlbnZpcm9ubWVudHMuIEkgYW0gdHJ5 aW5nIHRvIGZpbmQgb3V0IHdoeSwgYnV0IG5vdCBtYWtpbmcgbXVjaCBwcm9ncmVzcywgdG8gYmUg aG9uZXN0Ljxicj4KPGJyPgpNZWFud2hpbGUgSSB3b25kZXIgd2h5IHRoaXMgaXMgYSBmYWlsdXJl IHRvIHJ1biBPU1QgdGVzdHMsIGFzIHRoZSBSdWJ5IFNESyBpc24mIzM5O3QgdXNlZCBhdCBhbGwg aW4gdGhvc2UgdGVzdHMsIGFzIGZhciBhcyBJIGtub3cuPC9ibG9ja3F1b3RlPjwvZGl2PjwvZGl2 PjwvZGl2PjxkaXYgZGlyPSJhdXRvIj48YnI+PC9kaXY+PGRpdiBkaXI9ImF1dG8iPlRoaXMgaXMg bm90IGFjdHVhbGx5IGZhaWxpbmcgT1NULCBpdHMgbWFraW5nIENRIG5vdCBydW4gT1NUIGF0IGFs bCwgc2luY2UgaXQgdmVyaWZpZXMgYnVpbGRzIHBhc3MgYmVmb3JlIHRyeWluZy48L2Rpdj48ZGl2 IGRpcj0iYXV0byI+PGJyPjwvZGl2PjxkaXYgZGlyPSJhdXRvIj48c3BhbiBzdHlsZT0iZm9udC1m YW1pbHk6c2Fucy1zZXJpZiI+QmFyYWsgS29ycmVuPC9zcGFuPjxiciBzdHlsZT0iZm9udC1mYW1p bHk6c2Fucy1zZXJpZiI+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OnNhbnMtc2VyaWYiPjxhIGhy ZWY9Im1haWx0bzpia29ycmVuQHJlZGhhdC5jb20iPmJrb3JyZW5AcmVkaGF0LmNvbTwvYT48L3Nw YW4+PGJyIHN0eWxlPSJmb250LWZhbWlseTpzYW5zLXNlcmlmIj48c3BhbiBzdHlsZT0iZm9udC1m YW1pbHk6c2Fucy1zZXJpZiI+UkhDRSwgUkhDaSwgUkhWLURldk9wcyBUZWFtPC9zcGFuPjxiciBz dHlsZT0iZm9udC1mYW1pbHk6c2Fucy1zZXJpZiI+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OnNh bnMtc2VyaWYiPjxhIGhyZWY9Imh0dHBzOi8vaWZpcmViYWxsLndvcmRwcmVzcy5jb20vIj5odHRw czovL2lmaXJlYmFsbC53b3JkcHJlc3MuY29tLzwvYT48L3NwYW4+PGJyPjwvZGl2PjxkaXYgZGly PSJhdXRvIj48ZGl2IGNsYXNzPSJnbWFpbF9leHRyYSI+PGRpdiBjbGFzcz0iZ21haWxfcXVvdGUi PjxibG9ja3F1b3RlIGNsYXNzPSJxdW90ZSIgc3R5bGU9Im1hcmdpbjowIDAgMCAuOGV4O2JvcmRl ci1sZWZ0OjFweCAjY2NjIHNvbGlkO3BhZGRpbmctbGVmdDoxZXgiPjxibG9ja3F1b3RlIGNsYXNz PSJnbWFpbF9xdW90ZSIgc3R5bGU9Im1hcmdpbjowIDAgMCAuOGV4O2JvcmRlci1sZWZ0OjFweCAj Y2NjIHNvbGlkO3BhZGRpbmctbGVmdDoxZXgiPjxkaXYgY2xhc3M9ImVsaWRlZC10ZXh0Ij48Ymxv Y2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MCAwIDAgLjhleDtib3Jk ZXItbGVmdDoxcHggI2NjYyBzb2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij48L2Jsb2NrcXVvdGU+PC9k aXY+PC9ibG9ja3F1b3RlPjwvYmxvY2txdW90ZT48L2Rpdj48L2Rpdj48L2Rpdj48ZGl2IGRpcj0i YXV0byI+PGRpdiBjbGFzcz0iZ21haWxfZXh0cmEiPjxkaXYgY2xhc3M9ImdtYWlsX3F1b3RlIj48 YmxvY2txdW90ZSBjbGFzcz0icXVvdGUiIHN0eWxlPSJtYXJnaW46MCAwIDAgLjhleDtib3JkZXIt bGVmdDoxcHggI2NjYyBzb2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij48YmxvY2txdW90ZSBjbGFzcz0i Z21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MCAwIDAgLjhleDtib3JkZXItbGVmdDoxcHggI2Nj YyBzb2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij48ZGl2IGNsYXNzPSJlbGlkZWQtdGV4dCI+PGJsb2Nr cXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjAgMCAwIC44ZXg7Ym9yZGVy LWxlZnQ6MXB4ICNjY2Mgc29saWQ7cGFkZGluZy1sZWZ0OjFleCI+PGJyPjwvYmxvY2txdW90ZT48 L2Rpdj48L2Jsb2NrcXVvdGU+PC9ibG9ja3F1b3RlPjwvZGl2PjwvZGl2PjwvZGl2PjwvZGl2Pgo= --===============0761872571488581844==--