Odd errors on vdsm_unit_tests
Jenkins used to send email notifications about unit tests failures to vdsm-patches@lists.fedorahosted.org at least until March 20. Come to think of it, the emails seized to have content on March 5. Now they are not sent at all. Beyond this, we have a very odd non-deterministic failure http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1976/console (except below). Could the test be run with an unusual locale? Does anybody have another idea? Could someone (Giuseppe? David?) log into this fedora18-vm01 slave and try to reproduce the issue from the command line? Regards, Dan. ====================================================================== ERROR: test_deviceCustomProperties (hooksTests.TestHooks) ---------------------------------------------------------------------- Traceback (most recent call last): File "/jenkins-workspaces/vdsm_unit_tests_gerrit/tests/hooksTests.py", line 125, in test_deviceCustomProperties params={'customProperty': ' rocks!'}) File "/jenkins-workspaces/vdsm_unit_tests_gerrit/vdsm/hooks.py", line 72, in _runHooksDir scriptenv[k] = unicode(v).encode('utf-8') UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 7: ordinal not in range(128) ====================================================================== ERROR: test_runHooksDir (hooksTests.TestHooks) ---------------------------------------------------------------------- Traceback (most recent call last): File "/jenkins-workspaces/vdsm_unit_tests_gerrit/tests/hooksTests.py", line 72, in test_runHooksDir res = hooks._runHooksDir(DOMXML, dirName) File "/jenkins-workspaces/vdsm_unit_tests_gerrit/vdsm/hooks.py", line 72, in _runHooksDir scriptenv[k] = unicode(v).encode('utf-8') UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 7: ordinal not in range(128)
On Thu 11 Apr 2013 11:22:00 PM CEST, Dan Kenigsberg wrote:
Jenkins used to send email notifications about unit tests failures to vdsm-patches@lists.fedorahosted.org at least until March 20.
Come to think of it, the emails seized to have content on March 5.
Now they are not sent at all.
I think I'm not on that list, I'll sign up and look at it, see if I can figure out something.
Beyond this, we have a very odd non-deterministic failure http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1976/console (except below).
Could the test be run with an unusual locale? Does anybody have another idea?
Could someone (Giuseppe? David?) log into this fedora18-vm01 slave and try to reproduce the issue from the command line?
Regards, Dan.
====================================================================== ERROR: test_deviceCustomProperties (hooksTests.TestHooks) ---------------------------------------------------------------------- Traceback (most recent call last): File "/jenkins-workspaces/vdsm_unit_tests_gerrit/tests/hooksTests.py", line 125, in test_deviceCustomProperties params={'customProperty': ' rocks!'}) File "/jenkins-workspaces/vdsm_unit_tests_gerrit/vdsm/hooks.py", line 72, in _runHooksDir scriptenv[k] = unicode(v).encode('utf-8') UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 7: ordinal not in range(128)
====================================================================== ERROR: test_runHooksDir (hooksTests.TestHooks) ---------------------------------------------------------------------- Traceback (most recent call last): File "/jenkins-workspaces/vdsm_unit_tests_gerrit/tests/hooksTests.py", line 72, in test_runHooksDir res = hooks._runHooksDir(DOMXML, dirName) File "/jenkins-workspaces/vdsm_unit_tests_gerrit/vdsm/hooks.py", line 72, in _runHooksDir scriptenv[k] = unicode(v).encode('utf-8') UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 7: ordinal not in range(128)
About that error, yes, Me and Antoni Segura (apuimedo) troubleshot it, the problem is that one of the developers has non ascii letter on it's name, passing it to the author tag on gerrit, that loads it on GERRIT_*_AUTHOR env variable. Not sure if antoni is working on a patch or notified someone though. -- David Caro Red Hat Czech s.r.o. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.cz.redhat.com Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno, Czech Republic RHT Global #: 82-62605
On Thu 11 Apr 2013 11:36:27 PM CEST, David Caro wrote:
On Thu 11 Apr 2013 11:22:00 PM CEST, Dan Kenigsberg wrote:
Jenkins used to send email notifications about unit tests failures to vdsm-patches@lists.fedorahosted.org at least until March 20.
Come to think of it, the emails seized to have content on March 5.
Now they are not sent at all.
I think I'm not on that list, I'll sign up and look at it, see if I can figure out something.
Beyond this, we have a very odd non-deterministic failure http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1976/console (except below).
Could the test be run with an unusual locale? Does anybody have another idea?
Could someone (Giuseppe? David?) log into this fedora18-vm01 slave and try to reproduce the issue from the command line?
Regards, Dan.
====================================================================== ERROR: test_deviceCustomProperties (hooksTests.TestHooks) ---------------------------------------------------------------------- Traceback (most recent call last): File "/jenkins-workspaces/vdsm_unit_tests_gerrit/tests/hooksTests.py", line 125, in test_deviceCustomProperties params={'customProperty': ' rocks!'}) File "/jenkins-workspaces/vdsm_unit_tests_gerrit/vdsm/hooks.py", line 72, in _runHooksDir scriptenv[k] = unicode(v).encode('utf-8') UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 7: ordinal not in range(128)
====================================================================== ERROR: test_runHooksDir (hooksTests.TestHooks) ---------------------------------------------------------------------- Traceback (most recent call last): File "/jenkins-workspaces/vdsm_unit_tests_gerrit/tests/hooksTests.py", line 72, in test_runHooksDir res = hooks._runHooksDir(DOMXML, dirName) File "/jenkins-workspaces/vdsm_unit_tests_gerrit/vdsm/hooks.py", line 72, in _runHooksDir scriptenv[k] = unicode(v).encode('utf-8') UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 7: ordinal not in range(128)
About that error, yes, Me and Antoni Segura (apuimedo) troubleshot it, the problem is that one of the developers has non ascii letter on it's name, passing it to the author tag on gerrit, that loads it on GERRIT_*_AUTHOR env variable. Not sure if antoni is working on a patch or notified someone though.
-- David Caro
Red Hat Czech s.r.o. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.cz.redhat.com Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno, Czech Republic RHT Global #: 82-62605 _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Forgot to mention, the error that bugs me is this one: http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1962/console We had several of those today and it all seemed to stop when I set the core dump limit to 'unlimited' (ulimit -c). I saw that before with one patch but subsequent patches worked, so I did not investigate further. As far as I know, the problem is that the test testIsSuperUp somehow end up killing the jenkins slave process (if you run it manually you get a message that says 'killed' in the console). That test seems to mess with threads and processes and maybe it just identifies the slave process as the one to stop (just a wild guess :S). -- David Caro Red Hat Czech s.r.o. Continuous Integration Engineer - EMEA ENG Virtualization R&D Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.cz.redhat.com Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno, Czech Republic RHT Global #: 82-62605
On Thu, Apr 11, 2013 at 11:47:52PM +0200, David Caro wrote:
On Thu 11 Apr 2013 11:36:27 PM CEST, David Caro wrote:
On Thu 11 Apr 2013 11:22:00 PM CEST, Dan Kenigsberg wrote:
Jenkins used to send email notifications about unit tests failures to vdsm-patches@lists.fedorahosted.org at least until March 20.
Come to think of it, the emails seized to have content on March 5.
Now they are not sent at all.
I think I'm not on that list, I'll sign up and look at it, see if I can figure out something.
Beyond this, we have a very odd non-deterministic failure http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1976/console (except below).
Could the test be run with an unusual locale? Does anybody have another idea?
Could someone (Giuseppe? David?) log into this fedora18-vm01 slave and try to reproduce the issue from the command line?
Regards, Dan.
====================================================================== ERROR: test_deviceCustomProperties (hooksTests.TestHooks) ---------------------------------------------------------------------- Traceback (most recent call last): File "/jenkins-workspaces/vdsm_unit_tests_gerrit/tests/hooksTests.py", line 125, in test_deviceCustomProperties params={'customProperty': ' rocks!'}) File "/jenkins-workspaces/vdsm_unit_tests_gerrit/vdsm/hooks.py", line 72, in _runHooksDir scriptenv[k] = unicode(v).encode('utf-8') UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 7: ordinal not in range(128)
====================================================================== ERROR: test_runHooksDir (hooksTests.TestHooks) ---------------------------------------------------------------------- Traceback (most recent call last): File "/jenkins-workspaces/vdsm_unit_tests_gerrit/tests/hooksTests.py", line 72, in test_runHooksDir res = hooks._runHooksDir(DOMXML, dirName) File "/jenkins-workspaces/vdsm_unit_tests_gerrit/vdsm/hooks.py", line 72, in _runHooksDir scriptenv[k] = unicode(v).encode('utf-8') UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 7: ordinal not in range(128)
About that error, yes, Me and Antoni Segura (apuimedo) troubleshot it, the problem is that one of the developers has non ascii letter on it's name, passing it to the author tag on gerrit, that loads it on GERRIT_*_AUTHOR env variable. Not sure if antoni is working on a patch or notified someone though.
-- David Caro
Red Hat Czech s.r.o. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.cz.redhat.com Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno, Czech Republic RHT Global #: 82-62605 _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Forgot to mention, the error that bugs me is this one:
http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1962/console
We had several of those today and it all seemed to stop when I set the core dump limit to 'unlimited' (ulimit -c). I saw that before with one patch but subsequent patches worked, so I did not investigate further. As far as I know, the problem is that the test testIsSuperUp somehow end up killing the jenkins slave process (if you run it manually you get a message that says 'killed' in the console). That test seems to mess with threads and processes and maybe it just identifies the slave process as the one to stop (just a wild guess :S).
Yeah, I've noticed this bug in testIsSuperUp, and Yaniv Bronheim is aware of it. Dan.
Right, If supervdsmServer.py could not start when executing (wrong import for example), testIsSuperUp (first ut of supervdsmTests) does not return and you end up with "killed".. It happens because we had logic that after 3 retries of communicating with supervdsmServer fail we kill the running process http://gerrit.ovirt.org/11932 (Normally vdsm process, in this case the testrunner process is killed). First, http://gerrit.ovirt.org/#/c/13804/ - this fix the issue that causes supervdsmServer to fail. Second, http://gerrit.ovirt.org/#/c/13878/ - check if supervdsmServer starts as expected, if not the test will straightly fail. Third, I'll submit a change for the logic of communicating with supervdsmServer.. I prefer to avoid the panic call. at least i can monkeyPatch it during the tests. Thanks, Yaniv. ----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "David Caro" <dcaroest@redhat.com>, "Yaniv Bronheim" <ybronhei@redhat.com> Cc: "Giuseppe Vallarelli" <gvallare@redhat.com>, infra@ovirt.org Sent: Sunday, April 14, 2013 11:28:54 AM Subject: Re: Odd errors on vdsm_unit_tests
On Thu, Apr 11, 2013 at 11:47:52PM +0200, David Caro wrote:
On Thu 11 Apr 2013 11:36:27 PM CEST, David Caro wrote:
On Thu 11 Apr 2013 11:22:00 PM CEST, Dan Kenigsberg wrote:
Jenkins used to send email notifications about unit tests failures to vdsm-patches@lists.fedorahosted.org at least until March 20.
Come to think of it, the emails seized to have content on March 5.
Now they are not sent at all.
I think I'm not on that list, I'll sign up and look at it, see if I can figure out something.
Beyond this, we have a very odd non-deterministic failure http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1976/console (except below).
Could the test be run with an unusual locale? Does anybody have another idea?
Could someone (Giuseppe? David?) log into this fedora18-vm01 slave and try to reproduce the issue from the command line?
Regards, Dan.
====================================================================== ERROR: test_deviceCustomProperties (hooksTests.TestHooks) ---------------------------------------------------------------------- Traceback (most recent call last): File "/jenkins-workspaces/vdsm_unit_tests_gerrit/tests/hooksTests.py", line 125, in test_deviceCustomProperties params={'customProperty': ' rocks!'}) File "/jenkins-workspaces/vdsm_unit_tests_gerrit/vdsm/hooks.py", line 72, in _runHooksDir scriptenv[k] = unicode(v).encode('utf-8') UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 7: ordinal not in range(128)
====================================================================== ERROR: test_runHooksDir (hooksTests.TestHooks) ---------------------------------------------------------------------- Traceback (most recent call last): File "/jenkins-workspaces/vdsm_unit_tests_gerrit/tests/hooksTests.py", line 72, in test_runHooksDir res = hooks._runHooksDir(DOMXML, dirName) File "/jenkins-workspaces/vdsm_unit_tests_gerrit/vdsm/hooks.py", line 72, in _runHooksDir scriptenv[k] = unicode(v).encode('utf-8') UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 7: ordinal not in range(128)
About that error, yes, Me and Antoni Segura (apuimedo) troubleshot it, the problem is that one of the developers has non ascii letter on it's name, passing it to the author tag on gerrit, that loads it on GERRIT_*_AUTHOR env variable. Not sure if antoni is working on a patch or notified someone though.
-- David Caro
Red Hat Czech s.r.o. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.cz.redhat.com Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno, Czech Republic RHT Global #: 82-62605 _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Forgot to mention, the error that bugs me is this one:
http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/1962/console
We had several of those today and it all seemed to stop when I set the core dump limit to 'unlimited' (ulimit -c). I saw that before with one patch but subsequent patches worked, so I did not investigate further. As far as I know, the problem is that the test testIsSuperUp somehow end up killing the jenkins slave process (if you run it manually you get a message that says 'killed' in the console). That test seems to mess with threads and processes and maybe it just identifies the slave process as the one to stop (just a wild guess :S).
Yeah, I've noticed this bug in testIsSuperUp, and Yaniv Bronheim is aware of it.
Dan.
participants (3)
-
Dan Kenigsberg -
David Caro -
Yaniv Bronheim