Hi everyone, We have the test-repo_ovirt_experimental_master job failing since build 717: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/717/ System tests are also starting to fail: http://jenkins.ovirt.org/job/ovirt_master_system-tests/380/ Here's the list of patches which appeared in build 717: https://gerrit.ovirt.org/#/c/61384/ - Sandro Bonazzola - packaging: spec: drop default defattr https://gerrit.ovirt.org/#/c/62026/ - Yaniv Bronhaim - Using %{_libdir} macro instead of /usr/lib https://gerrit.ovirt.org/#/c/62028/ - Yaniv Bronhaim - Require python2-devel specifically to avoid python3 pkg https://gerrit.ovirt.org/#/c/62049/ - Fabian Deutsch - Revert "imgbase: Drop journal support" https://gerrit.ovirt.org/#/c/62050/ - Fabian Deutsch - cli: Add journal logging The error is related to secondary storage domain addition, it sounds like a known bug but now it seems to be triggered every time. I'm not sure how we can identify the source of the regression since these tests run on repos of already built RPMs, so the complete set has to be re-built with a specific patch and tested to find out the offending one. Regards, Evgheni Dereveanchin
On Wed, Aug 10, 2016 at 2:51 PM, Evgheni Dereveanchin <ederevea@redhat.com> wrote:
Hi everyone,
We have the test-repo_ovirt_experimental_master job failing since build 717: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/717/
System tests are also starting to fail: http://jenkins.ovirt.org/job/ovirt_master_system-tests/380/
Here's the list of patches which appeared in build 717: https://gerrit.ovirt.org/#/c/61384/ - Sandro Bonazzola - packaging: spec: drop default defattr https://gerrit.ovirt.org/#/c/62026/ - Yaniv Bronhaim - Using %{_libdir} macro instead of /usr/lib https://gerrit.ovirt.org/#/c/62028/ - Yaniv Bronhaim - Require python2-devel specifically to avoid python3 pkg https://gerrit.ovirt.org/#/c/62049/ - Fabian Deutsch - Revert "imgbase: Drop journal support" https://gerrit.ovirt.org/#/c/62050/ - Fabian Deutsch - cli: Add journal logging
On VDSM log order is: * 0a10329 - Require python2-devel specifically to avoid python3 pkg (27 ore fa) <Yaniv Bronhaim> * bdf3b9a - packaging: spec: drop default defattr (27 ore fa) <Sandro Bonazzola> * 271dd9b - Using %{_libdir} macro instead of /usr/lib (27 ore fa) <Yaniv Bronhaim> So let's start with: https://gerrit.ovirt.org/#/c/62026/ - Yaniv Bronhaim - Using %{_libdir} macro instead of /usr/lib Before merge: Build Successful: http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/1017/ : SUCCESS http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/5127/ : SUCCESS Once merged: Build Failed http://jenkins.ovirt.org/job/vdsm_master_check-merged-fc24-x86_64/174/ : FAILURE http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-el7-ppc64le/53/ : FAILURE http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/21/ : FAILURE http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-fc24-x86_64/151/ : SUCCESS http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-el7-x86_64/649/ : SUCCESS Let's exclude the ppc64le job for now. FC24 failed on functional testing: *00:06:43.766* # Deploy environment: *00:06:43.767* * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: *00:10:05.208* - STDERR*00:10:05.208* Failed to synchronize cache for repo 'localsync', disabling.*00:10:05.208* Error: Unable to find a match.*00:10:05.208* *00:10:05.209* * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: ERROR (in 0:03:21)*00:10:05.211* Error while running thread*00:10:05.211* Traceback (most recent call last):*00:10:05.211* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue*00:10:05.211* queue.put({'return': func()})*00:10:05.211* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host*00:10:05.211* host.name(),*00:10:05.211* RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24*00:10:05.213* # Deploy environment: ERROR (in 0:03:21)*00:10:05.498* @ Deploy oVirt environment: ERROR (in 0:03:21)*00:10:05.499* Error occured, aborting*00:10:05.499* Traceback (most recent call last):*00:10:05.499* File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 258, in do_run*00:10:05.499* self.cli_plugins[args.ovirtverb].do_run(args)*00:10:05.499* File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 180, in do_run*00:10:05.499* self._do_run(**vars(args))*00:10:05.499* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 488, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 499, in wrapper*00:10:05.500* return func(*args, prefix=prefix, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 184, in do_deploy*00:10:05.500* prefix.deploy()*00:10:05.500* File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 64, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.501* File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 351, in deploy*00:10:05.501* return super(OvirtPrefix, self).deploy()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper*00:10:05.501* return func(*args, **kwargs)*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1258, in deploy*00:10:05.501* self._deploy_host, self.virt_env.get_vms().values()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 95, in invoke_in_parallel*00:10:05.501* vt.join_all()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue*00:10:05.502* queue.put({'return': func()})*00:10:05.502* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host*00:10:05.502* host.name(),*00:10:05.502* RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24 el7 failed on functional testing too: *00:06:43.734* @ Deploy oVirt environment: *00:06:43.739* # ovirt-role metadata entry will be soon deprecated, instead you should use the vm-provider entry in the domain definiton and set it no one of: ovirt-node, ovirt-engine, ovirt-host*00:06:43.766* # Deploy environment: *00:06:43.767* * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: *00:10:05.208* - STDERR*00:10:05.208* Failed to synchronize cache for repo 'localsync', disabling.*00:10:05.208* Error: Unable to find a match.*00:10:05.208* *00:10:05.209* * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: ERROR (in 0:03:21)*00:10:05.211* Error while running thread*00:10:05.211* Traceback (most recent call last):*00:10:05.211* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue*00:10:05.211* queue.put({'return': func()})*00:10:05.211* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host*00:10:05.211* host.name(),*00:10:05.211* RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24*00:10:05.213* # Deploy environment: ERROR (in 0:03:21)*00:10:05.498* @ Deploy oVirt environment: ERROR (in 0:03:21)*00:10:05.499* Error occured, aborting*00:10:05.499* Traceback (most recent call last):*00:10:05.499* File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 258, in do_run*00:10:05.499* self.cli_plugins[args.ovirtverb].do_run(args)*00:10:05.499* File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 180, in do_run*00:10:05.499* self._do_run(**vars(args))*00:10:05.499* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 488, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 499, in wrapper*00:10:05.500* return func(*args, prefix=prefix, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 184, in do_deploy*00:10:05.500* prefix.deploy()*00:10:05.500* File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 64, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.501* File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 351, in deploy*00:10:05.501* return super(OvirtPrefix, self).deploy()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper*00:10:05.501* return func(*args, **kwargs)*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1258, in deploy*00:10:05.501* self._deploy_host, self.virt_env.get_vms().values()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 95, in invoke_in_parallel*00:10:05.501* vt.join_all()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue*00:10:05.502* queue.put({'return': func()})*00:10:05.502* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host*00:10:05.502* host.name(),*00:10:05.502* RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24 Can you please check why above error happened?
The error is related to secondary storage domain addition, it sounds like a known bug but now it seems to be triggered every time.
I'm not sure how we can identify the source of the regression since these tests run on repos of already built RPMs, so the complete set has to be re-built with a specific patch and tested to find out the offending one.
Regards, Evgheni Dereveanchin
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
Sandro, check-merged and build-artifacts failed jobs are not related to Yaniv's patch. check-merged-fc24 : known issue, Gal Ben Haim is checking it. check-merged-el7 and build artifacts el7-ppc jobs were created recently (apparently by mistake) and should be removed. On Wed, Aug 10, 2016 at 4:11 PM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
On Wed, Aug 10, 2016 at 2:51 PM, Evgheni Dereveanchin <ederevea@redhat.com
wrote:
Hi everyone,
We have the test-repo_ovirt_experimental_master job failing since build 717: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/717/
System tests are also starting to fail: http://jenkins.ovirt.org/job/ovirt_master_system-tests/380/
Here's the list of patches which appeared in build 717: https://gerrit.ovirt.org/#/c/61384/ - Sandro Bonazzola - packaging: spec: drop default defattr https://gerrit.ovirt.org/#/c/62026/ - Yaniv Bronhaim - Using %{_libdir} macro instead of /usr/lib https://gerrit.ovirt.org/#/c/62028/ - Yaniv Bronhaim - Require python2-devel specifically to avoid python3 pkg https://gerrit.ovirt.org/#/c/62049/ - Fabian Deutsch - Revert "imgbase: Drop journal support" https://gerrit.ovirt.org/#/c/62050/ - Fabian Deutsch - cli: Add journal logging
On VDSM log order is:
* 0a10329 - Require python2-devel specifically to avoid python3 pkg (27 ore fa) <Yaniv Bronhaim> * bdf3b9a - packaging: spec: drop default defattr (27 ore fa) <Sandro Bonazzola> * 271dd9b - Using %{_libdir} macro instead of /usr/lib (27 ore fa) <Yaniv Bronhaim>
So let's start with: https://gerrit.ovirt.org/#/c/62026/ - Yaniv Bronhaim - Using %{_libdir} macro instead of /usr/lib
Before merge: Build Successful:
http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/1017/ : SUCCESS http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/5127/ : SUCCESS
Once merged: Build Failed http://jenkins.ovirt.org/job/vdsm_master_check-merged-fc24-x86_64/174/ : FAILURE http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-el7-ppc64le/53/ : FAILURE http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/21/ : FAILURE http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-fc24-x86_64/151/ : SUCCESS http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-el7-x86_64/649/ : SUCCESS
Let's exclude the ppc64le job for now. FC24 failed on functional testing:
*00:06:43.766* # Deploy environment: *00:06:43.767* * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: *00:10:05.208* - STDERR*00:10:05.208* Failed to synchronize cache for repo 'localsync', disabling.*00:10:05.208* Error: Unable to find a match.*00:10:05.208* *00:10:05.209* * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: ERROR (in 0:03:21)*00:10:05.211* Error while running thread*00:10:05.211* Traceback (most recent call last):*00:10:05.211* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue*00:10:05.211* queue.put({'return': func()})*00:10:05.211* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host*00:10:05.211* host.name(),*00:10:05.211* RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24*00:10:05.213* # Deploy environment: ERROR (in 0:03:21)*00:10:05.498* @ Deploy oVirt environment: ERROR (in 0:03:21)*00:10:05.499* Error occured, aborting*00:10:05.499* Traceback (most recent call last):*00:10:05.499* File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 258, in do_run*00:10:05.499* self.cli_plugins[args.ovirtverb].do_run(args)*00:10:05.499* File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 180, in do_run*00:10:05.499* self._do_run(**vars(args))*00:10:05.499* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 488, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 499, in wrapper*00:10:05.500* return func(*args, prefix=prefix, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 184, in do_deploy*00:10:05.500* prefix.deploy()*00:10:05.500* File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 64, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.501* File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 351, in deploy*00:10:05.501* return super(OvirtPrefix, self).deploy()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper*00:10:05.501* return func(*args, **kwargs)*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1258, in deploy*00:10:05.501* self._deploy_host, self.virt_env.get_vms().values()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 95, in invoke_in_parallel*00:10:05.501* vt.join_all()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue*00:10:05.502* queue.put({'return': func()})*00:10:05.502* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host*00:10:05.502* host.name(),*00:10:05.502* RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24
el7 failed on functional testing too:
*00:06:43.734* @ Deploy oVirt environment: *00:06:43.739* # ovirt-role metadata entry will be soon deprecated, instead you should use the vm-provider entry in the domain definiton and set it no one of: ovirt-node, ovirt-engine, ovirt-host*00:06:43.766* # Deploy environment: *00:06:43.767* * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: *00:10:05.208* - STDERR*00:10:05.208* Failed to synchronize cache for repo 'localsync', disabling.*00:10:05.208* Error: Unable to find a match.*00:10:05.208* *00:10:05.209* * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: ERROR (in 0:03:21)*00:10:05.211* Error while running thread*00:10:05.211* Traceback (most recent call last):*00:10:05.211* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue*00:10:05.211* queue.put({'return': func()})*00:10:05.211* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host*00:10:05.211* host.name(),*00:10:05.211* RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24*00:10:05.213* # Deploy environment: ERROR (in 0:03:21)*00:10:05.498* @ Deploy oVirt environment: ERROR (in 0:03:21)*00:10:05.499* Error occured, aborting*00:10:05.499* Traceback (most recent call last):*00:10:05.499* File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 258, in do_run*00:10:05.499* self.cli_plugins[args.ovirtverb].do_run(args)*00:10:05.499* File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 180, in do_run*00:10:05.499* self._do_run(**vars(args))*00:10:05.499* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 488, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 499, in wrapper*00:10:05.500* return func(*args, prefix=prefix, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 184, in do_deploy*00:10:05.500* prefix.deploy()*00:10:05.500* File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 64, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.501* File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 351, in deploy*00:10:05.501* return super(OvirtPrefix, self).deploy()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper*00:10:05.501* return func(*args, **kwargs)*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1258, in deploy*00:10:05.501* self._deploy_host, self.virt_env.get_vms().values()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 95, in invoke_in_parallel*00:10:05.501* vt.join_all()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue*00:10:05.502* queue.put({'return': func()})*00:10:05.502* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host*00:10:05.502* host.name(),*00:10:05.502* RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24
Can you please check why above error happened?
The error is related to secondary storage domain addition, it sounds like a known bug but now it seems to be triggered every time.
I'm not sure how we can identify the source of the regression since these tests run on repos of already built RPMs, so the complete set has to be re-built with a specific patch and tested to find out the offending one.
Regards, Evgheni Dereveanchin
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Irit Goihman Software Engineer Red Hat Israel Ltd.
On Wed, Aug 10, 2016 at 3:27 PM, Irit Goihman <igoihman@redhat.com> wrote:
Sandro, check-merged and build-artifacts failed jobs are not related to Yaniv's patch. check-merged-fc24 : known issue, Gal Ben Haim is checking it. check-merged-el7 and build artifacts el7-ppc jobs were created recently (apparently by mistake) and should be removed.
So, basically we have no check-merged job for vdsm master right?
On Wed, Aug 10, 2016 at 4:11 PM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
On Wed, Aug 10, 2016 at 2:51 PM, Evgheni Dereveanchin < ederevea@redhat.com> wrote:
Hi everyone,
We have the test-repo_ovirt_experimental_master job failing since build 717: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/717/
System tests are also starting to fail: http://jenkins.ovirt.org/job/ovirt_master_system-tests/380/
Here's the list of patches which appeared in build 717: https://gerrit.ovirt.org/#/c/61384/ - Sandro Bonazzola - packaging: spec: drop default defattr https://gerrit.ovirt.org/#/c/62026/ - Yaniv Bronhaim - Using %{_libdir} macro instead of /usr/lib https://gerrit.ovirt.org/#/c/62028/ - Yaniv Bronhaim - Require python2-devel specifically to avoid python3 pkg https://gerrit.ovirt.org/#/c/62049/ - Fabian Deutsch - Revert "imgbase: Drop journal support" https://gerrit.ovirt.org/#/c/62050/ - Fabian Deutsch - cli: Add journal logging
On VDSM log order is:
* 0a10329 - Require python2-devel specifically to avoid python3 pkg (27 ore fa) <Yaniv Bronhaim> * bdf3b9a - packaging: spec: drop default defattr (27 ore fa) <Sandro Bonazzola> * 271dd9b - Using %{_libdir} macro instead of /usr/lib (27 ore fa) <Yaniv Bronhaim>
So let's start with: https://gerrit.ovirt.org/#/c/62026/ - Yaniv Bronhaim - Using %{_libdir} macro instead of /usr/lib
Before merge: Build Successful:
http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/1017/ : SUCCESS http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/5127/ : SUCCESS
Once merged: Build Failed http://jenkins.ovirt.org/job/vdsm_master_check-merged-fc24-x86_64/174/ : FAILURE http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-el7-ppc64le/53/ : FAILURE http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/21/ : FAILURE http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-fc24-x86_64/151/ : SUCCESS http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-el7-x86_64/649/ : SUCCESS
Let's exclude the ppc64le job for now. FC24 failed on functional testing:
*00:06:43.766* # Deploy environment: *00:06:43.767* * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: *00:10:05.208* - STDERR*00:10:05.208* Failed to synchronize cache for repo 'localsync', disabling.*00:10:05.208* Error: Unable to find a match.*00:10:05.208* *00:10:05.209* * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: ERROR (in 0:03:21)*00:10:05.211* Error while running thread*00:10:05.211* Traceback (most recent call last):*00:10:05.211* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue*00:10:05.211* queue.put({'return': func()})*00:10:05.211* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host*00:10:05.211* host.name(),*00:10:05.211* RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24*00:10:05.213* # Deploy environment: ERROR (in 0:03:21)*00:10:05.498* @ Deploy oVirt environment: ERROR (in 0:03:21)*00:10:05.499* Error occured, aborting*00:10:05.499* Traceback (most recent call last):*00:10:05.499* File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 258, in do_run*00:10:05.499* self.cli_plugins[args.ovirtverb].do_run(args)*00:10:05.499* File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 180, in do_run*00:10:05.499* self._do_run(**vars(args))*00:10:05.499* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 488, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 499, in wrapper*00:10:05.500* return func(*args, prefix=prefix, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 184, in do_deploy*00:10:05.500* prefix.deploy()*00:10:05.500* File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 64, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.501* File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 351, in deploy*00:10:05.501* return super(OvirtPrefix, self).deploy()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper*00:10:05.501* return func(*args, **kwargs)*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1258, in deploy*00:10:05.501* self._deploy_host, self.virt_env.get_vms().values()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 95, in invoke_in_parallel*00:10:05.501* vt.join_all()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue*00:10:05.502* queue.put({'return': func()})*00:10:05.502* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host*00:10:05.502* host.name(),*00:10:05.502* RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24
el7 failed on functional testing too:
*00:06:43.734* @ Deploy oVirt environment: *00:06:43.739* # ovirt-role metadata entry will be soon deprecated, instead you should use the vm-provider entry in the domain definiton and set it no one of: ovirt-node, ovirt-engine, ovirt-host*00:06:43.766* # Deploy environment: *00:06:43.767* * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: *00:10:05.208* - STDERR*00:10:05.208* Failed to synchronize cache for repo 'localsync', disabling.*00:10:05.208* Error: Unable to find a match.*00:10:05.208* *00:10:05.209* * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: ERROR (in 0:03:21)*00:10:05.211* Error while running thread*00:10:05.211* Traceback (most recent call last):*00:10:05.211* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue*00:10:05.211* queue.put({'return': func()})*00:10:05.211* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host*00:10:05.211* host.name(),*00:10:05.211* RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24*00:10:05.213* # Deploy environment: ERROR (in 0:03:21)*00:10:05.498* @ Deploy oVirt environment: ERROR (in 0:03:21)*00:10:05.499* Error occured, aborting*00:10:05.499* Traceback (most recent call last):*00:10:05.499* File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 258, in do_run*00:10:05.499* self.cli_plugins[args.ovirtverb].do_run(args)*00:10:05.499* File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 180, in do_run*00:10:05.499* self._do_run(**vars(args))*00:10:05.499* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 488, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 499, in wrapper*00:10:05.500* return func(*args, prefix=prefix, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 184, in do_deploy*00:10:05.500* prefix.deploy()*00:10:05.500* File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 64, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.501* File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 351, in deploy*00:10:05.501* return super(OvirtPrefix, self).deploy()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper*00:10:05.501* return func(*args, **kwargs)*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1258, in deploy*00:10:05.501* self._deploy_host, self.virt_env.get_vms().values()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 95, in invoke_in_parallel*00:10:05.501* vt.join_all()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue*00:10:05.502* queue.put({'return': func()})*00:10:05.502* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host*00:10:05.502* host.name(),*00:10:05.502* RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24
Can you please check why above error happened?
The error is related to secondary storage domain addition, it sounds like a known bug but now it seems to be triggered every time.
I'm not sure how we can identify the source of the regression since these tests run on repos of already built RPMs, so the complete set has to be re-built with a specific patch and tested to find out the offending one.
Regards, Evgheni Dereveanchin
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Irit Goihman Software Engineer Red Hat Israel Ltd.
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
On Wed, Aug 10, 2016 at 3:32 PM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
On Wed, Aug 10, 2016 at 3:27 PM, Irit Goihman <igoihman@redhat.com> wrote:
Sandro, check-merged and build-artifacts failed jobs are not related to Yaniv's patch. check-merged-fc24 : known issue, Gal Ben Haim is checking it.
Gal, can you elaborateo n what's going on there?
check-merged-el7 and build artifacts el7-ppc jobs were created recently
(apparently by mistake) and should be removed.
So, basically we have no check-merged job for vdsm master right?
On Wed, Aug 10, 2016 at 4:11 PM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
On Wed, Aug 10, 2016 at 2:51 PM, Evgheni Dereveanchin < ederevea@redhat.com> wrote:
Hi everyone,
We have the test-repo_ovirt_experimental_master job failing since build 717: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/717/
System tests are also starting to fail: http://jenkins.ovirt.org/job/ovirt_master_system-tests/380/
Here's the list of patches which appeared in build 717: https://gerrit.ovirt.org/#/c/61384/ - Sandro Bonazzola - packaging: spec: drop default defattr https://gerrit.ovirt.org/#/c/62026/ - Yaniv Bronhaim - Using %{_libdir} macro instead of /usr/lib https://gerrit.ovirt.org/#/c/62028/ - Yaniv Bronhaim - Require python2-devel specifically to avoid python3 pkg https://gerrit.ovirt.org/#/c/62049/ - Fabian Deutsch - Revert "imgbase: Drop journal support" https://gerrit.ovirt.org/#/c/62050/ - Fabian Deutsch - cli: Add journal logging
On VDSM log order is:
* 0a10329 - Require python2-devel specifically to avoid python3 pkg (27 ore fa) <Yaniv Bronhaim> * bdf3b9a - packaging: spec: drop default defattr (27 ore fa) <Sandro Bonazzola> * 271dd9b - Using %{_libdir} macro instead of /usr/lib (27 ore fa) <Yaniv Bronhaim>
So let's start with: https://gerrit.ovirt.org/#/c/62026/ - Yaniv Bronhaim - Using %{_libdir} macro instead of /usr/lib
Before merge: Build Successful:
http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/1017/ : SUCCESS http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/5127/ : SUCCESS
Once merged: Build Failed http://jenkins.ovirt.org/job/vdsm_master_check-merged-fc24-x86_64/174/ : FAILURE http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-el7-ppc64le/53/ : FAILURE http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/21/ : FAILURE http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-fc2 4-x86_64/151/ : SUCCESS http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-el7-x86_64/649/ : SUCCESS
Let's exclude the ppc64le job for now. FC24 failed on functional testing:
*00:06:43.766* # Deploy environment: *00:06:43.767* * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: *00:10:05.208* - STDERR*00:10:05.208* Failed to synchronize cache for repo 'localsync', disabling.*00:10:05.208* Error: Unable to find a match.*00:10:05.208* *00:10:05.209* * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: ERROR (in 0:03:21)*00:10:05.211* Error while running thread*00:10:05.211* Traceback (most recent call last):*00:10:05.211* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue*00:10:05.211* queue.put({'return': func()})*00:10:05.211* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host*00:10:05.211* host.name(),*00:10:05.211* RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24*00:10:05.213* # Deploy environment: ERROR (in 0:03:21)*00:10:05.498* @ Deploy oVirt environment: ERROR (in 0:03:21)*00:10:05.499* Error occured, aborting*00:10:05.499* Traceback (most recent call last):*00:10:05.499* File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 258, in do_run*00:10:05.499* self.cli_plugins[args.ovirtverb].do_run(args)*00:10:05.499* File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 180, in do_run*00:10:05.499* self._do_run(**vars(args))*00:10:05.499* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 488, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 499, in wrapper*00:10:05.500* return func(*args, prefix=prefix, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 184, in do_deploy*00:10:05.500* prefix.deploy()*00:10:05.500* File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 64, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.501* File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 351, in deploy*00:10:05.501* return super(OvirtPrefix, self).deploy()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper*00:10:05.501* return func(*args, **kwargs)*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1258, in deploy*00:10:05.501* self._deploy_host, self.virt_env.get_vms().values()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 95, in invoke_in_parallel*00:10:05.501* vt.join_all()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue*00:10:05.502* queue.put({'return': func()})*00:10:05.502* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host*00:10:05.502* host.name(),*00:10:05.502* RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24
el7 failed on functional testing too:
*00:06:43.734* @ Deploy oVirt environment: *00:06:43.739* # ovirt-role metadata entry will be soon deprecated, instead you should use the vm-provider entry in the domain definiton and set it no one of: ovirt-node, ovirt-engine, ovirt-host*00:06:43.766* # Deploy environment: *00:06:43.767* * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: *00:10:05.208* - STDERR*00:10:05.208* Failed to synchronize cache for repo 'localsync', disabling.*00:10:05.208* Error: Unable to find a match.*00:10:05.208* *00:10:05.209* * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: ERROR (in 0:03:21)*00:10:05.211* Error while running thread*00:10:05.211* Traceback (most recent call last):*00:10:05.211* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue*00:10:05.211* queue.put({'return': func()})*00:10:05.211* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host*00:10:05.211* host.name(),*00:10:05.211* RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24*00:10:05.213* # Deploy environment: ERROR (in 0:03:21)*00:10:05.498* @ Deploy oVirt environment: ERROR (in 0:03:21)*00:10:05.499* Error occured, aborting*00:10:05.499* Traceback (most recent call last):*00:10:05.499* File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 258, in do_run*00:10:05.499* self.cli_plugins[args.ovirtverb].do_run(args)*00:10:05.499* File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 180, in do_run*00:10:05.499* self._do_run(**vars(args))*00:10:05.499* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 488, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 499, in wrapper*00:10:05.500* return func(*args, prefix=prefix, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 184, in do_deploy*00:10:05.500* prefix.deploy()*00:10:05.500* File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 64, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.501* File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 351, in deploy*00:10:05.501* return super(OvirtPrefix, self).deploy()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper*00:10:05.501* return func(*args, **kwargs)*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1258, in deploy*00:10:05.501* self._deploy_host, self.virt_env.get_vms().values()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 95, in invoke_in_parallel*00:10:05.501* vt.join_all()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue*00:10:05.502* queue.put({'return': func()})*00:10:05.502* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host*00:10:05.502* host.name(),*00:10:05.502* RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24
Can you please check why above error happened?
The error is related to secondary storage domain addition, it sounds like a known bug but now it seems to be triggered every time.
I'm not sure how we can identify the source of the regression since these tests run on repos of already built RPMs, so the complete set has to be re-built with a specific patch and tested to find out the offending one.
Regards, Evgheni Dereveanchin
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Irit Goihman Software Engineer Red Hat Israel Ltd.
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
On Wed, Aug 10, 2016 at 4:32 PM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
On Wed, Aug 10, 2016 at 3:27 PM, Irit Goihman <igoihman@redhat.com> wrote:
Sandro, check-merged and build-artifacts failed jobs are not related to Yaniv's patch. check-merged-fc24 : known issue, Gal Ben Haim is checking it. check-merged-el7 and build artifacts el7-ppc jobs were created recently (apparently by mistake) and should be removed.
So, basically we have no check-merged job for vdsm master right?
We have check-merged fc24 job which fails occasionally and not stable right now.
On Wed, Aug 10, 2016 at 4:11 PM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
On Wed, Aug 10, 2016 at 2:51 PM, Evgheni Dereveanchin < ederevea@redhat.com> wrote:
Hi everyone,
We have the test-repo_ovirt_experimental_master job failing since build 717: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/717/
System tests are also starting to fail: http://jenkins.ovirt.org/job/ovirt_master_system-tests/380/
Here's the list of patches which appeared in build 717: https://gerrit.ovirt.org/#/c/61384/ - Sandro Bonazzola - packaging: spec: drop default defattr https://gerrit.ovirt.org/#/c/62026/ - Yaniv Bronhaim - Using %{_libdir} macro instead of /usr/lib https://gerrit.ovirt.org/#/c/62028/ - Yaniv Bronhaim - Require python2-devel specifically to avoid python3 pkg https://gerrit.ovirt.org/#/c/62049/ - Fabian Deutsch - Revert "imgbase: Drop journal support" https://gerrit.ovirt.org/#/c/62050/ - Fabian Deutsch - cli: Add journal logging
On VDSM log order is:
* 0a10329 - Require python2-devel specifically to avoid python3 pkg (27 ore fa) <Yaniv Bronhaim> * bdf3b9a - packaging: spec: drop default defattr (27 ore fa) <Sandro Bonazzola> * 271dd9b - Using %{_libdir} macro instead of /usr/lib (27 ore fa) <Yaniv Bronhaim>
So let's start with: https://gerrit.ovirt.org/#/c/62026/ - Yaniv Bronhaim - Using %{_libdir} macro instead of /usr/lib
Before merge: Build Successful:
http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/1017/ : SUCCESS http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/5127/ : SUCCESS
Once merged: Build Failed http://jenkins.ovirt.org/job/vdsm_master_check-merged-fc24-x86_64/174/ : FAILURE http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-el7-ppc64le/53/ : FAILURE http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/21/ : FAILURE http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-fc2 4-x86_64/151/ : SUCCESS http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-el7-x86_64/649/ : SUCCESS
Let's exclude the ppc64le job for now. FC24 failed on functional testing:
*00:06:43.766* # Deploy environment: *00:06:43.767* * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: *00:10:05.208* - STDERR*00:10:05.208* Failed to synchronize cache for repo 'localsync', disabling.*00:10:05.208* Error: Unable to find a match.*00:10:05.208* *00:10:05.209* * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: ERROR (in 0:03:21)*00:10:05.211* Error while running thread*00:10:05.211* Traceback (most recent call last):*00:10:05.211* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue*00:10:05.211* queue.put({'return': func()})*00:10:05.211* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host*00:10:05.211* host.name(),*00:10:05.211* RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24*00:10:05.213* # Deploy environment: ERROR (in 0:03:21)*00:10:05.498* @ Deploy oVirt environment: ERROR (in 0:03:21)*00:10:05.499* Error occured, aborting*00:10:05.499* Traceback (most recent call last):*00:10:05.499* File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 258, in do_run*00:10:05.499* self.cli_plugins[args.ovirtverb].do_run(args)*00:10:05.499* File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 180, in do_run*00:10:05.499* self._do_run(**vars(args))*00:10:05.499* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 488, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 499, in wrapper*00:10:05.500* return func(*args, prefix=prefix, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 184, in do_deploy*00:10:05.500* prefix.deploy()*00:10:05.500* File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 64, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.501* File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 351, in deploy*00:10:05.501* return super(OvirtPrefix, self).deploy()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper*00:10:05.501* return func(*args, **kwargs)*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1258, in deploy*00:10:05.501* self._deploy_host, self.virt_env.get_vms().values()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 95, in invoke_in_parallel*00:10:05.501* vt.join_all()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue*00:10:05.502* queue.put({'return': func()})*00:10:05.502* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host*00:10:05.502* host.name(),*00:10:05.502* RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24
el7 failed on functional testing too:
*00:06:43.734* @ Deploy oVirt environment: *00:06:43.739* # ovirt-role metadata entry will be soon deprecated, instead you should use the vm-provider entry in the domain definiton and set it no one of: ovirt-node, ovirt-engine, ovirt-host*00:06:43.766* # Deploy environment: *00:06:43.767* * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: *00:10:05.208* - STDERR*00:10:05.208* Failed to synchronize cache for repo 'localsync', disabling.*00:10:05.208* Error: Unable to find a match.*00:10:05.208* *00:10:05.209* * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: ERROR (in 0:03:21)*00:10:05.211* Error while running thread*00:10:05.211* Traceback (most recent call last):*00:10:05.211* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue*00:10:05.211* queue.put({'return': func()})*00:10:05.211* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host*00:10:05.211* host.name(),*00:10:05.211* RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24*00:10:05.213* # Deploy environment: ERROR (in 0:03:21)*00:10:05.498* @ Deploy oVirt environment: ERROR (in 0:03:21)*00:10:05.499* Error occured, aborting*00:10:05.499* Traceback (most recent call last):*00:10:05.499* File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 258, in do_run*00:10:05.499* self.cli_plugins[args.ovirtverb].do_run(args)*00:10:05.499* File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 180, in do_run*00:10:05.499* self._do_run(**vars(args))*00:10:05.499* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 488, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 499, in wrapper*00:10:05.500* return func(*args, prefix=prefix, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 184, in do_deploy*00:10:05.500* prefix.deploy()*00:10:05.500* File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.500* File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 64, in wrapper*00:10:05.500* return func(*args, **kwargs)*00:10:05.501* File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 351, in deploy*00:10:05.501* return super(OvirtPrefix, self).deploy()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper*00:10:05.501* return func(*args, **kwargs)*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1258, in deploy*00:10:05.501* self._deploy_host, self.virt_env.get_vms().values()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 95, in invoke_in_parallel*00:10:05.501* vt.join_all()*00:10:05.501* File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue*00:10:05.502* queue.put({'return': func()})*00:10:05.502* File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host*00:10:05.502* host.name(),*00:10:05.502* RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24
Can you please check why above error happened?
The error is related to secondary storage domain addition, it sounds like a known bug but now it seems to be triggered every time.
I'm not sure how we can identify the source of the regression since these tests run on repos of already built RPMs, so the complete set has to be re-built with a specific patch and tested to find out the offending one.
Regards, Evgheni Dereveanchin
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Irit Goihman Software Engineer Red Hat Israel Ltd.
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
-- Irit Goihman Software Engineer Red Hat Israel Ltd.
On Wed, Aug 10, 2016 at 3:51 PM, Evgheni Dereveanchin <ederevea@redhat.com> wrote:
Hi everyone,
We have the test-repo_ovirt_experimental_master job failing since build 717: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/717/
System tests are also starting to fail: http://jenkins.ovirt.org/job/ovirt_master_system-tests/380/
Here's the list of patches which appeared in build 717: https://gerrit.ovirt.org/#/c/61384/ - Sandro Bonazzola - packaging: spec: drop default defattr https://gerrit.ovirt.org/#/c/62026/ - Yaniv Bronhaim - Using %{_libdir} macro instead of /usr/lib https://gerrit.ovirt.org/#/c/62028/ - Yaniv Bronhaim - Require python2-devel specifically to avoid python3 pkg https://gerrit.ovirt.org/#/c/62049/ - Fabian Deutsch - Revert "imgbase: Drop journal support" https://gerrit.ovirt.org/#/c/62050/ - Fabian Deutsch - cli: Add journal logging
The error is related to secondary storage domain addition,
Note that I (ab)use the secondary storage domains addition (as it takes time) to do other things. For example, run ovirt-log-collector. So one needs to look at the error carefully. Y.
it sounds like a known bug but now it seems to be triggered every time.
I'm not sure how we can identify the source of the regression since these tests run on repos of already built RPMs, so the complete set has to be re-built with a specific patch and tested to find out the offending one.
Regards, Evgheni Dereveanchin
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On Wed, Aug 10, 2016 at 11:26 PM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Wed, Aug 10, 2016 at 3:51 PM, Evgheni Dereveanchin <ederevea@redhat.com> wrote:
Hi everyone,
We have the test-repo_ovirt_experimental_master job failing since build 717: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/717/
System tests are also starting to fail: http://jenkins.ovirt.org/job/ovirt_master_system-tests/380/
Here's the list of patches which appeared in build 717: https://gerrit.ovirt.org/#/c/61384/ - Sandro Bonazzola - packaging: spec: drop default defattr https://gerrit.ovirt.org/#/c/62026/ - Yaniv Bronhaim - Using %{_libdir} macro instead of /usr/lib
This patch breakss block storage - vms with disks on block storage will fail with this error: Could not open '/rhev/data-center/...': Permission denied We have this https://gerrit.ovirt.org/#/c/62196/, should be resolved tomorrow.
https://gerrit.ovirt.org/#/c/62028/ - Yaniv Bronhaim - Require python2-devel specifically to avoid python3 pkg https://gerrit.ovirt.org/#/c/62049/ - Fabian Deutsch - Revert "imgbase: Drop journal support" https://gerrit.ovirt.org/#/c/62050/ - Fabian Deutsch - cli: Add journal logging
The error is related to secondary storage domain addition,
Note that I (ab)use the secondary storage domains addition (as it takes time) to do other things. For example, run ovirt-log-collector. So one needs to look at the error carefully. Y.
it sounds like a known bug but now it seems to be triggered every time.
I'm not sure how we can identify the source of the regression since these tests run on repos of already built RPMs, so the complete set has to be re-built with a specific patch and tested to find out the offending one.
Regards, Evgheni Dereveanchin
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
participants (5)
-
Evgheni Dereveanchin -
Irit Goihman -
Nir Soffer -
Sandro Bonazzola -
Yaniv Kaul