On Wed, Aug 10, 2016 at 2:51 PM, Evgheni Dereveanchin <ederevea@redhat.com> wrote:
Hi everyone,

We have the test-repo_ovirt_experimental_master job failing since build 717:
http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/717/

System tests are also starting to fail:
http://jenkins.ovirt.org/job/ovirt_master_system-tests/380/

Here's the list of patches which appeared in build 717:
https://gerrit.ovirt.org/#/c/61384/ - Sandro Bonazzola - packaging: spec: drop default defattr
https://gerrit.ovirt.org/#/c/62026/ - Yaniv Bronhaim   - Using %{_libdir} macro instead of /usr/lib
https://gerrit.ovirt.org/#/c/62028/ - Yaniv Bronhaim   - Require python2-devel specifically to avoid python3 pkg
https://gerrit.ovirt.org/#/c/62049/ - Fabian Deutsch   - Revert "imgbase: Drop journal support"
https://gerrit.ovirt.org/#/c/62050/ - Fabian Deutsch   - cli: Add journal logging


On VDSM log order is:

* 0a10329 - Require python2-devel specifically to avoid python3 pkg (27 ore fa) <Yaniv Bronhaim>
* bdf3b9a - packaging: spec: drop default defattr (27 ore fa) <Sandro Bonazzola>
* 271dd9b - Using %{_libdir} macro instead of /usr/lib (27 ore fa) <Yaniv Bronhaim>

So let's start with: https://gerrit.ovirt.org/#/c/62026/ - Yaniv Bronhaim   - Using %{_libdir} macro instead of /usr/lib
Before merge: Build Successful:
http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/1017/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/5127/ : SUCCESS

Once merged: Build Failed
http://jenkins.ovirt.org/job/vdsm_master_check-merged-fc24-x86_64/174/ : FAILURE
http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-el7-ppc64le/53/ : FAILURE
http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/21/ : FAILURE
http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-fc24-x86_64/151/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-el7-x86_64/649/ : SUCCESS

Let's exclude the ppc64le job for now.
FC24 failed on functional testing:
00:06:43.766   # Deploy environment: 
00:06:43.767     * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: 
00:10:05.208       - STDERR
00:10:05.208 Failed to synchronize cache for repo 'localsync', disabling.
00:10:05.208 Error: Unable to find a match.
00:10:05.208 
00:10:05.209     * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: ERROR (in 0:03:21)
00:10:05.211 Error while running thread
00:10:05.211 Traceback (most recent call last):
00:10:05.211   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue
00:10:05.211     queue.put({'return': func()})
00:10:05.211   File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host
00:10:05.211     host.name(),
00:10:05.211 RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24
00:10:05.213   # Deploy environment: ERROR (in 0:03:21)
00:10:05.498 @ Deploy oVirt environment: ERROR (in 0:03:21)
00:10:05.499 Error occured, aborting
00:10:05.499 Traceback (most recent call last):
00:10:05.499   File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 258, in do_run
00:10:05.499     self.cli_plugins[args.ovirtverb].do_run(args)
00:10:05.499   File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 180, in do_run
00:10:05.499     self._do_run(**vars(args))
00:10:05.499   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 488, in wrapper
00:10:05.500     return func(*args, **kwargs)
00:10:05.500   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 499, in wrapper
00:10:05.500     return func(*args, prefix=prefix, **kwargs)
00:10:05.500   File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 184, in do_deploy
00:10:05.500     prefix.deploy()
00:10:05.500   File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper
00:10:05.500     return func(*args, **kwargs)
00:10:05.500   File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 64, in wrapper
00:10:05.500     return func(*args, **kwargs)
00:10:05.501   File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 351, in deploy
00:10:05.501     return super(OvirtPrefix, self).deploy()
00:10:05.501   File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper
00:10:05.501     return func(*args, **kwargs)
00:10:05.501   File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1258, in deploy
00:10:05.501     self._deploy_host, self.virt_env.get_vms().values()
00:10:05.501   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 95, in invoke_in_parallel
00:10:05.501     vt.join_all()
00:10:05.501   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue
00:10:05.502     queue.put({'return': func()})
00:10:05.502   File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host
00:10:05.502     host.name(),
00:10:05.502 RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24

el7 failed on functional testing too:

00:06:43.734 @ Deploy oVirt environment: 
00:06:43.739   # ovirt-role metadata entry will be soon deprecated, instead you should use the vm-provider entry in the domain definiton and set it no one of: ovirt-node, ovirt-engine, ovirt-host
00:06:43.766   # Deploy environment: 
00:06:43.767     * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: 
00:10:05.208       - STDERR
00:10:05.208 Failed to synchronize cache for repo 'localsync', disabling.
00:10:05.208 Error: Unable to find a match.
00:10:05.208 
00:10:05.209     * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: ERROR (in 0:03:21)
00:10:05.211 Error while running thread
00:10:05.211 Traceback (most recent call last):
00:10:05.211   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue
00:10:05.211     queue.put({'return': func()})
00:10:05.211   File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host
00:10:05.211     host.name(),
00:10:05.211 RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24
00:10:05.213   # Deploy environment: ERROR (in 0:03:21)
00:10:05.498 @ Deploy oVirt environment: ERROR (in 0:03:21)
00:10:05.499 Error occured, aborting
00:10:05.499 Traceback (most recent call last):
00:10:05.499   File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 258, in do_run
00:10:05.499     self.cli_plugins[args.ovirtverb].do_run(args)
00:10:05.499   File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 180, in do_run
00:10:05.499     self._do_run(**vars(args))
00:10:05.499   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 488, in wrapper
00:10:05.500     return func(*args, **kwargs)
00:10:05.500   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 499, in wrapper
00:10:05.500     return func(*args, prefix=prefix, **kwargs)
00:10:05.500   File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 184, in do_deploy
00:10:05.500     prefix.deploy()
00:10:05.500   File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper
00:10:05.500     return func(*args, **kwargs)
00:10:05.500   File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 64, in wrapper
00:10:05.500     return func(*args, **kwargs)
00:10:05.501   File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 351, in deploy
00:10:05.501     return super(OvirtPrefix, self).deploy()
00:10:05.501   File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper
00:10:05.501     return func(*args, **kwargs)
00:10:05.501   File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1258, in deploy
00:10:05.501     self._deploy_host, self.virt_env.get_vms().values()
00:10:05.501   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 95, in invoke_in_parallel
00:10:05.501     vt.join_all()
00:10:05.501   File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue
00:10:05.502     queue.put({'return': func()})
00:10:05.502   File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host
00:10:05.502     host.name(),
00:10:05.502 RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24


Can you please check why above error happened?

 
The error is related to secondary storage domain addition,
it sounds like a known bug but now it seems to be triggered every time.

I'm not sure how we can identify the source of the regression
since these tests run on repos of already built RPMs, so the
complete set has to be re-built with a specific patch and tested
to find out the offending one.

Regards,
Evgheni Dereveanchin

_______________________________________________
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel



--
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com