<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 10, 2016 at 2:51 PM, Evgheni Dereveanchin <span dir="ltr"><<a href="mailto:ederevea@redhat.com" target="_blank">ederevea@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi everyone,<br>
<br>
We have the test-repo_ovirt_experimental_<wbr>master job failing since build 717:<br>
<a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/717/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>test-repo_ovirt_experimental_<wbr>master/717/</a><br>
<br>
System tests are also starting to fail:<br>
<a href="http://jenkins.ovirt.org/job/ovirt_master_system-tests/380/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>ovirt_master_system-tests/380/</a><br>
<br>
Here's the list of patches which appeared in build 717:<br>
<a href="https://gerrit.ovirt.org/#/c/61384/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>61384/</a> - Sandro Bonazzola - packaging: spec: drop default defattr<br>
<a href="https://gerrit.ovirt.org/#/c/62026/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>62026/</a> - Yaniv Bronhaim - Using %{_libdir} macro instead of /usr/lib<br>
<a href="https://gerrit.ovirt.org/#/c/62028/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>62028/</a> - Yaniv Bronhaim - Require python2-devel specifically to avoid python3 pkg<br>
<a href="https://gerrit.ovirt.org/#/c/62049/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>62049/</a> - Fabian Deutsch - Revert "imgbase: Drop journal support"<br>
<a href="https://gerrit.ovirt.org/#/c/62050/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>62050/</a> - Fabian Deutsch - cli: Add journal logging<br>
<br></blockquote><div><br></div><div>On VDSM log order is:</div><div><br></div><div><div>* 0a10329 - Require python2-devel specifically to avoid python3 pkg (27 ore fa) <Yaniv Bronhaim></div><div>* bdf3b9a - packaging: spec: drop default defattr (27 ore fa) <Sandro Bonazzola></div><div>* 271dd9b - Using %{_libdir} macro instead of /usr/lib (27 ore fa) <Yaniv Bronhaim></div><div><br></div></div><div>So let's start with: <a href="https://gerrit.ovirt.org/#/c/62026/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>62026/</a> - Yaniv Bronhaim - Using %{_libdir} macro instead of /usr/lib</div><div><span style="color:rgb(53,53,53);font-family:sans-serif;white-space:pre-wrap">Before merge: Build Successful:</span></div><div><a href="http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/1017/" target="_blank" style="font-family:sans-serif;white-space:pre-wrap;color:rgb(6,84,172);text-decoration:none">http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/1017/</a><span style="color:rgb(53,53,53);font-family:sans-serif;white-space:pre-wrap"> : SUCCESS</span></div><div><a href="http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/5127/" target="_blank" style="font-family:sans-serif;white-space:pre-wrap;color:rgb(6,84,172);text-decoration:none">http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/5127/</a><span style="color:rgb(53,53,53);font-family:sans-serif;white-space:pre-wrap"> : SUCCESS</span></div><div><br></div><div>Once merged: <span style="color:rgb(53,53,53);font-family:sans-serif;white-space:pre-wrap">Build Failed </span></div><div><a href="http://jenkins.ovirt.org/job/vdsm_master_check-merged-fc24-x86_64/174/" target="_blank" style="font-family:sans-serif;white-space:pre-wrap;color:rgb(6,84,172);text-decoration:none">http://jenkins.ovirt.org/job/vdsm_master_check-merged-fc24-x86_64/174/</a><span style="color:rgb(53,53,53);font-family:sans-serif;white-space:pre-wrap"> : FAILURE</span></div><div><a href="http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-el7-ppc64le/53/" target="_blank" style="font-family:sans-serif;white-space:pre-wrap;color:rgb(6,84,172);text-decoration:none">http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-el7-ppc64le/53/</a><span style="color:rgb(53,53,53);font-family:sans-serif;white-space:pre-wrap"> : FAILURE</span></div><div><a href="http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/21/" target="_blank" style="font-family:sans-serif;white-space:pre-wrap;color:rgb(6,84,172);text-decoration:none">http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/21/</a><span style="color:rgb(53,53,53);font-family:sans-serif;white-space:pre-wrap"> : FAILURE</span></div><div><a href="http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-fc24-x86_64/151/" target="_blank" style="font-family:sans-serif;white-space:pre-wrap;color:rgb(6,84,172);text-decoration:none">http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-fc24-x86_64/151/</a><span style="color:rgb(53,53,53);font-family:sans-serif;white-space:pre-wrap"> : SUCCESS</span></div><div><a href="http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-el7-x86_64/649/" target="_blank" style="font-family:sans-serif;white-space:pre-wrap;color:rgb(6,84,172);text-decoration:none">http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-el7-x86_64/649/</a><span style="color:rgb(53,53,53);font-family:sans-serif;white-space:pre-wrap"> : SUCCESS</span></div><div><br></div><div>Let's exclude the ppc64le job for now.</div><div>FC24 failed on functional testing:</div><div><pre class="" style="white-space:pre-wrap;word-wrap:break-word;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:13px"><span class="" style=""><b style="">00:06:43.766</b> </span><span style="color:rgb(0,205,205)"> # Deploy environment: </span>
<span class="" style=""><b style="">00:06:43.767</b> </span><span style="color:rgb(0,205,205)"> * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: </span>
<span class="" style=""><b style="">00:10:05.208</b> </span><span style="color:rgb(205,0,0)"> - STDERR
<span class="" style=""><b style="">00:10:05.208</b> </span>Failed to synchronize cache for repo 'localsync', disabling.
<span class="" style=""><b style="">00:10:05.208</b> </span>Error: Unable to find a match.
<span class="" style=""><b style="">00:10:05.208</b> </span></span>
<span class="" style=""><b style="">00:10:05.209</b> </span><span style="color:rgb(0,205,205)"> * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: </span><span style="color:rgb(205,0,0)">ERROR</span> (in 0:03:21)
<span class="" style=""><b style="">00:10:05.211</b> </span><span style="color:rgb(205,0,0)">Error while running thread
<span class="" style=""><b style="">00:10:05.211</b> </span>Traceback (most recent call last):
<span class="" style=""><b style="">00:10:05.211</b> </span> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue
<span class="" style=""><b style="">00:10:05.211</b> </span> queue.put({'return': func()})
<span class="" style=""><b style="">00:10:05.211</b> </span> File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host
<span class="" style=""><b style="">00:10:05.211</b> </span> <a href="http://host.name">host.name</a>(),
<span class="" style=""><b style="">00:10:05.211</b> </span>RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24</span>
<span class="" style=""><b style="">00:10:05.213</b> </span><span style="color:rgb(0,205,205)"> # Deploy environment: </span><span style="color:rgb(205,0,0)">ERROR</span> (in 0:03:21)
<span class="" style=""><b style="">00:10:05.498</b> </span><span style="color:rgb(0,205,205)">@ Deploy oVirt environment: </span><span style="color:rgb(205,0,0)">ERROR</span> (in 0:03:21)
<span class="" style=""><b style="">00:10:05.499</b> </span><span style="color:rgb(205,0,0)">Error occured, aborting
<span class="" style=""><b style="">00:10:05.499</b> </span>Traceback (most recent call last):
<span class="" style=""><b style="">00:10:05.499</b> </span> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 258, in do_run
<span class="" style=""><b style="">00:10:05.499</b> </span> self.cli_plugins[args.ovirtverb].do_run(args)
<span class="" style=""><b style="">00:10:05.499</b> </span> File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 180, in do_run
<span class="" style=""><b style="">00:10:05.499</b> </span> self._do_run(**vars(args))
<span class="" style=""><b style="">00:10:05.499</b> </span> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 488, in wrapper
<span class="" style=""><b style="">00:10:05.500</b> </span> return func(*args, **kwargs)
<span class="" style=""><b style="">00:10:05.500</b> </span> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 499, in wrapper
<span class="" style=""><b style="">00:10:05.500</b> </span> return func(*args, prefix=prefix, **kwargs)
<span class="" style=""><b style="">00:10:05.500</b> </span> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 184, in do_deploy
<span class="" style=""><b style="">00:10:05.500</b> </span> prefix.deploy()
<span class="" style=""><b style="">00:10:05.500</b> </span> File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper
<span class="" style=""><b style="">00:10:05.500</b> </span> return func(*args, **kwargs)
<span class="" style=""><b style="">00:10:05.500</b> </span> File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 64, in wrapper
<span class="" style=""><b style="">00:10:05.500</b> </span> return func(*args, **kwargs)
<span class="" style=""><b style="">00:10:05.501</b> </span> File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 351, in deploy
<span class="" style=""><b style="">00:10:05.501</b> </span> return super(OvirtPrefix, self).deploy()
<span class="" style=""><b style="">00:10:05.501</b> </span> File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper
<span class="" style=""><b style="">00:10:05.501</b> </span> return func(*args, **kwargs)
<span class="" style=""><b style="">00:10:05.501</b> </span> File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1258, in deploy
<span class="" style=""><b style="">00:10:05.501</b> </span> self._deploy_host, self.virt_env.get_vms().values()
<span class="" style=""><b style="">00:10:05.501</b> </span> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 95, in invoke_in_parallel
<span class="" style=""><b style="">00:10:05.501</b> </span> vt.join_all()
<span class="" style=""><b style="">00:10:05.501</b> </span> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue
<span class="" style=""><b style="">00:10:05.502</b> </span> queue.put({'return': func()})
<span class="" style=""><b style="">00:10:05.502</b> </span> File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host
<span class="" style=""><b style="">00:10:05.502</b> </span> <a href="http://host.name">host.name</a>(),
<span class="" style=""><b style="">00:10:05.502</b> </span>RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24</span></pre><pre class="" style="white-space:pre-wrap;word-wrap:break-word;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:13px"><span style="color:rgb(205,0,0)"><br></span></pre>el7 failed on functional testing too:</div><div><br><pre class="" style="white-space:pre-wrap;word-wrap:break-word;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:13px"><pre class="" style="white-space:pre-wrap;word-wrap:break-word;margin-top:0px;margin-bottom:0px"><span class="" style=""><b style="">00:06:43.734</b> </span><span style="color:rgb(0,205,205)">@ Deploy oVirt environment: </span>
<span class="" style=""><b style="">00:06:43.739</b> </span><span style="color:rgb(205,205,0)"> # ovirt-role metadata entry will be soon deprecated, instead you should use the vm-provider entry in the domain definiton and set it no one of: ovirt-node, ovirt-engine, ovirt-host</span>
<span class="" style=""><b style="">00:06:43.766</b> </span><span style="color:rgb(0,205,205)"> # Deploy environment: </span>
<span class="" style=""><b style="">00:06:43.767</b> </span><span style="color:rgb(0,205,205)"> * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: </span>
<span class="" style=""><b style="">00:10:05.208</b> </span><span style="color:rgb(205,0,0)"> - STDERR
<span class="" style=""><b style="">00:10:05.208</b> </span>Failed to synchronize cache for repo 'localsync', disabling.
<span class="" style=""><b style="">00:10:05.208</b> </span>Error: Unable to find a match.
<span class="" style=""><b style="">00:10:05.208</b> </span></span>
<span class="" style=""><b style="">00:10:05.209</b> </span><span style="color:rgb(0,205,205)"> * [Thread-2] Deploy VM vdsm_functional_tests_host-fc24: </span><span style="color:rgb(205,0,0)">ERROR</span> (in 0:03:21)
<span class="" style=""><b style="">00:10:05.211</b> </span><span style="color:rgb(205,0,0)">Error while running thread
<span class="" style=""><b style="">00:10:05.211</b> </span>Traceback (most recent call last):
<span class="" style=""><b style="">00:10:05.211</b> </span> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue
<span class="" style=""><b style="">00:10:05.211</b> </span> queue.put({'return': func()})
<span class="" style=""><b style="">00:10:05.211</b> </span> File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host
<span class="" style=""><b style="">00:10:05.211</b> </span> <a href="http://host.name">host.name</a>(),
<span class="" style=""><b style="">00:10:05.211</b> </span>RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24</span>
<span class="" style=""><b style="">00:10:05.213</b> </span><span style="color:rgb(0,205,205)"> # Deploy environment: </span><span style="color:rgb(205,0,0)">ERROR</span> (in 0:03:21)
<span class="" style=""><b style="">00:10:05.498</b> </span><span style="color:rgb(0,205,205)">@ Deploy oVirt environment: </span><span style="color:rgb(205,0,0)">ERROR</span> (in 0:03:21)
<span class="" style=""><b style="">00:10:05.499</b> </span><span style="color:rgb(205,0,0)">Error occured, aborting
<span class="" style=""><b style="">00:10:05.499</b> </span>Traceback (most recent call last):
<span class="" style=""><b style="">00:10:05.499</b> </span> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 258, in do_run
<span class="" style=""><b style="">00:10:05.499</b> </span> self.cli_plugins[args.ovirtverb].do_run(args)
<span class="" style=""><b style="">00:10:05.499</b> </span> File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 180, in do_run
<span class="" style=""><b style="">00:10:05.499</b> </span> self._do_run(**vars(args))
<span class="" style=""><b style="">00:10:05.499</b> </span> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 488, in wrapper
<span class="" style=""><b style="">00:10:05.500</b> </span> return func(*args, **kwargs)
<span class="" style=""><b style="">00:10:05.500</b> </span> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 499, in wrapper
<span class="" style=""><b style="">00:10:05.500</b> </span> return func(*args, prefix=prefix, **kwargs)
<span class="" style=""><b style="">00:10:05.500</b> </span> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 184, in do_deploy
<span class="" style=""><b style="">00:10:05.500</b> </span> prefix.deploy()
<span class="" style=""><b style="">00:10:05.500</b> </span> File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper
<span class="" style=""><b style="">00:10:05.500</b> </span> return func(*args, **kwargs)
<span class="" style=""><b style="">00:10:05.500</b> </span> File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 64, in wrapper
<span class="" style=""><b style="">00:10:05.500</b> </span> return func(*args, **kwargs)
<span class="" style=""><b style="">00:10:05.501</b> </span> File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 351, in deploy
<span class="" style=""><b style="">00:10:05.501</b> </span> return super(OvirtPrefix, self).deploy()
<span class="" style=""><b style="">00:10:05.501</b> </span> File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper
<span class="" style=""><b style="">00:10:05.501</b> </span> return func(*args, **kwargs)
<span class="" style=""><b style="">00:10:05.501</b> </span> File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1258, in deploy
<span class="" style=""><b style="">00:10:05.501</b> </span> self._deploy_host, self.virt_env.get_vms().values()
<span class="" style=""><b style="">00:10:05.501</b> </span> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 95, in invoke_in_parallel
<span class="" style=""><b style="">00:10:05.501</b> </span> vt.join_all()
<span class="" style=""><b style="">00:10:05.501</b> </span> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue
<span class="" style=""><b style="">00:10:05.502</b> </span> queue.put({'return': func()})
<span class="" style=""><b style="">00:10:05.502</b> </span> File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1251, in _deploy_host
<span class="" style=""><b style="">00:10:05.502</b> </span> <a href="http://host.name">host.name</a>(),
<span class="" style=""><b style="">00:10:05.502</b> </span>RuntimeError: /home/jenkins/workspace/vdsm_master_check-merged-el7-x86_64/vdsm/automation/vdsm_functional/default/scripts/_home_jenkins_workspace_vdsm_master_check-merged-el7-x86_64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-fc24</span></pre></pre><pre class="" style="white-space:pre-wrap;word-wrap:break-word;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:13px"><span style="color:rgb(205,0,0)"><br></span></pre><pre class="" style="white-space:pre-wrap;word-wrap:break-word;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:13px"><span style="color:rgb(205,0,0)"><br></span></pre>Can you please check why above error happened?</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
The error is related to secondary storage domain addition,<br>
it sounds like a known bug but now it seems to be triggered every time.<br>
<br>
I'm not sure how we can identify the source of the regression<br>
since these tests run on repos of already built RPMs, so the<br>
complete set has to be re-built with a specific patch and tested<br>
to find out the offending one.<br>
<br>
Regards,<br>
Evgheni Dereveanchin<br>
<br>
______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr">Sandro Bonazzola<br>Better technology. Faster innovation. Powered by community collaboration.<br>See how it works at <a href="http://redhat.com" target="_blank">redhat.com</a><br></div></div></div></div>
</div></div>