<div dir="ltr">Sandro,<div>check-merged and build-artifacts failed jobs are not related to Yaniv's patch.</div><div>check-merged-fc24 : known issue, Gal Ben Haim is checking it.</div><div>check-merged-el7 and build artifacts el7-ppc jobs were created recently (apparently by mistake) and should be removed.<br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 10, 2016 at 4:11 PM, Sandro Bonazzola <span dir="ltr"><<a href="mailto:sbonazzo@redhat.com" target="_blank">sbonazzo@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Wed, Aug 10, 2016 at 2:51 PM, Evgheni Dereveanchin <span dir="ltr"><<a href="mailto:ederevea@redhat.com" target="_blank">ederevea@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">Hi everyone,<br>
<br>
We have the test-repo_ovirt_experimental_m<wbr>aster job failing since build 717:<br>
<a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/717/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/t<wbr>est-repo_ovirt_experimental_ma<wbr>ster/717/</a><br>
<br>
System tests are also starting to fail:<br>
<a href="http://jenkins.ovirt.org/job/ovirt_master_system-tests/380/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/o<wbr>virt_master_system-tests/380/</a><br>
<br>
Here's the list of patches which appeared in build 717:<br>
<a href="https://gerrit.ovirt.org/#/c/61384/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/6<wbr>1384/</a> - Sandro Bonazzola - packaging: spec: drop default defattr<br>
<a href="https://gerrit.ovirt.org/#/c/62026/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/6<wbr>2026/</a> - Yaniv Bronhaim - Using %{_libdir} macro instead of /usr/lib<br>
<a href="https://gerrit.ovirt.org/#/c/62028/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/6<wbr>2028/</a> - Yaniv Bronhaim - Require python2-devel specifically to avoid python3 pkg<br>
<a href="https://gerrit.ovirt.org/#/c/62049/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/6<wbr>2049/</a> - Fabian Deutsch - Revert "imgbase: Drop journal support"<br>
<a href="https://gerrit.ovirt.org/#/c/62050/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/6<wbr>2050/</a> - Fabian Deutsch - cli: Add journal logging<br>
<br></blockquote><div><br></div></span><div>On VDSM log order is:</div><div><br></div><div><div>* 0a10329 - Require python2-devel specifically to avoid python3 pkg (27 ore fa) <Yaniv Bronhaim></div><div>* bdf3b9a - packaging: spec: drop default defattr (27 ore fa) <Sandro Bonazzola></div><div>* 271dd9b - Using %{_libdir} macro instead of /usr/lib (27 ore fa) <Yaniv Bronhaim></div><div><br></div></div><div>So let's start with: <a href="https://gerrit.ovirt.org/#/c/62026/" rel="noreferrer" target="_blank">https://gerrit.ovirt.<wbr>org/#/c/62026/</a> - Yaniv Bronhaim - Using %{_libdir} macro instead of /usr/lib </div></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><span style="color:rgb(53,53,53);font-family:sans-serif;white-space:pre-wrap">Before merge: Build Successful:</span></div><div><a href="http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/1017/" style="font-family:sans-serif;white-space:pre-wrap;color:rgb(6,84,172);text-decoration:none" target="_blank">http://jenkins.ovirt.org/job/<wbr>vdsm_master_check-patch-fc24-<wbr>x86_64/1017/</a><span style="color:rgb(53,53,53);font-family:sans-serif;white-space:pre-wrap"> : SUCCESS</span></div><div><a href="http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/5127/" style="font-family:sans-serif;white-space:pre-wrap;color:rgb(6,84,172);text-decoration:none" target="_blank">http://jenkins.ovirt.org/job/<wbr>vdsm_master_check-patch-el7-<wbr>x86_64/5127/</a><span style="color:rgb(53,53,53);font-family:sans-serif;white-space:pre-wrap"> : SUCCESS</span></div><div><br></div><div>Once merged: <span style="color:rgb(53,53,53);font-family:sans-serif;white-space:pre-wrap">Build Failed </span></div><div><a href="http://jenkins.ovirt.org/job/vdsm_master_check-merged-fc24-x86_64/174/" style="font-family:sans-serif;white-space:pre-wrap;color:rgb(6,84,172);text-decoration:none" target="_blank">http://jenkins.ovirt.org/job/<wbr>vdsm_master_check-merged-fc24-<wbr>x86_64/174/</a><span style="color:rgb(53,53,53);font-family:sans-serif;white-space:pre-wrap"> : FAILURE</span></div><div><a href="http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-el7-ppc64le/53/" style="font-family:sans-serif;white-space:pre-wrap;color:rgb(6,84,172);text-decoration:none" target="_blank">http://jenkins.ovirt.org/job/<wbr>vdsm_master_build-artifacts-<wbr>el7-ppc64le/53/</a><span style="color:rgb(53,53,53);font-family:sans-serif;white-space:pre-wrap"> : FAILURE</span></div><div><a href="http://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/21/" style="font-family:sans-serif;white-space:pre-wrap;color:rgb(6,84,172);text-decoration:none" target="_blank">http://jenkins.ovirt.org/job/<wbr>vdsm_master_check-merged-el7-<wbr>x86_64/21/</a><span style="color:rgb(53,53,53);font-family:sans-serif;white-space:pre-wrap"> : FAILURE</span></div><div><a href="http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-fc24-x86_64/151/" style="font-family:sans-serif;white-space:pre-wrap;color:rgb(6,84,172);text-decoration:none" target="_blank">http://jenkins.ovirt.org/job/<wbr>vdsm_master_build-artifacts-<wbr>fc24-x86_64/151/</a><span style="color:rgb(53,53,53);font-family:sans-serif;white-space:pre-wrap"> : SUCCESS</span></div><div><a href="http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-el7-x86_64/649/" style="font-family:sans-serif;white-space:pre-wrap;color:rgb(6,84,172);text-decoration:none" target="_blank">http://jenkins.ovirt.org/job/<wbr>vdsm_master_build-artifacts-<wbr>el7-x86_64/649/</a><span style="color:rgb(53,53,53);font-family:sans-serif;white-space:pre-wrap"> : SUCCESS</span></div><div><br></div><div>Let's exclude the ppc64le job for now.</div><div>FC24 failed on functional testing:</div><div><pre style="white-space:pre-wrap;word-wrap:break-word;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:13px"><span><b>00:06:43.766</b> </span><span style="color:rgb(0,205,205)"> # Deploy environment: </span>
<span><b>00:06:43.767</b> </span><span style="color:rgb(0,205,205)"> * [Thread-2] Deploy VM vdsm_functional_tests_host-<wbr>fc24: </span>
<span><b>00:10:05.208</b> </span><span style="color:rgb(205,0,0)"> - STDERR
<span><b>00:10:05.208</b> </span>Failed to synchronize cache for repo 'localsync', disabling.
<span><b>00:10:05.208</b> </span>Error: Unable to find a match.
<span><b>00:10:05.208</b> </span></span>
<span><b>00:10:05.209</b> </span><span style="color:rgb(0,205,205)"> * [Thread-2] Deploy VM vdsm_functional_tests_host-<wbr>fc24: </span><span style="color:rgb(205,0,0)">ERROR</span> (in 0:03:21)
<span><b>00:10:05.211</b> </span><span style="color:rgb(205,0,0)">Error while running thread
<span><b>00:10:05.211</b> </span>Traceback (most recent call last):
<span><b>00:10:05.211</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/lago/utils.py", line 53, in _ret_via_queue
<span><b>00:10:05.211</b> </span> queue.put({'return': func()})
<span><b>00:10:05.211</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/lago/prefix.py", line 1251, in _deploy_host
<span><b>00:10:05.211</b> </span> <a href="http://host.name" target="_blank">host.name</a>(),
<span><b>00:10:05.211</b> </span>RuntimeError: /home/jenkins/workspace/vdsm_<wbr>master_check-merged-el7-x86_<wbr>64/vdsm/automation/vdsm_<wbr>functional/default/scripts/_<wbr>home_jenkins_workspace_vdsm_<wbr>master_check-merged-el7-x86_<wbr>64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-<wbr>fc24</span>
<span><b>00:10:05.213</b> </span><span style="color:rgb(0,205,205)"> # Deploy environment: </span><span style="color:rgb(205,0,0)">ERROR</span> (in 0:03:21)
<span><b>00:10:05.498</b> </span><span style="color:rgb(0,205,205)">@ Deploy oVirt environment: </span><span style="color:rgb(205,0,0)">ERROR</span> (in 0:03:21)
<span><b>00:10:05.499</b> </span><span style="color:rgb(205,0,0)">Error occured, aborting
<span><b>00:10:05.499</b> </span>Traceback (most recent call last):
<span><b>00:10:05.499</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/ovirtlago/cmd.py", line 258, in do_run
<span><b>00:10:05.499</b> </span> self.cli_plugins[args.<wbr>ovirtverb].do_run(args)
<span><b>00:10:05.499</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/lago/plugins/cli.py", line 180, in do_run
<span><b>00:10:05.499</b> </span> self._do_run(**vars(args))
<span><b>00:10:05.499</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/lago/utils.py", line 488, in wrapper
<span><b>00:10:05.500</b> </span> return func(*args, **kwargs)
<span><b>00:10:05.500</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/lago/utils.py", line 499, in wrapper
<span><b>00:10:05.500</b> </span> return func(*args, prefix=prefix, **kwargs)
<span><b>00:10:05.500</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/ovirtlago/cmd.py", line 184, in do_deploy
<span><b>00:10:05.500</b> </span> prefix.deploy()
<span><b>00:10:05.500</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/lago/log_utils.py", line 615, in wrapper
<span><b>00:10:05.500</b> </span> return func(*args, **kwargs)
<span><b>00:10:05.500</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/ovirtlago/reposetup.<wbr>py", line 64, in wrapper
<span><b>00:10:05.500</b> </span> return func(*args, **kwargs)
<span><b>00:10:05.501</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/ovirtlago/__init__.<wbr>py", line 351, in deploy
<span><b>00:10:05.501</b> </span> return super(OvirtPrefix, self).deploy()
<span><b>00:10:05.501</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/lago/log_utils.py", line 615, in wrapper
<span><b>00:10:05.501</b> </span> return func(*args, **kwargs)
<span><b>00:10:05.501</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/lago/prefix.py", line 1258, in deploy
<span><b>00:10:05.501</b> </span> self._deploy_host, self.virt_env.get_vms().<wbr>values()
<span><b>00:10:05.501</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/lago/utils.py", line 95, in invoke_in_parallel
<span><b>00:10:05.501</b> </span> vt.join_all()
<span><b>00:10:05.501</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/lago/utils.py", line 53, in _ret_via_queue
<span><b>00:10:05.502</b> </span> queue.put({'return': func()})
<span><b>00:10:05.502</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/lago/prefix.py", line 1251, in _deploy_host
<span><b>00:10:05.502</b> </span> <a href="http://host.name" target="_blank">host.name</a>(),
<span><b>00:10:05.502</b> </span>RuntimeError: /home/jenkins/workspace/vdsm_<wbr>master_check-merged-el7-x86_<wbr>64/vdsm/automation/vdsm_<wbr>functional/default/scripts/_<wbr>home_jenkins_workspace_vdsm_<wbr>master_check-merged-el7-x86_<wbr>64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-<wbr>fc24</span></pre><pre style="white-space:pre-wrap;word-wrap:break-word;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:13px"><span style="color:rgb(205,0,0)"><br></span></pre>el7 failed on functional testing too:</div><div><br><pre style="white-space:pre-wrap;word-wrap:break-word;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:13px"><pre style="white-space:pre-wrap;word-wrap:break-word;margin-top:0px;margin-bottom:0px"><span><b>00:06:43.734</b> </span><span style="color:rgb(0,205,205)">@ Deploy oVirt environment: </span>
<span><b>00:06:43.739</b> </span><span style="color:rgb(205,205,0)"> # ovirt-role metadata entry will be soon deprecated, instead you should use the vm-provider entry in the domain definiton and set it no one of: ovirt-node, ovirt-engine, ovirt-host</span>
<span><b>00:06:43.766</b> </span><span style="color:rgb(0,205,205)"> # Deploy environment: </span>
<span><b>00:06:43.767</b> </span><span style="color:rgb(0,205,205)"> * [Thread-2] Deploy VM vdsm_functional_tests_host-<wbr>fc24: </span>
<span><b>00:10:05.208</b> </span><span style="color:rgb(205,0,0)"> - STDERR
<span><b>00:10:05.208</b> </span>Failed to synchronize cache for repo 'localsync', disabling.
<span><b>00:10:05.208</b> </span>Error: Unable to find a match.
<span><b>00:10:05.208</b> </span></span>
<span><b>00:10:05.209</b> </span><span style="color:rgb(0,205,205)"> * [Thread-2] Deploy VM vdsm_functional_tests_host-<wbr>fc24: </span><span style="color:rgb(205,0,0)">ERROR</span> (in 0:03:21)
<span><b>00:10:05.211</b> </span><span style="color:rgb(205,0,0)">Error while running thread
<span><b>00:10:05.211</b> </span>Traceback (most recent call last):
<span><b>00:10:05.211</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/lago/utils.py", line 53, in _ret_via_queue
<span><b>00:10:05.211</b> </span> queue.put({'return': func()})
<span><b>00:10:05.211</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/lago/prefix.py", line 1251, in _deploy_host
<span><b>00:10:05.211</b> </span> <a href="http://host.name" target="_blank">host.name</a>(),
<span><b>00:10:05.211</b> </span>RuntimeError: /home/jenkins/workspace/vdsm_<wbr>master_check-merged-el7-x86_<wbr>64/vdsm/automation/vdsm_<wbr>functional/default/scripts/_<wbr>home_jenkins_workspace_vdsm_<wbr>master_check-merged-el7-x86_<wbr>64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-<wbr>fc24</span>
<span><b>00:10:05.213</b> </span><span style="color:rgb(0,205,205)"> # Deploy environment: </span><span style="color:rgb(205,0,0)">ERROR</span> (in 0:03:21)
<span><b>00:10:05.498</b> </span><span style="color:rgb(0,205,205)">@ Deploy oVirt environment: </span><span style="color:rgb(205,0,0)">ERROR</span> (in 0:03:21)
<span><b>00:10:05.499</b> </span><span style="color:rgb(205,0,0)">Error occured, aborting
<span><b>00:10:05.499</b> </span>Traceback (most recent call last):
<span><b>00:10:05.499</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/ovirtlago/cmd.py", line 258, in do_run
<span><b>00:10:05.499</b> </span> self.cli_plugins[args.<wbr>ovirtverb].do_run(args)
<span><b>00:10:05.499</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/lago/plugins/cli.py", line 180, in do_run
<span><b>00:10:05.499</b> </span> self._do_run(**vars(args))
<span><b>00:10:05.499</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/lago/utils.py", line 488, in wrapper
<span><b>00:10:05.500</b> </span> return func(*args, **kwargs)
<span><b>00:10:05.500</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/lago/utils.py", line 499, in wrapper
<span><b>00:10:05.500</b> </span> return func(*args, prefix=prefix, **kwargs)
<span><b>00:10:05.500</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/ovirtlago/cmd.py", line 184, in do_deploy
<span><b>00:10:05.500</b> </span> prefix.deploy()
<span><b>00:10:05.500</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/lago/log_utils.py", line 615, in wrapper
<span><b>00:10:05.500</b> </span> return func(*args, **kwargs)
<span><b>00:10:05.500</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/ovirtlago/reposetup.<wbr>py", line 64, in wrapper
<span><b>00:10:05.500</b> </span> return func(*args, **kwargs)
<span><b>00:10:05.501</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/ovirtlago/__init__.<wbr>py", line 351, in deploy
<span><b>00:10:05.501</b> </span> return super(OvirtPrefix, self).deploy()
<span><b>00:10:05.501</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/lago/log_utils.py", line 615, in wrapper
<span><b>00:10:05.501</b> </span> return func(*args, **kwargs)
<span><b>00:10:05.501</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/lago/prefix.py", line 1258, in deploy
<span><b>00:10:05.501</b> </span> self._deploy_host, self.virt_env.get_vms().<wbr>values()
<span><b>00:10:05.501</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/lago/utils.py", line 95, in invoke_in_parallel
<span><b>00:10:05.501</b> </span> vt.join_all()
<span><b>00:10:05.501</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/lago/utils.py", line 53, in _ret_via_queue
<span><b>00:10:05.502</b> </span> queue.put({'return': func()})
<span><b>00:10:05.502</b> </span> File "/usr/lib/python2.7/site-<wbr>packages/lago/prefix.py", line 1251, in _deploy_host
<span><b>00:10:05.502</b> </span> <a href="http://host.name" target="_blank">host.name</a>(),
<span><b>00:10:05.502</b> </span>RuntimeError: /home/jenkins/workspace/vdsm_<wbr>master_check-merged-el7-x86_<wbr>64/vdsm/automation/vdsm_<wbr>functional/default/scripts/_<wbr>home_jenkins_workspace_vdsm_<wbr>master_check-merged-el7-x86_<wbr>64_vdsm_automation_deploy.sh failed with status 1 on vdsm_functional_tests_host-<wbr>fc24</span></pre></pre><pre style="white-space:pre-wrap;word-wrap:break-word;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:13px"><span style="color:rgb(205,0,0)"><br></span></pre><pre style="white-space:pre-wrap;word-wrap:break-word;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:13px"><span style="color:rgb(205,0,0)"><br></span></pre>Can you please check why above error happened?</div><span class=""><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
The error is related to secondary storage domain addition,<br>
it sounds like a known bug but now it seems to be triggered every time.<br>
<br>
I'm not sure how we can identify the source of the regression<br>
since these tests run on repos of already built RPMs, so the<br>
complete set has to be re-built with a specific patch and tested<br>
to find out the offending one.<br>
<br>
Regards,<br>
Evgheni Dereveanchin<br>
<br>
______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/devel</a><br>
</blockquote></span></div><span class=""><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr">Sandro Bonazzola<br>Better technology. Faster innovation. Powered by community collaboration.<br>See how it works at <a href="http://redhat.com" target="_blank">redhat.com</a><br></div></div></div></div>
</font></span></div></div>
<br>______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Irit Goihman<div>Software Engineer</div><div>Red Hat Israel Ltd.<br></div></div></div>
</div></div></div>