<div dir="ltr">FYI,<div><br></div><div>I opened a bug [1] to track this issue since I don't see any attempts to resolve the issue on the thread, hopefully a bug will get more attention.</div><div>Opened on VDSM since we see the libvirt error there, feel free to move product/team.</div><div><br></div><div><br></div><div>[1] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1401303">https://bugzilla.redhat.com/show_bug.cgi?id=1401303</a><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Dec 4, 2016 at 1:23 PM, Eyal Edri <span dir="ltr"><<a href="mailto:eedri@redhat.com" target="_blank">eedri@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Not sure if relevant, but Juan posted a fix for SDK4 last time it happened ( but different failure on log-collector ):<div><br></div><div><a href="https://gerrit.ovirt.org/#/c/67213/" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>67213/</a><br></div><div><br></div><div><span style="color:rgb(53,53,53);font-family:monospace;white-space:pre-wrap">* Added `urandom` to the `RngSource` enumerated type.
</span><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Dec 4, 2016 at 9:17 AM, Eyal Edri <span dir="ltr"><<a href="mailto:eedri@redhat.com" target="_blank">eedri@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">And its still failing from Friday, <div>Since we don't have official Centos 7.3 repos yet ( hopefully we'll have it this week, but as of this moment its not published yet ) , we have to either revert the offending patch</div><div>or send a quick fix.</div><div><br></div><div>Right now all experimental flows for master are not working and nightly rpms are not refreshed with new RPMs. </div><div><br></div><div><br></div></div><div class="gmail_extra"><div><div class="m_-8590852049412765294h5"><br><div class="gmail_quote">On Fri, Dec 2, 2016 at 9:41 PM, Yaniv Kaul <span dir="ltr"><<a href="mailto:ykaul@redhat.com" target="_blank">ykaul@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><span><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Dec 2, 2016 2:11 PM, "Anton Marchukov" <<a href="mailto:amarchuk@redhat.com" target="_blank">amarchuk@redhat.com</a>> wrote:<br type="attribution"><blockquote class="m_-8590852049412765294m_-8368919148765653837m_4397371983421905186quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hello Martin.<div><br></div><div>Do by outdated you mean the old libvirt? If so that is that livirt available in CentOS 7.2? There is no 7.3 yet.</div></div></blockquote></div></div></div><div dir="auto"><br></div></span><div dir="auto">Right, this is the issue. </div><div dir="auto">Y. </div><div dir="auto"><br></div><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="m_-8590852049412765294m_-8368919148765653837m_4397371983421905186quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><font color="#888888"><div><br></div><div>Anton.</div></font></div><div class="m_-8590852049412765294m_-8368919148765653837m_4397371983421905186elided-text"><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_-8590852049412765294m_-8368919148765653837h5">On Fri, Dec 2, 2016 at 1:07 PM, Martin Polednik <span dir="ltr"><<a href="mailto:mpolednik@redhat.com" target="_blank">mpolednik@redhat.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="m_-8590852049412765294m_-8368919148765653837m_4397371983421905186m_3873929572442477785HOEnZb"><div class="m_-8590852049412765294m_-8368919148765653837m_4397371983421905186m_3873929572442477785h5"><div><div class="m_-8590852049412765294m_-8368919148765653837h5">On 02/12/16 10:55 +0100, Anton Marchukov wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="m_-8590852049412765294m_-8368919148765653837h5">
Hello All.<br>
<br>
Engine log can be viewed here:<br>
<br>
<a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/3838/artifact/exported-artifacts/basic_suite_master.sh-el7/exported-artifacts/test_logs/basic-suite-master/post-004_basic_sanity.py/lago-basic-suite-master-engine/_var_log_ovirt-engine/engine.log" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/t<wbr>est-repo_ovirt_experimental_ma<wbr>ster/3838/artifact/exported-ar<wbr>tifacts/basic_suite_master.sh-<wbr>el7/exported-artifacts/test_lo<wbr>gs/basic-suite-master/post-004<wbr>_basic_sanity.py/lago-basic-su<wbr>ite-master-engine/_var_log_ovi<wbr>rt-engine/engine.log</a><br>
<br>
I see the following exception there:<br>
<br>
2016-12-02 04:29:24,030-05 DEBUG<br>
[org.ovirt.vdsm.jsonrpc.client<wbr>.internal.ResponseWorker]<br>
(ResponseWorker) [83b6b5d] Message received: {"jsonrpc": "2.0", "id":<br>
"ec254aad-441b-47e7-a644-aebdd<wbr>cc1d62c", "result": true}<br>
2016-12-02 04:29:24,030-05 ERROR<br>
[org.ovirt.vdsm.jsonrpc.client<wbr>.JsonRpcClient] (ResponseWorker)<br>
[83b6b5d] Not able to update response for<br>
"ec254aad-441b-47e7-a644-aebdd<wbr>cc1d62c"<br>
2016-12-02 04:29:24,041-05 DEBUG<br>
[org.ovirt.engine.core.utils.t<wbr>imer.FixedDelayJobListener]<br>
(DefaultQuartzScheduler3) [47a31d72] Rescheduling<br>
DEFAULT.org.ovirt.engine.core.<wbr>bll.gluster.GlusterSyncJob.ref<wbr>reshLightWeightData#-922337203<wbr>6854775775<br>
as there is no unfired trigger.<br>
2016-12-02 04:29:24,024-05 DEBUG<br>
[org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.PollVDSCommand] (default<br>
task-12) [d932871a-af4f-4fc9-9ee5-f7a01<wbr>26a7b85] Exception:<br>
org.ovirt.engine.core.vdsbroke<wbr>r.vdsbroker.VDSNetworkExceptio<wbr>n:<br>
VDSGenericException: VDSNetworkException: Timeout during xml-rpc call<br>
at org.ovirt.engine.core.vdsbroke<wbr>r.vdsbroker.FutureVDSCommand.g<wbr>et(FutureVDSCommand.java:73)<br>
[vdsbroker.jar:]<br>
<br>
...<br>
<br></div></div><div><div class="m_-8590852049412765294m_-8368919148765653837h5">
2016-12-02 04:29:24,042-05 ERROR<br>
[org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.PollVDSCommand] (default<br>
task-12) [d932871a-af4f-4fc9-9ee5-f7a01<wbr>26a7b85] Timeout waiting for<br>
VDSM response: Internal timeout occured<br>
2016-12-02 04:29:24,044-05 DEBUG<br>
[org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand]<br>
(default task-12) [d932871a-af4f-4fc9-9ee5-f7a01<wbr>26a7b85] START,<br>
GetCapabilitiesVDSCommand(Host<wbr>Name = lago-basic-suite-master-host0,<br>
VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true',<br>
hostId='5eb7019e-28a3-4f93-918<wbr>8-685b6c64a2f5',<br>
vds='Host[lago-basic-suite-mas<wbr>ter-host0,5eb7019e-28a3-4f93-9<wbr>188-685b6c64a2f5]'}),<br>
log id: 58f448b8<br>
2016-12-02 04:29:24,044-05 DEBUG<br>
[org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.stomp.impl.Message] (default<br>
task-12) [d932871a-af4f-4fc9-9ee5-f7a01<wbr>26a7b85] SEND<br>
destination:jms.topic.vdsm_req<wbr>uests<br>
reply-to:jms.topic.vdsm_respon<wbr>ses<br>
content-length:105<br>
<br>
<br>
Please note that this runs on localhost with local bridge. So it is not<br>
likely to be network itself.<br>
</div></div></blockquote>
<br></div></div><div><div class="m_-8590852049412765294m_-8368919148765653837h5">
The main issue I see is that the VM run command has actually failed<br>
due to libvirt no accepting /dev/urandom as RNG source[1]. This was<br>
done as engine patch and according to git log, posted around Mon Nov<br>
28. Also adding Jakub - this should either not happen from engine's<br>
point of view or the lago host is outdated.<br>
<br>
[1]<br>
<a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/3838/artifact/exported-artifacts/basic_suite_master.sh-el7/exported-artifacts/test_logs/basic-suite-master/post-004_basic_sanity.py/lago-basic-suite-master-host0/_var_log_vdsm/vdsm.log" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/t<wbr>est-repo_ovirt_experimental_ma<wbr>ster/3838/artifact/exported-ar<wbr>tifacts/basic_suite_master.sh-<wbr>el7/exported-artifacts/test_lo<wbr>gs/basic-suite-master/post-004<wbr>_basic_sanity.py/lago-basic-su<wbr>ite-master-host0/_var_log_vdsm<wbr>/vdsm.log</a><div class="m_-8590852049412765294m_-8368919148765653837m_4397371983421905186m_3873929572442477785HOEnZb"><div class="m_-8590852049412765294m_-8368919148765653837m_4397371983421905186m_3873929572442477785h5"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Anton.<br>
<br>
On Fri, Dec 2, 2016 at 10:43 AM, Anton Marchukov <<a href="mailto:amarchuk@redhat.com" target="_blank">amarchuk@redhat.com</a>><br>
wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
FYI. Experimental flow for master currently fails to run a VM. The tests<br>
times out while waiting for 180 seconds:<br>
<br>
<a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/t<wbr>est-repo_ovirt_experimental_</a><br>
master/3838/testReport/(root)/<wbr>004_basic_sanity/vm_run/<br>
<br>
This is reproducible over 23 runs of this happened tonight, sounds like a<br>
regression to me:<br>
<br>
<a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/t<wbr>est-repo_ovirt_experimental_ma<wbr>ster/</a><br>
<br>
I will update here with additional information once I find it.<br>
<br>
Last successful run was with this patch:<br>
<br>
<a href="https://gerrit.ovirt.org/#/c/66416/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/6<wbr>6416/</a> (vdsm: API: move vm parameters fixup<br>
in a method)<br>
<br>
Known to start failing around this patch:<br>
<br>
<a href="https://gerrit.ovirt.org/#/c/67647/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/6<wbr>7647/</a> (vdsmapi: fix a typo in string<br>
formatting)<br>
<br>
Please notes that we do not have gating implemented yet, so everything<br>
that was merged in between those patches might have caused this (not<br>
necessary in vdsm project).<br>
<br>
Anton.<br>
--<br>
Anton Marchukov<br>
Senior Software Engineer - RHEV CI - Red Hat<br>
<br>
<br>
</blockquote>
<br>
<br>
-- <br>
Anton Marchukov<br>
Senior Software Engineer - RHEV CI - Red Hat<br>
</blockquote>
<br>
</div></div><div class="m_-8590852049412765294m_-8368919148765653837m_4397371983421905186m_3873929572442477785HOEnZb"><div class="m_-8590852049412765294m_-8368919148765653837m_4397371983421905186m_3873929572442477785h5"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/devel</a><br>
</blockquote>
<br>
</div></div></div></div></blockquote></div><div><div class="m_-8590852049412765294m_-8368919148765653837h5"><br><br clear="all"><div><br></div>-- <br><div class="m_-8590852049412765294m_-8368919148765653837m_4397371983421905186m_3873929572442477785gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><span><font color="#888888"><div>Anton Marchukov<br>Senior Software Engineer - <span><span><font color="#888888"><span><span><font color="#888888"><span><span><font color="#888888">RHEV CI - </font></span></span></font></span></span>Red Hat</font></span></span><br><br></div></font></span></div></div></div></div>
</div></div></div>
</div><div><div class="m_-8590852049412765294m_-8368919148765653837h5"><br>______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/devel</a><br></div></div></blockquote></div><br></div></div></div>
<br>______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/devel</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br></div></div><span><div class="m_-8590852049412765294m_-8368919148765653837gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Eyal Edri<br>Associate Manager</div><div>RHV DevOps<br>EMEA ENG Virtualization R&D<br>Red Hat Israel<br><br>phone: <a href="tel:%2B972-9-7692018" value="+97297692018" target="_blank">+972-9-7692018</a><br>irc: eedri (on #tlv #rhev-dev #rhev-integ)</div></div></div></div></div></div></div>
</span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="m_-8590852049412765294gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Eyal Edri<br>Associate Manager</div><div>RHV DevOps<br>EMEA ENG Virtualization R&D<br>Red Hat Israel<br><br>phone: <a href="tel:%2B972-9-7692018" value="+97297692018" target="_blank">+972-9-7692018</a><br>irc: eedri (on #tlv #rhev-dev #rhev-integ)</div></div></div></div></div></div></div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Eyal Edri<br>Associate Manager</div><div>RHV DevOps<br>EMEA ENG Virtualization R&D<br>Red Hat Israel<br><br>phone: +972-9-7692018<br>irc: eedri (on #tlv #rhev-dev #rhev-integ)</div></div></div></div></div></div></div>
</div>