<div dir="ltr">Some more info on changes that were merged during Friday:<div><br>And from the looks of it [1] looks the culprit, I'd like to avoid another db upgrade issue with revert, anyone from dev can please handle this? </div><div><br></div><div>[1] <a href="https://gerrit.ovirt.org/#/c/67470/">https://gerrit.ovirt.org/#/c/67470/</a></div><div><br></div><div><table class="gmail-com-google-gerrit-client-change-ChangeScreen_BinderImpl_GenCss_style-infoTable" style="color:rgb(53,53,53);font-family:sans-serif"><tbody><tr><td class="gmail-com-google-gerrit-client-change-ChangeScreen_BinderImpl_GenCss_style-commitColumn" style="font-family:sans-serif;padding:0px;vertical-align:top;width:560px"><div class="gmail-com-google-gerrit-client-change-CommitBox_BinderImpl_GenCss_style-collapsed"><div class="gmail-com-google-gerrit-client-change-CommitBox_BinderImpl_GenCss_style-scroll" style="width:560px;border-right:1px solid rgb(238,238,238);border-bottom:1px solid rgb(238,238,238);height:250px;overflow:auto;zoom:1"><div style="zoom:1"><div class="gmail-com-google-gerrit-client-change-CommitBox_BinderImpl_GenCss_style-text" style="font-family:monospace;white-space:pre">core: New VM has RND device by default
Script adds urandom rng device to Blank template and all predefined
instance types. This causes that new VMs will inherit such RNG device.
Custom instance types are not changed. The assumption is that if they
were created without a RNG device, it was an intentional decision.
Change-Id: I93a51b67c0e8bff06152d9fe7a4315efd509774d
Bug-Url: <a href="https://bugzilla.redhat.com/1337101" target="_blank" style="color:rgb(6,84,172);text-decoration:none">https://bugzilla.redhat.com/1337101</a>
Signed-off-by: Jakub Niedermertl <<a href="mailto:jniederm@redhat.com">jniederm@redhat.com</a></div></div></div></div></td></tr></tbody></table></div><div><br></div><div>ovirt-engine changelog<br><div><br></div><div><div>* d9fcf3b - engine: Reduce the severity of check-upgrade log (3 days ago) Moti Asayag <<a href="mailto:masayag@redhat.com">masayag@redhat.com</a>></div><div>* 4616f4d - core: New VM has RND device by default (3 days ago) Jakub Niedermertl <<a href="mailto:jniederm@redhat.com">jniederm@redhat.com</a>></div><div>* c95365a - core: Fix of NPE when creating new instance type (3 days ago) Jakub Niedermertl <<a href="mailto:jniederm@redhat.com">jniederm@redhat.com</a>></div><div>* 2ff5c4d - restapi: Reflecting template RNG settings to new VM (3 days ago) Jakub Niedermertl <<a href="mailto:jniederm@redhat.com">jniederm@redhat.com</a>></div><div>* f8bdfa0 - frontend: use authz name instead of profile name for sysprep (3 days ago) Ondra Machacek <<a href="mailto:omachace@redhat.com">omachace@redhat.com</a>></div></div><div><br></div><div><br></div><div>vdsm changelog:<br></div></div><div><br></div><div><div><div>* a149cb7 - Adding simple client for sending gauge metrics to statsd port using udp (3 days ago) Yaniv Bronhaim <<a href="mailto:ybronhei@redhat.com">ybronhei@redhat.com</a>></div><div>* d5c00b9 - API: move vm parameters fixup in a method (3 days ago) Francesco Romani <<a href="mailto:fromani@redhat.com">fromani@redhat.com</a>></div><div>* 8180dfb - hostdev: add test for massive number of devices (3 days ago) Martin Polednik <<a href="mailto:mpolednik@redhat.com">mpolednik@redhat.com</a>></div><div>* f904734 - Remove the usage of clientIF from GlusterApi (3 days ago) Ramesh Nachimuthu <<a href="mailto:rnachimu@redhat.com">rnachimu@redhat.com</a>></div><div>* 36c0ce6 - rename method wrapApiMethod to _wrap_api_method (3 days ago) Ramesh Nachimuthu <<a href="mailto:rnachimu@redhat.com">rnachimu@redhat.com</a>></div><div>* 3383158 - vmfakecon: optimize HostDeviceStub (3 days ago) Martin Polednik <<a href="mailto:mpolednik@redhat.com">mpolednik@redhat.com</a>></div><div>* 316893c - hostdev: use *c*ElementTree (3 days ago) Martin Polednik <<a href="mailto:mpolednik@redhat.com">mpolednik@redhat.com</a>></div><div>* c56619e - client: document ConnectionError exception (3 days ago) Irit Goihman <<a href="mailto:igoihman@redhat.com">igoihman@redhat.com</a>></div><div>* f5d605e - py3: take Queue from six.moves (3 days ago) Dan Kenigsberg <<a href="mailto:danken@redhat.com">danken@redhat.com</a>></div></div></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Dec 4, 2016 at 3:34 PM, Eyal Edri <span dir="ltr"><<a href="mailto:eedri@redhat.com" target="_blank">eedri@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">FYI,<div><br></div><div>I opened a bug [1] to track this issue since I don't see any attempts to resolve the issue on the thread, hopefully a bug will get more attention.</div><div>Opened on VDSM since we see the libvirt error there, feel free to move product/team.</div><div><br></div><div><br></div><div>[1] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1401303" target="_blank">https://bugzilla.redhat.<wbr>com/show_bug.cgi?id=1401303</a><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Dec 4, 2016 at 1:23 PM, Eyal Edri <span dir="ltr"><<a href="mailto:eedri@redhat.com" target="_blank">eedri@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Not sure if relevant, but Juan posted a fix for SDK4 last time it happened ( but different failure on log-collector ):<div><br></div><div><a href="https://gerrit.ovirt.org/#/c/67213/" target="_blank">https://gerrit.ovirt.org/#/c/6<wbr>7213/</a><br></div><div><br></div><div><span style="color:rgb(53,53,53);font-family:monospace;white-space:pre-wrap">* Added `urandom` to the `RngSource` enumerated type.
</span><br></div></div><div class="m_313783804965705482HOEnZb"><div class="m_313783804965705482h5"><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Dec 4, 2016 at 9:17 AM, Eyal Edri <span dir="ltr"><<a href="mailto:eedri@redhat.com" target="_blank">eedri@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">And its still failing from Friday, <div>Since we don't have official Centos 7.3 repos yet ( hopefully we'll have it this week, but as of this moment its not published yet ) , we have to either revert the offending patch</div><div>or send a quick fix.</div><div><br></div><div>Right now all experimental flows for master are not working and nightly rpms are not refreshed with new RPMs. </div><div><br></div><div><br></div></div><div class="gmail_extra"><div><div class="m_313783804965705482m_-8590852049412765294h5"><br><div class="gmail_quote">On Fri, Dec 2, 2016 at 9:41 PM, Yaniv Kaul <span dir="ltr"><<a href="mailto:ykaul@redhat.com" target="_blank">ykaul@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><span><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Dec 2, 2016 2:11 PM, "Anton Marchukov" <<a href="mailto:amarchuk@redhat.com" target="_blank">amarchuk@redhat.com</a>> wrote:<br type="attribution"><blockquote class="m_313783804965705482m_-8590852049412765294m_-8368919148765653837m_4397371983421905186quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hello Martin.<div><br></div><div>Do by outdated you mean the old libvirt? If so that is that livirt available in CentOS 7.2? There is no 7.3 yet.</div></div></blockquote></div></div></div><div dir="auto"><br></div></span><div dir="auto">Right, this is the issue. </div><div dir="auto">Y. </div><div dir="auto"><br></div><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="m_313783804965705482m_-8590852049412765294m_-8368919148765653837m_4397371983421905186quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><font color="#888888"><div><br></div><div>Anton.</div></font></div><div class="m_313783804965705482m_-8590852049412765294m_-8368919148765653837m_4397371983421905186elided-text"><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_313783804965705482m_-8590852049412765294m_-8368919148765653837h5">On Fri, Dec 2, 2016 at 1:07 PM, Martin Polednik <span dir="ltr"><<a href="mailto:mpolednik@redhat.com" target="_blank">mpolednik@redhat.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="m_313783804965705482m_-8590852049412765294m_-8368919148765653837m_4397371983421905186m_3873929572442477785HOEnZb"><div class="m_313783804965705482m_-8590852049412765294m_-8368919148765653837m_4397371983421905186m_3873929572442477785h5"><div><div class="m_313783804965705482m_-8590852049412765294m_-8368919148765653837h5">On 02/12/16 10:55 +0100, Anton Marchukov wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="m_313783804965705482m_-8590852049412765294m_-8368919148765653837h5">
Hello All.<br>
<br>
Engine log can be viewed here:<br>
<br>
<a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/3838/artifact/exported-artifacts/basic_suite_master.sh-el7/exported-artifacts/test_logs/basic-suite-master/post-004_basic_sanity.py/lago-basic-suite-master-engine/_var_log_ovirt-engine/engine.log" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/t<wbr>est-repo_ovirt_experimental_ma<wbr>ster/3838/artifact/exported-ar<wbr>tifacts/basic_suite_master.sh-<wbr>el7/exported-artifacts/test_lo<wbr>gs/basic-suite-master/post-004<wbr>_basic_sanity.py/lago-basic-su<wbr>ite-master-engine/_var_log_ovi<wbr>rt-engine/engine.log</a><br>
<br>
I see the following exception there:<br>
<br>
2016-12-02 04:29:24,030-05 DEBUG<br>
[org.ovirt.vdsm.jsonrpc.client<wbr>.internal.ResponseWorker]<br>
(ResponseWorker) [83b6b5d] Message received: {"jsonrpc": "2.0", "id":<br>
"ec254aad-441b-47e7-a644-aebdd<wbr>cc1d62c", "result": true}<br>
2016-12-02 04:29:24,030-05 ERROR<br>
[org.ovirt.vdsm.jsonrpc.client<wbr>.JsonRpcClient] (ResponseWorker)<br>
[83b6b5d] Not able to update response for<br>
"ec254aad-441b-47e7-a644-aebdd<wbr>cc1d62c"<br>
2016-12-02 04:29:24,041-05 DEBUG<br>
[org.ovirt.engine.core.utils.t<wbr>imer.FixedDelayJobListener]<br>
(DefaultQuartzScheduler3) [47a31d72] Rescheduling<br>
DEFAULT.org.ovirt.engine.core.<wbr>bll.gluster.GlusterSyncJob.ref<wbr>reshLightWeightData#-922337203<wbr>6854775775<br>
as there is no unfired trigger.<br>
2016-12-02 04:29:24,024-05 DEBUG<br>
[org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.PollVDSCommand] (default<br>
task-12) [d932871a-af4f-4fc9-9ee5-f7a01<wbr>26a7b85] Exception:<br>
org.ovirt.engine.core.vdsbroke<wbr>r.vdsbroker.VDSNetworkExceptio<wbr>n:<br>
VDSGenericException: VDSNetworkException: Timeout during xml-rpc call<br>
at org.ovirt.engine.core.vdsbroke<wbr>r.vdsbroker.FutureVDSCommand.g<wbr>et(FutureVDSCommand.java:73)<br>
[vdsbroker.jar:]<br>
<br>
...<br>
<br></div></div><div><div class="m_313783804965705482m_-8590852049412765294m_-8368919148765653837h5">
2016-12-02 04:29:24,042-05 ERROR<br>
[org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.PollVDSCommand] (default<br>
task-12) [d932871a-af4f-4fc9-9ee5-f7a01<wbr>26a7b85] Timeout waiting for<br>
VDSM response: Internal timeout occured<br>
2016-12-02 04:29:24,044-05 DEBUG<br>
[org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand]<br>
(default task-12) [d932871a-af4f-4fc9-9ee5-f7a01<wbr>26a7b85] START,<br>
GetCapabilitiesVDSCommand(Host<wbr>Name = lago-basic-suite-master-host0,<br>
VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true',<br>
hostId='5eb7019e-28a3-4f93-918<wbr>8-685b6c64a2f5',<br>
vds='Host[lago-basic-suite-mas<wbr>ter-host0,5eb7019e-28a3-4f93-9<wbr>188-685b6c64a2f5]'}),<br>
log id: 58f448b8<br>
2016-12-02 04:29:24,044-05 DEBUG<br>
[org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.stomp.impl.Message] (default<br>
task-12) [d932871a-af4f-4fc9-9ee5-f7a01<wbr>26a7b85] SEND<br>
destination:jms.topic.vdsm_req<wbr>uests<br>
reply-to:jms.topic.vdsm_respon<wbr>ses<br>
content-length:105<br>
<br>
<br>
Please note that this runs on localhost with local bridge. So it is not<br>
likely to be network itself.<br>
</div></div></blockquote>
<br></div></div><div><div class="m_313783804965705482m_-8590852049412765294m_-8368919148765653837h5">
The main issue I see is that the VM run command has actually failed<br>
due to libvirt no accepting /dev/urandom as RNG source[1]. This was<br>
done as engine patch and according to git log, posted around Mon Nov<br>
28. Also adding Jakub - this should either not happen from engine's<br>
point of view or the lago host is outdated.<br>
<br>
[1]<br>
<a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/3838/artifact/exported-artifacts/basic_suite_master.sh-el7/exported-artifacts/test_logs/basic-suite-master/post-004_basic_sanity.py/lago-basic-suite-master-host0/_var_log_vdsm/vdsm.log" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/t<wbr>est-repo_ovirt_experimental_ma<wbr>ster/3838/artifact/exported-ar<wbr>tifacts/basic_suite_master.sh-<wbr>el7/exported-artifacts/test_lo<wbr>gs/basic-suite-master/post-004<wbr>_basic_sanity.py/lago-basic-su<wbr>ite-master-host0/_var_log_vdsm<wbr>/vdsm.log</a><div class="m_313783804965705482m_-8590852049412765294m_-8368919148765653837m_4397371983421905186m_3873929572442477785HOEnZb"><div class="m_313783804965705482m_-8590852049412765294m_-8368919148765653837m_4397371983421905186m_3873929572442477785h5"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Anton.<br>
<br>
On Fri, Dec 2, 2016 at 10:43 AM, Anton Marchukov <<a href="mailto:amarchuk@redhat.com" target="_blank">amarchuk@redhat.com</a>><br>
wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
FYI. Experimental flow for master currently fails to run a VM. The tests<br>
times out while waiting for 180 seconds:<br>
<br>
<a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/t<wbr>est-repo_ovirt_experimental_</a><br>
master/3838/testReport/(root)/<wbr>004_basic_sanity/vm_run/<br>
<br>
This is reproducible over 23 runs of this happened tonight, sounds like a<br>
regression to me:<br>
<br>
<a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/t<wbr>est-repo_ovirt_experimental_ma<wbr>ster/</a><br>
<br>
I will update here with additional information once I find it.<br>
<br>
Last successful run was with this patch:<br>
<br>
<a href="https://gerrit.ovirt.org/#/c/66416/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/6<wbr>6416/</a> (vdsm: API: move vm parameters fixup<br>
in a method)<br>
<br>
Known to start failing around this patch:<br>
<br>
<a href="https://gerrit.ovirt.org/#/c/67647/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/6<wbr>7647/</a> (vdsmapi: fix a typo in string<br>
formatting)<br>
<br>
Please notes that we do not have gating implemented yet, so everything<br>
that was merged in between those patches might have caused this (not<br>
necessary in vdsm project).<br>
<br>
Anton.<br>
--<br>
Anton Marchukov<br>
Senior Software Engineer - RHEV CI - Red Hat<br>
<br>
<br>
</blockquote>
<br>
<br>
-- <br>
Anton Marchukov<br>
Senior Software Engineer - RHEV CI - Red Hat<br>
</blockquote>
<br>
</div></div><div class="m_313783804965705482m_-8590852049412765294m_-8368919148765653837m_4397371983421905186m_3873929572442477785HOEnZb"><div class="m_313783804965705482m_-8590852049412765294m_-8368919148765653837m_4397371983421905186m_3873929572442477785h5"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/devel</a><br>
</blockquote>
<br>
</div></div></div></div></blockquote></div><div><div class="m_313783804965705482m_-8590852049412765294m_-8368919148765653837h5"><br><br clear="all"><div><br></div>-- <br><div class="m_313783804965705482m_-8590852049412765294m_-8368919148765653837m_4397371983421905186m_3873929572442477785gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><span><font color="#888888"><div>Anton Marchukov<br>Senior Software Engineer - <span><span><font color="#888888"><span><span><font color="#888888"><span><span><font color="#888888">RHEV CI - </font></span></span></font></span></span>Red Hat</font></span></span><br><br></div></font></span></div></div></div></div>
</div></div></div>
</div><div><div class="m_313783804965705482m_-8590852049412765294m_-8368919148765653837h5"><br>______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/devel</a><br></div></div></blockquote></div><br></div></div></div>
<br>______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/devel</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br></div></div><span><div class="m_313783804965705482m_-8590852049412765294m_-8368919148765653837gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Eyal Edri<br>Associate Manager</div><div>RHV DevOps<br>EMEA ENG Virtualization R&D<br>Red Hat Israel<br><br>phone: <a href="tel:%2B972-9-7692018" value="+97297692018" target="_blank">+972-9-7692018</a><br>irc: eedri (on #tlv #rhev-dev #rhev-integ)</div></div></div></div></div></div></div>
</span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="m_313783804965705482m_-8590852049412765294gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Eyal Edri<br>Associate Manager</div><div>RHV DevOps<br>EMEA ENG Virtualization R&D<br>Red Hat Israel<br><br>phone: <a href="tel:%2B972-9-7692018" value="+97297692018" target="_blank">+972-9-7692018</a><br>irc: eedri (on #tlv #rhev-dev #rhev-integ)</div></div></div></div></div></div></div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="m_313783804965705482gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Eyal Edri<br>Associate Manager</div><div>RHV DevOps<br>EMEA ENG Virtualization R&D<br>Red Hat Israel<br><br>phone: <a href="tel:%2B972-9-7692018" value="+97297692018" target="_blank">+972-9-7692018</a><br>irc: eedri (on #tlv #rhev-dev #rhev-integ)</div></div></div></div></div></div></div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Eyal Edri<br>Associate Manager</div><div>RHV DevOps<br>EMEA ENG Virtualization R&D<br>Red Hat Israel<br><br>phone: +972-9-7692018<br>irc: eedri (on #tlv #rhev-dev #rhev-integ)</div></div></div></div></div></div></div>
</div>