Re: [ovirt-devel] Experimental Flow for Master Fails to Run a VM

On Dec 4, 2016 6:42 PM, "Arik Hadas" <ahadas(a)redhat.com&gt; wrote: Yaniv will try to lower the cluster level used in the system-tests to 4.0 - this is supposed to solve the issue. Done. Y. If it won't help (we will know it in about an hour), we'll add a db-script that changes the rng device of the blank template only. On Sun, Dec 4, 2016 at 3:34 PM, Eyal Edri <eedri(a)redhat.com&gt; wrote: > FYI, > > I opened a bug [1] to track this issue since I don't see any attempts to > resolve the issue on the thread, hopefully a bug will get more attention. > Opened on VDSM since we see the libvirt error there, feel free to move > product/team. > > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1401303 > > On Sun, Dec 4, 2016 at 1:23 PM, Eyal Edri <eedri(a)redhat.com&gt; wrote: > >> Not sure if relevant, but Juan posted a fix for SDK4 last time it >> happened ( but different failure on log-collector ): >> >> https://gerrit.ovirt.org/#/c/67213/ >> >> * Added `urandom` to the `RngSource` enumerated type. >> >> On Sun, Dec 4, 2016 at 9:17 AM, Eyal Edri <eedri(a)redhat.com&gt; wrote: >> >>> And its still failing from Friday, >>> Since we don't have official Centos 7.3 repos yet ( hopefully we'll >>> have it this week, but as of this moment its not published yet ) , we have >>> to either revert the offending patch >>> or send a quick fix. >>> >>> Right now all experimental flows for master are not working and nightly >>> rpms are not refreshed with new RPMs. >>> >>> >>> >>> On Fri, Dec 2, 2016 at 9:41 PM, Yaniv Kaul <ykaul(a)redhat.com&gt; wrote: >>> >>>> >>>> >>>> On Dec 2, 2016 2:11 PM, "Anton Marchukov" <amarchuk(a)redhat.com&gt; wrote: >>>> >>>> Hello Martin. >>>> >>>> Do by outdated you mean the old libvirt? If so that is that livirt >>>> available in CentOS 7.2? There is no 7.3 yet. >>>> >>>> >>>> Right, this is the issue. >>>> Y. >>>> >>>> >>>> Anton. >>>> >>>> On Fri, Dec 2, 2016 at 1:07 PM, Martin Polednik <mpolednik(a)redhat.com&gt; >>>> wrote: >>>> >>>>> On 02/12/16 10:55 +0100, Anton Marchukov wrote: >>>>> >>>>>> Hello All. >>>>>> >>>>>> Engine log can be viewed here: >>>>>> >>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ma >>>>>> ster/3838/artifact/exported-artifacts/basic_suite_master.sh- >>>>>> el7/exported-artifacts/test_logs/basic-suite-master/post-004 >>>>>> _basic_sanity.py/lago-basic-suite-master-engine/_var_log_ovi >>>>>> rt-engine/engine.log >>>>>> >>>>>> I see the following exception there: >>>>>> >>>>>> 2016-12-02 04:29:24,030-05 DEBUG >>>>>> [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] >>>>>> (ResponseWorker) [83b6b5d] Message received: {"jsonrpc": "2.0", "id": >>>>>> "ec254aad-441b-47e7-a644-aebddcc1d62c", "result": true} >>>>>> 2016-12-02 04:29:24,030-05 ERROR >>>>>> [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) >>>>>> [83b6b5d] Not able to update response for >>>>>> "ec254aad-441b-47e7-a644-aebddcc1d62c" >>>>>> 2016-12-02 04:29:24,041-05 DEBUG >>>>>> [org.ovirt.engine.core.utils.timer.FixedDelayJobListener] >>>>>> (DefaultQuartzScheduler3) [47a31d72] Rescheduling >>>>>> DEFAULT.org.ovirt.engine.core.bll.gluster.GlusterSyncJob.ref >>>>>> reshLightWeightData#-9223372036854775775 >>>>>> as there is no unfired trigger. >>>>>> 2016-12-02 04:29:24,024-05 DEBUG >>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default >>>>>> task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] Exception: >>>>>> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: >>>>>> VDSGenericException: VDSNetworkException: Timeout during xml-rpc call >>>>>> at org.ovirt.engine.core.vdsbroke >>>>>> r.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:73) >>>>>> [vdsbroker.jar:] >>>>>> >>>>>> ... >>>>>> >>>>>> 2016-12-02 04:29:24,042-05 ERROR >>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default >>>>>> task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] Timeout waiting for >>>>>> VDSM response: Internal timeout occured >>>>>> 2016-12-02 04:29:24,044-05 DEBUG >>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVD >>>>>> SCommand] >>>>>> (default task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] START, >>>>>> GetCapabilitiesVDSCommand(HostName = lago-basic-suite-master-host0, >>>>>> VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', >>>>>> hostId='5eb7019e-28a3-4f93-9188-685b6c64a2f5', >>>>>> vds='Host[lago-basic-suite-master-host0,5eb7019e-28a3-4f93-9 >>>>>> 188-685b6c64a2f5]'}), >>>>>> log id: 58f448b8 >>>>>> 2016-12-02 04:29:24,044-05 DEBUG >>>>>> [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (default >>>>>> task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] SEND >>>>>> destination:jms.topic.vdsm_requests >>>>>> reply-to:jms.topic.vdsm_responses >>>>>> content-length:105 >>>>>> >>>>>> >>>>>> Please note that this runs on localhost with local bridge. So it is >>>>>> not >>>>>> likely to be network itself. >>>>>> >>>>> >>>>> The main issue I see is that the VM run command has actually failed >>>>> due to libvirt no accepting /dev/urandom as RNG source[1]. This was >>>>> done as engine patch and according to git log, posted around Mon Nov >>>>> 28. Also adding Jakub - this should either not happen from engine's >>>>> point of view or the lago host is outdated. >>>>> >>>>> [1] >>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ma >>>>> ster/3838/artifact/exported-artifacts/basic_suite_master.sh- >>>>> el7/exported-artifacts/test_logs/basic-suite-master/post-004 >>>>> _basic_sanity.py/lago-basic-suite-master-host0/_var_log_vdsm/vdsm.log >>>>> >>>>> >>>>> Anton. >>>>>> >>>>>> On Fri, Dec 2, 2016 at 10:43 AM, Anton Marchukov < >>>>>> amarchuk(a)redhat.com&gt; >>>>>> wrote: >>>>>> >>>>>> FYI. Experimental flow for master currently fails to run a VM. The >>>>>>> tests >>>>>>> times out while waiting for 180 seconds: >>>>>>> >>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ >>>>>>> master/3838/testReport/(root)/004_basic_sanity/vm_run/ >>>>>>> >>>>>>> This is reproducible over 23 runs of this happened tonight, sounds >>>>>>> like a >>>>>>> regression to me: >>>>>>> >>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/ >>>>>>> >>>>>>> I will update here with additional information once I find it. >>>>>>> >>>>>>> Last successful run was with this patch: >>>>>>> >>>>>>> https://gerrit.ovirt.org/#/c/66416/ (vdsm: API: move vm parameters >>>>>>> fixup >>>>>>> in a method) >>>>>>> >>>>>>> Known to start failing around this patch: >>>>>>> >>>>>>> https://gerrit.ovirt.org/#/c/67647/ (vdsmapi: fix a typo in string >>>>>>> formatting) >>>>>>> >>>>>>> Please notes that we do not have gating implemented yet, so >>>>>>> everything >>>>>>> that was merged in between those patches might have caused this (not >>>>>>> necessary in vdsm project). >>>>>>> >>>>>>> Anton. >>>>>>> -- >>>>>>> Anton Marchukov >>>>>>> Senior Software Engineer - RHEV CI - Red Hat >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> Anton Marchukov >>>>>> Senior Software Engineer - RHEV CI - Red Hat >>>>>> >>>>> >>>>> _______________________________________________ >>>>>> Devel mailing list >>>>>> Devel(a)ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Anton Marchukov >>>> Senior Software Engineer - RHEV CI - Red Hat >>>> >>>> >>>> _______________________________________________ >>>> Devel mailing list >>>> Devel(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/devel >>>> >>>> >>>> >>>> _______________________________________________ >>>> Devel mailing list >>>> Devel(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/devel >>>> >>> >>> >>> >>> -- >>> Eyal Edri >>> Associate Manager >>> RHV DevOps >>> EMEA ENG Virtualization R&D >>> Red Hat Israel >>> >>> phone: +972-9-7692018 >>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>> >> >> >> >> -- >> Eyal Edri >> Associate Manager >> RHV DevOps >> EMEA ENG Virtualization R&D >> Red Hat Israel >> >> phone: +972-9-7692018 >> irc: eedri (on #tlv #rhev-dev #rhev-integ) >> > > > > -- > Eyal Edri > Associate Manager > RHV DevOps > EMEA ENG Virtualization R&D > Red Hat Israel > > phone: +972-9-7692018 > irc: eedri (on #tlv #rhev-dev #rhev-integ) > > _______________________________________________ > Devel mailing list > Devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel >