Re: [ovirt-devel] Experimental Flow for Master Fails to Run a VM

FYI, I opened a bug [1] to track this issue since I don't see any attempts to resolve the issue on the thread, hopefully a bug will get more attention. Opened on VDSM since we see the libvirt error there, feel free to move product/team. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1401303 On Sun, Dec 4, 2016 at 1:23 PM, Eyal Edri <eedri(a)redhat.com&gt; wrote: > Not sure if relevant, but Juan posted a fix for SDK4 last time it > happened ( but different failure on log-collector ): > > https://gerrit.ovirt.org/#/c/67213/ > > * Added `urandom` to the `RngSource` enumerated type. > > On Sun, Dec 4, 2016 at 9:17 AM, Eyal Edri <eedri(a)redhat.com&gt; wrote: > >> And its still failing from Friday, >> Since we don't have official Centos 7.3 repos yet ( hopefully we'll have >> it this week, but as of this moment its not published yet ) , we have to >> either revert the offending patch >> or send a quick fix. >> >> Right now all experimental flows for master are not working and nightly >> rpms are not refreshed with new RPMs. >> >> >> >> On Fri, Dec 2, 2016 at 9:41 PM, Yaniv Kaul <ykaul(a)redhat.com&gt; wrote: >> >>> >>> >>> On Dec 2, 2016 2:11 PM, "Anton Marchukov" <amarchuk(a)redhat.com&gt; wrote: >>> >>> Hello Martin. >>> >>> Do by outdated you mean the old libvirt? If so that is that livirt >>> available in CentOS 7.2? There is no 7.3 yet. >>> >>> >>> Right, this is the issue. >>> Y. >>> >>> >>> Anton. >>> >>> On Fri, Dec 2, 2016 at 1:07 PM, Martin Polednik <mpolednik(a)redhat.com&gt; >>> wrote: >>> >>>> On 02/12/16 10:55 +0100, Anton Marchukov wrote: >>>> >>>>> Hello All. >>>>> >>>>> Engine log can be viewed here: >>>>> >>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ma >>>>> ster/3838/artifact/exported-artifacts/basic_suite_master.sh- >>>>> el7/exported-artifacts/test_logs/basic-suite-master/post-004 >>>>> _basic_sanity.py/lago-basic-suite-master-engine/_var_log_ovi >>>>> rt-engine/engine.log >>>>> >>>>> I see the following exception there: >>>>> >>>>> 2016-12-02 04:29:24,030-05 DEBUG >>>>> [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] >>>>> (ResponseWorker) [83b6b5d] Message received: {"jsonrpc": "2.0", "id": >>>>> "ec254aad-441b-47e7-a644-aebddcc1d62c", "result": true} >>>>> 2016-12-02 04:29:24,030-05 ERROR >>>>> [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) >>>>> [83b6b5d] Not able to update response for >>>>> "ec254aad-441b-47e7-a644-aebddcc1d62c" >>>>> 2016-12-02 04:29:24,041-05 DEBUG >>>>> [org.ovirt.engine.core.utils.timer.FixedDelayJobListener] >>>>> (DefaultQuartzScheduler3) [47a31d72] Rescheduling >>>>> DEFAULT.org.ovirt.engine.core.bll.gluster.GlusterSyncJob.ref >>>>> reshLightWeightData#-9223372036854775775 >>>>> as there is no unfired trigger. >>>>> 2016-12-02 04:29:24,024-05 DEBUG >>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default >>>>> task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] Exception: >>>>> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: >>>>> VDSGenericException: VDSNetworkException: Timeout during xml-rpc call >>>>> at org.ovirt.engine.core.vdsbroke >>>>> r.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:73) >>>>> [vdsbroker.jar:] >>>>> >>>>> ... >>>>> >>>>> 2016-12-02 04:29:24,042-05 ERROR >>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default >>>>> task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] Timeout waiting for >>>>> VDSM response: Internal timeout occured >>>>> 2016-12-02 04:29:24,044-05 DEBUG >>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] >>>>> (default task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] START, >>>>> GetCapabilitiesVDSCommand(HostName = lago-basic-suite-master-host0, >>>>> VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', >>>>> hostId='5eb7019e-28a3-4f93-9188-685b6c64a2f5', >>>>> vds='Host[lago-basic-suite-master-host0,5eb7019e-28a3-4f93-9 >>>>> 188-685b6c64a2f5]'}), >>>>> log id: 58f448b8 >>>>> 2016-12-02 04:29:24,044-05 DEBUG >>>>> [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (default >>>>> task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] SEND >>>>> destination:jms.topic.vdsm_requests >>>>> reply-to:jms.topic.vdsm_responses >>>>> content-length:105 >>>>> >>>>> >>>>> Please note that this runs on localhost with local bridge. So it is >>>>> not >>>>> likely to be network itself. >>>>> >>>> >>>> The main issue I see is that the VM run command has actually failed >>>> due to libvirt no accepting /dev/urandom as RNG source[1]. This was >>>> done as engine patch and according to git log, posted around Mon Nov >>>> 28. Also adding Jakub - this should either not happen from engine's >>>> point of view or the lago host is outdated. >>>> >>>> [1] >>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ma >>>> ster/3838/artifact/exported-artifacts/basic_suite_master.sh- >>>> el7/exported-artifacts/test_logs/basic-suite-master/post-004 >>>> _basic_sanity.py/lago-basic-suite-master-host0/_var_log_vdsm/vdsm.log >>>> >>>> >>>> Anton. >>>>> >>>>> On Fri, Dec 2, 2016 at 10:43 AM, Anton Marchukov <amarchuk(a)redhat.com >>>>> > >>>>> wrote: >>>>> >>>>> FYI. Experimental flow for master currently fails to run a VM. The >>>>>> tests >>>>>> times out while waiting for 180 seconds: >>>>>> >>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ >>>>>> master/3838/testReport/(root)/004_basic_sanity/vm_run/ >>>>>> >>>>>> This is reproducible over 23 runs of this happened tonight, sounds >>>>>> like a >>>>>> regression to me: >>>>>> >>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/ >>>>>> >>>>>> I will update here with additional information once I find it. >>>>>> >>>>>> Last successful run was with this patch: >>>>>> >>>>>> https://gerrit.ovirt.org/#/c/66416/ (vdsm: API: move vm parameters >>>>>> fixup >>>>>> in a method) >>>>>> >>>>>> Known to start failing around this patch: >>>>>> >>>>>> https://gerrit.ovirt.org/#/c/67647/ (vdsmapi: fix a typo in string >>>>>> formatting) >>>>>> >>>>>> Please notes that we do not have gating implemented yet, so >>>>>> everything >>>>>> that was merged in between those patches might have caused this (not >>>>>> necessary in vdsm project). >>>>>> >>>>>> Anton. >>>>>> -- >>>>>> Anton Marchukov >>>>>> Senior Software Engineer - RHEV CI - Red Hat >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> Anton Marchukov >>>>> Senior Software Engineer - RHEV CI - Red Hat >>>>> >>>> >>>> _______________________________________________ >>>>> Devel mailing list >>>>> Devel(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>> >>>> >>>> >>> >>> >>> -- >>> Anton Marchukov >>> Senior Software Engineer - RHEV CI - Red Hat >>> >>> >>> _______________________________________________ >>> Devel mailing list >>> Devel(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/devel >>> >>> >>> >>> _______________________________________________ >>> Devel mailing list >>> Devel(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/devel >>> >> >> >> >> -- >> Eyal Edri >> Associate Manager >> RHV DevOps >> EMEA ENG Virtualization R&D >> Red Hat Israel >> >> phone: +972-9-7692018 >> irc: eedri (on #tlv #rhev-dev #rhev-integ) >> > > > > -- > Eyal Edri > Associate Manager > RHV DevOps > EMEA ENG Virtualization R&D > Red Hat Israel > > phone: +972-9-7692018 > irc: eedri (on #tlv #rhev-dev #rhev-integ) > -- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)