[ovirt-devel] Experimental Flow for Master Fails to Run a VM

Sun Dec 4 20:57:13 UTC 2016

tests are back to stable,  but with a cost of not testing 4.1 CL.

Iets hope we get centos 7.3 soon.

On Dec 4, 2016 22:41, "Yaniv Kaul" <ykaul at redhat.com> wrote:

>
>
> On Dec 4, 2016 6:42 PM, "Arik Hadas" <ahadas at redhat.com> wrote:
>
> Yaniv will try to lower the cluster level used in the system-tests to 4.0
> - this is supposed to solve the issue.
>
>
> Done.
> Y.
>
> If it won't help (we will know it in about an hour), we'll add a db-script
> that changes the rng device of the blank template only.
>
> On Sun, Dec 4, 2016 at 3:34 PM, Eyal Edri <eedri at redhat.com> wrote:
>
>> FYI,
>>
>> I opened a bug [1] to track this issue since I don't see any attempts to
>> resolve the issue on the thread, hopefully a bug will get more attention.
>> Opened on VDSM since we see the libvirt error there, feel free to move
>> product/team.
>>
>>
>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1401303
>>
>> On Sun, Dec 4, 2016 at 1:23 PM, Eyal Edri <eedri at redhat.com> wrote:
>>
>>> Not sure if relevant, but Juan posted a fix for SDK4 last time it
>>> happened ( but different failure on log-collector ):
>>>
>>> https://gerrit.ovirt.org/#/c/67213/
>>>
>>> * Added `urandom` to the `RngSource` enumerated type.
>>>
>>> On Sun, Dec 4, 2016 at 9:17 AM, Eyal Edri <eedri at redhat.com> wrote:
>>>
>>>> And its still failing from Friday,
>>>> Since we don't have official Centos 7.3 repos yet ( hopefully we'll
>>>> have it this week, but as of this moment its not published yet ) , we have
>>>> to either revert the offending patch
>>>> or send a quick fix.
>>>>
>>>> Right now all experimental flows for master are not working and nightly
>>>> rpms are not refreshed with new RPMs.
>>>>
>>>>
>>>>
>>>> On Fri, Dec 2, 2016 at 9:41 PM, Yaniv Kaul <ykaul at redhat.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Dec 2, 2016 2:11 PM, "Anton Marchukov" <amarchuk at redhat.com> wrote:
>>>>>
>>>>> Hello Martin.
>>>>>
>>>>> Do by outdated you mean the old libvirt? If so that is that livirt
>>>>> available in CentOS 7.2? There is no 7.3 yet.
>>>>>
>>>>>
>>>>> Right, this is the issue.
>>>>> Y.
>>>>>
>>>>>
>>>>> Anton.
>>>>>
>>>>> On Fri, Dec 2, 2016 at 1:07 PM, Martin Polednik <mpolednik at redhat.com>
>>>>> wrote:
>>>>>
>>>>>> On 02/12/16 10:55 +0100, Anton Marchukov wrote:
>>>>>>
>>>>>>> Hello All.
>>>>>>>
>>>>>>> Engine log can be viewed here:
>>>>>>>
>>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ma
>>>>>>> ster/3838/artifact/exported-artifacts/basic_suite_master.sh-
>>>>>>> el7/exported-artifacts/test_logs/basic-suite-master/post-004
>>>>>>> _basic_sanity.py/lago-basic-suite-master-engine/_var_log_ovi
>>>>>>> rt-engine/engine.log
>>>>>>>
>>>>>>> I see the following exception there:
>>>>>>>
>>>>>>> 2016-12-02 04:29:24,030-05 DEBUG
>>>>>>> [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker]
>>>>>>> (ResponseWorker) [83b6b5d] Message received: {"jsonrpc": "2.0", "id":
>>>>>>> "ec254aad-441b-47e7-a644-aebddcc1d62c", "result": true}
>>>>>>> 2016-12-02 04:29:24,030-05 ERROR
>>>>>>> [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker)
>>>>>>> [83b6b5d] Not able to update response for
>>>>>>> "ec254aad-441b-47e7-a644-aebddcc1d62c"
>>>>>>> 2016-12-02 04:29:24,041-05 DEBUG
>>>>>>> [org.ovirt.engine.core.utils.timer.FixedDelayJobListener]
>>>>>>> (DefaultQuartzScheduler3) [47a31d72] Rescheduling
>>>>>>> DEFAULT.org.ovirt.engine.core.bll.gluster.GlusterSyncJob.ref
>>>>>>> reshLightWeightData#-9223372036854775775
>>>>>>> as there is no unfired trigger.
>>>>>>> 2016-12-02 04:29:24,024-05 DEBUG
>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default
>>>>>>> task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] Exception:
>>>>>>> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
>>>>>>> VDSGenericException: VDSNetworkException: Timeout during xml-rpc call
>>>>>>>         at org.ovirt.engine.core.vdsbroke
>>>>>>> r.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:73)
>>>>>>> [vdsbroker.jar:]
>>>>>>>
>>>>>>> ...
>>>>>>>
>>>>>>> 2016-12-02 04:29:24,042-05 ERROR
>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default
>>>>>>> task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] Timeout waiting for
>>>>>>> VDSM response: Internal timeout occured
>>>>>>> 2016-12-02 04:29:24,044-05 DEBUG
>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVD
>>>>>>> SCommand]
>>>>>>> (default task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] START,
>>>>>>> GetCapabilitiesVDSCommand(HostName = lago-basic-suite-master-host0,
>>>>>>> VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
>>>>>>> hostId='5eb7019e-28a3-4f93-9188-685b6c64a2f5',
>>>>>>> vds='Host[lago-basic-suite-master-host0,5eb7019e-28a3-4f93-9
>>>>>>> 188-685b6c64a2f5]'}),
>>>>>>> log id: 58f448b8
>>>>>>> 2016-12-02 04:29:24,044-05 DEBUG
>>>>>>> [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (default
>>>>>>> task-12) [d932871a-af4f-4fc9-9ee5-f7a0126a7b85] SEND
>>>>>>> destination:jms.topic.vdsm_requests
>>>>>>> reply-to:jms.topic.vdsm_responses
>>>>>>> content-length:105
>>>>>>>
>>>>>>>
>>>>>>> Please note that this runs on localhost with local bridge. So it is
>>>>>>> not
>>>>>>> likely to be network itself.
>>>>>>>
>>>>>>
>>>>>> The main issue I see is that the VM run command has actually failed
>>>>>> due to libvirt no accepting /dev/urandom as RNG source[1]. This was
>>>>>> done as engine patch and according to git log, posted around Mon Nov
>>>>>> 28. Also adding Jakub - this should either not happen from engine's
>>>>>> point of view or the lago host is outdated.
>>>>>>
>>>>>> [1]
>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ma
>>>>>> ster/3838/artifact/exported-artifacts/basic_suite_master.sh-
>>>>>> el7/exported-artifacts/test_logs/basic-suite-master/post-004
>>>>>> _basic_sanity.py/lago-basic-suite-master-host0/_var_log_vdsm/vdsm.log
>>>>>>
>>>>>>
>>>>>> Anton.
>>>>>>>
>>>>>>> On Fri, Dec 2, 2016 at 10:43 AM, Anton Marchukov <
>>>>>>> amarchuk at redhat.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> FYI. Experimental flow for master currently fails to run a VM. The
>>>>>>>> tests
>>>>>>>> times out while waiting for 180 seconds:
>>>>>>>>
>>>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_
>>>>>>>> master/3838/testReport/(root)/004_basic_sanity/vm_run/
>>>>>>>>
>>>>>>>> This is reproducible over 23 runs of this happened tonight, sounds
>>>>>>>> like a
>>>>>>>> regression to me:
>>>>>>>>
>>>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/
>>>>>>>>
>>>>>>>> I will update here with additional information once I find it.
>>>>>>>>
>>>>>>>> Last successful run was with this patch:
>>>>>>>>
>>>>>>>> https://gerrit.ovirt.org/#/c/66416/ (vdsm: API: move vm parameters
>>>>>>>> fixup
>>>>>>>> in a method)
>>>>>>>>
>>>>>>>> Known to start failing around this patch:
>>>>>>>>
>>>>>>>> https://gerrit.ovirt.org/#/c/67647/ (vdsmapi: fix a typo in string
>>>>>>>> formatting)
>>>>>>>>
>>>>>>>> Please notes that we do not have gating implemented yet, so
>>>>>>>> everything
>>>>>>>> that was merged in between those patches might have caused this (not
>>>>>>>> necessary in vdsm project).
>>>>>>>>
>>>>>>>> Anton.
>>>>>>>> --
>>>>>>>> Anton Marchukov
>>>>>>>> Senior Software Engineer - RHEV CI - Red Hat
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Anton Marchukov
>>>>>>> Senior Software Engineer - RHEV CI - Red Hat
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>>> Devel mailing list
>>>>>>> Devel at ovirt.org
>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Anton Marchukov
>>>>> Senior Software Engineer - RHEV CI - Red Hat
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Devel mailing list
>>>>> Devel at ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Devel mailing list
>>>>> Devel at ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Eyal Edri
>>>> Associate Manager
>>>> RHV DevOps
>>>> EMEA ENG Virtualization R&D
>>>> Red Hat Israel
>>>>
>>>> phone: +972-9-7692018
>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>>
>>>
>>>
>>>
>>> --
>>> Eyal Edri
>>> Associate Manager
>>> RHV DevOps
>>> EMEA ENG Virtualization R&D
>>> Red Hat Israel
>>>
>>> phone: +972-9-7692018
>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>
>>
>>
>>
>> --
>> Eyal Edri
>> Associate Manager
>> RHV DevOps
>> EMEA ENG Virtualization R&D
>> Red Hat Israel
>>
>> phone: +972-9-7692018
>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>
>> _______________________________________________
>> Devel mailing list
>> Devel at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20161204/14e088f4/attachment-0001.html>