Re: [ovirt-devel] [ OST Failure Report ] [ oVirt master ] [ 03-07-2017 ] [ 006_migrations.migrate_vm ]

On Wed, Jul 5, 2017 at 10:02 AM, Yaniv Kaul <ykaul(a)redhat.com&gt; wrote: > > > On Wed, Jul 5, 2017 at 9:39 AM, Irit Goihman <igoihman(a)redhat.com&gt; wrote: > >> https://gerrit.ovirt.org/#/c/78536 was indeed the offending patch, the >> change was reverted and OST should pass now. >> > > - Do we know why? > - O-S-T seems to be a great tool for finding JSON-RPC/STOMP issues. I > suggest running it on every change related to these. > In addition, if we have collectd installed now, can't we add a test that will check if CPU/Memory consumption spike above the normal and fail before it reach actions like vm run/migration?

> > Y. > > >> >> On Tue, Jul 4, 2017 at 5:19 PM, Eyal Edri <eedri(a)redhat.com&gt; wrote: >> >>> Guys, >>> >>> I think we proved which vdsm works ( git sha1: 28558d7 ) and what was >>> the changelog since until the point it fails, so you have the list of >>> changes and steps to reproduce locally. >>> Which again, is reproducible on CI and locally, so please go over the >>> changes done or reproduce the problem locally and see the issue on a live >>> system. >>> >>> >>> >>> >>> >>> On Tue, Jul 4, 2017 at 5:07 PM, Piotr Kliczewski < >>> piotr.kliczewski(a)gmail.com&gt; wrote: >>> >>>> Looking at the last experimental job the reason of the failure is: >>>> >>>> 2017-07-04 09:39:10,491-04 ERROR >>>> [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default >>>> task-18) [] Operation Failed: [Cannot run VM. There is no host that >>>> satisfies current scheduling constraints. See below for details:, The host >>>> lago-basic-suite-master-host0 did not satisfy internal filter CPUOverloaded >>>> because its CPU is too loaded.] >>>> >>>> Do we think that vdsm increased its cpu consumption recently? >>>> >>>> On Tue, Jul 4, 2017 at 3:54 PM, Irit Goihman <igoihman(a)redhat.com&gt; >>>> wrote: >>>> >>>>> I've checked vdsm logs and couldn't find anything related to my >>>>> change. >>>>> I'll run OST without my changes and see if it runs successfully. >>>>> >>>>> On Tue, Jul 4, 2017 at 4:49 PM, Eyal Edri <eedri(a)redhat.com&gt; wrote: >>>>> >>>>>> >>>>>> >>>>>> On Tue, Jul 4, 2017 at 4:29 PM, Dafna Ron <dron(a)redhat.com&gt; wrote: >>>>>> >>>>>>> This issue is reproduced locally as well. >>>>>>> >>>>>>> you can run the following to reproduce locally >>>>>>> >>>>>>> ./run_suite.sh -s >>>>>>> http://jenkins.ovirt.org/job/vdsm_master_build-artifacts-el7-x86_64/2694/ >>>>>>> basic-suite-master >>>>>>> >>>>>>> you will have the environment still running which would allow to >>>>>>> view the live environment. >>>>>>> if you have any issues please ping me and I will help any way I >>>>>>> can. >>>>>>> >>>>>>> Thanks, >>>>>>> Dafna >>>>>>> >>>>>>> >>>>>>> >>>>>> Here is the list of changes done from the vdsm that is verified ( in >>>>>> tested now ) to HEAD: >>>>>> >>>>>> * 74b2276 - (HEAD -> master, origin/master, origin/HEAD) stomp: add >>>>>> integration tests for client reconnect (6 hours ago) Irit Goihman < >>>>>> igoihman(a)redhat.com&gt; >>>>>> * 2a2f6cd - stomp: set default heartbeat values and add grace period >>>>>> (6 hours ago) Irit Goihman <igoihman(a)redhat.com&gt; >>>>>> * 56c306a - tests: Make random uuid test repeatable (17 hours ago) >>>>>> Nir Soffer <nsoffer(a)redhat.com&gt; >>>>>> * 864d4e3 - python3: Fix UUID packing/unpacking on python 3 (17 >>>>>> hours ago) Nir Soffer <nsoffer(a)redhat.com&gt; >>>>>> * 4ac4221 - python3: Improve uuid packing tests (17 hours ago) Nir >>>>>> Soffer <nsoffer(a)redhat.com&gt; >>>>>> * d264c8d - python3: Run misc_test in python 3 (17 hours ago) Nir >>>>>> Soffer <nsoffer(a)redhat.com&gt; >>>>>> * f923b0b - storage: Added disk type change logging (18 hours ago) >>>>>> Denis Chaplygin <dchaplyg(a)redhat.com&gt; >>>>>> * f1d54a1 - net: Unneeded newline is added when updating only the >>>>>> mtu (25 hours ago) Edward Haas <edwardh(a)redhat.com&gt; >>>>>> * 9056d61 - virt: metadata: remove dead code (26 hours ago) >>>>>> Francesco Romani <fromani(a)redhat.com&gt; >>>>>> * 08982b4 - virt: network: use core.find_device_guest_address (31 >>>>>> hours ago) Francesco Romani <fromani(a)redhat.com&gt; >>>>>> * 62e2bc5 - python3: Run qcow2_test on python 3 (2 days ago) Nir >>>>>> Soffer <nsoffer(a)redhat.com&gt; >>>>>> * 42f5efb - stomp: implement client reconnect (2 days ago) Irit >>>>>> Goihman <igoihman(a)redhat.com&gt; >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 07/04/2017 01:35 PM, Barak Korren wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 4 July 2017 at 14:32, Irit Goihman <igoihman(a)redhat.com&gt; wrote: >>>>>>> >>>>>>>> https://gerrit.ovirt.org/#/c/78536 broke network functional tests >>>>>>>> but a fix was merged today: https://gerrit.ovirt.org/#/c/78925/ >>>>>>>> >>>>>>>> I tried to run OST with my fix yesterday and still encountered the >>>>>>>> same failures. >>>>>>>> >>>>>>> >>>>>>> Here is a reproducer of the failure with the fix patch: >>>>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1061/ >>>>>>> >>>>>>> So that was not it probably... >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Barak Korren >>>>>>> RHV DevOps team , RHCE, RHCi >>>>>>> Red Hat EMEA >>>>>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Devel mailing listDevel@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/devel >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Devel mailing list >>>>>>> Devel(a)ovirt.org >>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Eyal edri >>>>>> >>>>>> >>>>>> ASSOCIATE MANAGER >>>>>> >>>>>> RHV DevOps >>>>>> >>>>>> EMEA VIRTUALIZATION R&D >>>>>> >>>>>> >>>>>> Red Hat EMEA <https://www.redhat.com/> >>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>>>>> <https://redhat.com/trusted> >>>>>> phone: +972-9-7692018 <+972%209-769-2018> >>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> IRIT GOIHMAN >>>>> >>>>> SOFTWARE ENGINEER >>>>> >>>>> EMEA VIRTUALIZATION R&D >>>>> >>>>> Red Hat EMEA <https://www.redhat.com/> >>>>> >>>>> <https://red.ht/sig> >>>>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >>>>> @redhatnews <https://twitter.com/redhatnews> Red Hat >>>>> <https://www.linkedin.com/company/red-hat> Red Hat >>>>> <https://www.facebook.com/RedHatInc> >>>>> >>>>> _______________________________________________ >>>>> Devel mailing list >>>>> Devel(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>> >>>> >>>> >>> >>> >>> -- >>> >>> Eyal edri >>> >>> >>> ASSOCIATE MANAGER >>> >>> RHV DevOps >>> >>> EMEA VIRTUALIZATION R&D >>> >>> >>> Red Hat EMEA <https://www.redhat.com/> >>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>> <https://redhat.com/trusted> >>> phone: +972-9-7692018 <+972%209-769-2018> >>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>> >> >> >> >> -- >> >> IRIT GOIHMAN >> >> SOFTWARE ENGINEER >> >> EMEA VIRTUALIZATION R&D >> >> Red Hat EMEA <https://www.redhat.com/> >> >> <https://red.ht/sig> >> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >> @redhatnews <https://twitter.com/redhatnews> Red Hat >> <https://www.linkedin.com/company/red-hat> Red Hat >> <https://www.facebook.com/RedHatInc> >> >> _______________________________________________ >> Devel mailing list >> Devel(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/devel >> > > -- Eyal edri ASSOCIATE MANAGER RHV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)