[ovirt-devel] [ OST Failure Report ] [ oVirt master ] [ 03-07-2017 ] [ 006_migrations.migrate_vm ]

Wed Jul 5 07:18:11 UTC 2017

On Wed, Jul 5, 2017 at 10:02 AM, Yaniv Kaul <ykaul at redhat.com> wrote:

>
>
> On Wed, Jul 5, 2017 at 9:39 AM, Irit Goihman <igoihman at redhat.com> wrote:
>
>> https://gerrit.ovirt.org/#/c/78536 was indeed the offending patch, the
>> change was reverted and OST should pass now.
>>
>
> - Do we know why?
>

High CPU load caused by reactor thread which is triggered after heartbeats
timeout has exceeded.  I'm still testing it and trying to find the root
cause.

- O-S-T seems to be a great tool for finding JSON-RPC/STOMP issues. I
> suggest running it on every change related to these.
>
> This will be part of verification from now on.

> Y.
>
>
>>
>> On Tue, Jul 4, 2017 at 5:19 PM, Eyal Edri <eedri at redhat.com> wrote:
>>
>>> Guys,
>>>
>>> I think we proved which vdsm works ( git sha1: 28558d7 ) and what was
>>> the changelog since until the point it fails, so you have the list of
>>> changes and steps to reproduce locally.
>>> Which again, is reproducible on CI and locally, so please go over the
>>> changes done or reproduce the problem locally and see the issue on a live
>>> system.
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Jul 4, 2017 at 5:07 PM, Piotr Kliczewski <
>>> piotr.kliczewski at gmail.com> wrote:
>>>
>>>> Looking at the last experimental job the reason of the failure is:
>>>>
>>>> 2017-07-04 09:39:10,491-04 ERROR [org.ovirt.engine.api.restapi.
>>>> resource.AbstractBackendResource] (default task-18) [] Operation
>>>> Failed: [Cannot run VM. There is no host that satisfies current scheduling
>>>> constraints. See below for details:, The host lago-basic-suite-master-host0
>>>> did not satisfy internal filter CPUOverloaded because its CPU is too
>>>> loaded.]
>>>>
>>>> Do we think that vdsm increased its cpu consumption recently?
>>>>
>>>> On Tue, Jul 4, 2017 at 3:54 PM, Irit Goihman <igoihman at redhat.com>
>>>> wrote:
>>>>
>>>>> I've checked vdsm logs and couldn't find anything related to my change.
>>>>> I'll run OST without my changes and see if it runs successfully.
>>>>>
>>>>> On Tue, Jul 4, 2017 at 4:49 PM, Eyal Edri <eedri at redhat.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Jul 4, 2017 at 4:29 PM, Dafna Ron <dron at redhat.com> wrote:
>>>>>>
>>>>>>> This issue is reproduced locally as well.
>>>>>>>
>>>>>>> you can run the following to reproduce locally
>>>>>>>
>>>>>>> ./run_suite.sh -s http://jenkins.ovirt.org/job/v
>>>>>>> dsm_master_build-artifacts-el7-x86_64/2694/ basic-suite-master
>>>>>>>
>>>>>>> you will have the environment still running which would allow to
>>>>>>> view the live environment.
>>>>>>> if you have any issues please ping me and I will help any way I can.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Dafna
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Here is the list of changes done from the vdsm that is verified ( in
>>>>>> tested now ) to HEAD:
>>>>>>
>>>>>> * 74b2276 - (HEAD -> master, origin/master, origin/HEAD) stomp: add
>>>>>> integration tests for client reconnect (6 hours ago) Irit Goihman <
>>>>>> igoihman at redhat.com>
>>>>>> * 2a2f6cd - stomp: set default heartbeat values and add grace period
>>>>>> (6 hours ago) Irit Goihman <igoihman at redhat.com>
>>>>>> * 56c306a - tests: Make random uuid test repeatable (17 hours ago)
>>>>>> Nir Soffer <nsoffer at redhat.com>
>>>>>> * 864d4e3 - python3: Fix UUID packing/unpacking on python 3 (17 hours
>>>>>> ago) Nir Soffer <nsoffer at redhat.com>
>>>>>> * 4ac4221 - python3: Improve uuid packing tests (17 hours ago) Nir
>>>>>> Soffer <nsoffer at redhat.com>
>>>>>> * d264c8d - python3: Run misc_test in python 3 (17 hours ago) Nir
>>>>>> Soffer <nsoffer at redhat.com>
>>>>>> * f923b0b - storage: Added disk type change logging (18 hours ago)
>>>>>> Denis Chaplygin <dchaplyg at redhat.com>
>>>>>> * f1d54a1 - net: Unneeded newline is added when updating only the mtu
>>>>>> (25 hours ago) Edward Haas <edwardh at redhat.com>
>>>>>> * 9056d61 - virt: metadata: remove dead code (26 hours ago) Francesco
>>>>>> Romani <fromani at redhat.com>
>>>>>> * 08982b4 - virt: network: use core.find_device_guest_address (31
>>>>>> hours ago) Francesco Romani <fromani at redhat.com>
>>>>>> * 62e2bc5 - python3: Run qcow2_test on python 3 (2 days ago) Nir
>>>>>> Soffer <nsoffer at redhat.com>
>>>>>> * 42f5efb - stomp: implement client reconnect (2 days ago) Irit
>>>>>> Goihman <igoihman at redhat.com>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 07/04/2017 01:35 PM, Barak Korren wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 4 July 2017 at 14:32, Irit Goihman <igoihman at redhat.com> wrote:
>>>>>>>
>>>>>>>> https://gerrit.ovirt.org/#/c/78536 broke network functional tests
>>>>>>>> but a fix was merged today: https://gerrit.ovirt.org/#/c/78925/
>>>>>>>>
>>>>>>>> I tried to run OST with my fix yesterday and still encountered the
>>>>>>>> same failures.
>>>>>>>>
>>>>>>>
>>>>>>> Here is a reproducer of the failure with the fix patch:
>>>>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1061/
>>>>>>>
>>>>>>> So that was not it probably...
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Barak Korren
>>>>>>> RHV DevOps team , RHCE, RHCi
>>>>>>> Red Hat EMEA
>>>>>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Devel mailing listDevel at ovirt.orghttp://lists.ovirt.org/mailman/listinfo/devel
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Devel mailing list
>>>>>>> Devel at ovirt.org
>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Eyal edri
>>>>>>
>>>>>>
>>>>>> ASSOCIATE MANAGER
>>>>>>
>>>>>> RHV DevOps
>>>>>>
>>>>>> EMEA VIRTUALIZATION R&D
>>>>>>
>>>>>>
>>>>>> Red Hat EMEA <https://www.redhat.com/>
>>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED.
>>>>>> <https://redhat.com/trusted>
>>>>>> phone: +972-9-7692018 <+972%209-769-2018>
>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> IRIT GOIHMAN
>>>>>
>>>>> SOFTWARE ENGINEER
>>>>>
>>>>> EMEA VIRTUALIZATION R&D
>>>>>
>>>>> Red Hat EMEA <https://www.redhat.com/>
>>>>>
>>>>> <https://red.ht/sig>
>>>>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
>>>>> @redhatnews <https://twitter.com/redhatnews>   Red Hat
>>>>> <https://www.linkedin.com/company/red-hat>   Red Hat
>>>>> <https://www.facebook.com/RedHatInc>
>>>>>
>>>>> _______________________________________________
>>>>> Devel mailing list
>>>>> Devel at ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Eyal edri
>>>
>>>
>>> ASSOCIATE MANAGER
>>>
>>> RHV DevOps
>>>
>>> EMEA VIRTUALIZATION R&D
>>>
>>>
>>> Red Hat EMEA <https://www.redhat.com/>
>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED.
>>> <https://redhat.com/trusted>
>>> phone: +972-9-7692018 <+972%209-769-2018>
>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>
>>
>>
>>
>> --
>>
>> IRIT GOIHMAN
>>
>> SOFTWARE ENGINEER
>>
>> EMEA VIRTUALIZATION R&D
>>
>> Red Hat EMEA <https://www.redhat.com/>
>>
>> <https://red.ht/sig>
>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
>> @redhatnews <https://twitter.com/redhatnews>   Red Hat
>> <https://www.linkedin.com/company/red-hat>   Red Hat
>> <https://www.facebook.com/RedHatInc>
>>
>> _______________________________________________
>> Devel mailing list
>> Devel at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>>
>
>

-- 

IRIT GOIHMAN

SOFTWARE ENGINEER

EMEA VIRTUALIZATION R&D

Red Hat EMEA <https://www.redhat.com/>

<https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
@redhatnews <https://twitter.com/redhatnews>   Red Hat
<https://www.linkedin.com/company/red-hat>   Red Hat
<https://www.facebook.com/RedHatInc>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20170705/3b3ff2ed/attachment-0001.html>