[ovirt-devel] Re: master CQ: random failures for add_master_storage_domain

Milan Zamazal <mzamazal(a)redhat.com&gt; writes: Is anybody working on any fix of the failure? >> Thanks, >> michal >> >>> >>>> while the other host is still in status Installing (so it cannot be used >>>> for fencing- hence the fence action failure). >>>> The vdsm then goes back up in few moments, but engine, expecting the host >>>> is up all the time, meanwhile fails doing an operation that requires host >>>> to be up. >>>> >>>> [1] >>>> https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/15829/arti... >>>> >>>> On Fri, Sep 13, 2019 at 5:18 PM Dusan Fodor <dfodor(a)redhat.com&gt; wrote: >>>> >>>>> For brave investigators, similar issue in later stage of the same test can >>>>> be found here [1]. Same symptom of fence action fail, but this time it >>>>> causes failure for adding storage itself: >>>>> *2019-09-12 09:53:32,571-04 ERROR >>>>> [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default >>>>> task-1) [] Operation Failed: [Cannot attach Storage. There is no active >>>>> Host in the Data Center.]* >>>>> >>>>> [1] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/15821 >>>>> >>>>> On Fri, Sep 13, 2019 at 5:09 PM Dusan Fodor <dfodor(a)redhat.com&gt; wrote: >>>>> >>>>>> Hello all, >>>>>> lately i witnessed multiple failures for add_master_storage_domain test, >>>>>> which were not related to changes themselves, nor any infra issue. One >>>>>> example can be found here [1]. >>>>>> After investigation with huge help of Milan, issue is that Host falls >>>>>> from up state to whatever-but-not-up suddenly. >>>>>> >>>>>> >>>>>> 1. add_storage_domain picks a random host that is in up state >>>>>> 2. meantime engine starts fence action for it, so probably something >>>>>> gone bad with the host; the fence action fails with: >>>>>> *[org.ovirt.engine.core.bll.pm.FenceProxyLocator] >>>>>> (EE-ManagedThreadFactory-engineScheduled-Thread-38) [6692895f] Can not run >>>>>> fence action on host 'lago-basic-suite-master-host-0', no suitable proxy >>>>>> host was found.* >>>>>> 3. test fails on not being able to attach the domain to non-up >>>>>> host: >>>>>> *[org.ovirt.engine.api.restapi.resource.AbstractBackendResource] >>>>>> (default task-1) [] Operation Failed: [Cannot add storage server connection >>>>>> when Host status is not up]* >>>>>> >>>>>> For better orientation in failed job's engine log [1], fence action for >>>>>> host fails at >>>>>> :46:12,842-04 >>>>>> engine learns it cannot connect storage to host at >>>>>> :46:16,105-04 >>>>>> >>>>>> The test itself add_master_storage_domain starts at ~ :46:13,753 >>>>>> (according to lago log). >>>>>> >>>>>> Could you please check this? >>>>>> Thanks >>>>>> >>>>>> [1] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/15829 >>>>>> [2] >>>>>> https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/15829/arti... >>>>>> >>>>>> >>>> _______________________________________________ >>>> Devel mailing list -- devel(a)ovirt.org >>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>>> List Archives: >>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/MMH7DGCH24G... >>> _______________________________________________ >>> Devel mailing list -- devel(a)ovirt.org >>> To unsubscribe send an email to devel-leave(a)ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: >>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/KQY5JULWUDT...