[ovirt-devel] Re: [URGENT] Re: Re: [ OST Failure Report ] [ oVirt Master ] [ 27-02-2019 ] [ 002_bootstrap.get_host_devices]

On Fri, 1 Mar 2019, 16:23 Greg Sheremeta, <gshereme(a)redhat.com&gt; wrote: > > > On Fri, Mar 1, 2019 at 9:20 AM Dafna Ron <dron(a)redhat.com&gt; wrote: > >> >> >> On Fri, Mar 1, 2019 at 2:07 PM Dan Kenigsberg <danken(a)redhat.com&gt; wrote: >> >>> >>> >>> On Fri, 1 Mar 2019, 15:48 Dafna Ron, <dron(a)redhat.com&gt; wrote: >>> >>>> Hi Dan, >>>> >>>> our intention is never to skip tests and I would not merge a SkipTest >>>> patch if someone actually shows interest in resolving this. >>>> if you notice, this was reported a few days ago, after CI determined >>>> that this is not an infra related issue and since then, but aside from >>>> Michal asking for logs and getting a link to them, no other communication >>>> from dev that suggest that this is being checked was sent - even after I >>>> added [URGENT] and asked if there is any update. >>>> As we have done in the past, when I see that there is no cooperation >>>> from the developers to resolve a sporadic test failure, then I add a skip >>>> test until someone in dev has time to resolve the issue >>>> >>> >>> I believe that this is wrong. You should not merge a Skip before having >>> a phone call with the relevant team lead. >>> >> >> If we had any info on who to speak with we would probably do so - in >> cases like this, where there are multiple projects failures and its >> sporadic, its hard for us to pin someone down. >> Also, please note that the CI team is here to maintain the >> infrastructure that allows developers to run the tests and I don't think >> that we should be chasing after the developers to fix regressions (either >> in the code or in the tests) and that this should be dev's responsibility. >> > > I can see both sides of the argument, but developers need to do a better > job of being more responsive to Dafna. She is always extremely diligent > about reporting these, including follow-ups, and sometimes she doesn't get > replies. > Note that I want her to do less, not more: identify a suspected team lead, and tell him/her it is their job to fix it. That's all. Phone call? I don't answer my phone unless it's my wife (phone spam in the > US is unrelenting). > So she should IRC/WhatsApp/gChat you if she thinks you are responsible for a failure. I think the bigger issue is that everyone (ok, most people) filters devel > and other lists into a label and hardly checks it. IMO that needs to stop. > True. I think that late-evening phone call can help make it happen.

Or maybe we need a dedicated ost-failures list that people don't filter. > The same people who filter out URGENT messages from Dafna would easily filter a new list. It might be easier for them. But none of them should be a maintainer or team lead.

That sounds a little better to me than a single dev contact. > Each *test* has a natural dev contact. Host devices are virt, so the contract here is Michal+Ryan until proven differently or suspected differently by Dafna. > Greg > > >> >> I think that it may be time to have a dev contact which would be helping >> to debug and assign issues such as these and I will raise that in a >> different thread. >> >>> >>> - not ideal but having different project fail sporadically is not ideal >>>> as well >>>> >>>> As I see that you guys are now responding I have no reason to merge >>>> the patch - and that is what we all want. >>>> >>>> I can say that since its effecting more then one project, and from >>>> past experience, its either related to the OST test itself or to the >>>> API/SDK. >>>> I think that Evgheni ruled out an infra issue and suggested that the >>>> issue is actually related to the API - so not sure its strictly related to >>>> Ryan. >>>> >>> >>> I'll trust you here. Though since the failing test is Ryan's, it is >>> Ryan's job to find what breaks it, and pass the buck. >>> >>> If you believe it is a bug in oVirt ask, then you should call mperina, >>> too. >>> >>> >>>> Thanks, >>>> Dafna >>>> >>>> >>>> >>>> On Fri, Mar 1, 2019 at 1:27 PM Dan Kenigsberg <danken(a)redhat.com&gt; >>>> wrote: >>>> >>>>> I think this would be a bad idea, Dafna. >>>>> >>>>> OST exists to find bug, not to skip tests. >>>>> Michal said that he be working on it. It's time to call him (or Ryan >>>>> or Shmuel) to hear his finding, not to kill his tests. >>>>> >>>>> If you do skip his test, you should file an oVirt bug about it. >>>>> >>>>> On Fri, 1 Mar 2019, 11:46 Dafna Ron, <dron(a)redhat.com&gt; wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> As this is failing on average 3 patches a day (from different >>>>>> projects) and it seems no one is working on debugging this issue, I will >>>>>> add a patch to skip this test today. >>>>>> >>>>>> Thanks, >>>>>> Dafna >>>>>> >>>>>> >>>>>> On Thu, Feb 28, 2019 at 2:12 PM Dafna Ron <dron(a)redhat.com&gt; wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> We are still getting random failures on this test. >>>>>>> Is anyone working on this? >>>>>>> >>>>>>> Thanks, >>>>>>> Dafna >>>>>>> >>>>>>> >>>>>>> On Wed, Feb 27, 2019 at 9:10 PM Nir Soffer <nsoffer(a)redhat.com&gt; >>>>>>> wrote: >>>>>>> >>>>>>>> On Wed, Feb 27, 2019 at 4:31 PM Dan Kenigsberg <danken(a)redhat.com&gt; >>>>>>>> wrote: >>>>>>>> >>>>>>>>> HTH >>>>>>>>> >>>>>>>>> >>>>>>>>> https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13163/arti... >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13163/arti... >>>>>>>>> >>>>>>>>> >>>>>>>> Hers is another failure: >>>>>>>> >>>>>>>> https://jenkins.ovirt.org/job/ovirt-system-tests_manual/4169/testReport/j... >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Feb 27, 2019 at 4:24 PM Michal Skrivanek < >>>>>>>>> michal.skrivanek(a)redhat.com&gt; wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 27 Feb 2019, at 14:07, Galit Rosenthal <grosenth(a)redhat.com&gt; >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> we are failing basic suite master (vdsm, ovirt-engine-sdk ...) >>>>>>>>>> >>>>>>>>>> It happens sporadically. >>>>>>>>>> >>>>>>>>>> We disqualify that this is an infra issue. >>>>>>>>>> >>>>>>>>>> Can you please have a look at the issue? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ERROR: >>>>>>>>>> >>>>>>>>>> <testcase classname="002_bootstrap" name="get_host_devices" time="0.077"> >>>>>>>>>> <error type="exceptions.RuntimeError" message="*Could not find block_vda_1 device in host devices*: "><![CDATA[Traceback (most recent call last): >>>>>>>>>> File "/usr/lib64/python2.7/unittest/case.py", line 369, in run >>>>>>>>>> testMethod() >>>>>>>>>> File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest >>>>>>>>>> self.test(*self.arg) >>>>>>>>>> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test >>>>>>>>>> test() >>>>>>>>>> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper >>>>>>>>>> return func(get_test_prefix(), *args, **kwargs) >>>>>>>>>> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper >>>>>>>>>> prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs >>>>>>>>>> File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 1016, in get_host_devices >>>>>>>>>> raise RuntimeError('Could not find block_vda_1 device in host devices: {}'.format(device_list)) >>>>>>>>>> RuntimeError: Could not find block_vda_1 device in host devices: >>>>>>>>>> ]]></error> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Galit >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Examples, Full logs: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> where can I find full logs? It would be interesting to see >>>>>>>>>> hosts’ logs as well as engine.log >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> michal >>>>>>>>>> >>>>>>>>>> [1] https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/ovirt... >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [2] >>>>>>>>>> >>>>>>>>>> https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/ovirt... >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Infra mailing list -- infra(a)ovirt.org >>>>>>>>>> To unsubscribe send an email to infra-leave(a)ovirt.org >>>>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>>>> oVirt Code of Conduct: >>>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>> List Archives: >>>>>>>>>> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/WXW6II3NYEX... >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> GALIT ROSENTHAL >>>>>>>>>> >>>>>>>>>> SOFTWARE ENGINEER >>>>>>>>>> Red Hat >>>>>>>>>> >>>>>>>>>> <https://www.redhat.com/> >>>>>>>>>> >>>>>>>>>> galit(a)gmail.com T: 972-9-7692230 >>>>>>>>>> <https://red.ht/sig> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>>>> oVirt Code of Conduct: >>>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>> List Archives: >>>>>>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/HF4MB2VMJKB... >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>>>> oVirt Code of Conduct: >>>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>> List Archives: >>>>>>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/YY4MXJ44Q6T... >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>>> oVirt Code of Conduct: >>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>> List Archives: >>>>>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/KUHLEKMAVQ7... >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>> oVirt Code of Conduct: >>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>> List Archives: >>>>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/6HHUAPUKLVL... >>>>>>>> >>>>>>> _______________________________________________ >> Devel mailing list -- devel(a)ovirt.org >> To unsubscribe send an email to devel-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/ERLW4GNWISM... >> > > > -- > > GREG SHEREMETA > > SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX > > Red Hat NA > > <https://www.redhat.com/> > > gshereme(a)redhat.com IRC: gshereme > <https://red.ht/sig> > _______________________________________________ Devel mailing list -- devel(a)ovirt.org To unsubscribe send an email to devel-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/SLTCK7D6QG2...