HTH https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13163/artifac... https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13163/artifac... On Wed, Feb 27, 2019 at 4:24 PM Michal Skrivanek < michal.skrivanek@redhat.com> wrote:

On 27 Feb 2019, at 14:07, Galit Rosenthal <grosenth@redhat.com> wrote:

Hi,

we are failing basic suite master (vdsm, ovirt-engine-sdk ...)

It happens sporadically.

We disqualify that this is an infra issue.

Can you please have a look at the issue?

ERROR:

<testcase classname="002_bootstrap" name="get_host_devices" time="0.077"> <error type="exceptions.RuntimeError" message="*Could not find block_vda_1 device in host devices*: "><![CDATA[Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 1016, in get_host_devices raise RuntimeError('Could not find block_vda_1 device in host devices: {}'.format(device_list)) RuntimeError: Could not find block_vda_1 device in host devices: ]]></error>

Thanks,

Galit

Examples, Full logs:

where can I find full logs? It would be interesting to see hosts’ logs as well as engine.log

Thanks, michal

[1] https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/ovirt-ma...

[2]

https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/ovirt-ma...

_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/WXW6II3NYEXHIS...

--

GALIT ROSENTHAL

SOFTWARE ENGINEER Red Hat

<https://www.redhat.com/>

galit@gmail.com T: 972-9-7692230 <https://red.ht/sig> _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/HF4MB2VMJKBO4O...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/YY4MXJ44Q6TJQR...

Hi, We are still getting random failures on this test. Is anyone working on this? Thanks, Dafna On Wed, Feb 27, 2019 at 9:10 PM Nir Soffer <nsoffer@redhat.com> wrote:

On Wed, Feb 27, 2019 at 4:31 PM Dan Kenigsberg <danken@redhat.com> wrote:

...

HTH

https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13163/artifac...

https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13163/artifac...

Hers is another failure:

https://jenkins.ovirt.org/job/ovirt-system-tests_manual/4169/testReport/juni...

...

On Wed, Feb 27, 2019 at 4:24 PM Michal Skrivanek < michal.skrivanek@redhat.com> wrote:

...

On 27 Feb 2019, at 14:07, Galit Rosenthal <grosenth@redhat.com> wrote:

Hi,

we are failing basic suite master (vdsm, ovirt-engine-sdk ...)

It happens sporadically.

We disqualify that this is an infra issue.

Can you please have a look at the issue?

ERROR:

<testcase classname="002_bootstrap" name="get_host_devices" time="0.077"> <error type="exceptions.RuntimeError" message="*Could not find block_vda_1 device in host devices*: "><![CDATA[Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 1016, in get_host_devices raise RuntimeError('Could not find block_vda_1 device in host devices: {}'.format(device_list)) RuntimeError: Could not find block_vda_1 device in host devices: ]]></error>

Thanks,

Galit

Examples, Full logs:

where can I find full logs? It would be interesting to see hosts’ logs as well as engine.log

Thanks, michal

[1] https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/ovirt-ma...

[2]

https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/ovirt-ma...

_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/WXW6II3NYEXHIS...

--

GALIT ROSENTHAL

SOFTWARE ENGINEER Red Hat

<https://www.redhat.com/>

galit@gmail.com T: 972-9-7692230 <https://red.ht/sig> _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/HF4MB2VMJKBO4O...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/YY4MXJ44Q6TJQR...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/KUHLEKMAVQ7NPX...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/6HHUAPUKLVLWSA...

I think this would be a bad idea, Dafna. OST exists to find bug, not to skip tests. Michal said that he be working on it. It's time to call him (or Ryan or Shmuel) to hear his finding, not to kill his tests. If you do skip his test, you should file an oVirt bug about it. On Fri, 1 Mar 2019, 11:46 Dafna Ron, <dron@redhat.com> wrote:

Hi,

As this is failing on average 3 patches a day (from different projects) and it seems no one is working on debugging this issue, I will add a patch to skip this test today.

Thanks, Dafna

On Thu, Feb 28, 2019 at 2:12 PM Dafna Ron <dron@redhat.com> wrote:

...

Hi,

We are still getting random failures on this test. Is anyone working on this?

Thanks, Dafna

On Wed, Feb 27, 2019 at 9:10 PM Nir Soffer <nsoffer@redhat.com> wrote:

...

On Wed, Feb 27, 2019 at 4:31 PM Dan Kenigsberg <danken@redhat.com> wrote:

...

HTH

https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13163/artifac...

https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13163/artifac...

Hers is another failure:

https://jenkins.ovirt.org/job/ovirt-system-tests_manual/4169/testReport/juni...

...

On Wed, Feb 27, 2019 at 4:24 PM Michal Skrivanek < michal.skrivanek@redhat.com> wrote:

...

On 27 Feb 2019, at 14:07, Galit Rosenthal <grosenth@redhat.com> wrote:

Hi,

we are failing basic suite master (vdsm, ovirt-engine-sdk ...)

It happens sporadically.

We disqualify that this is an infra issue.

Can you please have a look at the issue?

ERROR:

<testcase classname="002_bootstrap" name="get_host_devices" time="0.077"> <error type="exceptions.RuntimeError" message="*Could not find block_vda_1 device in host devices*: "><![CDATA[Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 1016, in get_host_devices raise RuntimeError('Could not find block_vda_1 device in host devices: {}'.format(device_list)) RuntimeError: Could not find block_vda_1 device in host devices: ]]></error>

Thanks,

Galit

Examples, Full logs:

where can I find full logs? It would be interesting to see hosts’ logs as well as engine.log

Thanks, michal

[1] https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/ovirt-ma...

[2]

https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/ovirt-ma...

_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/WXW6II3NYEXHIS...

--

GALIT ROSENTHAL

SOFTWARE ENGINEER Red Hat

<https://www.redhat.com/>

galit@gmail.com T: 972-9-7692230 <https://red.ht/sig> _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/HF4MB2VMJKBO4O...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/YY4MXJ44Q6TJQR...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/KUHLEKMAVQ7NPX...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/6HHUAPUKLVLWSA...

On Fri, 1 Mar 2019, 15:48 Dafna Ron, <dron@redhat.com> wrote:

Hi Dan,

our intention is never to skip tests and I would not merge a SkipTest patch if someone actually shows interest in resolving this. if you notice, this was reported a few days ago, after CI determined that this is not an infra related issue and since then, but aside from Michal asking for logs and getting a link to them, no other communication from dev that suggest that this is being checked was sent - even after I added [URGENT] and asked if there is any update. As we have done in the past, when I see that there is no cooperation from the developers to resolve a sporadic test failure, then I add a skip test until someone in dev has time to resolve the issue

I believe that this is wrong. You should not merge a Skip before having a phone call with the relevant team lead. - not ideal but having different project fail sporadically is not ideal as

well

As I see that you guys are now responding I have no reason to merge the patch - and that is what we all want.

I can say that since its effecting more then one project, and from past experience, its either related to the OST test itself or to the API/SDK. I think that Evgheni ruled out an infra issue and suggested that the issue is actually related to the API - so not sure its strictly related to Ryan.

I'll trust you here. Though since the failing test is Ryan's, it is Ryan's job to find what breaks it, and pass the buck. If you believe it is a bug in oVirt ask, then you should call mperina, too.

Thanks, Dafna

On Fri, Mar 1, 2019 at 1:27 PM Dan Kenigsberg <danken@redhat.com> wrote:

...

I think this would be a bad idea, Dafna.

OST exists to find bug, not to skip tests. Michal said that he be working on it. It's time to call him (or Ryan or Shmuel) to hear his finding, not to kill his tests.

If you do skip his test, you should file an oVirt bug about it.

On Fri, 1 Mar 2019, 11:46 Dafna Ron, <dron@redhat.com> wrote:

...

Hi,

As this is failing on average 3 patches a day (from different projects) and it seems no one is working on debugging this issue, I will add a patch to skip this test today.

Thanks, Dafna

On Thu, Feb 28, 2019 at 2:12 PM Dafna Ron <dron@redhat.com> wrote:

...

Hi,

We are still getting random failures on this test. Is anyone working on this?

Thanks, Dafna

On Wed, Feb 27, 2019 at 9:10 PM Nir Soffer <nsoffer@redhat.com> wrote:

...

On Wed, Feb 27, 2019 at 4:31 PM Dan Kenigsberg <danken@redhat.com> wrote:

...

HTH

https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13163/artifac...

https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13163/artifac...

Hers is another failure:

https://jenkins.ovirt.org/job/ovirt-system-tests_manual/4169/testReport/juni...

...

On Wed, Feb 27, 2019 at 4:24 PM Michal Skrivanek < michal.skrivanek@redhat.com> wrote:

> > > On 27 Feb 2019, at 14:07, Galit Rosenthal <grosenth@redhat.com> > wrote: > > Hi, > > we are failing basic suite master (vdsm, ovirt-engine-sdk ...) > > It happens sporadically. > > We disqualify that this is an infra issue. > > Can you please have a look at the issue? > > > ERROR: > > <testcase classname="002_bootstrap" name="get_host_devices" time="0.077"> > <error type="exceptions.RuntimeError" message="*Could not find block_vda_1 device in host devices*: "><![CDATA[Traceback (most recent call last): > File "/usr/lib64/python2.7/unittest/case.py", line 369, in run > testMethod() > File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest > self.test(*self.arg) > File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test > test() > File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper > return func(get_test_prefix(), *args, **kwargs) > File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper > prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs > File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 1016, in get_host_devices > raise RuntimeError('Could not find block_vda_1 device in host devices: {}'.format(device_list)) > RuntimeError: Could not find block_vda_1 device in host devices: > ]]></error> > > > Thanks, > > Galit > > > Examples, Full logs: > > > where can I find full logs? It would be interesting to see hosts’ > logs as well as engine.log > > Thanks, > michal > > [1] https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/ovirt-ma... > > > [2] > > https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/ovirt-ma... > > > > _______________________________________________ > Infra mailing list -- infra@ovirt.org > To unsubscribe send an email to infra-leave@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/WXW6II3NYEXHIS... > > > -- > > GALIT ROSENTHAL > > SOFTWARE ENGINEER > Red Hat > > <https://www.redhat.com/> > > galit@gmail.com T: 972-9-7692230 > <https://red.ht/sig> > _______________________________________________ > Devel mailing list -- devel@ovirt.org > To unsubscribe send an email to devel-leave@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/devel@ovirt.org/message/HF4MB2VMJKBO4O... > > > _______________________________________________ > Devel mailing list -- devel@ovirt.org > To unsubscribe send an email to devel-leave@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/devel@ovirt.org/message/YY4MXJ44Q6TJQR... > _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/KUHLEKMAVQ7NPX...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/6HHUAPUKLVLWSA...

On Fri, Mar 1, 2019 at 9:20 AM Dafna Ron <dron@redhat.com> wrote:

On Fri, Mar 1, 2019 at 2:07 PM Dan Kenigsberg <danken@redhat.com> wrote:

...

On Fri, 1 Mar 2019, 15:48 Dafna Ron, <dron@redhat.com> wrote:

...

Hi Dan,

our intention is never to skip tests and I would not merge a SkipTest patch if someone actually shows interest in resolving this. if you notice, this was reported a few days ago, after CI determined that this is not an infra related issue and since then, but aside from Michal asking for logs and getting a link to them, no other communication from dev that suggest that this is being checked was sent - even after I added [URGENT] and asked if there is any update. As we have done in the past, when I see that there is no cooperation from the developers to resolve a sporadic test failure, then I add a skip test until someone in dev has time to resolve the issue

I believe that this is wrong. You should not merge a Skip before having a phone call with the relevant team lead.

If we had any info on who to speak with we would probably do so - in cases like this, where there are multiple projects failures and its sporadic, its hard for us to pin someone down. Also, please note that the CI team is here to maintain the infrastructure that allows developers to run the tests and I don't think that we should be chasing after the developers to fix regressions (either in the code or in the tests) and that this should be dev's responsibility.

I can see both sides of the argument, but developers need to do a better job of being more responsive to Dafna. She is always extremely diligent about reporting these, including follow-ups, and sometimes she doesn't get replies. Phone call? I don't answer my phone unless it's my wife (phone spam in the US is unrelenting). I think the bigger issue is that everyone (ok, most people) filters devel and other lists into a label and hardly checks it. IMO that needs to stop. Or maybe we need a dedicated ost-failures list that people don't filter. That sounds a little better to me than a single dev contact. Greg

I think that it may be time to have a dev contact which would be helping to debug and assign issues such as these and I will raise that in a different thread.

...

- not ideal but having different project fail sporadically is not ideal

...

as well

As I see that you guys are now responding I have no reason to merge the patch - and that is what we all want.

I can say that since its effecting more then one project, and from past experience, its either related to the OST test itself or to the API/SDK. I think that Evgheni ruled out an infra issue and suggested that the issue is actually related to the API - so not sure its strictly related to Ryan.

I'll trust you here. Though since the failing test is Ryan's, it is Ryan's job to find what breaks it, and pass the buck.

If you believe it is a bug in oVirt ask, then you should call mperina, too.

...

Thanks, Dafna

On Fri, Mar 1, 2019 at 1:27 PM Dan Kenigsberg <danken@redhat.com> wrote:

...

I think this would be a bad idea, Dafna.

OST exists to find bug, not to skip tests. Michal said that he be working on it. It's time to call him (or Ryan or Shmuel) to hear his finding, not to kill his tests.

If you do skip his test, you should file an oVirt bug about it.

On Fri, 1 Mar 2019, 11:46 Dafna Ron, <dron@redhat.com> wrote:

...

Hi,

As this is failing on average 3 patches a day (from different projects) and it seems no one is working on debugging this issue, I will add a patch to skip this test today.

Thanks, Dafna

On Thu, Feb 28, 2019 at 2:12 PM Dafna Ron <dron@redhat.com> wrote:

...

Hi,

We are still getting random failures on this test. Is anyone working on this?

Thanks, Dafna

On Wed, Feb 27, 2019 at 9:10 PM Nir Soffer <nsoffer@redhat.com> wrote:

> On Wed, Feb 27, 2019 at 4:31 PM Dan Kenigsberg <danken@redhat.com> > wrote: > >> HTH >> >> >> https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13163/artifac... >> >> >> >> https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13163/artifac... >> >> > Hers is another failure: > > https://jenkins.ovirt.org/job/ovirt-system-tests_manual/4169/testReport/juni... > > > >> >> On Wed, Feb 27, 2019 at 4:24 PM Michal Skrivanek < >> michal.skrivanek@redhat.com> wrote: >> >>> >>> >>> On 27 Feb 2019, at 14:07, Galit Rosenthal <grosenth@redhat.com> >>> wrote: >>> >>> Hi, >>> >>> we are failing basic suite master (vdsm, ovirt-engine-sdk ...) >>> >>> It happens sporadically. >>> >>> We disqualify that this is an infra issue. >>> >>> Can you please have a look at the issue? >>> >>> >>> ERROR: >>> >>> <testcase classname="002_bootstrap" name="get_host_devices" time="0.077"> >>> <error type="exceptions.RuntimeError" message="*Could not find block_vda_1 device in host devices*: "><![CDATA[Traceback (most recent call last): >>> File "/usr/lib64/python2.7/unittest/case.py", line 369, in run >>> testMethod() >>> File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest >>> self.test(*self.arg) >>> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test >>> test() >>> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper >>> return func(get_test_prefix(), *args, **kwargs) >>> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper >>> prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs >>> File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 1016, in get_host_devices >>> raise RuntimeError('Could not find block_vda_1 device in host devices: {}'.format(device_list)) >>> RuntimeError: Could not find block_vda_1 device in host devices: >>> ]]></error> >>> >>> >>> Thanks, >>> >>> Galit >>> >>> >>> Examples, Full logs: >>> >>> >>> where can I find full logs? It would be interesting to see hosts’ >>> logs as well as engine.log >>> >>> Thanks, >>> michal >>> >>> [1] https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/ovirt-ma... >>> >>> >>> [2] >>> >>> https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/ovirt-ma... >>> >>> >>> >>> _______________________________________________ >>> Infra mailing list -- infra@ovirt.org >>> To unsubscribe send an email to infra-leave@ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: >>> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/WXW6II3NYEXHIS... >>> >>> >>> -- >>> >>> GALIT ROSENTHAL >>> >>> SOFTWARE ENGINEER >>> Red Hat >>> >>> <https://www.redhat.com/> >>> >>> galit@gmail.com T: 972-9-7692230 >>> <https://red.ht/sig> >>> _______________________________________________ >>> Devel mailing list -- devel@ovirt.org >>> To unsubscribe send an email to devel-leave@ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: >>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/HF4MB2VMJKBO4O... >>> >>> >>> _______________________________________________ >>> Devel mailing list -- devel@ovirt.org >>> To unsubscribe send an email to devel-leave@ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: >>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/YY4MXJ44Q6TJQR... >>> >> _______________________________________________ >> Devel mailing list -- devel@ovirt.org >> To unsubscribe send an email to devel-leave@ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/KUHLEKMAVQ7NPX... >> > _______________________________________________ > Devel mailing list -- devel@ovirt.org > To unsubscribe send an email to devel-leave@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/devel@ovirt.org/message/6HHUAPUKLVLWSA... > _______________________________________________

Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/ERLW4GNWISM5VJ...

-- GREG SHEREMETA SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX Red Hat NA <https://www.redhat.com/> gshereme@redhat.com IRC: gshereme <https://red.ht/sig>

On Fri, Mar 1, 2019 at 10:10 AM Dan Kenigsberg <danken@redhat.com> wrote:

On Fri, 1 Mar 2019, 16:23 Greg Sheremeta, <gshereme@redhat.com> wrote:

...

On Fri, Mar 1, 2019 at 9:20 AM Dafna Ron <dron@redhat.com> wrote:

...

On Fri, Mar 1, 2019 at 2:07 PM Dan Kenigsberg <danken@redhat.com> wrote:

...

On Fri, 1 Mar 2019, 15:48 Dafna Ron, <dron@redhat.com> wrote:

...

Hi Dan,

our intention is never to skip tests and I would not merge a SkipTest patch if someone actually shows interest in resolving this. if you notice, this was reported a few days ago, after CI determined that this is not an infra related issue and since then, but aside from Michal asking for logs and getting a link to them, no other communication from dev that suggest that this is being checked was sent - even after I added [URGENT] and asked if there is any update. As we have done in the past, when I see that there is no cooperation from the developers to resolve a sporadic test failure, then I add a skip test until someone in dev has time to resolve the issue

I believe that this is wrong. You should not merge a Skip before having a phone call with the relevant team lead.

If we had any info on who to speak with we would probably do so - in cases like this, where there are multiple projects failures and its sporadic, its hard for us to pin someone down. Also, please note that the CI team is here to maintain the infrastructure that allows developers to run the tests and I don't think that we should be chasing after the developers to fix regressions (either in the code or in the tests) and that this should be dev's responsibility.

I can see both sides of the argument, but developers need to do a better job of being more responsive to Dafna. She is always extremely diligent about reporting these, including follow-ups, and sometimes she doesn't get replies.

Note that I want her to do less, not more: identify a suspected team lead, and tell him/her it is their job to fix it. That's all.

Phone call? I don't answer my phone unless it's my wife (phone spam in the

...

US is unrelenting).

So she should IRC/WhatsApp/gChat you if she thinks you are responsible for a failure.

I think the bigger issue is that everyone (ok, most people) filters devel

...

and other lists into a label and hardly checks it. IMO that needs to stop.

True. I think that late-evening phone call can help make it happen.

WhatsApp/gChat/IRC are easy enough if there's an issue. I get few enough phonecalls from Israeli country codes that I'd answer anyway, but there's less tracking available.

Or maybe we need a dedicated ost-failures list that people don't filter.

...

The same people who filter out URGENT messages from Dafna would easily filter a new list. It might be easier for them. But none of them should be a maintainer or team lead.

+1 We really don't need a dedicated list, since all of us read URGENT messages (and I read all of the messages from Dafna). Just on PTO yesterday/today/Monday, so there's a delay in reading the mail if I don't have cellular service. Thanks for handling, Michal.

That sounds a little better to me than a single dev contact.

...

Each *test* has a natural dev contact. Host devices are virt, so the contract here is Michal+Ryan until proven differently or suspected differently by Dafna.

...

Greg

...

I think that it may be time to have a dev contact which would be helping to debug and assign issues such as these and I will raise that in a different thread.

...

- not ideal but having different project fail sporadically is not ideal

...

as well

As I see that you guys are now responding I have no reason to merge the patch - and that is what we all want.

I can say that since its effecting more then one project, and from past experience, its either related to the OST test itself or to the API/SDK. I think that Evgheni ruled out an infra issue and suggested that the issue is actually related to the API - so not sure its strictly related to Ryan.

I'll trust you here. Though since the failing test is Ryan's, it is Ryan's job to find what breaks it, and pass the buck.

If you believe it is a bug in oVirt ask, then you should call mperina, too.

...

Thanks, Dafna

On Fri, Mar 1, 2019 at 1:27 PM Dan Kenigsberg <danken@redhat.com> wrote:

...

I think this would be a bad idea, Dafna.

OST exists to find bug, not to skip tests. Michal said that he be working on it. It's time to call him (or Ryan or Shmuel) to hear his finding, not to kill his tests.

If you do skip his test, you should file an oVirt bug about it.

On Fri, 1 Mar 2019, 11:46 Dafna Ron, <dron@redhat.com> wrote:

> Hi, > > As this is failing on average 3 patches a day (from different > projects) and it seems no one is working on debugging this issue, I will > add a patch to skip this test today. > > Thanks, > Dafna > > > On Thu, Feb 28, 2019 at 2:12 PM Dafna Ron <dron@redhat.com> wrote: > >> Hi, >> >> We are still getting random failures on this test. >> Is anyone working on this? >> >> Thanks, >> Dafna >> >> >> On Wed, Feb 27, 2019 at 9:10 PM Nir Soffer <nsoffer@redhat.com> >> wrote: >> >>> On Wed, Feb 27, 2019 at 4:31 PM Dan Kenigsberg <danken@redhat.com> >>> wrote: >>> >>>> HTH >>>> >>>> >>>> https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13163/artifac... >>>> >>>> >>>> >>>> https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13163/artifac... >>>> >>>> >>> Hers is another failure: >>> >>> https://jenkins.ovirt.org/job/ovirt-system-tests_manual/4169/testReport/juni... >>> >>> >>> >>>> >>>> On Wed, Feb 27, 2019 at 4:24 PM Michal Skrivanek < >>>> michal.skrivanek@redhat.com> wrote: >>>> >>>>> >>>>> >>>>> On 27 Feb 2019, at 14:07, Galit Rosenthal <grosenth@redhat.com> >>>>> wrote: >>>>> >>>>> Hi, >>>>> >>>>> we are failing basic suite master (vdsm, ovirt-engine-sdk ...) >>>>> >>>>> It happens sporadically. >>>>> >>>>> We disqualify that this is an infra issue. >>>>> >>>>> Can you please have a look at the issue? >>>>> >>>>> >>>>> ERROR: >>>>> >>>>> <testcase classname="002_bootstrap" name="get_host_devices" time="0.077"> >>>>> <error type="exceptions.RuntimeError" message="*Could not find block_vda_1 device in host devices*: "><![CDATA[Traceback (most recent call last): >>>>> File "/usr/lib64/python2.7/unittest/case.py", line 369, in run >>>>> testMethod() >>>>> File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest >>>>> self.test(*self.arg) >>>>> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test >>>>> test() >>>>> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper >>>>> return func(get_test_prefix(), *args, **kwargs) >>>>> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper >>>>> prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs >>>>> File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 1016, in get_host_devices >>>>> raise RuntimeError('Could not find block_vda_1 device in host devices: {}'.format(device_list)) >>>>> RuntimeError: Could not find block_vda_1 device in host devices: >>>>> ]]></error> >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Galit >>>>> >>>>> >>>>> Examples, Full logs: >>>>> >>>>> >>>>> where can I find full logs? It would be interesting to see >>>>> hosts’ logs as well as engine.log >>>>> >>>>> Thanks, >>>>> michal >>>>> >>>>> [1] https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/ovirt-ma... >>>>> >>>>> >>>>> [2] >>>>> >>>>> https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/ovirt-ma... >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Infra mailing list -- infra@ovirt.org >>>>> To unsubscribe send an email to infra-leave@ovirt.org >>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>> oVirt Code of Conduct: >>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>> List Archives: >>>>> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/WXW6II3NYEXHIS... >>>>> >>>>> >>>>> -- >>>>> >>>>> GALIT ROSENTHAL >>>>> >>>>> SOFTWARE ENGINEER >>>>> Red Hat >>>>> >>>>> <https://www.redhat.com/> >>>>> >>>>> galit@gmail.com T: 972-9-7692230 >>>>> <https://red.ht/sig> >>>>> _______________________________________________ >>>>> Devel mailing list -- devel@ovirt.org >>>>> To unsubscribe send an email to devel-leave@ovirt.org >>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>> oVirt Code of Conduct: >>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>> List Archives: >>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/HF4MB2VMJKBO4O... >>>>> >>>>> >>>>> _______________________________________________ >>>>> Devel mailing list -- devel@ovirt.org >>>>> To unsubscribe send an email to devel-leave@ovirt.org >>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>> oVirt Code of Conduct: >>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>> List Archives: >>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/YY4MXJ44Q6TJQR... >>>>> >>>> _______________________________________________ >>>> Devel mailing list -- devel@ovirt.org >>>> To unsubscribe send an email to devel-leave@ovirt.org >>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>> oVirt Code of Conduct: >>>> https://www.ovirt.org/community/about/community-guidelines/ >>>> List Archives: >>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/KUHLEKMAVQ7NPX... >>>> >>> _______________________________________________ >>> Devel mailing list -- devel@ovirt.org >>> To unsubscribe send an email to devel-leave@ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: >>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/6HHUAPUKLVLWSA... >>> >> _______________________________________________

Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/ERLW4GNWISM5VJ...

--

GREG SHEREMETA

SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX

Red Hat NA

<https://www.redhat.com/>

gshereme@redhat.com IRC: gshereme <https://red.ht/sig>

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/SLTCK7D6QG2XBX...

-- Ryan Barry Associate Manager - RHV Virt/SLA rbarry@redhat.com M: +16518159306 IM: rbarry <https://red.ht/sig>

On Fri, Mar 1, 2019, 15:38 Dan Kenigsberg <danken@redhat.com> wrote:

I think this would be a bad idea, Dafna.

OST exists to find bug, not to skip tests. Michal said that he be working on it. It's time to call him (or Ryan or Shmuel) to hear his finding, not to kill his tests.

Sorry, but Dafna won't be calling to anyone just to make sure they read urgent emails or act on OST failures. if its important enough, anyone from the relevant team can act upon it faster, if they can't and its affecting CI stability, they can revert the Skip when they have time to address and debug it.

If you do skip his test, you should file an oVirt bug about it.

On Fri, 1 Mar 2019, 11:46 Dafna Ron, <dron@redhat.com> wrote:

...

Hi,

As this is failing on average 3 patches a day (from different projects) and it seems no one is working on debugging this issue, I will add a patch to skip this test today.

Thanks, Dafna

On Thu, Feb 28, 2019 at 2:12 PM Dafna Ron <dron@redhat.com> wrote:

...

Hi,

We are still getting random failures on this test. Is anyone working on this?

Thanks, Dafna

On Wed, Feb 27, 2019 at 9:10 PM Nir Soffer <nsoffer@redhat.com> wrote:

...

On Wed, Feb 27, 2019 at 4:31 PM Dan Kenigsberg <danken@redhat.com> wrote:

...

HTH

https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13163/artifac...

https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13163/artifac...

Hers is another failure:

https://jenkins.ovirt.org/job/ovirt-system-tests_manual/4169/testReport/juni...

...

On Wed, Feb 27, 2019 at 4:24 PM Michal Skrivanek < michal.skrivanek@redhat.com> wrote:

...

On 27 Feb 2019, at 14:07, Galit Rosenthal <grosenth@redhat.com> wrote:

Hi,

we are failing basic suite master (vdsm, ovirt-engine-sdk ...)

It happens sporadically.

We disqualify that this is an infra issue.

Can you please have a look at the issue?

ERROR:

<testcase classname="002_bootstrap" name="get_host_devices" time="0.077"> <error type="exceptions.RuntimeError" message="*Could not find block_vda_1 device in host devices*: "><![CDATA[Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", line 1016, in get_host_devices raise RuntimeError('Could not find block_vda_1 device in host devices: {}'.format(device_list)) RuntimeError: Could not find block_vda_1 device in host devices: ]]></error>

Thanks,

Galit

Examples, Full logs:

where can I find full logs? It would be interesting to see hosts’ logs as well as engine.log

Thanks, michal

[1] https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/ovirt-ma...

[2]

https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/ovirt-ma...

_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/WXW6II3NYEXHIS...

--

GALIT ROSENTHAL

SOFTWARE ENGINEER Red Hat

<https://www.redhat.com/>

galit@gmail.com T: 972-9-7692230 <https://red.ht/sig> _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/HF4MB2VMJKBO4O...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/YY4MXJ44Q6TJQR...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/KUHLEKMAVQ7NPX...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/6HHUAPUKLVLWSA...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7EWV5YVRX5XJW2...