Il giorno ven 22 mar 2019 alle ore 12:42 Dan Kenigsberg <
danken@redhat.com> ha scritto:
Il giorno ven 22 mar 2019 alle ore 11:14 Dan Kenigsberg <
danken@redhat.com> ha scritto:
Il giorno ven 22 mar 2019 alle ore 10:52 Dan Kenigsberg <
danken@redhat.com> ha scritto:
Yes, I'm repeating myself.
SKIPPING TESTS IS BAD
I agree. And having the suite failing on a broken test skipping all the following tests is even worse.
This is why I would prefer the rest of the product being tested while someone take ownership of the broken test and fix it.
This is a good reason to rewrite OST with pytest, which continues on failure.
Patches are welcome :-)
This is not an empty gesture. The network suite came into being because of this issue (and others)
And a good reason to ping mperina on IRC to debug this. And a good reason not to merge new code.
It doesn't convince me that we should ignore the failure without due debugging.
Debugging in indeed needed but not on production system blocking the rest of the CI. Maintainer of the test can debug it on own test environment.
The product of this system are bugs. We found one. If you skip it, we all risk it being forgotten. Skipping should be rare, and happen only after the owner is found and admits that he is too busy/lazy to fix it now, and files a bug to fix it later.
We didn't found a bug in the product we are testing, we found a bug in the test that still need to be identified.
According to Dafna: "we are randomly failing on get_host_hooks test for at least 3 weeks. its not a specific branch or project and there are no commonalities that I can see,"
If it was a bug in the product I would have totally agreed with you, it couldn't have been ignored. I'm not saying to ignore this as well.
Being a bug in the test itself