RFC: Not Failing OST on Missing Artifact During Collection
Hello All. I faced it multiple times during local runs, e.g. first it was missing /tmp/otopi*. Now it fails for me on missing /var/lib/pgsql/upgrade_rh-postgresql95-postgresql.log and /etc/dnf on centos7 engine. It seems like any missing artifact during collection step will fail the test. I understand some potential benefit of it, but I suggest we do not fail it if we are unable to find any of the artifacts during collection step. Better just to issue a warning in the logs and go on. If somebody needs to test for particular log presence, I suggest it should be included as a test step instead. Wdyt? -- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
On Wednesday, January 29, 2020, Anton Marchukov <amarchuk@redhat.com> wrote:
Hello All.
I faced it multiple times during local runs, e.g. first it was missing /tmp/otopi*. Now it fails for me on missing /var/lib/pgsql/upgrade_rh-postgresql95-postgresql.log and /etc/dnf on centos7 engine. It seems like any missing artifact during collection step will fail the test.
I understand some potential benefit of it, but I suggest we do not fail it if we are unable to find any of the artifacts during collection step. Better just to issue a warning in the logs and go on. If somebody needs to test for particular log presence, I suggest it should be included as a test step instead.
Wdyt?
+1 A step further, Could be very helpful if we could select the suites for OST during its launch since many times we are only interested in a specific component. For example: only run storage suite and don't care for ui browser suite when checking a storage change. This way we can avoid OST being shut completely when only one suite is broken which is not relevant to the change being verified.
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community- guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/ message/J3JZ337UJMBRBGUBPUIRA2T5NMNNPGE6/
On 29 Jan 2020, at 18:52, Amit Bawer <abawer@redhat.com> wrote:
On Wednesday, January 29, 2020, Anton Marchukov <amarchuk@redhat.com <mailto:amarchuk@redhat.com>> wrote: Hello All.
I faced it multiple times during local runs, e.g. first it was missing /tmp/otopi*. Now it fails for me on missing /var/lib/pgsql/upgrade_rh-postgresql95-postgresql.log and /etc/dnf on centos7 engine. It seems like any missing artifact during collection step will fail the test.
I understand some potential benefit of it, but I suggest we do not fail it if we are unable to find any of the artifacts during collection step. Better just to issue a warning in the logs and go on. If somebody needs to test for particular log presence, I suggest it should be included as a test step instead.
Wdyt?
+1 A step further, Could be very helpful if we could select the suites for OST during its launch since many times we are only interested in a specific component. For example: only run storage suite and don't care for ui browser suite when checking a storage change. This way we can avoid OST being shut completely when only one suite is broken which is not relevant to the change being verified.
Maybe you confuse suites with test scenarios? basic-suite-master has several scenarious, like 004_basic_sanity.py or 100_basic_ui_sanity.py. There’s no separate suite for UI or storage. If you’re referring to the scenarios - they are part of elementary sanity tests and as such we do want to run them every time, all of them. if you want to run individual scenarios/tests you can always just run OST yourself locally and invoke it yourself Thanks, michal
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
_______________________________________________ Devel mailing list -- devel@ovirt.org <mailto:devel@ovirt.org> To unsubscribe send an email to devel-leave@ovirt.org <mailto:devel-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ <https://www.ovirt.org/site/privacy-policy/> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/J3JZ337UJMBRBG... <https://lists.ovirt.org/archives/list/devel@ovirt.org/message/J3JZ337UJMBRBGUBPUIRA2T5NMNNPGE6/> _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/C2VWXDHCDU6EKJ...
On Wed, Jan 29, 2020 at 9:39 PM Michal Skrivanek < michal.skrivanek@redhat.com> wrote:
On 29 Jan 2020, at 18:52, Amit Bawer <abawer@redhat.com> wrote:
On Wednesday, January 29, 2020, Anton Marchukov <amarchuk@redhat.com> wrote:
Hello All.
I faced it multiple times during local runs, e.g. first it was missing /tmp/otopi*. Now it fails for me on missing /var/lib/pgsql/upgrade_rh-postgresql95-postgresql.log and /etc/dnf on centos7 engine. It seems like any missing artifact during collection step will fail the test.
I understand some potential benefit of it, but I suggest we do not fail it if we are unable to find any of the artifacts during collection step. Better just to issue a warning in the logs and go on. If somebody needs to test for particular log presence, I suggest it should be included as a test step instead.
Wdyt?
+1
+1
A step further, Could be very helpful if we could select the suites for OST during its launch since many times we are only interested in a specific component. For example: only run storage suite and don't care for ui browser suite when checking a storage change. This way we can avoid OST being shut completely when only one suite is broken which is not relevant to the change being verified.
(In addition to below:) If some test scenarios often fail, we should fix them :-)
Maybe you confuse suites with test scenarios? basic-suite-master has several scenarious, like 004_basic_sanity.py or 100_basic_ui_sanity.py. There’s no separate suite for UI or storage. If you’re referring to the scenarios - they are part of elementary sanity tests and as such we do want to run them every time, all of them.
if you want to run individual scenarios/tests you can always just run OST yourself locally and invoke it yourself
Thanks, michal
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/J3JZ337UJMBRBG...
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/C2VWXDHCDU6EKJ...
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/PMF4JXKW72WXNR...
-- Didi
participants (4)
-
Amit Bawer -
Anton Marchukov -
Michal Skrivanek -
Yedidyah Bar David