Running basic-suite-master more often
Hi all, We currently run [1], despite its name, 3 times at night - at 01:00, 03:00, 05:00 (UTC+00, I think). What do you think about running it also during the day? This should help identify breakages caused by merging stuff during the day (most developers, I think, are in EMEA). I know we worked on gating in the past and gave up on this. I do not suggest to start that discussion again, nor to block anything any other way. Just run a bit more. Even once an hour will not add too much load IMO and can be useful. I also do not suggest to run per-patch (say, for the engine git repo) - this sometimes causes too much load, and sometimes is not enough - if a patch is not always failing a run, running more than once does add value. Best regards, [1] https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ -- Didi
Hi, On 1/20/21 10:44 AM, Yedidyah Bar David wrote:
Hi all,
We currently run [1], despite its name, 3 times at night - at 01:00, 03:00, 05:00 (UTC+00, I think). What do you think about running it also during the day? This should help identify breakages caused by merging stuff during the day (most developers, I think, are in EMEA). I don't think it will help with breakages caused by merging stuff - if you don't build RPMs out of the merged stuff it won't get into OST.
I know we worked on gating in the past and gave up on this. I do not suggest to start that discussion again, nor to block anything any other way. Just run a bit more. Even once an hour will not add too much load IMO and can be useful.
I also do not suggest to run per-patch (say, for the engine git repo) - this sometimes causes too much load, and sometimes is not enough - if a patch is not always failing a run, running more than once does add value.
This is actually something I've been thinking to get back to. Not in the same form we had it before. We'd need PSI-based runs that are deployed faster than current CI first. The other thing is I was thinking about making a "gating suite". Think of a trimmed-down version of basic suite that can run in 15-20 mins. But for this, we need the CI first and then some work on modularization of suites. Regards, Marcin
Best regards,
[1] https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/
On Tue, Jan 26, 2021 at 1:29 PM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Hi,
On 1/20/21 10:44 AM, Yedidyah Bar David wrote:
Hi all,
We currently run [1], despite its name, 3 times at night - at 01:00, 03:00, 05:00 (UTC+00, I think). What do you think about running it also during the day? This should help identify breakages caused by merging stuff during the day (most developers, I think, are in EMEA). I don't think it will help with breakages caused by merging stuff - if you don't build RPMs out of the merged stuff it won't get into OST.
I know we worked on gating in the past and gave up on this. I do not suggest to start that discussion again, nor to block anything any other way. Just run a bit more. Even once an hour will not add too much load IMO and can be useful.
Maybe run every 6 hours instead of every 2 hours during night?
I also do not suggest to run per-patch (say, for the engine git repo) - this sometimes causes too much load, and sometimes is not enough - if a patch is not always failing a run, running more than once does add value. This is actually something I've been thinking to get back to. Not in the same form we had it before. We'd need PSI-based runs that are deployed faster than current CI first. The other thing is I was thinking about making a "gating suite". Think of a trimmed-down version of basic suite that can run in 15-20 mins.
Yes please!
But for this, we need the CI first and then some work on modularization of suites.
I'm not sure about making it gating, but I think we need to make the basic suite thinner and move most tests to vertical based suites. Having a thin basic suite, maybe we could run 2 suites in parallel, basicSuite and suite needed for particular patch (e.g storage).
participants (3)
-
Marcin Sobczyk -
Nir Soffer -
Yedidyah Bar David