11:05 a.m.
On Wed, Feb 24, 2021 at 11:43 AM Milan Zamazal <mzamazal(a)redhat.com> wrote:
Yedidyah Bar David <didi(a)redhat.com> writes:
> Hi all,
>
> Right now, when we merge a patch e.g. to the engine (and many other
> projects), it can take up to several days until it is used by the
> hosted-engine ovirt-system-tests suite. Something similar will happen
> soon if/when we introduce suites that use ovirt-node.
>
> If I got it right:
> - Merge causes CI to build the engine - immediately, takes ~ 1 hour (say)
> - A publisher job [1] publishes it to resources.ovirt.org (daily,
> midnight (UTC))
> - The next run of an appliance build [2] includes it (daily, afternoon)
> - The next run of the publisher [1] publishes the appliance (daily, midnight)
> - The next run of ost-images [3] includes the appliance (daily,
> midnight, 2 hours after the publisher) (and publishes it immediately)
> - The next run of ost (e.g. [4]) will use it (daily, slightly *before*
> ost-images, but I guess we can change that. And this does not affect
> manual runs of OST, so can probably be ignored in the calculation, at
> least to some extent).
>
> So if I got it right, a patch merged to the engine in some morning,
> will be used by the nightly run of OST HE only after almost 3 days,
> and available for manual runs after 2 days. IMO that's too much time.
> I might be somewhat wrong, but not very, I think.
>
> One partial solution is to add automation .repos lines to relevant
> projects that will link at lastSuccessfulBuild (let's call it lastSB)
> of the more important projects they consume - e.g. appliance to use
> lastSB of engine+dwh+a few others, node to use lastSB of vdsm, etc.
> This will require more maintenance (adding/removing/fixing projects as
> needed) and cause some more load on CI (as now packages will be
> downloaded from it instead of from resources.ovirt.org).
>
> Another solution is to run relevant jobs (publisher/appliance/node)
> far more often - say, once every two hours.
One important thing to consider is an ability to run OST on our patches
at all. If there is (almost) always a newer build available then custom
repos added to OST runs, whether on Jenkins or locally, will be ignored
and we'll be unable to test our patches before they are merged.
Indeed. That's an important point. IIRC OST has a ticket specifically
addressing this issue.
> This will also add load, and might cause "perceived" instability - as
> things will likely fluctuate between green and red more often.
This doesn't sound very good, I perceive the things less than stable
already now.
Agreed.
I quoted "perceived" because I do not think they'll actually be less
stable.
Right now, when something critical is broken, we fix it, then manually
run some of the above jobs as needed, to quickly get back to business.
When we don't (often), some things simply remain broken for two days.
Running more often will simply notify us about breakage faster. If we
then fix, it will automatically propage the fix faster.
> I think I prefer the latter. What do you think?
Wouldn't it be possible to run the whole pipeline nightly (even if it
means e.g. running the publisher twice during the night)?
It will. But this will only fix the specific issue of appliance/node.
Running more often also simply gives feedback faster.
But I agree that perhaps we should wait with this until OST allows
using a custom repo reliably and easily.
Thanks,
--
Didi