Publish master more often
Hi all, Right now, when we merge a patch e.g. to the engine (and many other projects), it can take up to several days until it is used by the hosted-engine ovirt-system-tests suite. Something similar will happen soon if/when we introduce suites that use ovirt-node. If I got it right: - Merge causes CI to build the engine - immediately, takes ~ 1 hour (say) - A publisher job [1] publishes it to resources.ovirt.org (daily, midnight (UTC)) - The next run of an appliance build [2] includes it (daily, afternoon) - The next run of the publisher [1] publishes the appliance (daily, midnight) - The next run of ost-images [3] includes the appliance (daily, midnight, 2 hours after the publisher) (and publishes it immediately) - The next run of ost (e.g. [4]) will use it (daily, slightly *before* ost-images, but I guess we can change that. And this does not affect manual runs of OST, so can probably be ignored in the calculation, at least to some extent). So if I got it right, a patch merged to the engine in some morning, will be used by the nightly run of OST HE only after almost 3 days, and available for manual runs after 2 days. IMO that's too much time. I might be somewhat wrong, but not very, I think. One partial solution is to add automation .repos lines to relevant projects that will link at lastSuccessfulBuild (let's call it lastSB) of the more important projects they consume - e.g. appliance to use lastSB of engine+dwh+a few others, node to use lastSB of vdsm, etc. This will require more maintenance (adding/removing/fixing projects as needed) and cause some more load on CI (as now packages will be downloaded from it instead of from resources.ovirt.org). Another solution is to run relevant jobs (publisher/appliance/node) far more often - say, once every two hours. This will also add load, and might cause "perceived" instability - as things will likely fluctuate between green and red more often. I think I prefer the latter. What do you think? Thanks and best regards, [1] https://jenkins.ovirt.org/job/ovirt_master_publish-rpms_nightly/ [2] https://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-el8-x86... [3] https://jenkins.ovirt.org/job/ost-images_master_standard-poll-upstream-sourc... [4] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/ -- Didi
Yedidyah Bar David <didi@redhat.com> writes:
Hi all,
Right now, when we merge a patch e.g. to the engine (and many other projects), it can take up to several days until it is used by the hosted-engine ovirt-system-tests suite. Something similar will happen soon if/when we introduce suites that use ovirt-node.
If I got it right: - Merge causes CI to build the engine - immediately, takes ~ 1 hour (say) - A publisher job [1] publishes it to resources.ovirt.org (daily, midnight (UTC)) - The next run of an appliance build [2] includes it (daily, afternoon) - The next run of the publisher [1] publishes the appliance (daily, midnight) - The next run of ost-images [3] includes the appliance (daily, midnight, 2 hours after the publisher) (and publishes it immediately) - The next run of ost (e.g. [4]) will use it (daily, slightly *before* ost-images, but I guess we can change that. And this does not affect manual runs of OST, so can probably be ignored in the calculation, at least to some extent).
So if I got it right, a patch merged to the engine in some morning, will be used by the nightly run of OST HE only after almost 3 days, and available for manual runs after 2 days. IMO that's too much time. I might be somewhat wrong, but not very, I think.
One partial solution is to add automation .repos lines to relevant projects that will link at lastSuccessfulBuild (let's call it lastSB) of the more important projects they consume - e.g. appliance to use lastSB of engine+dwh+a few others, node to use lastSB of vdsm, etc. This will require more maintenance (adding/removing/fixing projects as needed) and cause some more load on CI (as now packages will be downloaded from it instead of from resources.ovirt.org).
Another solution is to run relevant jobs (publisher/appliance/node) far more often - say, once every two hours.
One important thing to consider is an ability to run OST on our patches at all. If there is (almost) always a newer build available then custom repos added to OST runs, whether on Jenkins or locally, will be ignored and we'll be unable to test our patches before they are merged.
This will also add load, and might cause "perceived" instability - as things will likely fluctuate between green and red more often.
This doesn't sound very good, I perceive the things less than stable already now.
I think I prefer the latter. What do you think?
Wouldn't it be possible to run the whole pipeline nightly (even if it means e.g. running the publisher twice during the night)?
Thanks and best regards,
[1] https://jenkins.ovirt.org/job/ovirt_master_publish-rpms_nightly/ [2] https://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-el8-x86... [3] https://jenkins.ovirt.org/job/ost-images_master_standard-poll-upstream-sourc... [4] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/
On Wed, Feb 24, 2021 at 11:43 AM Milan Zamazal <mzamazal@redhat.com> wrote:
Yedidyah Bar David <didi@redhat.com> writes:
Hi all,
Right now, when we merge a patch e.g. to the engine (and many other projects), it can take up to several days until it is used by the hosted-engine ovirt-system-tests suite. Something similar will happen soon if/when we introduce suites that use ovirt-node.
If I got it right: - Merge causes CI to build the engine - immediately, takes ~ 1 hour (say) - A publisher job [1] publishes it to resources.ovirt.org (daily, midnight (UTC)) - The next run of an appliance build [2] includes it (daily, afternoon) - The next run of the publisher [1] publishes the appliance (daily, midnight) - The next run of ost-images [3] includes the appliance (daily, midnight, 2 hours after the publisher) (and publishes it immediately) - The next run of ost (e.g. [4]) will use it (daily, slightly *before* ost-images, but I guess we can change that. And this does not affect manual runs of OST, so can probably be ignored in the calculation, at least to some extent).
So if I got it right, a patch merged to the engine in some morning, will be used by the nightly run of OST HE only after almost 3 days, and available for manual runs after 2 days. IMO that's too much time. I might be somewhat wrong, but not very, I think.
One partial solution is to add automation .repos lines to relevant projects that will link at lastSuccessfulBuild (let's call it lastSB) of the more important projects they consume - e.g. appliance to use lastSB of engine+dwh+a few others, node to use lastSB of vdsm, etc. This will require more maintenance (adding/removing/fixing projects as needed) and cause some more load on CI (as now packages will be downloaded from it instead of from resources.ovirt.org).
Another solution is to run relevant jobs (publisher/appliance/node) far more often - say, once every two hours.
One important thing to consider is an ability to run OST on our patches at all. If there is (almost) always a newer build available then custom repos added to OST runs, whether on Jenkins or locally, will be ignored and we'll be unable to test our patches before they are merged.
Indeed. That's an important point. IIRC OST has a ticket specifically addressing this issue.
This will also add load, and might cause "perceived" instability - as things will likely fluctuate between green and red more often.
This doesn't sound very good, I perceive the things less than stable already now.
Agreed. I quoted "perceived" because I do not think they'll actually be less stable. Right now, when something critical is broken, we fix it, then manually run some of the above jobs as needed, to quickly get back to business. When we don't (often), some things simply remain broken for two days. Running more often will simply notify us about breakage faster. If we then fix, it will automatically propage the fix faster.
I think I prefer the latter. What do you think?
Wouldn't it be possible to run the whole pipeline nightly (even if it means e.g. running the publisher twice during the night)?
It will. But this will only fix the specific issue of appliance/node. Running more often also simply gives feedback faster. But I agree that perhaps we should wait with this until OST allows using a custom repo reliably and easily. Thanks, -- Didi
On 2/24/21 11:05 AM, Yedidyah Bar David wrote:
On Wed, Feb 24, 2021 at 11:43 AM Milan Zamazal <mzamazal@redhat.com> wrote:
Yedidyah Bar David <didi@redhat.com> writes:
Hi all,
Right now, when we merge a patch e.g. to the engine (and many other projects), it can take up to several days until it is used by the hosted-engine ovirt-system-tests suite. Something similar will happen soon if/when we introduce suites that use ovirt-node.
If I got it right: - Merge causes CI to build the engine - immediately, takes ~ 1 hour (say) - A publisher job [1] publishes it to resources.ovirt.org (daily, midnight (UTC)) - The next run of an appliance build [2] includes it (daily, afternoon) - The next run of the publisher [1] publishes the appliance (daily, midnight) - The next run of ost-images [3] includes the appliance (daily, midnight, 2 hours after the publisher) (and publishes it immediately) - The next run of ost (e.g. [4]) will use it (daily, slightly *before* ost-images, but I guess we can change that. And this does not affect manual runs of OST, so can probably be ignored in the calculation, at least to some extent).
So if I got it right, a patch merged to the engine in some morning, will be used by the nightly run of OST HE only after almost 3 days, and available for manual runs after 2 days. IMO that's too much time. I might be somewhat wrong, but not very, I think.
One partial solution is to add automation .repos lines to relevant projects that will link at lastSuccessfulBuild (let's call it lastSB) of the more important projects they consume - e.g. appliance to use lastSB of engine+dwh+a few others, node to use lastSB of vdsm, etc. This will require more maintenance (adding/removing/fixing projects as needed) and cause some more load on CI (as now packages will be downloaded from it instead of from resources.ovirt.org).
Another solution is to run relevant jobs (publisher/appliance/node) far more often - say, once every two hours. One important thing to consider is an ability to run OST on our patches at all. If there is (almost) always a newer build available then custom repos added to OST runs, whether on Jenkins or locally, will be ignored and we'll be unable to test our patches before they are merged. Indeed. That's an important point. IIRC OST has a ticket specifically addressing this issue. Yes, we have:
https://gerrit.ovirt.org/#/c/ovirt-system-tests/+/113223/ and: https://issues.redhat.com/browse/RHV-41025 which is not implemented yet. The downside of upgrading to the latest RPMs from 'tested' repo is, as Milan mentioned, an increased chance that your own packages will not be used cause they're too old. The upside is that if someone breaks OST globally with i.e. some engine patch, and a fix for the problem is merged midday, upgrading to the latest RPMs will unblock the runs. If we don't upgrade, we'll have to wait for the nightly job to rebuild ost-images to include the fix. Rebuilding ost-images midday is an option, but it takes a lot of time, so in most cases one can simply wait till tomorrow... I want to fix this by implementing an option in OST's manual run (switched off by default) that will allow you to upgrade to the latest RPMs from 'tested'. That way one has ~24h for his/her patches to be fresh enough to be picked up by dnf. 'check-patch' jobs should always use latest RPMs from 'tested' IMO.
This will also add load, and might cause "perceived" instability - as things will likely fluctuate between green and red more often. This doesn't sound very good, I perceive the things less than stable already now. Agreed. I quoted "perceived" because I do not think they'll actually be less stable. Right now, when something critical is broken, we fix it, then manually run some of the above jobs as needed, to quickly get back to business. When we don't (often), some things simply remain broken for two days.
Running more often will simply notify us about breakage faster. If we then fix, it will automatically propage the fix faster.
Isn't upgrading the engine RPM on the appliance an option?
I think I prefer the latter. What do you think? Wouldn't it be possible to run the whole pipeline nightly (even if it means e.g. running the publisher twice during the night)? It will. But this will only fix the specific issue of appliance/node. Running more often also simply gives feedback faster.
But I agree that perhaps we should wait with this until OST allows using a custom repo reliably and easily.
Thanks,
On Wed, Feb 24, 2021 at 2:36 PM Marcin Sobczyk <msobczyk@redhat.com> wrote:
On 2/24/21 11:05 AM, Yedidyah Bar David wrote:
On Wed, Feb 24, 2021 at 11:43 AM Milan Zamazal <mzamazal@redhat.com> wrote:
Yedidyah Bar David <didi@redhat.com> writes:
Hi all,
Right now, when we merge a patch e.g. to the engine (and many other projects), it can take up to several days until it is used by the hosted-engine ovirt-system-tests suite. Something similar will happen soon if/when we introduce suites that use ovirt-node.
If I got it right: - Merge causes CI to build the engine - immediately, takes ~ 1 hour (say) - A publisher job [1] publishes it to resources.ovirt.org (daily, midnight (UTC)) - The next run of an appliance build [2] includes it (daily, afternoon) - The next run of the publisher [1] publishes the appliance (daily, midnight) - The next run of ost-images [3] includes the appliance (daily, midnight, 2 hours after the publisher) (and publishes it immediately) - The next run of ost (e.g. [4]) will use it (daily, slightly *before* ost-images, but I guess we can change that. And this does not affect manual runs of OST, so can probably be ignored in the calculation, at least to some extent).
So if I got it right, a patch merged to the engine in some morning, will be used by the nightly run of OST HE only after almost 3 days, and available for manual runs after 2 days. IMO that's too much time. I might be somewhat wrong, but not very, I think.
One partial solution is to add automation .repos lines to relevant projects that will link at lastSuccessfulBuild (let's call it lastSB) of the more important projects they consume - e.g. appliance to use lastSB of engine+dwh+a few others, node to use lastSB of vdsm, etc. This will require more maintenance (adding/removing/fixing projects as needed) and cause some more load on CI (as now packages will be downloaded from it instead of from resources.ovirt.org).
Another solution is to run relevant jobs (publisher/appliance/node) far more often - say, once every two hours. One important thing to consider is an ability to run OST on our patches at all. If there is (almost) always a newer build available then custom repos added to OST runs, whether on Jenkins or locally, will be ignored and we'll be unable to test our patches before they are merged. Indeed. That's an important point. IIRC OST has a ticket specifically addressing this issue. Yes, we have:
https://gerrit.ovirt.org/#/c/ovirt-system-tests/+/113223/
and:
https://issues.redhat.com/browse/RHV-41025
which is not implemented yet.
The downside of upgrading to the latest RPMs from 'tested' repo is, as Milan mentioned, an increased chance that your own packages will not be used cause they're too old. The upside is that if someone breaks OST globally with i.e. some engine patch, and a fix for the problem is merged midday, upgrading to the latest RPMs will unblock the runs. If we don't upgrade, we'll have to wait for the nightly job to rebuild ost-images to include the fix. Rebuilding ost-images midday is an option, but it takes a lot of time, so in most cases one can simply wait till tomorrow...
I want to fix this by implementing an option in OST's manual run (switched off by default) that will allow you to upgrade to the latest RPMs from 'tested'. That way one has ~24h for his/her patches to be fresh enough to be picked up by dnf.
'check-patch' jobs should always use latest RPMs from 'tested' IMO.
This will also add load, and might cause "perceived" instability - as things will likely fluctuate between green and red more often. This doesn't sound very good, I perceive the things less than stable already now. Agreed. I quoted "perceived" because I do not think they'll actually be less stable. Right now, when something critical is broken, we fix it, then manually run some of the above jobs as needed, to quickly get back to business. When we don't (often), some things simply remain broken for two days.
Running more often will simply notify us about breakage faster. If we then fix, it will automatically propage the fix faster.
Isn't upgrading the engine RPM on the appliance an option?
You mean, as part of the OST run itself? Generally speaking, 'hosted-engine --deploy' already does that, but in practice this does not work in CI. I didn't check recently why. Probably some configuration (repos) or missing proxy or something like that. It's done in a task called 'Update all packages' (something to search for in logs if you feel like it). It can be controlled from the CLI with he_offline_deployment [1], but I do not see anywhere that we use this in CI. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1851860
I think I prefer the latter. What do you think? Wouldn't it be possible to run the whole pipeline nightly (even if it means e.g. running the publisher twice during the night)? It will. But this will only fix the specific issue of appliance/node. Running more often also simply gives feedback faster.
But I agree that perhaps we should wait with this until OST allows using a custom repo reliably and easily.
Thanks,
-- Didi
Il giorno mer 24 feb 2021 alle ore 08:55 Yedidyah Bar David <didi@redhat.com> ha scritto:
Hi all,
Right now, when we merge a patch e.g. to the engine (and many other projects), it can take up to several days until it is used by the hosted-engine ovirt-system-tests suite. Something similar will happen soon if/when we introduce suites that use ovirt-node.
If I got it right: - Merge causes CI to build the engine - immediately, takes ~ 1 hour (say) - A publisher job [1] publishes it to resources.ovirt.org (daily, midnight (UTC)) - The next run of an appliance build [2] includes it (daily, afternoon) - The next run of the publisher [1] publishes the appliance (daily, midnight) - The next run of ost-images [3] includes the appliance (daily, midnight, 2 hours after the publisher) (and publishes it immediately) - The next run of ost (e.g. [4]) will use it (daily, slightly *before* ost-images, but I guess we can change that. And this does not affect manual runs of OST, so can probably be ignored in the calculation, at least to some extent).
So if I got it right, a patch merged to the engine in some morning, will be used by the nightly run of OST HE only after almost 3 days, and available for manual runs after 2 days. IMO that's too much time. I might be somewhat wrong, but not very, I think.
One partial solution is to add automation .repos lines to relevant projects that will link at lastSuccessfulBuild (let's call it lastSB) of the more important projects they consume - e.g. appliance to use lastSB of engine+dwh+a few others, node to use lastSB of vdsm, etc. This will require more maintenance (adding/removing/fixing projects as needed) and cause some more load on CI (as now packages will be downloaded from it instead of from resources.ovirt.org).
Another solution is to run relevant jobs (publisher/appliance/node) far more often - say, once every two hours. This will also add load, and might cause "perceived" instability - as things will likely fluctuate between green and red more often.
I think I prefer the latter. What do you think?
I think you may want to change ovirt-appliance to build from ovirt-release-master-tested instead of from ovirt-release-master. This way appliance is going to consume the repos as fresh as they passes OST chain. I think you may also want to trigger appliance build 4 times a day (it takes a couple of hours for a build so probably no big advantage doing it more often. The appliance is going to be sent to tested repo right after build. OST image should then get it from master-tested repo instead of master-nightly. This way you should get OST images updated 4 times a day.
Thanks and best regards,
[1] https://jenkins.ovirt.org/job/ovirt_master_publish-rpms_nightly/ [2] https://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-el8-x86... [3] https://jenkins.ovirt.org/job/ost-images_master_standard-poll-upstream-sourc... [4] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/ -- Didi _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/EY4V5JLE7MWN5L...
-- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. <https://mojo.redhat.com/docs/DOC-1199578>*
participants (4)
-
Marcin Sobczyk -
Milan Zamazal -
Sandro Bonazzola -
Yedidyah Bar David