On 12 Feb 2020, at 13:48, Dafna Ron <dron@redhat.com> wrote:

On Wed, Feb 12, 2020 at 6:38 PM Martin Perina <mperina@redhat.com> wrote:

On Wed, Feb 12, 2020 at 6:33 PM Dafna Ron <dron@redhat.com> wrote:

On Wed, Feb 12, 2020 at 5:03 PM Martin Perina <mperina@redhat.com> wrote:

Hi,

I'm probably missing something here, so please correct me if I'm wrong:

1. CQ is running OST to detect failures and if OST will not pass, RPM with the change will not land in tested repo, right?

unless someone manually moves packages into the tested repo - which caused all projects to fail on engine deploy. 

2. OST is still running engine on EL7, right?

yes

3. We have dropped EL7 builds from engine on Feb 6th https://gerrit.ovirt.org/106795

    So no matter what we push to engine repo, it will not pass the CQ, because it will still be testing the old broken version (because new builds will produce packages only for FC30 and EL8 repos). Am I right or missing something?

Actually, engine has been failing for more then 4 weeks already on unrelated issues. currently its failing on a package which is deprecated because the change has not passed CQ and the package that is tested is based on some un-merged change. so fixing the issues now, have nothing to do with the missing el8 packages and the longer you wait to fix the failures, the more regressions you will introduce and harder it will be to fix it.

At this point, I need to have patch https://gerrit.ovirt.org/#/c/106809/ run in CQ so I can see why its failing.

Well, there are other fixes after this patch which fixes known issues, so I'm not sure if only this fix will make the OST to run successfully.

Right now we are doing our best to make engine running on CentOS 8 (my personal estimation is Friday), so after that we should be able to merge OST patch https://gerrit.ovirt.org/106824 so OST will use EL8 for engine.

So unless I'm missing something it doesn't make sense to block merging to engine and its dependencies, because we need to make them running on EL8 and switch engine on EL8 in OST to unblock CQ. Am I right or am I missing something?

lets first get to a point where we are blocked by rl8 related issues instead of merging more changes which we later on cannot track.

Lets see what the change is failing as for now, you are blocking all other projects (not just engine and its deps).

Well maybe, but again I don't believe that we are able to stabilize OST before we get to engine EL8, because since last week there were merged several patches which might have help OST but also several fixes which broke engine on EL7. And all those fixes are mixed.

So from my point right now the biggest priority is to make engine running on EL8 asap and only afterwards focus on OST stabilization and fixing other less important dependencies

However we should (engine maintainers should) be cautious about merging other stuff unrelated to el8. If it’s not essential I would suggest to postpone merging exactly for that reason that we don’t have any validation beyond unit tests. 

OST is not related at all to this and its not about stabilizing OST but about stabilizing engine as these are code issues and not random OST failures.

It’s related in a sense that as soon as we get OST on el8 running we can sleep a bit better when merging engine patches.

The failures in cq are engine related and the rest of the projects are also failing because a faulty engine package has been introduced to the rest of the projects so everyone are merging "blind" because of the current failures.

You’re right about the other projects, but we can as well just go back to a previous build then. Just remove the last ovirt-engine manually?

you can continue merging if you like, but note that all the projects in CQ have no testing coverage at all at this time and you are merging without any CI testing. 

Regards,

Martin

On Wed, Feb 12, 2020 at 4:03 PM Dafna Ron <dron@redhat.com> wrote:

Hi all,

Engine has been failing for a while and someone has also moved the broken package manually to the tested repo which broke all projects in cq.

At this time, until we stabilise engine and get a new package into tested, all projects will continue failing cq and no new packages will be moved to the stable repo (tested).

if you continue to merge, we not only risk adding more regression but it also makes it more difficult to debug and fix as there are a lot of changes in the queue waiting to be tested and very little resources to run it.

Please stop merging on all projects until this is resolved.

Thanks,

Dafna

--

Martin Perina
Manager, Software Engineering
Red Hat Czech s.r.o.

--

Martin Perina
Manager, Software Engineering
Red Hat Czech s.r.o.

_______________________________________________
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/OJXQXGU4KG6WLFKNK2UAYPYXK4UQTGR6/