Hi, I'm probably missing something here, so please correct me if I'm wrong: 1. CQ is running OST to detect failures and if OST will not pass, RPM with the change will not land in tested repo, right?

2. OST is still running engine on EL7, right?

3. We have dropped EL7 builds from engine on Feb 6th https://gerrit.ovirt.org/106795 So no matter what we push to engine repo, it will not pass the CQ, because it will still be testing the old broken version (because new builds will produce packages only for FC30 and EL8 repos). Am I right or missing something? Actually, engine has been failing for more then 4 weeks already on

personal estimation is Friday), so after that we should be able to merge OST patch https://gerrit.ovirt.org/106824 so OST will use EL8 for engine.

So unless I'm missing something it doesn't make sense to block merging to engine and its dependencies, because we need to make them running on EL8 and switch engine on EL8 in OST to unblock CQ. Am I right or am I missing something?

Regards, Martin On Wed, Feb 12, 2020 at 4:03 PM Dafna Ron <dron(a)redhat.com&gt; wrote: > Hi all, > > Engine has been failing for a while and someone has also moved the broken > package manually to the tested repo which broke all projects in cq. > At this time, until we stabilise engine and get a new package into > tested, all projects will continue failing cq and no new packages will be > moved to the stable repo (tested). > if you continue to merge, we not only risk adding more regression but it > also makes it more difficult to debug and fix as there are a lot of changes > in the queue waiting to be tested and very little resources to run it. > > Please stop merging on all projects until this is resolved. > > Thanks, > Dafna > > > > -- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o.

On Wed, Feb 12, 2020 at 6:33 PM Dafna Ron <dron(a)redhat.com&gt; wrote: > > > On Wed, Feb 12, 2020 at 5:03 PM Martin Perina <mperina(a)redhat.com&gt; wrote: > >> Hi, >> >> I'm probably missing something here, so please correct me if I'm wrong: >> >> 1. CQ is running OST to detect failures and if OST will not pass, RPM >> with the change will not land in tested repo, right? >> > unless someone manually moves packages into the tested repo - which > caused all projects to fail on engine deploy. > >> >> 2. OST is still running engine on EL7, right? >> > yes > >> >> 3. We have dropped EL7 builds from engine on Feb 6th >> https://gerrit.ovirt.org/106795 >> So no matter what we push to engine repo, it will not pass the CQ, >> because it will still be testing the old broken version (because new builds >> will produce packages only for FC30 and EL8 repos). Am I right or missing >> something? >> >> Actually, engine has been failing for more then 4 weeks already on > unrelated issues. currently its failing on a package which is deprecated > because the change has not passed CQ and the package that is tested is > based on some un-merged change. so fixing the issues now, have nothing to > do with the missing el8 packages and the longer you wait to fix the > failures, the more regressions you will introduce and harder it will be to > fix it. > > At this point, I need to have patch https://gerrit.ovirt.org/#/c/106809/ > run in CQ so I can see why its failing. > Well, there are other fixes after this patch which fixes known issues, so I'm not sure if only this fix will make the OST to run successfully. > Right now we are doing our best to make engine running on CentOS 8 (my >> personal estimation is Friday), so after that we should be able to merge >> OST patch https://gerrit.ovirt.org/106824 so OST will use EL8 for >> engine. >> > >> So unless I'm missing something it doesn't make sense to block merging >> to engine and its dependencies, because we need to make them running on EL8 >> and switch engine on EL8 in OST to unblock CQ. Am I right or am I missing >> something? >> > > lets first get to a point where we are blocked by rl8 related issues > instead of merging more changes which we later on cannot track. > Lets see what the change is failing as for now, you are blocking all > other projects (not just engine and its deps). > Well maybe, but again I don't believe that we are able to stabilize OST before we get to engine EL8, because since last week there were merged several patches which might have help OST but also several fixes which broke engine on EL7. And all those fixes are mixed. So from my point right now the biggest priority is to make engine running on EL8 asap and only afterwards focus on OST stabilization and fixing other less important dependencies

Regards, Martin On Wed, Feb 12, 2020 at 4:03 PM Dafna Ron <dron(a)redhat.com&gt; wrote: > Hi all, > > Engine has been failing for a while and someone has also moved the broken > package manually to the tested repo which broke all projects in cq. > At this time, until we stabilise engine and get a new package into > tested, all projects will continue failing cq and no new packages will be > moved to the stable repo (tested). > if you continue to merge, we not only risk adding more regression but it > also makes it more difficult to debug and fix as there are a lot of changes > in the queue waiting to be tested and very little resources to run it. > > Please stop merging on all projects until this is resolved. > > Thanks, > Dafna > > > > -- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o.

-- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o.

On 12 Feb 2020, at 13:48, Dafna Ron <dron(a)redhat.com&gt; wrote: On Wed, Feb 12, 2020 at 6:38 PM Martin Perina <mperina(a)redhat.com&gt; wrote: > > > On Wed, Feb 12, 2020 at 6:33 PM Dafna Ron <dron(a)redhat.com&gt; wrote: > >> >> >> On Wed, Feb 12, 2020 at 5:03 PM Martin Perina <mperina(a)redhat.com&gt; >> wrote: >> >>> Hi, >>> >>> I'm probably missing something here, so please correct me if I'm wrong: >>> >>> 1. CQ is running OST to detect failures and if OST will not pass, RPM >>> with the change will not land in tested repo, right? >>> >> unless someone manually moves packages into the tested repo - which >> caused all projects to fail on engine deploy. >> >>> >>> 2. OST is still running engine on EL7, right? >>> >> yes >> >>> >>> 3. We have dropped EL7 builds from engine on Feb 6th >>> https://gerrit.ovirt.org/106795 >>> So no matter what we push to engine repo, it will not pass the CQ, >>> because it will still be testing the old broken version (because new builds >>> will produce packages only for FC30 and EL8 repos). Am I right or missing >>> something? >>> >>> Actually, engine has been failing for more then 4 weeks already on >> unrelated issues. currently its failing on a package which is deprecated >> because the change has not passed CQ and the package that is tested is >> based on some un-merged change. so fixing the issues now, have nothing to >> do with the missing el8 packages and the longer you wait to fix the >> failures, the more regressions you will introduce and harder it will be to >> fix it. >> >> At this point, I need to have patch https://gerrit.ovirt.org/#/c/106809/ >> run in CQ so I can see why its failing. >> > > Well, there are other fixes after this patch which fixes known issues, so > I'm not sure if only this fix will make the OST to run successfully. > > >> Right now we are doing our best to make engine running on CentOS 8 (my >>> personal estimation is Friday), so after that we should be able to merge >>> OST patch https://gerrit.ovirt.org/106824 so OST will use EL8 for >>> engine. >>> >> >>> So unless I'm missing something it doesn't make sense to block merging >>> to engine and its dependencies, because we need to make them running on EL8 >>> and switch engine on EL8 in OST to unblock CQ. Am I right or am I missing >>> something? >>> >> >> lets first get to a point where we are blocked by rl8 related issues >> instead of merging more changes which we later on cannot track. >> Lets see what the change is failing as for now, you are blocking all >> other projects (not just engine and its deps). >> > > Well maybe, but again I don't believe that we are able to stabilize OST > before we get to engine EL8, because since last week there were merged > several patches which might have help OST but also several fixes which > broke engine on EL7. And all those fixes are mixed. > > So from my point right now the biggest priority is to make engine running > on EL8 asap and only afterwards focus on OST stabilization and fixing other > less important dependencies > However we should (engine maintainers should) be cautious about merging other stuff unrelated to el8. If it’s not essential I would suggest to postpone merging exactly for that reason that we don’t have any validation beyond unit tests. OST is not related at all to this and its not about stabilizing OST but about stabilizing engine as these are code issues and not random OST failures. It’s related in a sense that as soon as we get OST on el8 running we can sleep a bit better when merging engine patches. The failures in cq are engine related and the rest of the projects are also failing because a faulty engine package has been introduced to the rest of the projects so everyone are merging "blind" because of the current failures. You’re right about the other projects, but we can as well just go back to a previous build then. Just remove the last ovirt-engine manually?

you can continue merging if you like, but note that all the projects in CQ have no testing coverage at all at this time and you are merging without any CI testing. > Regards, > Martin > > > On Wed, Feb 12, 2020 at 4:03 PM Dafna Ron <dron(a)redhat.com&gt; wrote: > >> Hi all, >> >> Engine has been failing for a while and someone has also moved the >> broken package manually to the tested repo which broke all projects in cq. >> At this time, until we stabilise engine and get a new package into >> tested, all projects will continue failing cq and no new packages will be >> moved to the stable repo (tested). >> if you continue to merge, we not only risk adding more regression but it >> also makes it more difficult to debug and fix as there are a lot of changes >> in the queue waiting to be tested and very little resources to run it. >> >> Please stop merging on all projects until this is resolved. >> >> Thanks, >> Dafna >> >> >> >> > > -- > Martin Perina > Manager, Software Engineering > Red Hat Czech s.r.o. > > > -- > Martin Perina > Manager, Software Engineering > Red Hat Czech s.r.o. > _______________________________________________ Devel mailing list -- devel(a)ovirt.org To unsubscribe send an email to devel-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/OJXQXGU4KG6...

On Thu, Feb 13, 2020 at 12:09 PM Dafna Ron <dron(a)redhat.com&gt; wrote: > > > > On Wed, Feb 12, 2020 at 11:37 PM Michal Skrivanek < michal.skrivanek(a)redhat.com&gt; wrote: >> >> >> >> On 12 Feb 2020, at 13:48, Dafna Ron <dron(a)redhat.com&gt; wrote: >> >> >> >> On Wed, Feb 12, 2020 at 6:38 PM Martin Perina <mperina(a)redhat.com&gt; wrote: >>> >>> >>> >>> On Wed, Feb 12, 2020 at 6:33 PM Dafna Ron <dron(a)redhat.com&gt; wrote: >>>> >>>> >>>> >>>> On Wed, Feb 12, 2020 at 5:03 PM Martin Perina <mperina(a)redhat.com&gt; wrote: >>>>> >>>>> Hi, >>>>> >>>>> I'm probably missing something here, so please correct me if I'm wrong: >>>>> >>>>> 1. CQ is running OST to detect failures and if OST will not pass, RPM with the change will not land in tested repo, right? >>>> >>>> unless someone manually moves packages into the tested repo - which caused all projects to fail on engine deploy. >>>>> >>>>> >>>>> 2. OST is still running engine on EL7, right? >>>> >>>> yes >>>>> >>>>> >>>>> 3. We have dropped EL7 builds from engine on Feb 6th https://gerrit.ovirt.org/106795 >>>>> So no matter what we push to engine repo, it will not pass the CQ, because it will still be testing the old broken version (because new builds will produce packages only for FC30 and EL8 repos). Am I right or missing something? >>>>> >>>> Actually, engine has been failing for more then 4 weeks already on unrelated issues. currently its failing on a package which is deprecated because the change has not passed CQ and the package that is tested is based on some un-merged change. so fixing the issues now, have nothing to do with the missing el8 packages and the longer you wait to fix the failures, the more regressions you will introduce and harder it will be to fix it. >>>> >>>> At this point, I need to have patch https://gerrit.ovirt.org/#/c/106809/ run in CQ so I can see why its failing. >>> >>> >>> Well, there are other fixes after this patch which fixes known issues, so I'm not sure if only this fix will make the OST to run successfully. >>> >>>> >>>>> Right now we are doing our best to make engine running on CentOS 8 (my personal estimation is Friday), so after that we should be able to merge OST patch https://gerrit.ovirt.org/106824 so OST will use EL8 for engine. >>>>> >>>>> >>>>> So unless I'm missing something it doesn't make sense to block merging to engine and its dependencies, because we need to make them running on EL8 and switch engine on EL8 in OST to unblock CQ. Am I right or am I missing something? >>>> >>>> >>>> lets first get to a point where we are blocked by rl8 related issues instead of merging more changes which we later on cannot track. >>>> Lets see what the change is failing as for now, you are blocking all other projects (not just engine and its deps). >>> >>> >>> Well maybe, but again I don't believe that we are able to stabilize OST before we get to engine EL8, because since last week there were merged several patches which might have help OST but also several fixes which broke engine on EL7. And all those fixes are mixed. >>> >>> So from my point right now the biggest priority is to make engine running on EL8 asap and only afterwards focus on OST stabilization and fixing other less important dependencies >> >> >> However we should (engine maintainers should) be cautious about merging other stuff unrelated to el8. If it’s not essential I would suggest to postpone merging exactly for that reason that we don’t have any validation beyond unit tests. >> >> >> OST is not related at all to this and its not about stabilizing OST but about stabilizing engine as these are code issues and not random OST failures. >> >> >> It’s related in a sense that as soon as we get OST on el8 running we can sleep a bit better when merging engine patches. >> >> The failures in cq are engine related and the rest of the projects are also failing because a faulty engine package has been introduced to the rest of the projects so everyone are merging "blind" because of the current failures. >> >> >> You’re right about the other projects, but we can as well just go back to a previous build then. Just remove the last ovirt-engine manually? > > > We do not have a roll back on the repo but perhaps Anton would have an idea. I do not see much point. This will give you a somewhat older el7 build. Newer ones do not exist anymore, because we do not build for el7 anymore. I agree that the best plan is to push strongly for moving to el8. I talked with Galit about this earlier today. She is working on a patch to OST to move the engine there to el8. As we agreed in private, there is no point in checking it against the current tested repo, because it's broken. Instead, it should be checked against the current engine master. If it passes, I think we can merge this patch to OST, and then hopefully CI/change-queue will be in a good-enough shape to at least help find real bugs and fix them :-) > >> >> you can continue merging if you like, but note that all the projects in CQ have no testing coverage at all at this time and you are merging without any CI testing. >> >> >> >>> >>> Regards, >>> Martin >>> >>> >>> On Wed, Feb 12, 2020 at 4:03 PM Dafna Ron <dron(a)redhat.com&gt; wrote: >>>> >>>> Hi all, >>>> >>>> Engine has been failing for a while and someone has also moved the broken package manually to the tested repo which broke all projects in cq. >>>> At this time, until we stabilise engine and get a new package into tested, all projects will continue failing cq and no new packages will be moved to the stable repo (tested). >>>> if you continue to merge, we not only risk adding more regression but it also makes it more difficult to debug and fix as there are a lot of changes in the queue waiting to be tested and very little resources to run it. >>>> >>>> Please stop merging on all projects until this is resolved. >>>> >>>> Thanks, >>>> Dafna >>>> >>>> >>>> >>> >>> >>> -- >>> Martin Perina >>> Manager, Software Engineering >>> Red Hat Czech s.r.o. >>> >>> >>> >>> -- >>> Martin Perina >>> Manager, Software Engineering >>> Red Hat Czech s.r.o. >> >> _______________________________________________ >> Devel mailing list -- devel(a)ovirt.org >> To unsubscribe send an email to devel-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/OJXQXGU4KG6... > > _______________________________________________ > Devel mailing list -- devel(a)ovirt.org > To unsubscribe send an email to devel-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/BD2UZXENQME... -- Didi