Ehud will switch it back to baremetals on the next week. The same will have to be done for OST CI itself. After that we can rebuild unused openshift nodes back to CI baremetals. And also convert el7 baremetals to el8 ones (I guess we will leave few and watch the load on them). Please expect some capacity drop during this time. On Thu, Nov 26, 2020 at 12:00 PM Marcin Sobczyk <msobczyk(a)redhat.com <mailto:msobczyk@redhat.com>> wrote: Hi, since all the important suites are working on el8 already, we're planning to drop all el7 OST runs with [1] very soon. This means we can finally say goodbye to py2 and other legacy stuff! We still need to move manual OST runs not to use containers for that to happen. This effort is tracked here [2]. Regards, Marcin [1] https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378> [2] https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145> -- Anton Marchukov Associate Manager - RHV DevOps - Red Hat

On 12/1/20 8:18 PM, Marcin Sobczyk wrote: > Hi, > > the patch that removed all el7 OST runs has been merged. > > Regards, Marcin > > On 11/26/20 12:04 PM, Anton Marchukov wrote: >> Ehud will switch it back to baremetals on the next week. The same will >> have to be done for OST CI itself. After that we can rebuild unused >> openshift nodes back to CI baremetals. And also convert el7 baremetals >> to el8 ones (I guess we will leave few and watch the load on them). >> Please expect some capacity drop during this time. Anton, Ehud, any progress on this? Thanks, Marcin >> >> On Thu, Nov 26, 2020 at 12:00 PM Marcin Sobczyk <msobczyk(a)redhat.com >> <mailto:msobczyk@redhat.com>> wrote: >> >> Hi, >> >> since all the important suites are working on el8 already, we're >> planning to drop all el7 OST runs with [1] very soon. >> >> This means we can finally say goodbye to py2 and other legacy stuff! >> >> We still need to move manual OST runs not to use containers for >> that to >> happen. >> This effort is tracked here [2]. >> >> Regards, Marcin >> >> [1] https://gerrit.ovirt.org/112378 < https://gerrit.ovirt.org/112378> >> [2] https://ovirt-jira.atlassian.net/browse/OST-145 >> <https://ovirt-jira.atlassian.net/browse/OST-145> >> >> >> >> -- >> Anton Marchukov >> Associate Manager - RHV DevOps - Red Hat

On 12/3/20 12:33 PM, Ehud Yonasi wrote: > Hey, > > If there are no more OST jobs that will use el7, Evgheni can start > rebuilding those nodes to el8. There shouldn't be. > Jenkins won't respawn the ost container anymore. So right now if I try to run a manual OST job it still seems to be using containers. Isn't [3] supposed to be changed? [3] https://github.com/oVirt/jenkins/blob/0a824788eda1e6d6755b451817e3964e1bf... > > On Thu, Dec 3, 2020 at 1:28 PM Marcin Sobczyk <msobczyk(a)redhat.com > <mailto:msobczyk@redhat.com>> wrote: > > > > On 12/1/20 8:18 PM, Marcin Sobczyk wrote: > > Hi, > > > > the patch that removed all el7 OST runs has been merged. > > > > Regards, Marcin > > > > On 11/26/20 12:04 PM, Anton Marchukov wrote: > >> Ehud will switch it back to baremetals on the next week. The > same will > >> have to be done for OST CI itself. After that we can rebuild unused > >> openshift nodes back to CI baremetals. And also convert el7 > baremetals > >> to el8 ones (I guess we will leave few and watch the load on them). > >> Please expect some capacity drop during this time. > Anton, Ehud, any progress on this? > > Thanks, Marcin > > >> > >> On Thu, Nov 26, 2020 at 12:00 PM Marcin Sobczyk > <msobczyk(a)redhat.com <mailto:msobczyk@redhat.com> > >> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>>> wrote: > >> > >> Hi, > >> > >> since all the important suites are working on el8 already, > we're > >> planning to drop all el7 OST runs with [1] very soon. > >> > >> This means we can finally say goodbye to py2 and other > legacy stuff! > >> > >> We still need to move manual OST runs not to use > containers for > >> that to > >> happen. > >> This effort is tracked here [2]. > >> > >> Regards, Marcin > >> > >> [1] https://gerrit.ovirt.org/112378 > <https://gerrit.ovirt.org/112378> <https://gerrit.ovirt.org/112378 > <https://gerrit.ovirt.org/112378>> > >> [2] https://ovirt-jira.atlassian.net/browse/OST-145 > <https://ovirt-jira.atlassian.net/browse/OST-145> > >> <https://ovirt-jira.atlassian.net/browse/OST-145 > <https://ovirt-jira.atlassian.net/browse/OST-145>> > >> > >> > >> > >> -- > >> Anton Marchukov > >> Associate Manager - RHV DevOps - Red Hat >

On 12/3/20 1:02 PM, Ehud Yonasi wrote: > Correct. It is now fixed with the manual ost job. Tried running a manual job a moment ago, but ended up with docker-related error [4].

[4] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... > > On Thu, Dec 3, 2020 at 1:54 PM Marcin Sobczyk <msobczyk(a)redhat.com > <mailto:msobczyk@redhat.com>> wrote: > > > > On 12/3/20 12:33 PM, Ehud Yonasi wrote: > > Hey, > > > > If there are no more OST jobs that will use el7, Evgheni can start > > rebuilding those nodes to el8. > There shouldn't be. > > > Jenkins won't respawn the ost container anymore. > So right now if I try to run a manual OST job it still seems to be > using > containers. > Isn't [3] supposed to be changed? > > [3] > https://github.com/oVirt/jenkins/blob/0a824788eda1e6d6755b451817e3964e1bf... > <https://github.com/oVirt/jenkins/blob/0a824788eda1e6d6755b451817e3964e1bf... > > > > > On Thu, Dec 3, 2020 at 1:28 PM Marcin Sobczyk > <msobczyk(a)redhat.com <mailto:msobczyk@redhat.com> > > <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>>> wrote: > > > > > > > > On 12/1/20 8:18 PM, Marcin Sobczyk wrote: > > > Hi, > > > > > > the patch that removed all el7 OST runs has been merged. > > > > > > Regards, Marcin > > > > > > On 11/26/20 12:04 PM, Anton Marchukov wrote: > > >> Ehud will switch it back to baremetals on the next week. The > > same will > > >> have to be done for OST CI itself. After that we can > rebuild unused > > >> openshift nodes back to CI baremetals. And also convert el7 > > baremetals > > >> to el8 ones (I guess we will leave few and watch the load > on them). > > >> Please expect some capacity drop during this time. > > Anton, Ehud, any progress on this? > > > > Thanks, Marcin > > > > >> > > >> On Thu, Nov 26, 2020 at 12:00 PM Marcin Sobczyk > > <msobczyk(a)redhat.com <mailto:msobczyk@redhat.com> > <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>> > > >> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com> > <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>>>> wrote: > > >> > > >> Hi, > > >> > > >> since all the important suites are working on el8 > already, > > we're > > >> planning to drop all el7 OST runs with [1] very soon. > > >> > > >> This means we can finally say goodbye to py2 and other > > legacy stuff! > > >> > > >> We still need to move manual OST runs not to use > > containers for > > >> that to > > >> happen. > > >> This effort is tracked here [2]. > > >> > > >> Regards, Marcin > > >> > > >> [1] https://gerrit.ovirt.org/112378 > <https://gerrit.ovirt.org/112378> > > <https://gerrit.ovirt.org/112378 > <https://gerrit.ovirt.org/112378>> > <https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378> > > <https://gerrit.ovirt.org/112378 > <https://gerrit.ovirt.org/112378>>> > > >> [2] https://ovirt-jira.atlassian.net/browse/OST-145 > <https://ovirt-jira.atlassian.net/browse/OST-145> > > <https://ovirt-jira.atlassian.net/browse/OST-145 > <https://ovirt-jira.atlassian.net/browse/OST-145>> > > >> <https://ovirt-jira.atlassian.net/browse/OST-145 > <https://ovirt-jira.atlassian.net/browse/OST-145> > > <https://ovirt-jira.atlassian.net/browse/OST-145 > <https://ovirt-jira.atlassian.net/browse/OST-145>>> > > >> > > >> > > >> > > >> -- > > >> Anton Marchukov > > >> Associate Manager - RHV DevOps - Red Hat > > > > _______________________________________________ Devel mailing list -- devel(a)ovirt.org To unsubscribe send an email to devel-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/4LSP4EO4V5S...

On 12/7/20 8:19 AM, Yedidyah Bar David wrote: > On Thu, Dec 3, 2020 at 5:16 PM Marcin Sobczyk <msobczyk(a)redhat.com&gt; wrote: >> >> On 12/3/20 1:02 PM, Ehud Yonasi wrote: >>> Correct. It is now fixed with the manual ost job. >> Tried running a manual job a moment ago, but ended up with >> docker-related error [4]. > Also, are the nightly runs [1] running in el8? Half of them still get > stuck on 'Create local VM' and timeout, which usually used to happen > on el7. AFAICS they're still using el7 containers: 02:55:24 Building remotely on openshift-integ-tests-container-wrrzs (el7 integ-tests-container) in workspace /home/jenkins/agent/workspace/ovirt-system-tests_he-basic-suite-master but I think fixing the manual runs is much more important right now.

Regards, Marcin > > Thanks, > > [1] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/ > >> [4] >> https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... >> >>> On Thu, Dec 3, 2020 at 1:54 PM Marcin Sobczyk <msobczyk(a)redhat.com >>> <mailto:msobczyk@redhat.com>> wrote: >>> >>> >>> >>> On 12/3/20 12:33 PM, Ehud Yonasi wrote: >>> > Hey, >>> > >>> > If there are no more OST jobs that will use el7, Evgheni can start >>> > rebuilding those nodes to el8. >>> There shouldn't be. >>> >>> > Jenkins won't respawn the ost container anymore. >>> So right now if I try to run a manual OST job it still seems to be >>> using >>> containers. >>> Isn't [3] supposed to be changed? >>> >>> [3] >>> https://github.com/oVirt/jenkins/blob/0a824788eda1e6d6755b451817e3964e1bf... >>> <https://github.com/oVirt/jenkins/blob/0a824788eda1e6d6755b451817e3964e1bf... >>> >>> > >>> > On Thu, Dec 3, 2020 at 1:28 PM Marcin Sobczyk >>> <msobczyk(a)redhat.com <mailto:msobczyk@redhat.com> >>> > <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>>> wrote: >>> > >>> > >>> > >>> > On 12/1/20 8:18 PM, Marcin Sobczyk wrote: >>> > > Hi, >>> > > >>> > > the patch that removed all el7 OST runs has been merged. >>> > > >>> > > Regards, Marcin >>> > > >>> > > On 11/26/20 12:04 PM, Anton Marchukov wrote: >>> > >> Ehud will switch it back to baremetals on the next week. The >>> > same will >>> > >> have to be done for OST CI itself. After that we can >>> rebuild unused >>> > >> openshift nodes back to CI baremetals. And also convert el7 >>> > baremetals >>> > >> to el8 ones (I guess we will leave few and watch the load >>> on them). >>> > >> Please expect some capacity drop during this time. >>> > Anton, Ehud, any progress on this? >>> > >>> > Thanks, Marcin >>> > >>> > >> >>> > >> On Thu, Nov 26, 2020 at 12:00 PM Marcin Sobczyk >>> > <msobczyk(a)redhat.com <mailto:msobczyk@redhat.com> >>> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>> >>> > >> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com> >>> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>>>> wrote: >>> > >> >>> > >> Hi, >>> > >> >>> > >> since all the important suites are working on el8 >>> already, >>> > we're >>> > >> planning to drop all el7 OST runs with [1] very soon. >>> > >> >>> > >> This means we can finally say goodbye to py2 and other >>> > legacy stuff! >>> > >> >>> > >> We still need to move manual OST runs not to use >>> > containers for >>> > >> that to >>> > >> happen. >>> > >> This effort is tracked here [2]. >>> > >> >>> > >> Regards, Marcin >>> > >> >>> > >> [1] https://gerrit.ovirt.org/112378 >>> <https://gerrit.ovirt.org/112378> >>> > <https://gerrit.ovirt.org/112378 >>> <https://gerrit.ovirt.org/112378>> >>> <https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378> >>> > <https://gerrit.ovirt.org/112378 >>> <https://gerrit.ovirt.org/112378>>> >>> > >> [2] https://ovirt-jira.atlassian.net/browse/OST-145 >>> <https://ovirt-jira.atlassian.net/browse/OST-145> >>> > <https://ovirt-jira.atlassian.net/browse/OST-145 >>> <https://ovirt-jira.atlassian.net/browse/OST-145>> >>> > >> <https://ovirt-jira.atlassian.net/browse/OST-145 >>> <https://ovirt-jira.atlassian.net/browse/OST-145> >>> > <https://ovirt-jira.atlassian.net/browse/OST-145 >>> <https://ovirt-jira.atlassian.net/browse/OST-145>>> >>> > >> >>> > >> >>> > >> >>> > >> -- >>> > >> Anton Marchukov >>> > >> Associate Manager - RHV DevOps - Red Hat >>> > >>> >>> >> _______________________________________________ >> Devel mailing list -- devel(a)ovirt.org >> To unsubscribe send an email to devel-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/4LSP4EO4V5S... > >

On 12/7/20 11:00 AM, Yedidyah Bar David wrote: > On Mon, Dec 7, 2020 at 11:48 AM Marcin Sobczyk <msobczyk(a)redhat.com&gt; wrote: >> >> >> On 12/7/20 8:19 AM, Yedidyah Bar David wrote: >>> On Thu, Dec 3, 2020 at 5:16 PM Marcin Sobczyk <msobczyk(a)redhat.com&gt; wrote: >>>> On 12/3/20 1:02 PM, Ehud Yonasi wrote: >>>>> Correct. It is now fixed with the manual ost job. >>>> Tried running a manual job a moment ago, but ended up with >>>> docker-related error [4]. >>> Also, are the nightly runs [1] running in el8? Half of them still get >>> stuck on 'Create local VM' and timeout, which usually used to happen >>> on el7. >> AFAICS they're still using el7 containers: >> >> 02:55:24 Building remotely on openshift-integ-tests-container-wrrzs (el7 >> integ-tests-container) in workspace >> /home/jenkins/agent/workspace/ovirt-system-tests_he-basic-suite-master >> >> but I think fixing the manual runs is much more important right now. > IMO both are important! ;-) > > We had he-basic-suite broken for more than 50% of the time, last year. > If we let it remain broken, we risk not noticing real bugs/regressions. Of course, I just meant we're actively working and running he suite regularly right now, so manual runs are a higher priority ATM. > > I also guess it's a simple fix, no? I think something like: > > https://gerrit.ovirt.org/c/jenkins/+/112527 > > (I'd change all el7 to el8 there, except for 4.3 suites, but perhaps better > split and do this later). Probably not - el8 chroot running on el7 kernel won't work, so we'd need to do node filtering, but I'll leave that to Ehud :) > >> Regards, Marcin >> >>> Thanks, >>> >>> [1] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/ >>> >>>> [4] >>>> https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... >>>> >>>>> On Thu, Dec 3, 2020 at 1:54 PM Marcin Sobczyk <msobczyk(a)redhat.com >>>>> <mailto:msobczyk@redhat.com>> wrote: >>>>> >>>>> >>>>> >>>>> On 12/3/20 12:33 PM, Ehud Yonasi wrote: >>>>> > Hey, >>>>> > >>>>> > If there are no more OST jobs that will use el7, Evgheni can start >>>>> > rebuilding those nodes to el8. >>>>> There shouldn't be. >>>>> >>>>> > Jenkins won't respawn the ost container anymore. >>>>> So right now if I try to run a manual OST job it still seems to be >>>>> using >>>>> containers. >>>>> Isn't [3] supposed to be changed? >>>>> >>>>> [3] >>>>> https://github.com/oVirt/jenkins/blob/0a824788eda1e6d6755b451817e3964e1bf... >>>>> < https://github.com/oVirt/jenkins/blob/0a824788eda1e6d6755b451817e3964e1bf... > >>>>> >>>>> > >>>>> > On Thu, Dec 3, 2020 at 1:28 PM Marcin Sobczyk >>>>> <msobczyk(a)redhat.com <mailto:msobczyk@redhat.com> >>>>> > <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>>> wrote: >>>>> > >>>>> > >>>>> > >>>>> > On 12/1/20 8:18 PM, Marcin Sobczyk wrote: >>>>> > > Hi, >>>>> > > >>>>> > > the patch that removed all el7 OST runs has been merged. >>>>> > > >>>>> > > Regards, Marcin >>>>> > > >>>>> > > On 11/26/20 12:04 PM, Anton Marchukov wrote: >>>>> > >> Ehud will switch it back to baremetals on the next week. The >>>>> > same will >>>>> > >> have to be done for OST CI itself. After that we can >>>>> rebuild unused >>>>> > >> openshift nodes back to CI baremetals. And also convert el7 >>>>> > baremetals >>>>> > >> to el8 ones (I guess we will leave few and watch the load >>>>> on them). >>>>> > >> Please expect some capacity drop during this time. >>>>> > Anton, Ehud, any progress on this? >>>>> > >>>>> > Thanks, Marcin >>>>> > >>>>> > >> >>>>> > >> On Thu, Nov 26, 2020 at 12:00 PM Marcin Sobczyk >>>>> > <msobczyk(a)redhat.com <mailto:msobczyk@redhat.com> >>>>> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>> >>>>> > >> <mailto:msobczyk@redhat.com <mailto: msobczyk(a)redhat.com&gt; >>>>> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>>>> wrote: >>>>> > >> >>>>> > >> Hi, >>>>> > >> >>>>> > >> since all the important suites are working on el8 >>>>> already, >>>>> > we're >>>>> > >> planning to drop all el7 OST runs with [1] very soon. >>>>> > >> >>>>> > >> This means we can finally say goodbye to py2 and other >>>>> > legacy stuff! >>>>> > >> >>>>> > >> We still need to move manual OST runs not to use >>>>> > containers for >>>>> > >> that to >>>>> > >> happen. >>>>> > >> This effort is tracked here [2]. >>>>> > >> >>>>> > >> Regards, Marcin >>>>> > >> >>>>> > >> [1] https://gerrit.ovirt.org/112378 >>>>> <https://gerrit.ovirt.org/112378> >>>>> > <https://gerrit.ovirt.org/112378 >>>>> <https://gerrit.ovirt.org/112378>> >>>>> <https://gerrit.ovirt.org/112378 < https://gerrit.ovirt.org/112378> >>>>> > <https://gerrit.ovirt.org/112378 >>>>> <https://gerrit.ovirt.org/112378>>> >>>>> > >> [2] https://ovirt-jira.atlassian.net/browse/OST-145 >>>>> <https://ovirt-jira.atlassian.net/browse/OST-145> >>>>> > <https://ovirt-jira.atlassian.net/browse/OST-145 >>>>> <https://ovirt-jira.atlassian.net/browse/OST-145>> >>>>> > >> <https://ovirt-jira.atlassian.net/browse/OST-145 >>>>> <https://ovirt-jira.atlassian.net/browse/OST-145> >>>>> > <https://ovirt-jira.atlassian.net/browse/OST-145 >>>>> <https://ovirt-jira.atlassian.net/browse/OST-145>>> >>>>> > >> >>>>> > >> >>>>> > >> >>>>> > >> -- >>>>> > >> Anton Marchukov >>>>> > >> Associate Manager - RHV DevOps - Red Hat >>>>> > >>>>> >>>>> >>>> _______________________________________________ >>>> Devel mailing list -- devel(a)ovirt.org >>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>>> List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/4LSP4EO4V5S... >>> > _______________________________________________ Devel mailing list -- devel(a)ovirt.org To unsubscribe send an email to devel-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/YAMBGKYODB5...