Dropping all el7 OST runs
Hi, since all the important suites are working on el8 already, we're planning to drop all el7 OST runs with [1] very soon. This means we can finally say goodbye to py2 and other legacy stuff! We still need to move manual OST runs not to use containers for that to happen. This effort is tracked here [2]. Regards, Marcin [1] https://gerrit.ovirt.org/112378 [2] https://ovirt-jira.atlassian.net/browse/OST-145
Ehud will switch it back to baremetals on the next week. The same will have to be done for OST CI itself. After that we can rebuild unused openshift nodes back to CI baremetals. And also convert el7 baremetals to el8 ones (I guess we will leave few and watch the load on them). Please expect some capacity drop during this time. On Thu, Nov 26, 2020 at 12:00 PM Marcin Sobczyk <msobczyk@redhat.com> wrote:
Hi,
since all the important suites are working on el8 already, we're planning to drop all el7 OST runs with [1] very soon.
This means we can finally say goodbye to py2 and other legacy stuff!
We still need to move manual OST runs not to use containers for that to happen. This effort is tracked here [2].
Regards, Marcin
[1] https://gerrit.ovirt.org/112378 [2] https://ovirt-jira.atlassian.net/browse/OST-145
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
Hi, the patch that removed all el7 OST runs has been merged. Regards, Marcin On 11/26/20 12:04 PM, Anton Marchukov wrote:
Ehud will switch it back to baremetals on the next week. The same will have to be done for OST CI itself. After that we can rebuild unused openshift nodes back to CI baremetals. And also convert el7 baremetals to el8 ones (I guess we will leave few and watch the load on them). Please expect some capacity drop during this time.
On Thu, Nov 26, 2020 at 12:00 PM Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com>> wrote:
Hi,
since all the important suites are working on el8 already, we're planning to drop all el7 OST runs with [1] very soon.
This means we can finally say goodbye to py2 and other legacy stuff!
We still need to move manual OST runs not to use containers for that to happen. This effort is tracked here [2].
Regards, Marcin
[1] https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378> [2] https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145>
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
On 12/1/20 8:18 PM, Marcin Sobczyk wrote:
Hi,
the patch that removed all el7 OST runs has been merged.
Regards, Marcin
On 11/26/20 12:04 PM, Anton Marchukov wrote:
Ehud will switch it back to baremetals on the next week. The same will have to be done for OST CI itself. After that we can rebuild unused openshift nodes back to CI baremetals. And also convert el7 baremetals to el8 ones (I guess we will leave few and watch the load on them). Please expect some capacity drop during this time. Anton, Ehud, any progress on this?
Thanks, Marcin
On Thu, Nov 26, 2020 at 12:00 PM Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com>> wrote:
Hi,
since all the important suites are working on el8 already, we're planning to drop all el7 OST runs with [1] very soon.
This means we can finally say goodbye to py2 and other legacy stuff!
We still need to move manual OST runs not to use containers for that to happen. This effort is tracked here [2].
Regards, Marcin
[1] https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378> [2] https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145>
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
Hey, If there are no more OST jobs that will use el7, Evgheni can start rebuilding those nodes to el8. Jenkins won't respawn the ost container anymore. On Thu, Dec 3, 2020 at 1:28 PM Marcin Sobczyk <msobczyk@redhat.com> wrote:
On 12/1/20 8:18 PM, Marcin Sobczyk wrote:
Hi,
the patch that removed all el7 OST runs has been merged.
Regards, Marcin
On 11/26/20 12:04 PM, Anton Marchukov wrote:
Ehud will switch it back to baremetals on the next week. The same will have to be done for OST CI itself. After that we can rebuild unused openshift nodes back to CI baremetals. And also convert el7 baremetals to el8 ones (I guess we will leave few and watch the load on them). Please expect some capacity drop during this time. Anton, Ehud, any progress on this?
Thanks, Marcin
On Thu, Nov 26, 2020 at 12:00 PM Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com>> wrote:
Hi,
since all the important suites are working on el8 already, we're planning to drop all el7 OST runs with [1] very soon.
This means we can finally say goodbye to py2 and other legacy
stuff!
We still need to move manual OST runs not to use containers for that to happen. This effort is tracked here [2].
Regards, Marcin
https://gerrit.ovirt.org/112378>
[2] https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145>
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
On 12/3/20 12:33 PM, Ehud Yonasi wrote:
Hey,
If there are no more OST jobs that will use el7, Evgheni can start rebuilding those nodes to el8. There shouldn't be.
Jenkins won't respawn the ost container anymore. So right now if I try to run a manual OST job it still seems to be using containers. Isn't [3] supposed to be changed?
[3] https://github.com/oVirt/jenkins/blob/0a824788eda1e6d6755b451817e3964e1bf14b...
On Thu, Dec 3, 2020 at 1:28 PM Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com>> wrote:
On 12/1/20 8:18 PM, Marcin Sobczyk wrote: > Hi, > > the patch that removed all el7 OST runs has been merged. > > Regards, Marcin > > On 11/26/20 12:04 PM, Anton Marchukov wrote: >> Ehud will switch it back to baremetals on the next week. The same will >> have to be done for OST CI itself. After that we can rebuild unused >> openshift nodes back to CI baremetals. And also convert el7 baremetals >> to el8 ones (I guess we will leave few and watch the load on them). >> Please expect some capacity drop during this time. Anton, Ehud, any progress on this?
Thanks, Marcin
>> >> On Thu, Nov 26, 2020 at 12:00 PM Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com> >> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>>> wrote: >> >> Hi, >> >> since all the important suites are working on el8 already, we're >> planning to drop all el7 OST runs with [1] very soon. >> >> This means we can finally say goodbye to py2 and other legacy stuff! >> >> We still need to move manual OST runs not to use containers for >> that to >> happen. >> This effort is tracked here [2]. >> >> Regards, Marcin >> >> [1] https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378> <https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378>> >> [2] https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145> >> <https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145>> >> >> >> >> -- >> Anton Marchukov >> Associate Manager - RHV DevOps - Red Hat
Correct. It is now fixed with the manual ost job. On Thu, Dec 3, 2020 at 1:54 PM Marcin Sobczyk <msobczyk@redhat.com> wrote:
On 12/3/20 12:33 PM, Ehud Yonasi wrote:
Hey,
If there are no more OST jobs that will use el7, Evgheni can start rebuilding those nodes to el8. There shouldn't be.
Jenkins won't respawn the ost container anymore. So right now if I try to run a manual OST job it still seems to be using containers. Isn't [3] supposed to be changed?
[3]
https://github.com/oVirt/jenkins/blob/0a824788eda1e6d6755b451817e3964e1bf14b...
On Thu, Dec 3, 2020 at 1:28 PM Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com>> wrote:
On 12/1/20 8:18 PM, Marcin Sobczyk wrote: > Hi, > > the patch that removed all el7 OST runs has been merged. > > Regards, Marcin > > On 11/26/20 12:04 PM, Anton Marchukov wrote: >> Ehud will switch it back to baremetals on the next week. The same will >> have to be done for OST CI itself. After that we can rebuild
unused
>> openshift nodes back to CI baremetals. And also convert el7 baremetals >> to el8 ones (I guess we will leave few and watch the load on
them).
>> Please expect some capacity drop during this time. Anton, Ehud, any progress on this?
Thanks, Marcin
>> >> On Thu, Nov 26, 2020 at 12:00 PM Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com> >> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>>> wrote: >> >> Hi, >> >> since all the important suites are working on el8 already, we're >> planning to drop all el7 OST runs with [1] very soon. >> >> This means we can finally say goodbye to py2 and other legacy stuff! >> >> We still need to move manual OST runs not to use containers for >> that to >> happen. >> This effort is tracked here [2]. >> >> Regards, Marcin >> >> [1] https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378> <https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378>> >> [2] https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145> >> <https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145>> >> >> >> >> -- >> Anton Marchukov >> Associate Manager - RHV DevOps - Red Hat
On 12/3/20 1:02 PM, Ehud Yonasi wrote:
Correct. It is now fixed with the manual ost job. Tried running a manual job a moment ago, but ended up with docker-related error [4].
[4] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...
On Thu, Dec 3, 2020 at 1:54 PM Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com>> wrote:
On 12/3/20 12:33 PM, Ehud Yonasi wrote: > Hey, > > If there are no more OST jobs that will use el7, Evgheni can start > rebuilding those nodes to el8. There shouldn't be.
> Jenkins won't respawn the ost container anymore. So right now if I try to run a manual OST job it still seems to be using containers. Isn't [3] supposed to be changed?
[3] https://github.com/oVirt/jenkins/blob/0a824788eda1e6d6755b451817e3964e1bf14b... <https://github.com/oVirt/jenkins/blob/0a824788eda1e6d6755b451817e3964e1bf14bfc/jobs/confs/projects/ovirt/system-tests.yaml#L98>
> > On Thu, Dec 3, 2020 at 1:28 PM Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com> > <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>>> wrote: > > > > On 12/1/20 8:18 PM, Marcin Sobczyk wrote: > > Hi, > > > > the patch that removed all el7 OST runs has been merged. > > > > Regards, Marcin > > > > On 11/26/20 12:04 PM, Anton Marchukov wrote: > >> Ehud will switch it back to baremetals on the next week. The > same will > >> have to be done for OST CI itself. After that we can rebuild unused > >> openshift nodes back to CI baremetals. And also convert el7 > baremetals > >> to el8 ones (I guess we will leave few and watch the load on them). > >> Please expect some capacity drop during this time. > Anton, Ehud, any progress on this? > > Thanks, Marcin > > >> > >> On Thu, Nov 26, 2020 at 12:00 PM Marcin Sobczyk > <msobczyk@redhat.com <mailto:msobczyk@redhat.com> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>> > >> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>>>> wrote: > >> > >> Hi, > >> > >> since all the important suites are working on el8 already, > we're > >> planning to drop all el7 OST runs with [1] very soon. > >> > >> This means we can finally say goodbye to py2 and other > legacy stuff! > >> > >> We still need to move manual OST runs not to use > containers for > >> that to > >> happen. > >> This effort is tracked here [2]. > >> > >> Regards, Marcin > >> > >> [1] https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378> > <https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378>> <https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378> > <https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378>>> > >> [2] https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145> > <https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145>> > >> <https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145> > <https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145>>> > >> > >> > >> > >> -- > >> Anton Marchukov > >> Associate Manager - RHV DevOps - Red Hat >
On Thu, Dec 3, 2020 at 5:16 PM Marcin Sobczyk <msobczyk@redhat.com> wrote:
On 12/3/20 1:02 PM, Ehud Yonasi wrote:
Correct. It is now fixed with the manual ost job. Tried running a manual job a moment ago, but ended up with docker-related error [4].
Also, are the nightly runs [1] running in el8? Half of them still get stuck on 'Create local VM' and timeout, which usually used to happen on el7. Thanks, [1] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/
[4] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...
On Thu, Dec 3, 2020 at 1:54 PM Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com>> wrote:
On 12/3/20 12:33 PM, Ehud Yonasi wrote: > Hey, > > If there are no more OST jobs that will use el7, Evgheni can start > rebuilding those nodes to el8. There shouldn't be.
> Jenkins won't respawn the ost container anymore. So right now if I try to run a manual OST job it still seems to be using containers. Isn't [3] supposed to be changed?
[3] https://github.com/oVirt/jenkins/blob/0a824788eda1e6d6755b451817e3964e1bf14b... <https://github.com/oVirt/jenkins/blob/0a824788eda1e6d6755b451817e3964e1bf14bfc/jobs/confs/projects/ovirt/system-tests.yaml#L98>
> > On Thu, Dec 3, 2020 at 1:28 PM Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com> > <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>>> wrote: > > > > On 12/1/20 8:18 PM, Marcin Sobczyk wrote: > > Hi, > > > > the patch that removed all el7 OST runs has been merged. > > > > Regards, Marcin > > > > On 11/26/20 12:04 PM, Anton Marchukov wrote: > >> Ehud will switch it back to baremetals on the next week. The > same will > >> have to be done for OST CI itself. After that we can rebuild unused > >> openshift nodes back to CI baremetals. And also convert el7 > baremetals > >> to el8 ones (I guess we will leave few and watch the load on them). > >> Please expect some capacity drop during this time. > Anton, Ehud, any progress on this? > > Thanks, Marcin > > >> > >> On Thu, Nov 26, 2020 at 12:00 PM Marcin Sobczyk > <msobczyk@redhat.com <mailto:msobczyk@redhat.com> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>> > >> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>>>> wrote: > >> > >> Hi, > >> > >> since all the important suites are working on el8 already, > we're > >> planning to drop all el7 OST runs with [1] very soon. > >> > >> This means we can finally say goodbye to py2 and other > legacy stuff! > >> > >> We still need to move manual OST runs not to use > containers for > >> that to > >> happen. > >> This effort is tracked here [2]. > >> > >> Regards, Marcin > >> > >> [1] https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378> > <https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378>> <https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378> > <https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378>>> > >> [2] https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145> > <https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145>> > >> <https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145> > <https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145>>> > >> > >> > >> > >> -- > >> Anton Marchukov > >> Associate Manager - RHV DevOps - Red Hat >
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/4LSP4EO4V5SCF5...
-- Didi
On 12/7/20 8:19 AM, Yedidyah Bar David wrote:
On Thu, Dec 3, 2020 at 5:16 PM Marcin Sobczyk <msobczyk@redhat.com> wrote:
On 12/3/20 1:02 PM, Ehud Yonasi wrote:
Correct. It is now fixed with the manual ost job. Tried running a manual job a moment ago, but ended up with docker-related error [4].
Also, are the nightly runs [1] running in el8? Half of them still get stuck on 'Create local VM' and timeout, which usually used to happen on el7. AFAICS they're still using el7 containers:
02:55:24 Building remotely on openshift-integ-tests-container-wrrzs (el7 integ-tests-container) in workspace /home/jenkins/agent/workspace/ovirt-system-tests_he-basic-suite-master but I think fixing the manual runs is much more important right now. Regards, Marcin
Thanks,
[1] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/
[4] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...
On Thu, Dec 3, 2020 at 1:54 PM Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com>> wrote:
On 12/3/20 12:33 PM, Ehud Yonasi wrote: > Hey, > > If there are no more OST jobs that will use el7, Evgheni can start > rebuilding those nodes to el8. There shouldn't be.
> Jenkins won't respawn the ost container anymore. So right now if I try to run a manual OST job it still seems to be using containers. Isn't [3] supposed to be changed?
[3] https://github.com/oVirt/jenkins/blob/0a824788eda1e6d6755b451817e3964e1bf14b... <https://github.com/oVirt/jenkins/blob/0a824788eda1e6d6755b451817e3964e1bf14bfc/jobs/confs/projects/ovirt/system-tests.yaml#L98>
> > On Thu, Dec 3, 2020 at 1:28 PM Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com> > <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>>> wrote: > > > > On 12/1/20 8:18 PM, Marcin Sobczyk wrote: > > Hi, > > > > the patch that removed all el7 OST runs has been merged. > > > > Regards, Marcin > > > > On 11/26/20 12:04 PM, Anton Marchukov wrote: > >> Ehud will switch it back to baremetals on the next week. The > same will > >> have to be done for OST CI itself. After that we can rebuild unused > >> openshift nodes back to CI baremetals. And also convert el7 > baremetals > >> to el8 ones (I guess we will leave few and watch the load on them). > >> Please expect some capacity drop during this time. > Anton, Ehud, any progress on this? > > Thanks, Marcin > > >> > >> On Thu, Nov 26, 2020 at 12:00 PM Marcin Sobczyk > <msobczyk@redhat.com <mailto:msobczyk@redhat.com> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>> > >> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>>>> wrote: > >> > >> Hi, > >> > >> since all the important suites are working on el8 already, > we're > >> planning to drop all el7 OST runs with [1] very soon. > >> > >> This means we can finally say goodbye to py2 and other > legacy stuff! > >> > >> We still need to move manual OST runs not to use > containers for > >> that to > >> happen. > >> This effort is tracked here [2]. > >> > >> Regards, Marcin > >> > >> [1] https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378> > <https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378>> <https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378> > <https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378>>> > >> [2] https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145> > <https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145>> > >> <https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145> > <https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145>>> > >> > >> > >> > >> -- > >> Anton Marchukov > >> Associate Manager - RHV DevOps - Red Hat >
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/4LSP4EO4V5SCF5...
On Mon, Dec 7, 2020 at 11:48 AM Marcin Sobczyk <msobczyk@redhat.com> wrote:
On 12/7/20 8:19 AM, Yedidyah Bar David wrote:
On Thu, Dec 3, 2020 at 5:16 PM Marcin Sobczyk <msobczyk@redhat.com> wrote:
On 12/3/20 1:02 PM, Ehud Yonasi wrote:
Correct. It is now fixed with the manual ost job. Tried running a manual job a moment ago, but ended up with docker-related error [4].
Also, are the nightly runs [1] running in el8? Half of them still get stuck on 'Create local VM' and timeout, which usually used to happen on el7. AFAICS they're still using el7 containers:
02:55:24 Building remotely on openshift-integ-tests-container-wrrzs (el7 integ-tests-container) in workspace /home/jenkins/agent/workspace/ovirt-system-tests_he-basic-suite-master
but I think fixing the manual runs is much more important right now.
IMO both are important! ;-) We had he-basic-suite broken for more than 50% of the time, last year. If we let it remain broken, we risk not noticing real bugs/regressions. I also guess it's a simple fix, no? I think something like: https://gerrit.ovirt.org/c/jenkins/+/112527 (I'd change all el7 to el8 there, except for 4.3 suites, but perhaps better split and do this later).
Regards, Marcin
Thanks,
[1] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/
[4] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...
On Thu, Dec 3, 2020 at 1:54 PM Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com>> wrote:
On 12/3/20 12:33 PM, Ehud Yonasi wrote: > Hey, > > If there are no more OST jobs that will use el7, Evgheni can start > rebuilding those nodes to el8. There shouldn't be.
> Jenkins won't respawn the ost container anymore. So right now if I try to run a manual OST job it still seems to be using containers. Isn't [3] supposed to be changed?
[3] https://github.com/oVirt/jenkins/blob/0a824788eda1e6d6755b451817e3964e1bf14b... <https://github.com/oVirt/jenkins/blob/0a824788eda1e6d6755b451817e3964e1bf14bfc/jobs/confs/projects/ovirt/system-tests.yaml#L98>
> > On Thu, Dec 3, 2020 at 1:28 PM Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com> > <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>>> wrote: > > > > On 12/1/20 8:18 PM, Marcin Sobczyk wrote: > > Hi, > > > > the patch that removed all el7 OST runs has been merged. > > > > Regards, Marcin > > > > On 11/26/20 12:04 PM, Anton Marchukov wrote: > >> Ehud will switch it back to baremetals on the next week. The > same will > >> have to be done for OST CI itself. After that we can rebuild unused > >> openshift nodes back to CI baremetals. And also convert el7 > baremetals > >> to el8 ones (I guess we will leave few and watch the load on them). > >> Please expect some capacity drop during this time. > Anton, Ehud, any progress on this? > > Thanks, Marcin > > >> > >> On Thu, Nov 26, 2020 at 12:00 PM Marcin Sobczyk > <msobczyk@redhat.com <mailto:msobczyk@redhat.com> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>> > >> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>>>> wrote: > >> > >> Hi, > >> > >> since all the important suites are working on el8 already, > we're > >> planning to drop all el7 OST runs with [1] very soon. > >> > >> This means we can finally say goodbye to py2 and other > legacy stuff! > >> > >> We still need to move manual OST runs not to use > containers for > >> that to > >> happen. > >> This effort is tracked here [2]. > >> > >> Regards, Marcin > >> > >> [1] https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378> > <https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378>> <https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378> > <https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378>>> > >> [2] https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145> > <https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145>> > >> <https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145> > <https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145>>> > >> > >> > >> > >> -- > >> Anton Marchukov > >> Associate Manager - RHV DevOps - Red Hat >
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/4LSP4EO4V5SCF5...
-- Didi
On 12/7/20 11:00 AM, Yedidyah Bar David wrote:
On Mon, Dec 7, 2020 at 11:48 AM Marcin Sobczyk <msobczyk@redhat.com> wrote:
On 12/7/20 8:19 AM, Yedidyah Bar David wrote:
On Thu, Dec 3, 2020 at 5:16 PM Marcin Sobczyk <msobczyk@redhat.com> wrote:
On 12/3/20 1:02 PM, Ehud Yonasi wrote:
Correct. It is now fixed with the manual ost job. Tried running a manual job a moment ago, but ended up with docker-related error [4]. Also, are the nightly runs [1] running in el8? Half of them still get stuck on 'Create local VM' and timeout, which usually used to happen on el7. AFAICS they're still using el7 containers:
02:55:24 Building remotely on openshift-integ-tests-container-wrrzs (el7 integ-tests-container) in workspace /home/jenkins/agent/workspace/ovirt-system-tests_he-basic-suite-master
but I think fixing the manual runs is much more important right now.
IMO both are important! ;-)
We had he-basic-suite broken for more than 50% of the time, last year. If we let it remain broken, we risk not noticing real bugs/regressions. Of course, I just meant we're actively working and running he suite regularly right now, so manual runs are a higher priority ATM.
I also guess it's a simple fix, no? I think something like:
https://gerrit.ovirt.org/c/jenkins/+/112527
(I'd change all el7 to el8 there, except for 4.3 suites, but perhaps better split and do this later).
Probably not - el8 chroot running on el7 kernel won't work, so we'd need to do node filtering, but I'll leave that to Ehud :)
Regards, Marcin
Thanks,
[1] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/
[4] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...
On Thu, Dec 3, 2020 at 1:54 PM Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com>> wrote:
On 12/3/20 12:33 PM, Ehud Yonasi wrote: > Hey, > > If there are no more OST jobs that will use el7, Evgheni can start > rebuilding those nodes to el8. There shouldn't be.
> Jenkins won't respawn the ost container anymore. So right now if I try to run a manual OST job it still seems to be using containers. Isn't [3] supposed to be changed?
[3] https://github.com/oVirt/jenkins/blob/0a824788eda1e6d6755b451817e3964e1bf14b... <https://github.com/oVirt/jenkins/blob/0a824788eda1e6d6755b451817e3964e1bf14bfc/jobs/confs/projects/ovirt/system-tests.yaml#L98>
> > On Thu, Dec 3, 2020 at 1:28 PM Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com> > <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>>> wrote: > > > > On 12/1/20 8:18 PM, Marcin Sobczyk wrote: > > Hi, > > > > the patch that removed all el7 OST runs has been merged. > > > > Regards, Marcin > > > > On 11/26/20 12:04 PM, Anton Marchukov wrote: > >> Ehud will switch it back to baremetals on the next week. The > same will > >> have to be done for OST CI itself. After that we can rebuild unused > >> openshift nodes back to CI baremetals. And also convert el7 > baremetals > >> to el8 ones (I guess we will leave few and watch the load on them). > >> Please expect some capacity drop during this time. > Anton, Ehud, any progress on this? > > Thanks, Marcin > > >> > >> On Thu, Nov 26, 2020 at 12:00 PM Marcin Sobczyk > <msobczyk@redhat.com <mailto:msobczyk@redhat.com> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>> > >> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>>>> wrote: > >> > >> Hi, > >> > >> since all the important suites are working on el8 already, > we're > >> planning to drop all el7 OST runs with [1] very soon. > >> > >> This means we can finally say goodbye to py2 and other > legacy stuff! > >> > >> We still need to move manual OST runs not to use > containers for > >> that to > >> happen. > >> This effort is tracked here [2]. > >> > >> Regards, Marcin > >> > >> [1] https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378> > <https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378>> <https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378> > <https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378>>> > >> [2] https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145> > <https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145>> > >> <https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145> > <https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145>>> > >> > >> > >> > >> -- > >> Anton Marchukov > >> Associate Manager - RHV DevOps - Red Hat >
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/4LSP4EO4V5SCF5...
It isn't only changing the nod filter. Also, the extra sources need some changes for el8. On Mon, Dec 7, 2020 at 12:15 PM Marcin Sobczyk <msobczyk@redhat.com> wrote:
On 12/7/20 11:00 AM, Yedidyah Bar David wrote:
On Mon, Dec 7, 2020 at 11:48 AM Marcin Sobczyk <msobczyk@redhat.com> wrote:
On 12/7/20 8:19 AM, Yedidyah Bar David wrote:
On Thu, Dec 3, 2020 at 5:16 PM Marcin Sobczyk <msobczyk@redhat.com>
wrote:
On 12/3/20 1:02 PM, Ehud Yonasi wrote:
Correct. It is now fixed with the manual ost job. Tried running a manual job a moment ago, but ended up with docker-related error [4]. Also, are the nightly runs [1] running in el8? Half of them still get stuck on 'Create local VM' and timeout, which usually used to happen on el7. AFAICS they're still using el7 containers:
02:55:24 Building remotely on openshift-integ-tests-container-wrrzs (el7 integ-tests-container) in workspace /home/jenkins/agent/workspace/ovirt-system-tests_he-basic-suite-master
but I think fixing the manual runs is much more important right now. IMO both are important! ;-)
We had he-basic-suite broken for more than 50% of the time, last year. If we let it remain broken, we risk not noticing real bugs/regressions. Of course, I just meant we're actively working and running he suite regularly right now, so manual runs are a higher priority ATM.
I also guess it's a simple fix, no? I think something like:
https://gerrit.ovirt.org/c/jenkins/+/112527
(I'd change all el7 to el8 there, except for 4.3 suites, but perhaps
better
split and do this later). Probably not - el8 chroot running on el7 kernel won't work, so we'd need to do node filtering, but I'll leave that to Ehud :)
Regards, Marcin
Thanks,
[1]
https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/
[4]
https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...
On Thu, Dec 3, 2020 at 1:54 PM Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com>> wrote:
On 12/3/20 12:33 PM, Ehud Yonasi wrote: > Hey, > > If there are no more OST jobs that will use el7, Evgheni can
start
> rebuilding those nodes to el8. There shouldn't be.
> Jenkins won't respawn the ost container anymore. So right now if I try to run a manual OST job it still seems
to be
using containers. Isn't [3] supposed to be changed?
[3]
https://github.com/oVirt/jenkins/blob/0a824788eda1e6d6755b451817e3964e1bf14b...
<
https://github.com/oVirt/jenkins/blob/0a824788eda1e6d6755b451817e3964e1bf14b...
> > On Thu, Dec 3, 2020 at 1:28 PM Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com> > <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>>>
wrote:
> > > > On 12/1/20 8:18 PM, Marcin Sobczyk wrote: > > Hi, > > > > the patch that removed all el7 OST runs has been
merged.
> > > > Regards, Marcin > > > > On 11/26/20 12:04 PM, Anton Marchukov wrote: > >> Ehud will switch it back to baremetals on the next
week. The
> same will > >> have to be done for OST CI itself. After that we can rebuild unused > >> openshift nodes back to CI baremetals. And also
convert el7
> baremetals > >> to el8 ones (I guess we will leave few and watch the
load
on them). > >> Please expect some capacity drop during this time. > Anton, Ehud, any progress on this? > > Thanks, Marcin > > >> > >> On Thu, Nov 26, 2020 at 12:00 PM Marcin Sobczyk > <msobczyk@redhat.com <mailto:msobczyk@redhat.com> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>> > >> <mailto:msobczyk@redhat.com <mailto:
msobczyk@redhat.com>
<mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>>>>
wrote:
> >> > >> Hi, > >> > >> since all the important suites are working on el8 already, > we're > >> planning to drop all el7 OST runs with [1] very
soon.
> >> > >> This means we can finally say goodbye to py2 and
other
> legacy stuff! > >> > >> We still need to move manual OST runs not to use > containers for > >> that to > >> happen. > >> This effort is tracked here [2]. > >> > >> Regards, Marcin > >> > >> [1] https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378> > <https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378>> <https://gerrit.ovirt.org/112378 <
https://gerrit.ovirt.org/112378>
> <https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378>>> > >> [2]
https://ovirt-jira.atlassian.net/browse/OST-145
<https://ovirt-jira.atlassian.net/browse/OST-145> > <https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145>> > >> <https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145> > <https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145>>> > >> > >> > >> > >> -- > >> Anton Marchukov > >> Associate Manager - RHV DevOps - Red Hat >
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/4LSP4EO4V5SCF5...
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/YAMBGKYODB5QVE...
-- Galit Rosenthal SOFTWARE ENGINEER Red Hat <https://www.redhat.com/> galit@redhat.com T: 972-9-7692230 <https://www.redhat.com/>
On 12/3/20 4:15 PM, Marcin Sobczyk wrote:
On 12/3/20 1:02 PM, Ehud Yonasi wrote:
Correct. It is now fixed with the manual ost job. Tried running a manual job a moment ago, but ended up with docker-related error [4].
[4] https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests... Hi Ehud, any updates on this? Manual OST runs are still broken.
Regards, Marcin
On Thu, Dec 3, 2020 at 1:54 PM Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com>> wrote:
On 12/3/20 12:33 PM, Ehud Yonasi wrote: > Hey, > > If there are no more OST jobs that will use el7, Evgheni can start > rebuilding those nodes to el8. There shouldn't be.
> Jenkins won't respawn the ost container anymore. So right now if I try to run a manual OST job it still seems to be using containers. Isn't [3] supposed to be changed?
[3] https://github.com/oVirt/jenkins/blob/0a824788eda1e6d6755b451817e3964e1bf14b... <https://github.com/oVirt/jenkins/blob/0a824788eda1e6d6755b451817e3964e1bf14bfc/jobs/confs/projects/ovirt/system-tests.yaml#L98>
> > On Thu, Dec 3, 2020 at 1:28 PM Marcin Sobczyk <msobczyk@redhat.com <mailto:msobczyk@redhat.com> > <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>>> wrote: > > > > On 12/1/20 8:18 PM, Marcin Sobczyk wrote: > > Hi, > > > > the patch that removed all el7 OST runs has been merged. > > > > Regards, Marcin > > > > On 11/26/20 12:04 PM, Anton Marchukov wrote: > >> Ehud will switch it back to baremetals on the next week. The > same will > >> have to be done for OST CI itself. After that we can rebuild unused > >> openshift nodes back to CI baremetals. And also convert el7 > baremetals > >> to el8 ones (I guess we will leave few and watch the load on them). > >> Please expect some capacity drop during this time. > Anton, Ehud, any progress on this? > > Thanks, Marcin > > >> > >> On Thu, Nov 26, 2020 at 12:00 PM Marcin Sobczyk > <msobczyk@redhat.com <mailto:msobczyk@redhat.com> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>> > >> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com> <mailto:msobczyk@redhat.com <mailto:msobczyk@redhat.com>>>> wrote: > >> > >> Hi, > >> > >> since all the important suites are working on el8 already, > we're > >> planning to drop all el7 OST runs with [1] very soon. > >> > >> This means we can finally say goodbye to py2 and other > legacy stuff! > >> > >> We still need to move manual OST runs not to use > containers for > >> that to > >> happen. > >> This effort is tracked here [2]. > >> > >> Regards, Marcin > >> > >> [1] https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378> > <https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378>> <https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378> > <https://gerrit.ovirt.org/112378 <https://gerrit.ovirt.org/112378>>> > >> [2] https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145> > <https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145>> > >> <https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145> > <https://ovirt-jira.atlassian.net/browse/OST-145 <https://ovirt-jira.atlassian.net/browse/OST-145>>> > >> > >> > >> > >> -- > >> Anton Marchukov > >> Associate Manager - RHV DevOps - Red Hat >
participants (5)
-
Anton Marchukov -
Ehud Yonasi -
Galit Rosenthal -
Marcin Sobczyk -
Yedidyah Bar David