Re: restricting check patch/merged parallel jobs

From what I saw, it was mostly ovirt-engine and vdsm jobs pending on the queue while other slaves are idle. we have over 40 slaves and we're about to add more, so I don't think that will be an issue and IMO 3 per job is not enough, especially if you get idle slaves.

We are thinking on a more dynamic approach of dynamic vm allocation on demand, so in the long run we'll have more control over it, for now i'm monitoring the queue size and slaves on a regular basis [1], so if anything will get blocked too much time we'll act and adjust accordingly. [1] http://graphite.phx.ovirt.org/dashboard/db/jenkins-monitoring On Wed, May 11, 2016 at 1:10 PM, Sandro Bonazzola <sbonazzo(a)redhat.com&gt; wrote: > > > On Tue, May 10, 2016 at 1:01 PM, Eyal Edri <eedri(a)redhat.com&gt; wrote: > >> Shlomi, >> Can you submit a patch to increase the limit to 6 for (i think all jobs >> are using the same yaml template) and we'll continue to monitor to queue >> and see if there is an improvement in the utilization of slaves? >> > > Issue was that long lasting jobs caused queue to increase too much. > Example: a patch set rebased on master and merged will cause triggering > of check-merged jobs, upgrade jobs, ...; running 6 instance of each of them > will cause all other projects to be queued for a lot of time. > > > >> >> E. >> >> On Tue, May 10, 2016 at 1:58 PM, David Caro <dcaro(a)redhat.com&gt; wrote: >> >>> On 05/10 13:54, Eyal Edri wrote: >>> > Is there any reason we're limiting the amount of check patch & check >>> merged >>> > jobs to run only 3 in parallel? >>> > >>> >>> We had some mess in the past where enabling parallel runs did not >>> really force >>> not using the same slave at the same time, I guess we never reenabled >>> them. >>> >>> > Each jobs runs in mock and on its own VM, anything presenting us from >>> > removing this limitation so we won't have idle slaves while other >>> jobs are >>> > in the queue? >>> > >>> > We can increase it at least to a higher level if we won't one >>> specific job >>> > to take over all slaves and starve other jobs, but i think >>> ovirt-engine >>> > jobs are probably the biggest consumer of ci, so the threshold should >>> be >>> > updated. >>> >>> +1 >>> >>> > >>> > -- >>> > Eyal Edri >>> > Associate Manager >>> > RHEV DevOps >>> > EMEA ENG Virtualization R&D >>> > Red Hat Israel >>> > >>> > phone: +972-9-7692018 >>> > irc: eedri (on #tlv #rhev-dev #rhev-integ) >>> >>> > _______________________________________________ >>> > Infra mailing list >>> > Infra(a)ovirt.org >>> > http://lists.ovirt.org/mailman/listinfo/infra >>> >>> >>> -- >>> David Caro >>> >>> Red Hat S.L. >>> Continuous Integration Engineer - EMEA ENG Virtualization R&D >>> >>> Tel.: +420 532 294 605 >>> Email: dcaro(a)redhat.com >>> IRC: dcaro|dcaroest@{freenode|oftc|redhat} >>> Web: www.redhat.com >>> RHT Global #: 82-62605 >>> >> >> >> >> -- >> Eyal Edri >> Associate Manager >> RHEV DevOps >> EMEA ENG Virtualization R&D >> Red Hat Israel >> >> phone: +972-9-7692018 >> irc: eedri (on #tlv #rhev-dev #rhev-integ) >> >> _______________________________________________ >> Infra mailing list >> Infra(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/infra >> >> > > > -- > Sandro Bonazzola > Better technology. Faster innovation. Powered by community collaboration. > See how it works at redhat.com > -- Eyal Edri Associate Manager RHEV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)