[ovirt-devel] Re: CI failing all the time

On Sun, Nov 28, 2021 at 10:59 AM Nir Soffer <nsoffer(a)redhat.com&gt; wrote: > > On Fri, Nov 26, 2021 at 12:57 PM Ehud Yonasi <eyonasi(a)redhat.com&gt; wrote: > > > > Hi Nir, > > > > There was a problem in jenkins to switch to the new k8s stack on IBM cloud, a restart was required and now it’s solved. > > The queue is increasing again, currently 12 jobs in the queue: Vdsm jobs requires el8 nodes, but we have only one: https://jenkins.ovirt.org/label/el8/ we also have only one el9stream node: https://jenkins.ovirt.org/label/el9stream/ I posted this to use all nodes in vdsm, instead of only the el8 nodes: https://gerrit.ovirt.org/c/vdsm/+/117826 It did not work in the old data center, but maybe the issue with the el9stream nodes is solved now.

> > Build Queue (12) > system-mk_mirrors_index-yum > part of vdsm_standard-check-patch #31073 vdsm [check-patch] > jenkins_master_check-patch-el7-ppc64le > jenkins_master_check-patch-el7-ppc64le > part of jenkins_standard-check-patch #5550 > part of jenkins_standard-check-patch #5548 > jenkins_master_check-patch-el7-ppc64le > jenkins_master_check-patch-el7-ppc64le > jenkins_master_check-patch-el7-ppc64le > jenkins_master_check-patch-el7-ppc64le > part of ovirt-appliance_standard-check-patch #385 ovirt-appliance > [check-patch] > lago-demo_master_github_timed-el7-x86_64 > > > > > > > > On 25 Nov 2021, at 21:53, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: > > > > On Thu, Nov 25, 2021 at 9:15 PM Michal Skrivanek > > <michal.skrivanek(a)redhat.com&gt; wrote: > > > > > > > > > > On 25. 11. 2021, at 16:45, Evgheni Dereveanchin <ederevea(a)redhat.com&gt; wrote: > > > > Hi Milan, > > > > Indeed, this is related to the Jenkins move: all pre-existing nodes are no longer usable so new ones have been created. > > We're looking into the reasons new nodes fail to initialize and bringing up new ones in parallel to help with processing the build queue. > > > > Sorry for the inconvenience. Please report any other issues if you see them as there may be quite some instability due to the rebuild. > > > > > > Hi Evgheni, > > the queue is now 53 builds long, and it seems nothing is building, perhaps there's no worker or the labelling is wrong? > > > > > > Looks like jobs are stuck for several hours > > https://jenkins.ovirt.org/job/vdsm_standard-check-patch/31036/ > > > > at the "loading code" stage: > > https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-... > > > > With this error: > > > > There are no nodes with the label ‘jenkins-vdsm_standard-check-patch-31036’ > > > > The label looks wrong. vdsm requires nodes with "el8" label. > > > > > > Thanks, > > michal > > > > > > Regards. > > Evgheni > > > > On Thu, Nov 25, 2021 at 4:24 PM Nir Soffer <nsoffer(a)redhat.com&gt; wrote: > > > > > > On Thu, Nov 25, 2021 at 3:48 PM Milan Zamazal <mzamazal(a)redhat.com&gt; wrote: > > > > > > Hi, > > > > all patches uploaded today I've seen (for Engine and Vdsm) fail due to > > problems with availability of packages when preparing the el7 > > environment. For example: > > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/15291/ > > > > Additionally, Vdsm tests are not run on PSI. > > > > Does anybody know what's wrong and how to fix it? Can it be related to > > the Jenkins move? > > > > > > Looking at > > https://jenkins.ovirt.org/ > > > > There are 33 jobs in the queue, and only 3 jobs running. > > > > Maybe we did not restore all the nodes? > > > > > > _______________________________________________ > > Devel mailing list -- devel(a)ovirt.org > > To unsubscribe send an email to devel-leave(a)ovirt.org > > Privacy Statement: https://www.ovirt.org/privacy-policy.html > > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > > List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/ZLH3ILKHC3Y... > > > > > > _______________________________________________ > > Devel mailing list -- devel(a)ovirt.org > > To unsubscribe send an email to devel-leave(a)ovirt.org > > Privacy Statement: https://www.ovirt.org/privacy-policy.html > > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > > List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/RZ7H4L26DU6... > > > > _______________________________________________ > > Devel mailing list -- devel(a)ovirt.org > > To unsubscribe send an email to devel-leave(a)ovirt.org > > Privacy Statement: https://www.ovirt.org/privacy-policy.html > > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > > List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/UK4KVTPITCA... > > > >