On 25. 11. 2021, at 16:45, Evgheni Dereveanchin <ederevea@redhat.com> wrote:

Hi Milan,

Indeed, this is related to the Jenkins move: all pre-existing nodes are no longer usable so new ones have been created. We're looking into the reasons new nodes fail to initialize and bringing up new ones in parallel to help with processing the build queue.

Sorry for the inconvenience. Please report any other issues if you see them as there may be quite some instability due to the rebuild.

Hi Evgheni, the queue is now 53 builds long, and it seems nothing is building, perhaps there's no worker or the labelling is wrong? Thanks, michal

Regards. Evgheni

On Thu, Nov 25, 2021 at 4:24 PM Nir Soffer <nsoffer@redhat.com <mailto:nsoffer@redhat.com>> wrote: On Thu, Nov 25, 2021 at 3:48 PM Milan Zamazal <mzamazal@redhat.com <mailto:mzamazal@redhat.com>> wrote:

...

Hi,

all patches uploaded today I've seen (for Engine and Vdsm) fail due to problems with availability of packages when preparing the el7 environment. For example: https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/15291/ <https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/15291/>

Additionally, Vdsm tests are not run on PSI.

Does anybody know what's wrong and how to fix it? Can it be related to the Jenkins move?

Looking at https://jenkins.ovirt.org/ <https://jenkins.ovirt.org/>

There are 33 jobs in the queue, and only 3 jobs running.

Maybe we did not restore all the nodes?

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/ZLH3ILKHC3YPZW...

Hi Nir, There was a problem in jenkins to switch to the new k8s stack on IBM cloud, a restart was required and now it’s solved.

On 25 Nov 2021, at 21:53, Nir Soffer <nsoffer@redhat.com> wrote:

On Thu, Nov 25, 2021 at 9:15 PM Michal Skrivanek <michal.skrivanek@redhat.com <mailto:michal.skrivanek@redhat.com>> wrote:

...

On 25. 11. 2021, at 16:45, Evgheni Dereveanchin <ederevea@redhat.com> wrote:

Hi Milan,

Indeed, this is related to the Jenkins move: all pre-existing nodes are no longer usable so new ones have been created. We're looking into the reasons new nodes fail to initialize and bringing up new ones in parallel to help with processing the build queue.

Sorry for the inconvenience. Please report any other issues if you see them as there may be quite some instability due to the rebuild.

Hi Evgheni, the queue is now 53 builds long, and it seems nothing is building, perhaps there's no worker or the labelling is wrong?

Looks like jobs are stuck for several hours https://jenkins.ovirt.org/job/vdsm_standard-check-patch/31036/ <https://jenkins.ovirt.org/job/vdsm_standard-check-patch/31036/>

at the "loading code" stage: https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pat... <https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/31036/pipeline>

With this error:

There are no nodes with the label ‘jenkins-vdsm_standard-check-patch-31036’

The label looks wrong. vdsm requires nodes with "el8" label.

...

Thanks, michal

Regards. Evgheni

On Thu, Nov 25, 2021 at 4:24 PM Nir Soffer <nsoffer@redhat.com> wrote:

...

On Thu, Nov 25, 2021 at 3:48 PM Milan Zamazal <mzamazal@redhat.com> wrote:

...

Hi,

all patches uploaded today I've seen (for Engine and Vdsm) fail due to problems with availability of packages when preparing the el7 environment. For example: https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/15291/

Additionally, Vdsm tests are not run on PSI.

Does anybody know what's wrong and how to fix it? Can it be related to the Jenkins move?

Looking at https://jenkins.ovirt.org/

There are 33 jobs in the queue, and only 3 jobs running.

Maybe we did not restore all the nodes?

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/ZLH3ILKHC3YPZW...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/RZ7H4L26DU6LAK...

_______________________________________________ Devel mailing list -- devel@ovirt.org <mailto:devel@ovirt.org> To unsubscribe send an email to devel-leave@ovirt.org <mailto:devel-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html <https://www.ovirt.org/privacy-policy.html> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/UK4KVTPITCAWHW... <https://lists.ovirt.org/archives/list/devel@ovirt.org/message/UK4KVTPITCAWHWUR5QPGN3RSNFYQSH34/>

On Fri, Nov 26, 2021 at 12:57 PM Ehud Yonasi <eyonasi@redhat.com> wrote:

Hi Nir,

There was a problem in jenkins to switch to the new k8s stack on IBM cloud, a restart was required and now it’s solved.

The queue is increasing again, currently 12 jobs in the queue: Build Queue (12) system-mk_mirrors_index-yum part of vdsm_standard-check-patch #31073 vdsm [check-patch] jenkins_master_check-patch-el7-ppc64le jenkins_master_check-patch-el7-ppc64le part of jenkins_standard-check-patch #5550 part of jenkins_standard-check-patch #5548 jenkins_master_check-patch-el7-ppc64le jenkins_master_check-patch-el7-ppc64le jenkins_master_check-patch-el7-ppc64le jenkins_master_check-patch-el7-ppc64le part of ovirt-appliance_standard-check-patch #385 ovirt-appliance [check-patch] lago-demo_master_github_timed-el7-x86_64

On 25 Nov 2021, at 21:53, Nir Soffer <nsoffer@redhat.com> wrote:

On Thu, Nov 25, 2021 at 9:15 PM Michal Skrivanek <michal.skrivanek@redhat.com> wrote:

On 25. 11. 2021, at 16:45, Evgheni Dereveanchin <ederevea@redhat.com> wrote:

Hi Milan,

Indeed, this is related to the Jenkins move: all pre-existing nodes are no longer usable so new ones have been created. We're looking into the reasons new nodes fail to initialize and bringing up new ones in parallel to help with processing the build queue.

Sorry for the inconvenience. Please report any other issues if you see them as there may be quite some instability due to the rebuild.

Hi Evgheni, the queue is now 53 builds long, and it seems nothing is building, perhaps there's no worker or the labelling is wrong?

Looks like jobs are stuck for several hours https://jenkins.ovirt.org/job/vdsm_standard-check-patch/31036/

at the "loading code" stage: https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pat...

With this error:

There are no nodes with the label ‘jenkins-vdsm_standard-check-patch-31036’

The label looks wrong. vdsm requires nodes with "el8" label.

Thanks, michal

Regards. Evgheni

On Thu, Nov 25, 2021 at 4:24 PM Nir Soffer <nsoffer@redhat.com> wrote:

On Thu, Nov 25, 2021 at 3:48 PM Milan Zamazal <mzamazal@redhat.com> wrote:

Hi,

all patches uploaded today I've seen (for Engine and Vdsm) fail due to problems with availability of packages when preparing the el7 environment. For example: https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/15291/

Additionally, Vdsm tests are not run on PSI.

Does anybody know what's wrong and how to fix it? Can it be related to the Jenkins move?

Looking at https://jenkins.ovirt.org/

There are 33 jobs in the queue, and only 3 jobs running.

Maybe we did not restore all the nodes?

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/ZLH3ILKHC3YPZW...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/RZ7H4L26DU6LAK...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/UK4KVTPITCAWHW...

On Sun, Nov 28, 2021 at 11:35 AM Nir Soffer <nsoffer@redhat.com> wrote:

On Sun, Nov 28, 2021 at 10:59 AM Nir Soffer <nsoffer@redhat.com> wrote:

...

On Fri, Nov 26, 2021 at 12:57 PM Ehud Yonasi <eyonasi@redhat.com> wrote:

...

Hi Nir,

There was a problem in jenkins to switch to the new k8s stack on IBM cloud, a restart was required and now it’s solved.

The queue is increasing again, currently 12 jobs in the queue:

Vdsm jobs requires el8 nodes, but we have only one: https://jenkins.ovirt.org/label/el8/

we also have only one el9stream node: https://jenkins.ovirt.org/label/el9stream/

I posted this to use all nodes in vdsm, instead of only the el8 nodes: https://gerrit.ovirt.org/c/vdsm/+/117826

It did not work in the old data center, but maybe the issue with the el9stream nodes is solved now.

It does not work yet, el9stream fails quickly in global_setup.sh: [2021-11-28T09:35:20.829Z] + sudo -n systemctl start libvirtd haveged firewalld [2021-11-28T09:35:21.086Z] Failed to start libvirtd.service: Unit libvirtd.service not found. [2021-11-28T09:35:21.086Z] Failed to start haveged.service: Unit haveged.service not found. See https://ovirt-jira.atlassian.net/browse/OVIRT-3126

...

Build Queue (12) system-mk_mirrors_index-yum part of vdsm_standard-check-patch #31073 vdsm [check-patch] jenkins_master_check-patch-el7-ppc64le jenkins_master_check-patch-el7-ppc64le part of jenkins_standard-check-patch #5550 part of jenkins_standard-check-patch #5548 jenkins_master_check-patch-el7-ppc64le jenkins_master_check-patch-el7-ppc64le jenkins_master_check-patch-el7-ppc64le jenkins_master_check-patch-el7-ppc64le part of ovirt-appliance_standard-check-patch #385 ovirt-appliance [check-patch] lago-demo_master_github_timed-el7-x86_64

...

On 25 Nov 2021, at 21:53, Nir Soffer <nsoffer@redhat.com> wrote:

On Thu, Nov 25, 2021 at 9:15 PM Michal Skrivanek <michal.skrivanek@redhat.com> wrote:

On 25. 11. 2021, at 16:45, Evgheni Dereveanchin <ederevea@redhat.com> wrote:

Hi Milan,

Indeed, this is related to the Jenkins move: all pre-existing nodes are no longer usable so new ones have been created. We're looking into the reasons new nodes fail to initialize and bringing up new ones in parallel to help with processing the build queue.

Sorry for the inconvenience. Please report any other issues if you see them as there may be quite some instability due to the rebuild.

Hi Evgheni, the queue is now 53 builds long, and it seems nothing is building, perhaps there's no worker or the labelling is wrong?

Looks like jobs are stuck for several hours https://jenkins.ovirt.org/job/vdsm_standard-check-patch/31036/

at the "loading code" stage: https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pat...

With this error:

There are no nodes with the label ‘jenkins-vdsm_standard-check-patch-31036’

The label looks wrong. vdsm requires nodes with "el8" label.

Thanks, michal

Regards. Evgheni

On Thu, Nov 25, 2021 at 4:24 PM Nir Soffer <nsoffer@redhat.com> wrote:

On Thu, Nov 25, 2021 at 3:48 PM Milan Zamazal <mzamazal@redhat.com> wrote:

Hi,

all patches uploaded today I've seen (for Engine and Vdsm) fail due to problems with availability of packages when preparing the el7 environment. For example: https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/15291/

Additionally, Vdsm tests are not run on PSI.

Does anybody know what's wrong and how to fix it? Can it be related to the Jenkins move?

Looking at https://jenkins.ovirt.org/

There are 33 jobs in the queue, and only 3 jobs running.

Maybe we did not restore all the nodes?

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/ZLH3ILKHC3YPZW...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/RZ7H4L26DU6LAK...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/UK4KVTPITCAWHW...

On Sun, Nov 28, 2021 at 10:59 AM Nir Soffer <nsoffer@redhat.com> wrote:

On Fri, Nov 26, 2021 at 12:57 PM Ehud Yonasi <eyonasi@redhat.com> wrote:

...

Hi Nir,

There was a problem in jenkins to switch to the new k8s stack on IBM cloud, a restart was required and now it’s solved.

The queue is increasing again, currently 12 jobs in the queue:

Build Queue (12) system-mk_mirrors_index-yum part of vdsm_standard-check-patch #31073 vdsm [check-patch] jenkins_master_check-patch-el7-ppc64le jenkins_master_check-patch-el7-ppc64le part of jenkins_standard-check-patch #5550 part of jenkins_standard-check-patch #5548 jenkins_master_check-patch-el7-ppc64le jenkins_master_check-patch-el7-ppc64le jenkins_master_check-patch-el7-ppc64le jenkins_master_check-patch-el7-ppc64le part of ovirt-appliance_standard-check-patch #385 ovirt-appliance [check-patch] lago-demo_master_github_timed-el7-x86_64

Queue size is 24 now Build Queue (24) ovirt-engine_standard-on-merge ovirt-engine_standard-on-merge standard-enqueue standard-enqueue ovirt-engine_standard-on-merge standard-enqueue ovirt-engine_standard-on-merge standard-enqueue part of ovirt-engine_standard-check-patch #15338 part of ovirt-engine_standard-check-patch #15337 part of ovirt-engine_standard-check-patch #15336 part of ovirt-engine_standard-check-patch #15335 jenkins_master_check-patch-el7-ppc64le jenkins_master_check-patch-el7-ppc64le part of jenkins_standard-check-patch #5550 part of jenkins_standard-check-patch #5548 jenkins_master_check-patch-el7-ppc64le jenkins_master_check-patch-el7-ppc64le jenkins_master_check-patch-el7-ppc64le jenkins_master_check-patch-el7-ppc64le part of ovirt-appliance_standard-check-patch #385 ovirt-appliance [check-patch] lago-demo_master_github_timed-el7-x86_64 part of ovirt-engine_standard-on-merge #2664 part of ovirt-engine_standard-check-patch #15334

...

On 25 Nov 2021, at 21:53, Nir Soffer <nsoffer@redhat.com> wrote:

On Thu, Nov 25, 2021 at 9:15 PM Michal Skrivanek <michal.skrivanek@redhat.com> wrote:

On 25. 11. 2021, at 16:45, Evgheni Dereveanchin <ederevea@redhat.com> wrote:

Hi Milan,

Indeed, this is related to the Jenkins move: all pre-existing nodes are no longer usable so new ones have been created. We're looking into the reasons new nodes fail to initialize and bringing up new ones in parallel to help with processing the build queue.

Sorry for the inconvenience. Please report any other issues if you see them as there may be quite some instability due to the rebuild.

Hi Evgheni, the queue is now 53 builds long, and it seems nothing is building, perhaps there's no worker or the labelling is wrong?

Looks like jobs are stuck for several hours https://jenkins.ovirt.org/job/vdsm_standard-check-patch/31036/

at the "loading code" stage: https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pat...

With this error:

There are no nodes with the label ‘jenkins-vdsm_standard-check-patch-31036’

The label looks wrong. vdsm requires nodes with "el8" label.

Thanks, michal

Regards. Evgheni

On Thu, Nov 25, 2021 at 4:24 PM Nir Soffer <nsoffer@redhat.com> wrote:

On Thu, Nov 25, 2021 at 3:48 PM Milan Zamazal <mzamazal@redhat.com> wrote:

Hi,

all patches uploaded today I've seen (for Engine and Vdsm) fail due to problems with availability of packages when preparing the el7 environment. For example: https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/15291/

Additionally, Vdsm tests are not run on PSI.

Does anybody know what's wrong and how to fix it? Can it be related to the Jenkins move?

Looking at https://jenkins.ovirt.org/

There are 33 jobs in the queue, and only 3 jobs running.

Maybe we did not restore all the nodes?

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/ZLH3ILKHC3YPZW...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/RZ7H4L26DU6LAK...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/UK4KVTPITCAWHW...