Hi, it looks like Jenkins CI jobs don't finish in any reasonable time. In case of Vdsm, at least jenkins-psi2 jobs run but they fail early with Cannot download x86_64/python3-ioprocess-1.4.2-1.202111071801.git53786ff.el8.x86_64.rpm: All mirrors were tried Is there a way to get fixed at least one of the runners? Thanks, Milan
I am getting other failed CI's in other repositories: *Api-model:* fatal: unable to access 'https://gerrit.ovirt.org/jenkins/': Could not resolve host: gerrit.ovirt.org; Unknown error Link: https://jenkins.ovirt.org/job/ovirt-engine-api-model_standard-check-patch/35... *OST:* failing on: Error: Unable to find a match: python3-ansible-runner Link: https://ovirt-devops-jenkins.upshift.rdu2.redhat.com/job/ovirt-system-tests_... Thanks, Saif On Thu, Dec 9, 2021 at 3:04 PM Milan Zamazal <mzamazal@redhat.com> wrote:
Hi,
it looks like Jenkins CI jobs don't finish in any reasonable time. In case of Vdsm, at least jenkins-psi2 jobs run but they fail early with
Cannot download x86_64/python3-ioprocess-1.4.2-1.202111071801.git53786ff.el8.x86_64.rpm: All mirrors were tried
Is there a way to get fixed at least one of the runners?
Thanks, Milan _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/SQ2DTNP63SQQYC...
On Thu, Dec 9, 2021 at 3:04 PM Milan Zamazal <mzamazal@redhat.com> wrote:
Hi,
it looks like Jenkins CI jobs don't finish in any reasonable time. In case of Vdsm, at least jenkins-psi2 jobs run but they fail early with
Cannot download x86_64/python3-ioprocess-1.4.2-1.202111071801.git53786ff.el8.x86_64.rpm: All mirrors were tried
Is there a way to get fixed at least one of the runners?
I'm using github CI now, no issue seen in the last week. But we have now a wrose issue, build-artifacts job fail randomly in one of the targets. This means there is no way to use the automatic OST run. Should we disable all vdsm tests on gerrit, and enable only x64_64 el8 build artifacts so we can at least get working OST?
On Tue, Dec 14, 2021 at 12:32 AM Nir Soffer <nsoffer@redhat.com> wrote:
On Thu, Dec 9, 2021 at 3:04 PM Milan Zamazal <mzamazal@redhat.com> wrote:
Hi,
it looks like Jenkins CI jobs don't finish in any reasonable time. In case of Vdsm, at least jenkins-psi2 jobs run but they fail early with
Cannot download x86_64/python3-ioprocess-1.4.2-1.202111071801.git53786ff.el8.x86_64.rpm: All mirrors were tried
Is there a way to get fixed at least one of the runners?
I'm using github CI now, no issue seen in the last week.
But we have now a wrose issue, build-artifacts job fail randomly in one of the targets. This means there is no way to use the automatic OST run.
Should we disable all vdsm tests on gerrit, and enable only x64_64 el8 build artifacts so we can at least get working OST?
I tried this here: https://gerrit.ovirt.org/c/vdsm/+/118022
On Tue, Dec 14, 2021 at 12:39 AM Nir Soffer <nsoffer@redhat.com> wrote:
On Tue, Dec 14, 2021 at 12:32 AM Nir Soffer <nsoffer@redhat.com> wrote:
On Thu, Dec 9, 2021 at 3:04 PM Milan Zamazal <mzamazal@redhat.com> wrote:
Hi,
it looks like Jenkins CI jobs don't finish in any reasonable time. In case of Vdsm, at least jenkins-psi2 jobs run but they fail early with
Cannot download x86_64/python3-ioprocess-1.4.2-1.202111071801.git53786ff.el8.x86_64.rpm: All mirrors were tried
Is there a way to get fixed at least one of the runners?
I'm using github CI now, no issue seen in the last week.
But we have now a wrose issue, build-artifacts job fail randomly in one of the targets. This means there is no way to use the automatic OST run.
Should we disable all vdsm tests on gerrit, and enable only x64_64 el8 build artifacts so we can at least get working OST?
I tried this here: https://gerrit.ovirt.org/c/vdsm/+/118022
Patch should be ready, I rebased by current thin-scratch-disks branch on top of it, and some OST jobs started.
On Tue, Dec 14, 2021 at 12:13 PM Nir Soffer <nsoffer@redhat.com> wrote: ...
Should we disable all vdsm tests on gerrit, and enable only x64_64 el8 build artifacts so we can at least get working OST?
I tried this here: https://gerrit.ovirt.org/c/vdsm/+/118022
Patch should be ready, I rebased by current thin-scratch-disks branch on top of it, and some OST jobs started.
Current state: https://gerrit.ovirt.org/q/topic:thin-scratch-disks+is:open 6 patches ready for merge. 3 OST runs started successfully 3 build artifacts failed (50% success rate) - I retriggered the failed runs Al the failures happened on el7-worker-02: - https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pat... - https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pat... - https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pat... Build history for this host - all builds fail in the last 4 days: https://jenkins.ovirt.org/computer/el7-worker-02/builds Build history for other hosts: https://jenkins.ovirt.org/computer/el7-worker-03/builds Nir
On Tue, Dec 14, 2021 at 1:09 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Tue, Dec 14, 2021 at 12:13 PM Nir Soffer <nsoffer@redhat.com> wrote: ...
Should we disable all vdsm tests on gerrit, and enable only x64_64 el8 build artifacts so we can at least get working OST?
I tried this here: https://gerrit.ovirt.org/c/vdsm/+/118022
Patch should be ready, I rebased by current thin-scratch-disks branch on top of it, and some OST jobs started.
Current state: https://gerrit.ovirt.org/q/topic:thin-scratch-disks+is:open
6 patches ready for merge. 3 OST runs started successfully 3 build artifacts failed (50% success rate) - I retriggered the failed runs
Al the failures happened on el7-worker-02: - https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pat... - https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pat... - https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pat...
Retriggered runs fail again on el7-worker-02: - https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pat... - https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pat...
Build history for this host - all builds fail in the last 4 days: https://jenkins.ovirt.org/computer/el7-worker-02/builds
Build history for other hosts: https://jenkins.ovirt.org/computer/el7-worker-03/builds
Nir
Hi, After digging to the issues we see that el7-worker-02 had dhcp configuration issues. There were changes in the dhcp configuration last week thus making the dns not resolvable. For further information about it see: https://issues.redhat.com/browse/CPDEVOPS-407 <https://issues.redhat.com/browse/CPDEVOPS-407> After rebooting the vm, it is now working. Regards, Ehud.
On 14 Dec 2021, at 13:45, Nir Soffer <nsoffer@redhat.com> wrote:
On Tue, Dec 14, 2021 at 1:09 PM Nir Soffer <nsoffer@redhat.com <mailto:nsoffer@redhat.com>> wrote:
On Tue, Dec 14, 2021 at 12:13 PM Nir Soffer <nsoffer@redhat.com> wrote: ...
Should we disable all vdsm tests on gerrit, and enable only x64_64 el8 build artifacts so we can at least get working OST?
I tried this here: https://gerrit.ovirt.org/c/vdsm/+/118022
Patch should be ready, I rebased by current thin-scratch-disks branch on top of it, and some OST jobs started.
Current state: https://gerrit.ovirt.org/q/topic:thin-scratch-disks+is:open
6 patches ready for merge. 3 OST runs started successfully 3 build artifacts failed (50% success rate) - I retriggered the failed runs
Al the failures happened on el7-worker-02: - https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pat... - https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pat... - https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pat...
Retriggered runs fail again on el7-worker-02: - https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pat... <https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/31493/pipeline> - https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-pat... <https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/31490/pipeline>
Build history for this host - all builds fail in the last 4 days: https://jenkins.ovirt.org/computer/el7-worker-02/builds <https://jenkins.ovirt.org/computer/el7-worker-02/builds>
Build history for other hosts: https://jenkins.ovirt.org/computer/el7-worker-03/builds <https://jenkins.ovirt.org/computer/el7-worker-03/builds>
Nir
Nir Soffer <nsoffer@redhat.com> writes:
I'm using github CI now, no issue seen in the last week.
But we have now a wrose issue, build-artifacts job fail randomly in one of the targets. This means there is no way to use the automatic OST run.
Should we disable all vdsm tests on gerrit, and enable only x64_64 el8 build artifacts so we can at least get working OST?
Since tests on gerrit are unusable, we should do it this way, good idea. Nevertheless manually testing patches by pushing them to GitHub is tiresome, we should move to GitHub completely ASAP. The only thing missing is OST, right?
On Tue, Dec 14, 2021 at 10:14 AM Milan Zamazal <mzamazal@redhat.com> wrote:
Nir Soffer <nsoffer@redhat.com> writes:
I'm using github CI now, no issue seen in the last week.
But we have now a wrose issue, build-artifacts job fail randomly in one of the targets. This means there is no way to use the automatic OST run.
Should we disable all vdsm tests on gerrit, and enable only x64_64 el8 build artifacts so we can at least get working OST?
Since tests on gerrit are unusable, we should do it this way, good idea.
Nevertheless manually testing patches by pushing them to GitHub is tiresome,
This is not that bad, I'm using this flow for a few years: while work needed: hack... push -f github # push to my github fork check the build in github git review # push to gerrit for review
we should move to GitHub completely ASAP. The only thing missing is OST, right?
Yes. I think what we need is a way to start OST build with a github pull request url: CUSTOM_REPOS=https://github.com/oVirt/{project}/pull/{7} Or if it is easier, a way to upload zip files to jenkins for running OST. Once we have something like this, we can move vdsm to github. Triggering OST builds automatically ("gating") is nice to have and can be done later. Nir
On Tuesday, 14 December 2021 09:44:44 CET Nir Soffer wrote:
On Tue, Dec 14, 2021 at 10:14 AM Milan Zamazal <mzamazal@redhat.com> wrote:
Nir Soffer <nsoffer@redhat.com> writes:
I'm using github CI now, no issue seen in the last week.
But we have now a wrose issue, build-artifacts job fail randomly in one of the targets. This means there is no way to use the automatic OST run.
Should we disable all vdsm tests on gerrit, and enable only x64_64 el8 build artifacts so we can at least get working OST?
Since tests on gerrit are unusable, we should do it this way, good idea.
Nevertheless manually testing patches by pushing them to GitHub is tiresome,
This is not that bad, I'm using this flow for a few years:
while work needed: hack... push -f github # push to my github fork check the build in github
git review # push to gerrit for review
which is basically the same workflow which we used when we run test also in Travis IIRC, so actually no change here ...
we should move to GitHub completely ASAP. The only thing missing is OST, right?
Yes. I think what we need is a way to start OST build with a github pull request url:
CUSTOM_REPOS=https://github.com/oVirt/{project}/pull/{7}
Or if it is easier, a way to upload zip files to jenkins for running OST.
If Jenkins wipes the workspace before/after each build, that won't be probably the most easy way to go. Github actions deliver the rpm in zip file, so we basically need * download this zip file before build and unzip * create local repo from it (vdsm.repo with baseurl=file://$PATH_TO_UNZIP_DIR) * use this local repo instead of previous Jenkins build artifact job URL So we just need to implement this or am I missing something and there are some issues with this approach and it's more complicated?
Once we have something like this, we can move vdsm to github.
Triggering OST builds automatically ("gating") is nice to have and can be done later.
Nir _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/Y6CO6UJG6SD4P IUTPWTMBDT7CC6DGDOH/
.On Tue, Dec 14, 2021 at 11:15 AM Vojtech Juranek <vjuranek@redhat.com> wrote: ...
we should move to GitHub completely ASAP. The only thing missing is OST, right?
Yes. I think what we need is a way to start OST build with a github pull request url:
CUSTOM_REPOS=https://github.com/oVirt/{project}/pull/{7}
Or if it is easier, a way to upload zip files to jenkins for running OST.
If Jenkins wipes the workspace before/after each build, that won't be probably the most easy way to go. Github actions deliver the rpm in zip file, so we basically need * download this zip file before build and unzip
This requires Personal Access Token: https://docs.github.com/en/authentication/keeping-your-account-and-data-secu... So we need to add a github users with PAT that can download the built artifacts. We need to find the artifacts in the build, likey using github API. For example: https://github.com/oVirt/ovirt-imageio/actions/runs/1573283335 The artifact we need is the centos-8.zip: https://github.com/oVirt/ovirt-imageio/suites/4639547952/artifacts/125747744 The URL does not tell us that this is the centos stream 8 artifact, so we need a way to locate it programatically.
* create local repo from it (vdsm.repo with baseurl=file://$PATH_TO_UNZIP_DIR) * use this local repo instead of previous Jenkins build artifact job URL
This is for a single host, all hosts in OST need to use the new repo. Another option is to have a web server serving built rpms, e.g. https://repo.ost.example.com/ And unzip each built rpm before the test to: /serverroot/93d9ee4f-4cda-403a-a29d-9a3669ee49af/ So we have this layout: /serverroot/ 93d9ee4f-4cda-403a-a29d-9a3669ee49af/ repodata/ *.rpm ... And start the OST job with: CUSTOM_REPOS=https://repo.ost.example.com/93d9ee4f-4cda-403a-a29d-9a3669ee49af When the OST job finish, delete /serverroot/93d9ee4f-4cda-403a-a29d-9a3669ee49af/ This can also work with multiple pull requests, e.g. testing both vdsm and engine pull request at the same time. Nir
participants (5)
-
Ehud Yonasi -
Milan Zamazal -
Nir Soffer -
Saif Abu Saleh -
Vojtech Juranek