Missing packages in oVirt repos, blocking 4.4.10 Vdsm build
Hi, CI is not working for me (at least) in ovirt-4.4.z Vdsm branch. As Sandro has found out, one part of that is the following: something or someone destroyed the tested repo, it's missing a huge amount of packages there this caused the nighlty reaper to also cleanup the ovirt-master-snapshot repo as no corresponding package was in tested repo we had a massive data loss there, can you please report it to devops / amarchuk 's team? I hope they have backups if not... we'll have to rebuild the missing packages. Could you please take a look and tell us what can be done about it? It is currently one of the issues blocking today's 4.4.10 Vdsm build. Thanks, Milan
Hello Milan. That’s unfortunate and sorry for that. Please check if you can use https://resources02.phx.ovirt.org for now to be unblocked. This is the old server. During migration some files were not migrated as there is really tons of old stuff, plus all those CI files are being distributed to mirrors that is not instantaneous and unneeded. We tried to check which ones are used and which are not. I think the plan is to have a separate file server for CI files as a permanent solution. CCing Denis also. Anton.
On 18. 11. 2021, at 9:44, Milan Zamazal <mzamazal@redhat.com> wrote:
Hi,
CI is not working for me (at least) in ovirt-4.4.z Vdsm branch. As Sandro has found out, one part of that is the following:
something or someone destroyed the tested repo, it's missing a huge amount of packages there this caused the nighlty reaper to also cleanup the ovirt-master-snapshot repo as no corresponding package was in tested repo we had a massive data loss there, can you please report it to devops / amarchuk 's team? I hope they have backups if not... we'll have to rebuild the missing packages.
Could you please take a look and tell us what can be done about it?
It is currently one of the issues blocking today's 4.4.10 Vdsm build.
Thanks, Milan
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
I have also opened https://issues.redhat.com/browse/CPDEVOPS-355 to track the permanent solution for this.
On 18. 11. 2021, at 10:10, Anton Marchukov <amarchuk@redhat.com> wrote:
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
Anton Marchukov <amarchuk@redhat.com> writes:
Hello Milan.
That’s unfortunate and sorry for that. Please check if you can use https://resources02.phx.ovirt.org for now to be unblocked. This is the old server.
Hi Anton, thank you for the suggestion. I tried but it looks to me that packages are missing there too. Am I mistaken or is it so really?
During migration some files were not migrated as there is really tons of old stuff, plus all those CI files are being distributed to mirrors that is not instantaneous and unneeded. We tried to check which ones are used and which are not. I think the plan is to have a separate file server for CI files as a permanent solution.
CCing Denis also.
Anton.
On 18. 11. 2021, at 9:44, Milan Zamazal <mzamazal@redhat.com> wrote:
Hi,
CI is not working for me (at least) in ovirt-4.4.z Vdsm branch. As Sandro has found out, one part of that is the following:
something or someone destroyed the tested repo, it's missing a huge amount of packages there this caused the nighlty reaper to also cleanup the ovirt-master-snapshot repo as no corresponding package was in tested repo we had a massive data loss there, can you please report it to devops / amarchuk 's team? I hope they have backups if not... we'll have to rebuild the missing packages.
Could you please take a look and tell us what can be done about it?
It is currently one of the issues blocking today's 4.4.10 Vdsm build.
Thanks, Milan
I am not aware about any on purpose changes on the old server. When was the last time it was working and which url is exactly missing? Anton.
On 18. 11. 2021, at 11:18, Milan Zamazal <mzamazal@redhat.com> wrote:
Anton Marchukov <amarchuk@redhat.com> writes:
Hello Milan.
That’s unfortunate and sorry for that. Please check if you can use https://resources02.phx.ovirt.org for now to be unblocked. This is the old server.
Hi Anton,
thank you for the suggestion. I tried but it looks to me that packages are missing there too. Am I mistaken or is it so really?
During migration some files were not migrated as there is really tons of old stuff, plus all those CI files are being distributed to mirrors that is not instantaneous and unneeded. We tried to check which ones are used and which are not. I think the plan is to have a separate file server for CI files as a permanent solution.
CCing Denis also.
Anton.
On 18. 11. 2021, at 9:44, Milan Zamazal <mzamazal@redhat.com> wrote:
Hi,
CI is not working for me (at least) in ovirt-4.4.z Vdsm branch. As Sandro has found out, one part of that is the following:
something or someone destroyed the tested repo, it's missing a huge amount of packages there this caused the nighlty reaper to also cleanup the ovirt-master-snapshot repo as no corresponding package was in tested repo we had a massive data loss there, can you please report it to devops / amarchuk 's team? I hope they have backups if not... we'll have to rebuild the missing packages.
Could you please take a look and tell us what can be done about it?
It is currently one of the issues blocking today's 4.4.10 Vdsm build.
Thanks, Milan
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
E.g. the “tested” I am aware off is here: https://resources02.phx.ovirt.org/repos/ovirt/tested/ It is also on the new server: https://resources.ovirt.org/repos/ovirt/tested/ Anton.
On 18. 11. 2021, at 11:21, Anton Marchukov <amarchuk@redhat.com> wrote:
I am not aware about any on purpose changes on the old server. When was the last time it was working and which url is exactly missing?
Anton.
On 18. 11. 2021, at 11:18, Milan Zamazal <mzamazal@redhat.com> wrote:
Anton Marchukov <amarchuk@redhat.com> writes:
Hello Milan.
That’s unfortunate and sorry for that. Please check if you can use https://resources02.phx.ovirt.org for now to be unblocked. This is the old server.
Hi Anton,
thank you for the suggestion. I tried but it looks to me that packages are missing there too. Am I mistaken or is it so really?
During migration some files were not migrated as there is really tons of old stuff, plus all those CI files are being distributed to mirrors that is not instantaneous and unneeded. We tried to check which ones are used and which are not. I think the plan is to have a separate file server for CI files as a permanent solution.
CCing Denis also.
Anton.
On 18. 11. 2021, at 9:44, Milan Zamazal <mzamazal@redhat.com> wrote:
Hi,
CI is not working for me (at least) in ovirt-4.4.z Vdsm branch. As Sandro has found out, one part of that is the following:
something or someone destroyed the tested repo, it's missing a huge amount of packages there this caused the nighlty reaper to also cleanup the ovirt-master-snapshot repo as no corresponding package was in tested repo we had a massive data loss there, can you please report it to devops / amarchuk 's team? I hope they have backups if not... we'll have to rebuild the missing packages.
Could you please take a look and tell us what can be done about it?
It is currently one of the issues blocking today's 4.4.10 Vdsm build.
Thanks, Milan
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
Adding also Ehud as it does not looks to be related to migration. Maybe some deploy job problem or just regular cleanup.
On 18. 11. 2021, at 11:24, Anton Marchukov <amarchuk@redhat.com> wrote:
E.g. the “tested” I am aware off is here:
https://resources02.phx.ovirt.org/repos/ovirt/tested/
It is also on the new server:
https://resources.ovirt.org/repos/ovirt/tested/
Anton.
On 18. 11. 2021, at 11:21, Anton Marchukov <amarchuk@redhat.com> wrote:
I am not aware about any on purpose changes on the old server. When was the last time it was working and which url is exactly missing?
Anton.
On 18. 11. 2021, at 11:18, Milan Zamazal <mzamazal@redhat.com> wrote:
Anton Marchukov <amarchuk@redhat.com> writes:
Hello Milan.
That’s unfortunate and sorry for that. Please check if you can use https://resources02.phx.ovirt.org for now to be unblocked. This is the old server.
Hi Anton,
thank you for the suggestion. I tried but it looks to me that packages are missing there too. Am I mistaken or is it so really?
During migration some files were not migrated as there is really tons of old stuff, plus all those CI files are being distributed to mirrors that is not instantaneous and unneeded. We tried to check which ones are used and which are not. I think the plan is to have a separate file server for CI files as a permanent solution.
CCing Denis also.
Anton.
On 18. 11. 2021, at 9:44, Milan Zamazal <mzamazal@redhat.com> wrote:
Hi,
CI is not working for me (at least) in ovirt-4.4.z Vdsm branch. As Sandro has found out, one part of that is the following:
something or someone destroyed the tested repo, it's missing a huge amount of packages there this caused the nighlty reaper to also cleanup the ovirt-master-snapshot repo as no corresponding package was in tested repo we had a massive data loss there, can you please report it to devops / amarchuk 's team? I hope they have backups if not... we'll have to rebuild the missing packages.
Could you please take a look and tell us what can be done about it?
It is currently one of the issues blocking today's 4.4.10 Vdsm build.
Thanks, Milan
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
Anton Marchukov <amarchuk@redhat.com> writes:
I am not aware about any on purpose changes on the old server. When was the last time it was working and which url is exactly missing?
Correct me if I'm wrong, but is it for example right that https://resources02.phx.ovirt.org/pub/ovirt-master-snapshot/rpm/el8/noarch/ is empty? I don't know when it was last working but we've experienced similar issues at least several days ago in some patches (don't know if related).
Anton.
On 18. 11. 2021, at 11:18, Milan Zamazal <mzamazal@redhat.com> wrote:
Anton Marchukov <amarchuk@redhat.com> writes:
Hello Milan.
That’s unfortunate and sorry for that. Please check if you can use https://resources02.phx.ovirt.org for now to be unblocked. This is the old server.
Hi Anton,
thank you for the suggestion. I tried but it looks to me that packages are missing there too. Am I mistaken or is it so really?
During migration some files were not migrated as there is really tons of old stuff, plus all those CI files are being distributed to mirrors that is not instantaneous and unneeded. We tried to check which ones are used and which are not. I think the plan is to have a separate file server for CI files as a permanent solution.
CCing Denis also.
Anton.
On 18. 11. 2021, at 9:44, Milan Zamazal <mzamazal@redhat.com> wrote:
Hi,
CI is not working for me (at least) in ovirt-4.4.z Vdsm branch. As Sandro has found out, one part of that is the following:
something or someone destroyed the tested repo, it's missing a huge amount of packages there this caused the nighlty reaper to also cleanup the ovirt-master-snapshot repo as no corresponding package was in tested repo we had a massive data loss there, can you please report it to devops / amarchuk 's team? I hope they have backups if not... we'll have to rebuild the missing packages.
Could you please take a look and tell us what can be done about it?
It is currently one of the issues blocking today's 4.4.10 Vdsm build.
Thanks, Milan
Adding Ehud in loop.
On 18. 11. 2021, at 11:29, Milan Zamazal <mzamazal@redhat.com> wrote:
Anton Marchukov <amarchuk@redhat.com> writes:
I am not aware about any on purpose changes on the old server. When was the last time it was working and which url is exactly missing?
Correct me if I'm wrong, but is it for example right that https://resources02.phx.ovirt.org/pub/ovirt-master-snapshot/rpm/el8/noarch/ is empty?
I don't know when it was last working but we've experienced similar issues at least several days ago in some patches (don't know if related).
Anton.
On 18. 11. 2021, at 11:18, Milan Zamazal <mzamazal@redhat.com> wrote:
Anton Marchukov <amarchuk@redhat.com> writes:
Hello Milan.
That’s unfortunate and sorry for that. Please check if you can use https://resources02.phx.ovirt.org for now to be unblocked. This is the old server.
Hi Anton,
thank you for the suggestion. I tried but it looks to me that packages are missing there too. Am I mistaken or is it so really?
During migration some files were not migrated as there is really tons of old stuff, plus all those CI files are being distributed to mirrors that is not instantaneous and unneeded. We tried to check which ones are used and which are not. I think the plan is to have a separate file server for CI files as a permanent solution.
CCing Denis also.
Anton.
On 18. 11. 2021, at 9:44, Milan Zamazal <mzamazal@redhat.com> wrote:
Hi,
CI is not working for me (at least) in ovirt-4.4.z Vdsm branch. As Sandro has found out, one part of that is the following:
something or someone destroyed the tested repo, it's missing a huge amount of packages there this caused the nighlty reaper to also cleanup the ovirt-master-snapshot repo as no corresponding package was in tested repo we had a massive data loss there, can you please report it to devops / amarchuk 's team? I hope they have backups if not... we'll have to rebuild the missing packages.
Could you please take a look and tell us what can be done about it?
It is currently one of the issues blocking today's 4.4.10 Vdsm build.
Thanks, Milan
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
The CI was switched to the new server: https://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el8/noarch/ is not empty. I guess the old one might be cleaned up by cron.
host resources.ovirt.org resources.ovirt.org has address 169.62.7.164
Can you please check your system uses the correct DNS name for resources and resolve to the correct IP? If it is then we need to know which exact packages are missing now on the new server (regular resources.ovirt.org) as this is where CI publishers are pushing stuff. Thanks.
On 18. 11. 2021, at 11:29, Milan Zamazal <mzamazal@redhat.com> wrote:
right that https://resources02.phx.ovirt.org/pub/ovirt-master-snapshot/rpm/el8/noarch/ is empty?
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
Anton Marchukov <amarchuk@redhat.com> writes:
The CI was switched to the new server:
https://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el8/noarch/
is not empty. I guess the old one might be cleaned up by cron.
OK.
host resources.ovirt.org resources.ovirt.org has address 169.62.7.164
Can you please check your system uses the correct DNS name for resources and resolve to the correct IP?
It does and I can see packages on the new server.
If it is then we need to know which exact packages are missing now on the new server (regular resources.ovirt.org) as this is where CI publishers are pushing stuff.
Let's try to make this working: https://gerrit.ovirt.org/c/vdsm/+/117699 I don't claim the repo changed there is necessarily the correct one, I can take whatever currently works :-). Currently it fails with: ERROR: Command failed: [2021-11-18T09:12:51.191Z] # /usr/bin/dnf --installroot /var/lib/mock/centos-stream-8-x86_64-80fb3df0df980170f23b5f62d18c8f56-5310/root/ --releasever 8 --setopt=deltarpm=False --allowerasing --disableplugin=local --disableplugin=spacewalk --disableplugin=versionlock --disableplugin=local --disableplugin=spacewalk --disableplugin=versionlock install tar gcc-c++ redhat-rpm-config redhat-release which xz sed make bzip2 gzip gcc coreutils unzip shadow-utils diffutils cpio bash gawk rpm-build info patch util-linux findutils grep python36 autoconf automake bash createrepo dnf dnf-utils e2fsprogs gcc gdb git iproute-tc iscsi-initiator-utils libguestfs-tools-c lshw lsof make NetworkManager nmstate nmstate-plugin-ovsdb openssl ovirt-imageio-client python3-augeas python3-blivet python3-dateutil python3-dbus python3-decorator python3-devel python3-dmidecode python3-ioprocess python3-libselinux python3-libvirt python3-magic python3-nose python3-pip python3-policycoreutils python3-pyyaml python3-requests python3-sanlock python3-six python3-yaml qemu-img rpm-build rpmlint sanlock sudo systemd systemd-udev xfsprogs --setopt=tsflags=nocontexts [2021-11-18T09:12:51.191Z] No matches found for the following disable plugin patterns: local, spacewalk, versionlock [2021-11-18T09:12:51.191Z] Last metadata expiration check: 0:00:07 ago on Thu Nov 18 09:12:37 2021. [2021-11-18T09:12:51.191Z] No match for argument: python3-ioprocess [2021-11-18T09:12:51.191Z] Error: Unable to find a match: python3-ioprocess (Note this is a different error than the one about missing ovirt-vmconsole etc. before switching from master to 4.4 repo.)
Hey, While trying to reproduce Milan’s vdsm patch, I am now getting errors that python3-ioprocess is not found. If it should exist on the ovirt-4.4, we need to create a release branch for it in the git repo. Currently I can see it exists on master branch, and for the missing packages I honestly didn’t find anything that is missing. If it is missing maybe it is not configured to be deployed in ovirt-4.4 tested yet.
On 18 Nov 2021, at 12:21, Anton Marchukov <amarchuk@redhat.com> wrote:
I am not aware about any on purpose changes on the old server. When was the last time it was working and which url is exactly missing?
Anton.
On 18. 11. 2021, at 11:18, Milan Zamazal <mzamazal@redhat.com> wrote:
Anton Marchukov <amarchuk@redhat.com> writes:
Hello Milan.
That’s unfortunate and sorry for that. Please check if you can use https://resources02.phx.ovirt.org for now to be unblocked. This is the old server.
Hi Anton,
thank you for the suggestion. I tried but it looks to me that packages are missing there too. Am I mistaken or is it so really?
During migration some files were not migrated as there is really tons of old stuff, plus all those CI files are being distributed to mirrors that is not instantaneous and unneeded. We tried to check which ones are used and which are not. I think the plan is to have a separate file server for CI files as a permanent solution.
CCing Denis also.
Anton.
On 18. 11. 2021, at 9:44, Milan Zamazal <mzamazal@redhat.com> wrote:
Hi,
CI is not working for me (at least) in ovirt-4.4.z Vdsm branch. As Sandro has found out, one part of that is the following:
something or someone destroyed the tested repo, it's missing a huge amount of packages there this caused the nighlty reaper to also cleanup the ovirt-master-snapshot repo as no corresponding package was in tested repo we had a massive data loss there, can you please report it to devops / amarchuk 's team? I hope they have backups if not... we'll have to rebuild the missing packages.
Could you please take a look and tell us what can be done about it?
It is currently one of the issues blocking today's 4.4.10 Vdsm build.
Thanks, Milan
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
_______________________________________________ Infra mailing list -- infra@ovirt.org <mailto:infra@ovirt.org> To unsubscribe send an email to infra-leave@ovirt.org <mailto:infra-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html <https://www.ovirt.org/privacy-policy.html> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/E5RBCMS4UWAWV4... <https://lists.ovirt.org/archives/list/infra@ovirt.org/message/E5RBCMS4UWAWV4MXREMJ3SMIDQN735CY/>
participants (3)
-
Anton Marchukov -
Ehud Yonasi -
Milan Zamazal