Doesn't seem to be we're over the unavailable mirrors issue with the new repos added to STD CI: Lets start working on mirroring the relevant repos to mirrors.phx.ovirt.org and get control over the update process. http://jenkins.ovirt.org/job/jenkins_master_check-patch-fc24-x86_64/16/artif... DEBUG util.py:421: https://mirrors.cat.pdx.edu/fedora/linux/updates/24/x86_64/repodata/repomd.x...: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: https://dl.fedoraproject.org/pub/fedora/linux/updates/24/x86_64/repodata/rep...: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: http://fedora.bhs.mirrors.ovh.net/linux/updates/24/x86_64/repodata/repomd.xm...: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: http://mirror.its.dal.ca/pub/fedora/linux/updates/24/x86_64/repodata/repomd....: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: https://mirror.csclub.uwaterloo.ca/fedora/linux/updates/24/x86_64/repodata/r...: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: One of the configured repositories failed (updates), DEBUG util.py:421: and yum doesn't have enough cached data to continue. At this point the only DEBUG util.py:421: safe thing yum can do is fail. There are a few ways to work "fix" this: DEBUG util.py:421: 1. Contact the upstream for the repository and get them to fix the problem. DEBUG util.py:421: 2. Reconfigure the baseurl/etc. for the repository, to point to a working DEBUG util.py:421: upstream. This is most often useful if you are using a newer DEBUG util.py:421: distribution release than is supported by the repository (and the DEBUG util.py:421: packages for the previous distribution release still work). DEBUG util.py:421: 3. Disable the repository, so yum won't use it by default. Yum will then DEBUG util.py:421: just ignore the repository until you permanently enable it again or use DEBUG util.py:421: --enablerepo for temporary usage: DEBUG util.py:421: yum-config-manager --disable updates DEBUG util.py:421: 4. Configure the failing repository to be skipped, if it is unavailable. DEBUG util.py:421: Note that yum will try to contact the repo. when it runs most commands, DEBUG util.py:421: so will have to try and fail each time (and thus. yum will be be much DEBUG util.py:421: slower). If it is a very temporary problem though, this is often a nice DEBUG util.py:421: compromise: DEBUG util.py:421: yum-config-manager --save --setopt=updates.skip_if_unavailable=true DEBUG util.py:421: failure: repodata/repomd.xml from updates: [Errno 256] No more mirrors to try. DEBUG util.py:421: http://mirror.math.princeton.edu/pub/fedora/linux/updates/24/x86_64/repodata...: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://linux.mirrors.es.net/fedora/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.cs.pitt.edu/fedora/linux/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirrors.mit.edu/fedora/linux/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://ftp.uci.edu/fedora/linux/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: https://mirror.chpc.utah.edu/pub/fedora/linux/updates/24/x86_64/repodata/rep...: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://repo.atlantic.net/fedora/linux/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.cs.princeton.edu/pub/mirrors/fedora/linux/updates/24/x86_64/re...: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.uoregon.edu/fedora/linux/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.cc.vt.edu/pub/fedora/linux/updates/24/x86_64/repodata/repomd.x...: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: https://mirrors.kernel.org/fedora/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: https://pubmirror2.math.uh.edu/fedora-buffet/fedora/linux/updates/24/x86_64/...: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.nexcess.net/fedora/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.sjc02.svwh.net/fedora/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.symnds.com/distributions/fedora/linux/updates/24/x86_64/repoda...: [Errno -1] repomd.xml does not -- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
Hi Eyal, This does not look like a mirror issue but a MetaLink one. MetaLink is a security plugin for yum [1] and I saw the same issue last week so MetaLink may have been broken at that time. Basically what it does is fetches a file like [2] and checks if the mirrors received from the mirrorlist are present there. Maybe it does some other checks, but the point of this is protection against stale/rogue mirrors. I think we should just disable MetaLink on our CI slaves as we don't rely on latest fedora software versions anyway. Regards, Evgheni Dereveanchin [1] https://fedoraproject.org/wiki/Features/YumMetalinks [2] https://mirrors.fedoraproject.org/metalink?repo=fedora-24&arch=x86_64 ----- Original Message ----- From: "Eyal Edri" <eedri@redhat.com> To: "infra" <infra@ovirt.org>, "Evgheni Dereveanchin" <ederevea@redhat.com> Sent: Sunday, 11 December, 2016 1:27:53 PM Subject: mirrors errors Doesn't seem to be we're over the unavailable mirrors issue with the new repos added to STD CI: Lets start working on mirroring the relevant repos to mirrors.phx.ovirt.org and get control over the update process. http://jenkins.ovirt.org/job/jenkins_master_check-patch-fc24-x86_64/16/artif... DEBUG util.py:421: https://mirrors.cat.pdx.edu/fedora/linux/updates/24/x86_64/repodata/repomd.x...: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: https://dl.fedoraproject.org/pub/fedora/linux/updates/24/x86_64/repodata/rep...: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: http://fedora.bhs.mirrors.ovh.net/linux/updates/24/x86_64/repodata/repomd.xm...: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: http://mirror.its.dal.ca/pub/fedora/linux/updates/24/x86_64/repodata/repomd....: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: https://mirror.csclub.uwaterloo.ca/fedora/linux/updates/24/x86_64/repodata/r...: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: One of the configured repositories failed (updates), DEBUG util.py:421: and yum doesn't have enough cached data to continue. At this point the only DEBUG util.py:421: safe thing yum can do is fail. There are a few ways to work "fix" this: DEBUG util.py:421: 1. Contact the upstream for the repository and get them to fix the problem. DEBUG util.py:421: 2. Reconfigure the baseurl/etc. for the repository, to point to a working DEBUG util.py:421: upstream. This is most often useful if you are using a newer DEBUG util.py:421: distribution release than is supported by the repository (and the DEBUG util.py:421: packages for the previous distribution release still work). DEBUG util.py:421: 3. Disable the repository, so yum won't use it by default. Yum will then DEBUG util.py:421: just ignore the repository until you permanently enable it again or use DEBUG util.py:421: --enablerepo for temporary usage: DEBUG util.py:421: yum-config-manager --disable updates DEBUG util.py:421: 4. Configure the failing repository to be skipped, if it is unavailable. DEBUG util.py:421: Note that yum will try to contact the repo. when it runs most commands, DEBUG util.py:421: so will have to try and fail each time (and thus. yum will be be much DEBUG util.py:421: slower). If it is a very temporary problem though, this is often a nice DEBUG util.py:421: compromise: DEBUG util.py:421: yum-config-manager --save --setopt=updates.skip_if_unavailable=true DEBUG util.py:421: failure: repodata/repomd.xml from updates: [Errno 256] No more mirrors to try. DEBUG util.py:421: http://mirror.math.princeton.edu/pub/fedora/linux/updates/24/x86_64/repodata...: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://linux.mirrors.es.net/fedora/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.cs.pitt.edu/fedora/linux/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirrors.mit.edu/fedora/linux/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://ftp.uci.edu/fedora/linux/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: https://mirror.chpc.utah.edu/pub/fedora/linux/updates/24/x86_64/repodata/rep...: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://repo.atlantic.net/fedora/linux/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.cs.princeton.edu/pub/mirrors/fedora/linux/updates/24/x86_64/re...: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.uoregon.edu/fedora/linux/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.cc.vt.edu/pub/fedora/linux/updates/24/x86_64/repodata/repomd.x...: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: https://mirrors.kernel.org/fedora/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: https://pubmirror2.math.uh.edu/fedora-buffet/fedora/linux/updates/24/x86_64/...: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.nexcess.net/fedora/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.sjc02.svwh.net/fedora/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.symnds.com/distributions/fedora/linux/updates/24/x86_64/repoda...: [Errno -1] repomd.xml does not -- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
On Mon, Dec 12, 2016 at 11:47 AM, Evgheni Dereveanchin <ederevea@redhat.com> wrote:
Hi Eyal,
This does not look like a mirror issue but a MetaLink one. MetaLink is a security plugin for yum [1] and I saw the same issue last week so MetaLink may have been broken at that time.
Basically what it does is fetches a file like [2] and checks if the mirrors received from the mirrorlist are present there. Maybe it does some other checks, but the point of this is protection against stale/rogue mirrors.
I think we should just disable MetaLink on our CI slaves as we don't rely on latest fedora software versions anyway.
If this is indeed the case, lets open a ticket on this and track it.
Regards, Evgheni Dereveanchin
[1] https://fedoraproject.org/wiki/Features/YumMetalinks [2] https://mirrors.fedoraproject.org/metalink?repo=fedora-24&arch=x86_64
----- Original Message ----- From: "Eyal Edri" <eedri@redhat.com> To: "infra" <infra@ovirt.org>, "Evgheni Dereveanchin" <ederevea@redhat.com
Sent: Sunday, 11 December, 2016 1:27:53 PM Subject: mirrors errors
Doesn't seem to be we're over the unavailable mirrors issue with the new repos added to STD CI:
Lets start working on mirroring the relevant repos to mirrors.phx.ovirt.org and get control over the update process.
http://jenkins.ovirt.org/job/jenkins_master_check-patch- fc24-x86_64/16/artifact/exported-artifacts/logs/ mocker-fedora-24-x86_64.fc24.init/root.log
DEBUG util.py:421: https://mirrors.cat.pdx.edu/fedora/linux/updates/24/x86_ 64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: https://dl.fedoraproject.org/pub/fedora/linux/updates/24/ x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: http://fedora.bhs.mirrors.ovh.net/linux/updates/24/x86_64/ repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: http://mirror.its.dal.ca/pub/fedora/linux/updates/24/x86_ 64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: https://mirror.csclub.uwaterloo.ca/fedora/linux/ updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: One of the configured repositories failed (updates), DEBUG util.py:421: and yum doesn't have enough cached data to continue. At this point the only DEBUG util.py:421: safe thing yum can do is fail. There are a few ways to work "fix" this: DEBUG util.py:421: 1. Contact the upstream for the repository and get them to fix the problem. DEBUG util.py:421: 2. Reconfigure the baseurl/etc. for the repository, to point to a working DEBUG util.py:421: upstream. This is most often useful if you are using a newer DEBUG util.py:421: distribution release than is supported by the repository (and the DEBUG util.py:421: packages for the previous distribution release still work). DEBUG util.py:421: 3. Disable the repository, so yum won't use it by default. Yum will then DEBUG util.py:421: just ignore the repository until you permanently enable it again or use DEBUG util.py:421: --enablerepo for temporary usage: DEBUG util.py:421: yum-config-manager --disable updates DEBUG util.py:421: 4. Configure the failing repository to be skipped, if it is unavailable. DEBUG util.py:421: Note that yum will try to contact the repo. when it runs most commands, DEBUG util.py:421: so will have to try and fail each time (and thus. yum will be be much DEBUG util.py:421: slower). If it is a very temporary problem though, this is often a nice DEBUG util.py:421: compromise: DEBUG util.py:421: yum-config-manager --save --setopt=updates.skip_if_unavailable=true DEBUG util.py:421: failure: repodata/repomd.xml from updates: [Errno 256] No more mirrors to try. DEBUG util.py:421: http://mirror.math.princeton.edu/pub/fedora/linux/updates/ 24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://linux.mirrors.es.net/fedora/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.cs.pitt.edu/fedora/linux/updates/24/x86_ 64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirrors.mit.edu/fedora/linux/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://ftp.uci.edu/fedora/linux/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: https://mirror.chpc.utah.edu/pub/fedora/linux/updates/24/ x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://repo.atlantic.net/fedora/linux/updates/24/x86_ 64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.cs.princeton.edu/pub/mirrors/fedora/linux/ updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.uoregon.edu/fedora/linux/updates/24/x86_ 64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.cc.vt.edu/pub/fedora/linux/updates/24/x86_ 64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: https://mirrors.kernel.org/fedora/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: https://pubmirror2.math.uh.edu/fedora-buffet/fedora/ linux/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.nexcess.net/fedora/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.sjc02.svwh.net/fedora/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.symnds.com/distributions/fedora/linux/ updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
Hello Eyal, All. I see that jobs (and probably all standard ci ones?) now uses default Fedora configs from mock_configs/fedora-24-x86_64.cfg [1] it relies on metalink that is central point that failed at that time, also it can generate random mirrorlist and hence effectively skips the list of mirrors that we select and I even not sure that it uses our squid proxy at all. There is no proxy configured at that mock config. The changes for mock were done at fedora-24-x86_64_proxied.cfg copies of mock configs and in fact the change for Fedora proxies were not yet merged [2] and even if it was merged the standrad ci will not use them since for some reason it now uses default config, so it even does not currently use our old reproxy solution for Fedora repos. Anybody have any insights of why is that? So my suggestion is to 1. Switch standrad ci back to our _proxied configs as that was before (or was not and we just did not note it???). 2. Merge the remaining patch for proxied config to use new proxy solution instead of reproproxy [2]. [1] https://gerrit.ovirt.org/gitweb?p=jenkins.git;a=blob;f=mock_configs/fedora-2... [2] https://gerrit.ovirt.org/#/c/67595/ Anton. On Mon, Dec 12, 2016 at 10:54 AM, Eyal Edri <eedri@redhat.com> wrote:
On Mon, Dec 12, 2016 at 11:47 AM, Evgheni Dereveanchin < ederevea@redhat.com> wrote:
Hi Eyal,
This does not look like a mirror issue but a MetaLink one. MetaLink is a security plugin for yum [1] and I saw the same issue last week so MetaLink may have been broken at that time.
Basically what it does is fetches a file like [2] and checks if the mirrors received from the mirrorlist are present there. Maybe it does some other checks, but the point of this is protection against stale/rogue mirrors.
I think we should just disable MetaLink on our CI slaves as we don't rely on latest fedora software versions anyway.
If this is indeed the case, lets open a ticket on this and track it.
Regards, Evgheni Dereveanchin
[1] https://fedoraproject.org/wiki/Features/YumMetalinks [2] https://mirrors.fedoraproject.org/metalink?repo=fedora-24&arch=x86_64
----- Original Message ----- From: "Eyal Edri" <eedri@redhat.com> To: "infra" <infra@ovirt.org>, "Evgheni Dereveanchin" < ederevea@redhat.com> Sent: Sunday, 11 December, 2016 1:27:53 PM Subject: mirrors errors
Doesn't seem to be we're over the unavailable mirrors issue with the new repos added to STD CI:
Lets start working on mirroring the relevant repos to mirrors.phx.ovirt.org and get control over the update process.
http://jenkins.ovirt.org/job/jenkins_master_check-patch-fc24 -x86_64/16/artifact/exported-artifacts/logs/mocker-fedora- 24-x86_64.fc24.init/root.log
DEBUG util.py:421: https://mirrors.cat.pdx.edu/fedora/linux/updates/24/x86_64/ repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: https://dl.fedoraproject.org/pub/fedora/linux/updates/24/x86 _64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: http://fedora.bhs.mirrors.ovh.net/linux/updates/24/x86_64/re podata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: http://mirror.its.dal.ca/pub/fedora/linux/updates/24/x86_64/ repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: https://mirror.csclub.uwaterloo.ca/fedora/linux/updates/24/ x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: One of the configured repositories failed (updates), DEBUG util.py:421: and yum doesn't have enough cached data to continue. At this point the only DEBUG util.py:421: safe thing yum can do is fail. There are a few ways to work "fix" this: DEBUG util.py:421: 1. Contact the upstream for the repository and get them to fix the problem. DEBUG util.py:421: 2. Reconfigure the baseurl/etc. for the repository, to point to a working DEBUG util.py:421: upstream. This is most often useful if you are using a newer DEBUG util.py:421: distribution release than is supported by the repository (and the DEBUG util.py:421: packages for the previous distribution release still work). DEBUG util.py:421: 3. Disable the repository, so yum won't use it by default. Yum will then DEBUG util.py:421: just ignore the repository until you permanently enable it again or use DEBUG util.py:421: --enablerepo for temporary usage: DEBUG util.py:421: yum-config-manager --disable updates DEBUG util.py:421: 4. Configure the failing repository to be skipped, if it is unavailable. DEBUG util.py:421: Note that yum will try to contact the repo. when it runs most commands, DEBUG util.py:421: so will have to try and fail each time (and thus. yum will be be much DEBUG util.py:421: slower). If it is a very temporary problem though, this is often a nice DEBUG util.py:421: compromise: DEBUG util.py:421: yum-config-manager --save --setopt=updates.skip_if_unavailable=true DEBUG util.py:421: failure: repodata/repomd.xml from updates: [Errno 256] No more mirrors to try. DEBUG util.py:421: http://mirror.math.princeton.edu/pub/fedora/linux/updates/24 /x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://linux.mirrors.es.net/fedora/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.cs.pitt.edu/fedora/linux/updates/24/x86_64/ repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirrors.mit.edu/fedora/linux/updates/24/x86_64/repodata/repomd.xml : [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://ftp.uci.edu/fedora/linux/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: https://mirror.chpc.utah.edu/pub/fedora/linux/updates/24/x86 _64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://repo.atlantic.net/fedora/linux/updates/24/x86_64/ repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.cs.princeton.edu/pub/mirrors/fedora/linux/upda tes/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.uoregon.edu/fedora/linux/updates/24/x86_64/ repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.cc.vt.edu/pub/fedora/linux/updates/24/x86_64/ repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: https://mirrors.kernel.org/fedora/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: https://pubmirror2.math.uh.edu/fedora-buffet/fedora/linux/ updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.nexcess.net/fedora/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.sjc02.svwh.net/fedora/updates/24/x86_64/repodata/repomd.xml : [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.symnds.com/distributions/fedora/linux/updates/ 24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.phx.ovirt.org/mailman/listinfo/infra
-- Anton Marchukov Senior Software Engineer - RHEV CI - Red Hat
Actually Evgheni found the reason for that [1]. 132 <https://gerrit.ovirt.org/gitweb?p=jenkins.git;a=blob;f=mock_configs/mock_genconfig;h=b15504a4139b21ce7df7d932b735ffcea3af0a55;hb=refs/heads/master#l132> if check_proxy(proxy_url, repo_url): 133 <https://gerrit.ovirt.org/gitweb?p=jenkins.git;a=blob;f=mock_configs/mock_genconfig;h=b15504a4139b21ce7df7d932b735ffcea3af0a55;hb=refs/heads/master#l133> base_config = proxied_config 134 <https://gerrit.ovirt.org/gitweb?p=jenkins.git;a=blob;f=mock_configs/mock_genconfig;h=b15504a4139b21ce7df7d932b735ffcea3af0a55;hb=refs/heads/master#l134> else: 135 <https://gerrit.ovirt.org/gitweb?p=jenkins.git;a=blob;f=mock_configs/mock_genconfig;h=b15504a4139b21ce7df7d932b735ffcea3af0a55;hb=refs/heads/master#l135> logging.warn("Not using proxied configuration") we verify repoproxy availability in mock_genconfig and if it is not available - use default configs instead of proxied. This essentially avoids all of our improvement patches. Guess that code is not needed anymore and can be safely removed. [1] https://gerrit.ovirt.org/gitweb?p=jenkins.git;a=blob;f=mock_configs/mock_gen... On Mon, Dec 12, 2016 at 4:18 PM, Anton Marchukov <amarchuk@redhat.com> wrote:
Hello Eyal, All.
I see that jobs (and probably all standard ci ones?) now uses default Fedora configs from mock_configs/fedora-24-x86_64.cfg [1] it relies on metalink that is central point that failed at that time, also it can generate random mirrorlist and hence effectively skips the list of mirrors that we select and I even not sure that it uses our squid proxy at all. There is no proxy configured at that mock config.
The changes for mock were done at fedora-24-x86_64_proxied.cfg copies of mock configs and in fact the change for Fedora proxies were not yet merged [2] and even if it was merged the standrad ci will not use them since for some reason it now uses default config, so it even does not currently use our old reproxy solution for Fedora repos.
Anybody have any insights of why is that?
So my suggestion is to
1. Switch standrad ci back to our _proxied configs as that was before (or was not and we just did not note it???). 2. Merge the remaining patch for proxied config to use new proxy solution instead of reproproxy [2].
[1] https://gerrit.ovirt.org/gitweb?p=jenkins.git;a=blob;f= mock_configs/fedora-24-x86_64.cfg;h=7bdfabba003ee7de83aea4d0aad0be f0d6d747d8;hb=refs/heads/master [2] https://gerrit.ovirt.org/#/c/67595/
Anton.
On Mon, Dec 12, 2016 at 10:54 AM, Eyal Edri <eedri@redhat.com> wrote:
On Mon, Dec 12, 2016 at 11:47 AM, Evgheni Dereveanchin < ederevea@redhat.com> wrote:
Hi Eyal,
This does not look like a mirror issue but a MetaLink one. MetaLink is a security plugin for yum [1] and I saw the same issue last week so MetaLink may have been broken at that time.
Basically what it does is fetches a file like [2] and checks if the mirrors received from the mirrorlist are present there. Maybe it does some other checks, but the point of this is protection against stale/rogue mirrors.
I think we should just disable MetaLink on our CI slaves as we don't rely on latest fedora software versions anyway.
If this is indeed the case, lets open a ticket on this and track it.
Regards, Evgheni Dereveanchin
[1] https://fedoraproject.org/wiki/Features/YumMetalinks [2] https://mirrors.fedoraproject.org/metalink?repo=fedora-24&ar ch=x86_64
----- Original Message ----- From: "Eyal Edri" <eedri@redhat.com> To: "infra" <infra@ovirt.org>, "Evgheni Dereveanchin" < ederevea@redhat.com> Sent: Sunday, 11 December, 2016 1:27:53 PM Subject: mirrors errors
Doesn't seem to be we're over the unavailable mirrors issue with the new repos added to STD CI:
Lets start working on mirroring the relevant repos to mirrors.phx.ovirt.org and get control over the update process.
http://jenkins.ovirt.org/job/jenkins_master_check-patch-fc24 -x86_64/16/artifact/exported-artifacts/logs/mocker-fedora-24 -x86_64.fc24.init/root.log
DEBUG util.py:421: https://mirrors.cat.pdx.edu/fedora/linux/updates/24/x86_64/r epodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: https://dl.fedoraproject.org/pub/fedora/linux/updates/24/x86 _64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: http://fedora.bhs.mirrors.ovh.net/linux/updates/24/x86_64/re podata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: http://mirror.its.dal.ca/pub/fedora/linux/updates/24/x86_64/ repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: https://mirror.csclub.uwaterloo.ca/fedora/linux/updates/24/x 86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: One of the configured repositories failed (updates), DEBUG util.py:421: and yum doesn't have enough cached data to continue. At this point the only DEBUG util.py:421: safe thing yum can do is fail. There are a few ways to work "fix" this: DEBUG util.py:421: 1. Contact the upstream for the repository and get them to fix the problem. DEBUG util.py:421: 2. Reconfigure the baseurl/etc. for the repository, to point to a working DEBUG util.py:421: upstream. This is most often useful if you are using a newer DEBUG util.py:421: distribution release than is supported by the repository (and the DEBUG util.py:421: packages for the previous distribution release still work). DEBUG util.py:421: 3. Disable the repository, so yum won't use it by default. Yum will then DEBUG util.py:421: just ignore the repository until you permanently enable it again or use DEBUG util.py:421: --enablerepo for temporary usage: DEBUG util.py:421: yum-config-manager --disable updates DEBUG util.py:421: 4. Configure the failing repository to be skipped, if it is unavailable. DEBUG util.py:421: Note that yum will try to contact the repo. when it runs most commands, DEBUG util.py:421: so will have to try and fail each time (and thus. yum will be be much DEBUG util.py:421: slower). If it is a very temporary problem though, this is often a nice DEBUG util.py:421: compromise: DEBUG util.py:421: yum-config-manager --save --setopt=updates.skip_if_unavailable=true DEBUG util.py:421: failure: repodata/repomd.xml from updates: [Errno 256] No more mirrors to try. DEBUG util.py:421: http://mirror.math.princeton.edu/pub/fedora/linux/updates/24 /x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://linux.mirrors.es.net/fedora/updates/24/x86_64/repodata/repomd.xml : [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.cs.pitt.edu/fedora/linux/updates/24/x86_64/rep odata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirrors.mit.edu/fedora/linux/updates/24/x86_64/repoda ta/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://ftp.uci.edu/fedora/linux/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: https://mirror.chpc.utah.edu/pub/fedora/linux/updates/24/x86 _64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://repo.atlantic.net/fedora/linux/updates/24/x86_64/repo data/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.cs.princeton.edu/pub/mirrors/fedora/linux/upda tes/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.uoregon.edu/fedora/linux/updates/24/x86_64/rep odata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.cc.vt.edu/pub/fedora/linux/updates/24/x86_64/r epodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: https://mirrors.kernel.org/fedora/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: https://pubmirror2.math.uh.edu/fedora-buffet/fedora/linux/up dates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.nexcess.net/fedora/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.sjc02.svwh.net/fedora/updates/24/x86_64/repoda ta/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.symnds.com/distributions/fedora/linux/updates/ 24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Infra mailing list Infra@ovirt.org http://lists.phx.ovirt.org/mailman/listinfo/infra
-- Anton Marchukov Senior Software Engineer - RHEV CI - Red Hat
-- Anton Marchukov Senior Software Engineer - RHEV CI - Red Hat
On 12 December 2016 at 17:27, Anton Marchukov <amarchuk@redhat.com> wrote:
Actually Evgheni found the reason for that [1].
...
we verify repoproxy availability in mock_genconfig and if it is not available - use default configs instead of proxied.
You could've asked me, I explained exactly this on one of the threads that discuss these failures. Its not using the default configs though - just the none-proxies ones, those conf files are managed by us as well.
This essentially avoids all of our improvement patches. Guess that code is not needed anymore and can be safely removed.
Neah, its good to have around so we're resilient to proxy server failures. Since the check is based on the URL in the *-proxed.conf file, it means that when you change it to not use repoproxy, this check will not use it as well.
On Mon, Dec 12, 2016 at 4:47 PM, Barak Korren <bkorren@redhat.com> wrote:
You could've asked me, I explained exactly this on one of the threads that discuss these failures. Its not using the default configs though - just the none-proxies ones, those conf files are managed by us as well.
That's exactly why I am writing this e-mail. So thanks for your reply and sorry for missing that mail thread. Neah, its good to have around so we're resilient to proxy server failures.
Since the check is based on the URL in the *-proxed.conf file, it means that when you change it to not use repoproxy, this check will not use it as well.
I see. Maybe the best thing we can do is to remove metalink from non proxied configs and put there the same set of mirrors configuration we use in proxied, but just without proxy. This should remove that point of failure on Fedora side while keeping this mitigation for our proxy to be down. -- Anton Marchukov Senior Software Engineer - RHEV CI - Red Hat
participants (4)
-
Anton Marchukov -
Barak Korren -
Evgheni Dereveanchin -
Eyal Edri