Hi Eyal, This does not look like a mirror issue but a MetaLink one. MetaLink is a security plugin for yum [1] and I saw the same issue last week so MetaLink may have been broken at that time. Basically what it does is fetches a file like [2] and checks if the mirrors received from the mirrorlist are present there. Maybe it does some other checks, but the point of this is protection against stale/rogue mirrors. I think we should just disable MetaLink on our CI slaves as we don't rely on latest fedora software versions anyway.

Regards, Evgheni Dereveanchin [1] https://fedoraproject.org/wiki/Features/YumMetalinks [2] https://mirrors.fedoraproject.org/metalink?repo=fedora-24&arch=x86_64 ----- Original Message ----- From: "Eyal Edri" <eedri(a)redhat.com&gt; To: "infra" <infra(a)ovirt.org&gt;, "Evgheni Dereveanchin" <ederevea(a)redhat.com > Sent: Sunday, 11 December, 2016 1:27:53 PM Subject: mirrors errors Doesn't seem to be we're over the unavailable mirrors issue with the new repos added to STD CI: Lets start working on mirroring the relevant repos to mirrors.phx.ovirt.org and get control over the update process. http://jenkins.ovirt.org/job/jenkins_master_check-patch- fc24-x86_64/16/artifact/exported-artifacts/logs/ mocker-fedora-24-x86_64.fc24.init/root.log DEBUG util.py:421: https://mirrors.cat.pdx.edu/fedora/linux/updates/24/x86_ 64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: https://dl.fedoraproject.org/pub/fedora/linux/updates/24/ x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: http://fedora.bhs.mirrors.ovh.net/linux/updates/24/x86_64/ repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: http://mirror.its.dal.ca/pub/fedora/linux/updates/24/x86_ 64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: https://mirror.csclub.uwaterloo.ca/fedora/linux/ updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: Trying other mirror. DEBUG util.py:421: One of the configured repositories failed (updates), DEBUG util.py:421: and yum doesn't have enough cached data to continue. At this point the only DEBUG util.py:421: safe thing yum can do is fail. There are a few ways to work "fix" this: DEBUG util.py:421: 1. Contact the upstream for the repository and get them to fix the problem. DEBUG util.py:421: 2. Reconfigure the baseurl/etc. for the repository, to point to a working DEBUG util.py:421: upstream. This is most often useful if you are using a newer DEBUG util.py:421: distribution release than is supported by the repository (and the DEBUG util.py:421: packages for the previous distribution release still work). DEBUG util.py:421: 3. Disable the repository, so yum won't use it by default. Yum will then DEBUG util.py:421: just ignore the repository until you permanently enable it again or use DEBUG util.py:421: --enablerepo for temporary usage: DEBUG util.py:421: yum-config-manager --disable updates DEBUG util.py:421: 4. Configure the failing repository to be skipped, if it is unavailable. DEBUG util.py:421: Note that yum will try to contact the repo. when it runs most commands, DEBUG util.py:421: so will have to try and fail each time (and thus. yum will be be much DEBUG util.py:421: slower). If it is a very temporary problem though, this is often a nice DEBUG util.py:421: compromise: DEBUG util.py:421: yum-config-manager --save --setopt=updates.skip_if_unavailable=true DEBUG util.py:421: failure: repodata/repomd.xml from updates: [Errno 256] No more mirrors to try. DEBUG util.py:421: http://mirror.math.princeton.edu/pub/fedora/linux/updates/ 24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://linux.mirrors.es.net/fedora/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.cs.pitt.edu/fedora/linux/updates/24/x86_ 64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirrors.mit.edu/fedora/linux/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://ftp.uci.edu/fedora/linux/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: https://mirror.chpc.utah.edu/pub/fedora/linux/updates/24/ x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://repo.atlantic.net/fedora/linux/updates/24/x86_ 64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.cs.princeton.edu/pub/mirrors/fedora/linux/ updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.uoregon.edu/fedora/linux/updates/24/x86_ 64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.cc.vt.edu/pub/fedora/linux/updates/24/x86_ 64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: https://mirrors.kernel.org/fedora/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: https://pubmirror2.math.uh.edu/fedora-buffet/fedora/ linux/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.nexcess.net/fedora/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.sjc02.svwh.net/fedora/updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for updates DEBUG util.py:421: http://mirror.symnds.com/distributions/fedora/linux/ updates/24/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not -- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

Hello Eyal, All. I see that jobs (and probably all standard ci ones?) now uses default Fedora configs from mock_configs/fedora-24-x86_64.cfg [1] it relies on metalink that is central point that failed at that time, also it can generate random mirrorlist and hence effectively skips the list of mirrors that we select and I even not sure that it uses our squid proxy at all. There is no proxy configured at that mock config. The changes for mock were done at fedora-24-x86_64_proxied.cfg copies of mock configs and in fact the change for Fedora proxies were not yet merged [2] and even if it was merged the standrad ci will not use them since for some reason it now uses default config, so it even does not currently use our old reproxy solution for Fedora repos. Anybody have any insights of why is that? So my suggestion is to 1. Switch standrad ci back to our _proxied configs as that was before (or was not and we just did not note it???). 2. Merge the remaining patch for proxied config to use new proxy solution instead of reproproxy [2]. [1] https://gerrit.ovirt.org/gitweb?p=jenkins.git;a=blob;f= mock_configs/fedora-24-x86_64.cfg;h=7bdfabba003ee7de83aea4d0aad0be f0d6d747d8;hb=refs/heads/master [2] https://gerrit.ovirt.org/#/c/67595/ Anton. On Mon, Dec 12, 2016 at 10:54 AM, Eyal Edri <eedri(a)redhat.com&gt; wrote: > > > On Mon, Dec 12, 2016 at 11:47 AM, Evgheni Dereveanchin < > ederevea(a)redhat.com&gt; wrote: > >> Hi Eyal, >> >> This does not look like a mirror issue but a MetaLink one. >> MetaLink is a security plugin for yum [1] and I saw the same >> issue last week so MetaLink may have been broken at that time. >> >> Basically what it does is fetches a file like [2] and checks >> if the mirrors received from the mirrorlist are present there. >> Maybe it does some other checks, but the point of this is >> protection against stale/rogue mirrors. >> >> I think we should just disable MetaLink on our CI slaves as >> we don't rely on latest fedora software versions anyway. >> > > If this is indeed the case, lets open a ticket on this and track it. > > >> >> Regards, >> Evgheni Dereveanchin >> >> [1] https://fedoraproject.org/wiki/Features/YumMetalinks >> [2] https://mirrors.fedoraproject.org/metalink?repo=fedora-24&ar >> ch=x86_64 >> >> ----- Original Message ----- >> From: "Eyal Edri" <eedri(a)redhat.com&gt; >> To: "infra" <infra(a)ovirt.org&gt;, "Evgheni Dereveanchin" < >> ederevea(a)redhat.com&gt; >> Sent: Sunday, 11 December, 2016 1:27:53 PM >> Subject: mirrors errors >> >> Doesn't seem to be we're over the unavailable mirrors issue with the new >> repos added to STD CI: >> >> Lets start working on mirroring the relevant repos to >> mirrors.phx.ovirt.org >> and get control over the update process. >> >> http://jenkins.ovirt.org/job/jenkins_master_check-patch-fc24 >> -x86_64/16/artifact/exported-artifacts/logs/mocker-fedora-24 >> -x86_64.fc24.init/root.log >> >> DEBUG util.py:421: >> https://mirrors.cat.pdx.edu/fedora/linux/updates/24/x86_64/r >> epodata/repomd.xml: >> [Errno -1] repomd.xml does not match metalink for updates >> DEBUG util.py:421: Trying other mirror. >> DEBUG util.py:421: >> https://dl.fedoraproject.org/pub/fedora/linux/updates/24/x86 >> _64/repodata/repomd.xml: >> [Errno -1] repomd.xml does not match metalink for updates >> DEBUG util.py:421: Trying other mirror. >> DEBUG util.py:421: >> http://fedora.bhs.mirrors.ovh.net/linux/updates/24/x86_64/re >> podata/repomd.xml: >> [Errno -1] repomd.xml does not match metalink for updates >> DEBUG util.py:421: Trying other mirror. >> DEBUG util.py:421: >> http://mirror.its.dal.ca/pub/fedora/linux/updates/24/x86_64/ >> repodata/repomd.xml: >> [Errno -1] repomd.xml does not match metalink for updates >> DEBUG util.py:421: Trying other mirror. >> DEBUG util.py:421: >> https://mirror.csclub.uwaterloo.ca/fedora/linux/updates/24/x >> 86_64/repodata/repomd.xml: >> [Errno -1] repomd.xml does not match metalink for updates >> DEBUG util.py:421: Trying other mirror. >> DEBUG util.py:421: One of the configured repositories failed (updates), >> DEBUG util.py:421: and yum doesn't have enough cached data to >> continue. At this point the only >> DEBUG util.py:421: safe thing yum can do is fail. There are a few >> ways to work "fix" this: >> DEBUG util.py:421: 1. Contact the upstream for the repository >> and get them to fix the problem. >> DEBUG util.py:421: 2. Reconfigure the baseurl/etc. for the >> repository, to point to a working >> DEBUG util.py:421: upstream. This is most often useful if you >> are using a newer >> DEBUG util.py:421: distribution release than is supported by >> the repository (and the >> DEBUG util.py:421: packages for the previous distribution >> release still work). >> DEBUG util.py:421: 3. Disable the repository, so yum won't use >> it by default. Yum will then >> DEBUG util.py:421: just ignore the repository until you >> permanently enable it again or use >> DEBUG util.py:421: --enablerepo for temporary usage: >> DEBUG util.py:421: yum-config-manager --disable updates >> DEBUG util.py:421: 4. Configure the failing repository to be >> skipped, if it is unavailable. >> DEBUG util.py:421: Note that yum will try to contact the >> repo. when it runs most commands, >> DEBUG util.py:421: so will have to try and fail each time >> (and thus. yum will be be much >> DEBUG util.py:421: slower). If it is a very temporary problem >> though, this is often a nice >> DEBUG util.py:421: compromise: >> DEBUG util.py:421: yum-config-manager --save >> --setopt=updates.skip_if_unavailable=true >> DEBUG util.py:421: failure: repodata/repomd.xml from updates: [Errno >> 256] No more mirrors to try. >> DEBUG util.py:421: >> http://mirror.math.princeton.edu/pub/fedora/linux/updates/24 >> /x86_64/repodata/repomd.xml: >> [Errno -1] repomd.xml does not match metalink for updates >> DEBUG util.py:421: >> http://linux.mirrors.es.net/fedora/updates/24/x86_64/repodata/repomd.xml >> : >> [Errno -1] repomd.xml does not match metalink for updates >> DEBUG util.py:421: >> http://mirror.cs.pitt.edu/fedora/linux/updates/24/x86_64/rep >> odata/repomd.xml: >> [Errno -1] repomd.xml does not match metalink for updates >> DEBUG util.py:421: >> http://mirrors.mit.edu/fedora/linux/updates/24/x86_64/repoda >> ta/repomd.xml: >> [Errno -1] repomd.xml does not match metalink for updates >> DEBUG util.py:421: >> http://ftp.uci.edu/fedora/linux/updates/24/x86_64/repodata/repomd.xml: >> [Errno -1] repomd.xml does not match metalink for updates >> DEBUG util.py:421: >> https://mirror.chpc.utah.edu/pub/fedora/linux/updates/24/x86 >> _64/repodata/repomd.xml: >> [Errno -1] repomd.xml does not match metalink for updates >> DEBUG util.py:421: >> http://repo.atlantic.net/fedora/linux/updates/24/x86_64/repo >> data/repomd.xml: >> [Errno -1] repomd.xml does not match metalink for updates >> DEBUG util.py:421: >> http://mirror.cs.princeton.edu/pub/mirrors/fedora/linux/upda >> tes/24/x86_64/repodata/repomd.xml: >> [Errno -1] repomd.xml does not match metalink for updates >> DEBUG util.py:421: >> http://mirror.uoregon.edu/fedora/linux/updates/24/x86_64/rep >> odata/repomd.xml: >> [Errno -1] repomd.xml does not match metalink for updates >> DEBUG util.py:421: >> http://mirror.cc.vt.edu/pub/fedora/linux/updates/24/x86_64/r >> epodata/repomd.xml: >> [Errno -1] repomd.xml does not match metalink for updates >> DEBUG util.py:421: >> https://mirrors.kernel.org/fedora/updates/24/x86_64/repodata/repomd.xml: >> [Errno -1] repomd.xml does not match metalink for updates >> DEBUG util.py:421: >> https://pubmirror2.math.uh.edu/fedora-buffet/fedora/linux/up >> dates/24/x86_64/repodata/repomd.xml: >> [Errno -1] repomd.xml does not match metalink for updates >> DEBUG util.py:421: >> http://mirror.nexcess.net/fedora/updates/24/x86_64/repodata/repomd.xml: >> [Errno -1] repomd.xml does not match metalink for updates >> DEBUG util.py:421: >> http://mirror.sjc02.svwh.net/fedora/updates/24/x86_64/repoda >> ta/repomd.xml: >> [Errno -1] repomd.xml does not match metalink for updates >> DEBUG util.py:421: >> http://mirror.symnds.com/distributions/fedora/linux/updates/ >> 24/x86_64/repodata/repomd.xml: >> [Errno -1] repomd.xml does not >> >> >> >> -- >> Eyal Edri >> Associate Manager >> RHV DevOps >> EMEA ENG Virtualization R&D >> Red Hat Israel >> >> phone: +972-9-7692018 >> irc: eedri (on #tlv #rhev-dev #rhev-integ) >> > > > > -- > Eyal Edri > Associate Manager > RHV DevOps > EMEA ENG Virtualization R&D > Red Hat Israel > > phone: +972-9-7692018 > irc: eedri (on #tlv #rhev-dev #rhev-integ) > > _______________________________________________ > Infra mailing list > Infra(a)ovirt.org > http://lists.phx.ovirt.org/mailman/listinfo/infra > > -- Anton Marchukov Senior Software Engineer - RHEV CI - Red Hat

You could've asked me, I explained exactly this on one of the threads that discuss these failures. Its not using the default configs though - just the none-proxies ones, those conf files are managed by us as well.

Since the check is based on the URL in the *-proxed.conf file, it means that when you change it to not use repoproxy, this check will not use it as well.