<div dir="ltr">Actually Evgheni found the reason for that [1].<div><br></div><div><div class="gmail-pre" style="font-family:monospace;white-space:pre;color:rgb(0,0,0)"><a id="gmail-l132" href="https://gerrit.ovirt.org/gitweb?p=jenkins.git;a=blob;f=mock_configs/mock_genconfig;h=b15504a4139b21ce7df7d932b735ffcea3af0a55;hb=refs/heads/master#l132" class="gmail-linenr" style="color:rgb(153,153,153);text-decoration:none"> 132</a> if check_proxy(proxy_url, repo_url):</div><div class="gmail-pre" style="font-family:monospace;white-space:pre;color:rgb(0,0,0)"><a id="gmail-l133" href="https://gerrit.ovirt.org/gitweb?p=jenkins.git;a=blob;f=mock_configs/mock_genconfig;h=b15504a4139b21ce7df7d932b735ffcea3af0a55;hb=refs/heads/master#l133" class="gmail-linenr" style="color:rgb(153,153,153);text-decoration:none"> 133</a> base_config = proxied_config</div><div class="gmail-pre" style="font-family:monospace;white-space:pre;color:rgb(0,0,0)"><a id="gmail-l134" href="https://gerrit.ovirt.org/gitweb?p=jenkins.git;a=blob;f=mock_configs/mock_genconfig;h=b15504a4139b21ce7df7d932b735ffcea3af0a55;hb=refs/heads/master#l134" class="gmail-linenr" style="color:rgb(153,153,153);text-decoration:none"> 134</a> else:</div><div class="gmail-pre" style="font-family:monospace;white-space:pre;color:rgb(0,0,0)"><a id="gmail-l135" href="https://gerrit.ovirt.org/gitweb?p=jenkins.git;a=blob;f=mock_configs/mock_genconfig;h=b15504a4139b21ce7df7d932b735ffcea3af0a55;hb=refs/heads/master#l135" class="gmail-linenr" style="color:rgb(153,153,153);text-decoration:none"> 135</a> logging.warn("Not using proxied configuration")</div></div><br>we verify repoproxy availability in mock_genconfig and if it is not available - use default configs instead of proxied. This essentially avoids all of our improvement patches. Guess that code is not needed anymore and can be safely removed.<div><br><div><div>[1] <a href="https://gerrit.ovirt.org/gitweb?p=jenkins.git;a=blob;f=mock_configs/mock_genconfig;h=b15504a4139b21ce7df7d932b735ffcea3af0a55;hb=refs/heads/master#l132">https://gerrit.ovirt.org/gitweb?p=jenkins.git;a=blob;f=mock_configs/mock_genconfig;h=b15504a4139b21ce7df7d932b735ffcea3af0a55;hb=refs/heads/master#l132</a> </div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Dec 12, 2016 at 4:18 PM, Anton Marchukov <span dir="ltr"><<a href="mailto:amarchuk@redhat.com" target="_blank">amarchuk@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hello Eyal, All.<div><br></div><div>I see that jobs (and probably all standard ci ones?) now uses default Fedora configs from mock_configs/fedora-24-x86_64.<wbr>cfg [1]</div><div>it relies on metalink that is central point that failed at that time, also it can generate random mirrorlist and hence effectively skips the list of mirrors that we select and I even not sure that it uses our squid proxy at all. There is no proxy configured at that mock config.</div><div><br></div><div>The changes for mock were done at fedora-24-x86_64_proxied.cfg copies of mock configs and in fact the change for Fedora proxies were not yet merged [2] and even if it was merged the standrad ci will not use them since for some reason it now uses default config, so it even does not currently use our old reproxy solution for Fedora repos.</div><div><br></div><div>Anybody have any insights of why is that?</div><div><br></div><div>So my suggestion is to</div><div><br></div><div>1. Switch standrad ci back to our _proxied configs as that was before (or was not and we just did not note it???).</div><div>2. Merge the remaining patch for proxied config to use new proxy solution instead of reproproxy [2].</div><div><br></div><div>[1] <a href="https://gerrit.ovirt.org/gitweb?p=jenkins.git;a=blob;f=mock_configs/fedora-24-x86_64.cfg;h=7bdfabba003ee7de83aea4d0aad0bef0d6d747d8;hb=refs/heads/master" target="_blank">https://gerrit.ovirt.org/<wbr>gitweb?p=jenkins.git;a=blob;f=<wbr>mock_configs/fedora-24-x86_64.<wbr>cfg;h=<wbr>7bdfabba003ee7de83aea4d0aad0be<wbr>f0d6d747d8;hb=refs/heads/<wbr>master</a></div><div>[2] <a href="https://gerrit.ovirt.org/#/c/67595/" target="_blank">https://gerrit.ovirt.org/#<wbr>/c/67595/</a></div><div><br></div><div>Anton.</div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Mon, Dec 12, 2016 at 10:54 AM, Eyal Edri <span dir="ltr"><<a href="mailto:eedri@redhat.com" target="_blank">eedri@redhat.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Mon, Dec 12, 2016 at 11:47 AM, Evgheni Dereveanchin <span dir="ltr"><<a href="mailto:ederevea@redhat.com" target="_blank">ederevea@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Eyal,<br>
<br>
This does not look like a mirror issue but a MetaLink one.<br>
MetaLink is a security plugin for yum [1] and I saw the same<br>
issue last week so MetaLink may have been broken at that time.<br>
<br>
Basically what it does is fetches a file like [2] and checks<br>
if the mirrors received from the mirrorlist are present there.<br>
Maybe it does some other checks, but the point of this is<br>
protection against stale/rogue mirrors.<br>
<br>
I think we should just disable MetaLink on our CI slaves as<br>
we don't rely on latest fedora software versions anyway.<br></blockquote><div><br></div></span><div>If this is indeed the case, lets open a ticket on this and track it.</div><div><div class="m_-3469960155461611802h5"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Regards,<br>
Evgheni Dereveanchin<br>
<br>
[1] <a href="https://fedoraproject.org/wiki/Features/YumMetalinks" rel="noreferrer" target="_blank">https://fedoraproject.org/wiki<wbr>/Features/YumMetalinks</a><br>
[2] <a href="https://mirrors.fedoraproject.org/metalink?repo=fedora-24&arch=x86_64" rel="noreferrer" target="_blank">https://mirrors.fedoraproject.<wbr>org/metalink?repo=fedora-24&ar<wbr>ch=x86_64</a><br>
<div class="m_-3469960155461611802m_-2671757763082516107HOEnZb"><div class="m_-3469960155461611802m_-2671757763082516107h5"><br>
----- Original Message -----<br>
From: "Eyal Edri" <<a href="mailto:eedri@redhat.com" target="_blank">eedri@redhat.com</a>><br>
To: "infra" <<a href="mailto:infra@ovirt.org" target="_blank">infra@ovirt.org</a>>, "Evgheni Dereveanchin" <<a href="mailto:ederevea@redhat.com" target="_blank">ederevea@redhat.com</a>><br>
Sent: Sunday, 11 December, 2016 1:27:53 PM<br>
Subject: mirrors errors<br>
<br>
Doesn't seem to be we're over the unavailable mirrors issue with the new<br>
repos added to STD CI:<br>
<br>
Lets start working on mirroring the relevant repos to <a href="http://mirrors.phx.ovirt.org" rel="noreferrer" target="_blank">mirrors.phx.ovirt.org</a><br>
and get control over the update process.<br>
<br>
<a href="http://jenkins.ovirt.org/job/jenkins_master_check-patch-fc24-x86_64/16/artifact/exported-artifacts/logs/mocker-fedora-24-x86_64.fc24.init/root.log" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/j<wbr>enkins_master_check-patch-fc24<wbr>-x86_64/16/artifact/exported-a<wbr>rtifacts/logs/mocker-fedora-24<wbr>-x86_64.fc24.init/root.log</a><br>
<br>
DEBUG util.py:421:<br>
<a href="https://mirrors.cat.pdx.edu/fedora/linux/updates/24/x86_64/repodata/repomd.xml" rel="noreferrer" target="_blank">https://mirrors.cat.pdx.edu/fe<wbr>dora/linux/updates/24/x86_64/r<wbr>epodata/repomd.xml</a>:<br>
[Errno -1] repomd.xml does not match metalink for updates<br>
DEBUG util.py:421: Trying other mirror.<br>
DEBUG util.py:421:<br>
<a href="https://dl.fedoraproject.org/pub/fedora/linux/updates/24/x86_64/repodata/repomd.xml" rel="noreferrer" target="_blank">https://dl.fedoraproject.org/p<wbr>ub/fedora/linux/updates/24/x86<wbr>_64/repodata/repomd.xml</a>:<br>
[Errno -1] repomd.xml does not match metalink for updates<br>
DEBUG util.py:421: Trying other mirror.<br>
DEBUG util.py:421:<br>
<a href="http://fedora.bhs.mirrors.ovh.net/linux/updates/24/x86_64/repodata/repomd.xml" rel="noreferrer" target="_blank">http://fedora.bhs.mirrors.ovh.<wbr>net/linux/updates/24/x86_64/re<wbr>podata/repomd.xml</a>:<br>
[Errno -1] repomd.xml does not match metalink for updates<br>
DEBUG util.py:421: Trying other mirror.<br>
DEBUG util.py:421:<br>
<a href="http://mirror.its.dal.ca/pub/fedora/linux/updates/24/x86_64/repodata/repomd.xml" rel="noreferrer" target="_blank">http://mirror.its.dal.ca/pub/f<wbr>edora/linux/updates/24/x86_64/<wbr>repodata/repomd.xml</a>:<br>
[Errno -1] repomd.xml does not match metalink for updates<br>
DEBUG util.py:421: Trying other mirror.<br>
DEBUG util.py:421:<br>
<a href="https://mirror.csclub.uwaterloo.ca/fedora/linux/updates/24/x86_64/repodata/repomd.xml" rel="noreferrer" target="_blank">https://mirror.csclub.uwaterlo<wbr>o.ca/fedora/linux/updates/24/x<wbr>86_64/repodata/repomd.xml</a>:<br>
[Errno -1] repomd.xml does not match metalink for updates<br>
DEBUG util.py:421: Trying other mirror.<br>
DEBUG util.py:421: One of the configured repositories failed (updates),<br>
DEBUG util.py:421: and yum doesn't have enough cached data to<br>
continue. At this point the only<br>
DEBUG util.py:421: safe thing yum can do is fail. There are a few<br>
ways to work "fix" this:<br>
DEBUG util.py:421: 1. Contact the upstream for the repository<br>
and get them to fix the problem.<br>
DEBUG util.py:421: 2. Reconfigure the baseurl/etc. for the<br>
repository, to point to a working<br>
DEBUG util.py:421: upstream. This is most often useful if you<br>
are using a newer<br>
DEBUG util.py:421: distribution release than is supported by<br>
the repository (and the<br>
DEBUG util.py:421: packages for the previous distribution<br>
release still work).<br>
DEBUG util.py:421: 3. Disable the repository, so yum won't use<br>
it by default. Yum will then<br>
DEBUG util.py:421: just ignore the repository until you<br>
permanently enable it again or use<br>
DEBUG util.py:421: --enablerepo for temporary usage:<br>
DEBUG util.py:421: yum-config-manager --disable updates<br>
DEBUG util.py:421: 4. Configure the failing repository to be<br>
skipped, if it is unavailable.<br>
DEBUG util.py:421: Note that yum will try to contact the<br>
repo. when it runs most commands,<br>
DEBUG util.py:421: so will have to try and fail each time<br>
(and thus. yum will be be much<br>
DEBUG util.py:421: slower). If it is a very temporary problem<br>
though, this is often a nice<br>
DEBUG util.py:421: compromise:<br>
DEBUG util.py:421: yum-config-manager --save<br>
--setopt=updates.skip_if_unava<wbr>ilable=true<br>
DEBUG util.py:421: failure: repodata/repomd.xml from updates: [Errno<br>
256] No more mirrors to try.<br>
DEBUG util.py:421:<br>
<a href="http://mirror.math.princeton.edu/pub/fedora/linux/updates/24/x86_64/repodata/repomd.xml" rel="noreferrer" target="_blank">http://mirror.math.princeton.e<wbr>du/pub/fedora/linux/updates/24<wbr>/x86_64/repodata/repomd.xml</a>:<br>
[Errno -1] repomd.xml does not match metalink for updates<br>
DEBUG util.py:421:<br>
<a href="http://linux.mirrors.es.net/fedora/updates/24/x86_64/repodata/repomd.xml" rel="noreferrer" target="_blank">http://linux.mirrors.es.net/fe<wbr>dora/updates/24/x86_64/repodat<wbr>a/repomd.xml</a>:<br>
[Errno -1] repomd.xml does not match metalink for updates<br>
DEBUG util.py:421:<br>
<a href="http://mirror.cs.pitt.edu/fedora/linux/updates/24/x86_64/repodata/repomd.xml" rel="noreferrer" target="_blank">http://mirror.cs.pitt.edu/fedo<wbr>ra/linux/updates/24/x86_64/rep<wbr>odata/repomd.xml</a>:<br>
[Errno -1] repomd.xml does not match metalink for updates<br>
DEBUG util.py:421:<br>
<a href="http://mirrors.mit.edu/fedora/linux/updates/24/x86_64/repodata/repomd.xml" rel="noreferrer" target="_blank">http://mirrors.mit.edu/fedora/<wbr>linux/updates/24/x86_64/repoda<wbr>ta/repomd.xml</a>:<br>
[Errno -1] repomd.xml does not match metalink for updates<br>
DEBUG util.py:421:<br>
<a href="http://ftp.uci.edu/fedora/linux/updates/24/x86_64/repodata/repomd.xml" rel="noreferrer" target="_blank">http://ftp.uci.edu/fedora/linu<wbr>x/updates/24/x86_64/repodata/r<wbr>epomd.xml</a>:<br>
[Errno -1] repomd.xml does not match metalink for updates<br>
DEBUG util.py:421:<br>
<a href="https://mirror.chpc.utah.edu/pub/fedora/linux/updates/24/x86_64/repodata/repomd.xml" rel="noreferrer" target="_blank">https://mirror.chpc.utah.edu/p<wbr>ub/fedora/linux/updates/24/x86<wbr>_64/repodata/repomd.xml</a>:<br>
[Errno -1] repomd.xml does not match metalink for updates<br>
DEBUG util.py:421:<br>
<a href="http://repo.atlantic.net/fedora/linux/updates/24/x86_64/repodata/repomd.xml" rel="noreferrer" target="_blank">http://repo.atlantic.net/fedor<wbr>a/linux/updates/24/x86_64/repo<wbr>data/repomd.xml</a>:<br>
[Errno -1] repomd.xml does not match metalink for updates<br>
DEBUG util.py:421:<br>
<a href="http://mirror.cs.princeton.edu/pub/mirrors/fedora/linux/updates/24/x86_64/repodata/repomd.xml" rel="noreferrer" target="_blank">http://mirror.cs.princeton.edu<wbr>/pub/mirrors/fedora/linux/upda<wbr>tes/24/x86_64/repodata/repomd.<wbr>xml</a>:<br>
[Errno -1] repomd.xml does not match metalink for updates<br>
DEBUG util.py:421:<br>
<a href="http://mirror.uoregon.edu/fedora/linux/updates/24/x86_64/repodata/repomd.xml" rel="noreferrer" target="_blank">http://mirror.uoregon.edu/fedo<wbr>ra/linux/updates/24/x86_64/rep<wbr>odata/repomd.xml</a>:<br>
[Errno -1] repomd.xml does not match metalink for updates<br>
DEBUG util.py:421:<br>
<a href="http://mirror.cc.vt.edu/pub/fedora/linux/updates/24/x86_64/repodata/repomd.xml" rel="noreferrer" target="_blank">http://mirror.cc.vt.edu/pub/fe<wbr>dora/linux/updates/24/x86_64/r<wbr>epodata/repomd.xml</a>:<br>
[Errno -1] repomd.xml does not match metalink for updates<br>
DEBUG util.py:421:<br>
<a href="https://mirrors.kernel.org/fedora/updates/24/x86_64/repodata/repomd.xml" rel="noreferrer" target="_blank">https://mirrors.kernel.org/fed<wbr>ora/updates/24/x86_64/repodata<wbr>/repomd.xml</a>:<br>
[Errno -1] repomd.xml does not match metalink for updates<br>
DEBUG util.py:421:<br>
<a href="https://pubmirror2.math.uh.edu/fedora-buffet/fedora/linux/updates/24/x86_64/repodata/repomd.xml" rel="noreferrer" target="_blank">https://pubmirror2.math.uh.edu<wbr>/fedora-buffet/fedora/linux/up<wbr>dates/24/x86_64/repodata/repom<wbr>d.xml</a>:<br>
[Errno -1] repomd.xml does not match metalink for updates<br>
DEBUG util.py:421:<br>
<a href="http://mirror.nexcess.net/fedora/updates/24/x86_64/repodata/repomd.xml" rel="noreferrer" target="_blank">http://mirror.nexcess.net/fedo<wbr>ra/updates/24/x86_64/repodata/<wbr>repomd.xml</a>:<br>
[Errno -1] repomd.xml does not match metalink for updates<br>
DEBUG util.py:421:<br>
<a href="http://mirror.sjc02.svwh.net/fedora/updates/24/x86_64/repodata/repomd.xml" rel="noreferrer" target="_blank">http://mirror.sjc02.svwh.net/f<wbr>edora/updates/24/x86_64/repoda<wbr>ta/repomd.xml</a>:<br>
[Errno -1] repomd.xml does not match metalink for updates<br>
DEBUG util.py:421:<br>
<a href="http://mirror.symnds.com/distributions/fedora/linux/updates/24/x86_64/repodata/repomd.xml" rel="noreferrer" target="_blank">http://mirror.symnds.com/distr<wbr>ibutions/fedora/linux/updates/<wbr>24/x86_64/repodata/repomd.xml</a>:<br>
[Errno -1] repomd.xml does not<br>
<br>
<br>
<br>
--<br>
Eyal Edri<br>
Associate Manager<br>
RHV DevOps<br>
EMEA ENG Virtualization R&D<br>
Red Hat Israel<br>
<br>
phone: <a href="tel:%2B972-9-7692018" value="+97297692018" target="_blank">+972-9-7692018</a><br>
irc: eedri (on #tlv #rhev-dev #rhev-integ)<br>
</div></div></blockquote></div></div></div><div><div class="m_-3469960155461611802h5"><br><br clear="all"><div><br></div>-- <br><div class="m_-3469960155461611802m_-2671757763082516107gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Eyal Edri<br>Associate Manager</div><div>RHV DevOps<br>EMEA ENG Virtualization R&D<br>Red Hat Israel<br><br>phone: <a href="tel:%2B972-9-7692018" value="+97297692018" target="_blank">+972-9-7692018</a><br>irc: eedri (on #tlv #rhev-dev #rhev-integ)</div></div></div></div></div></div></div>
</div></div></div></div>
<br></div></div><span class="">______________________________<wbr>_________________<br>
Infra mailing list<br>
<a href="mailto:Infra@ovirt.org" target="_blank">Infra@ovirt.org</a><br>
<a href="http://lists.phx.ovirt.org/mailman/listinfo/infra" rel="noreferrer" target="_blank">http://lists.phx.ovirt.org/mai<wbr>lman/listinfo/infra</a><br>
<br></span></blockquote></div><span class="HOEnZb"><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div class="m_-3469960155461611802gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><span><font color="#888888"><div>Anton Marchukov<br>Senior Software Engineer - <span><span><font color="#888888"><span><span><font color="#888888"><span><span><font color="#888888">RHEV CI - </font></span></span></font></span></span>Red Hat</font></span></span><br><br></div></font></span></div></div></div></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><span><font color="#888888"><div>Anton Marchukov<br>Senior Software Engineer - <span><span><font color="#888888"><span><span><font color="#888888"><span><span><font color="#888888">RHEV CI - </font></span></span></font></span></span>Red Hat</font></span></span><br><br></div></font></span></div></div></div></div>
</div>