<div dir="ltr">Hello Barak.<div><br></div><div>But why this should be handled on infra side? Was that infra code that produced two RPMs with same name and version and different content? If not then I would bug it against whoever code is creating such RPMs and it then should be rebuild with at least rpm release incremented and hence does not require cache invalidation.</div><div><br></div><div>Any reason we are not doing that?</div><div><br></div><div>Anton.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Dec 23, 2016 at 7:54 PM, Barak Korren <span dir="ltr"><<a href="mailto:bkorren@redhat.com" target="_blank">bkorren@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi infra team members!<br>
<br>
As you may know, we've recently changed out proxied Mock configuration<br>
so that the 'http_proxy' environment variable gets defined inside the<br>
Mock environment. This was in an effort to make 'pip', 'curl' and<br>
'wget' commands go through our PHX proxy. As it turns out, this also<br>
have unforeseen influence on yum tools.<br>
<br>
Now, when it come to yum, as it is used inside the mock environmet, we<br>
long has the proxied configuration hard-wiring it to use the proxy by<br>
setting it in "yum.conf". However, so far, yum tools (Such as<br>
reposync) that brought their own configuration, essentially bypassed<br>
the "yum.conf" file and hence were not using the proxy.<br>
<br>
Well, now it turns out that 'yum' and the derived tools also respect<br>
the 'http_proxy' environment variable [1]:<br>
<br>
10.2. Configuring Proxy Server Access for a Single User<br>
<br>
To enable proxy access for a specific user, add the lines in the example box<br>
below to the user's shell profile. For the default bash shell, the<br>
profile is<br>
the file ~/.bash_profile. The settings below enable yum to use the proxy<br>
server <a href="http://mycache.mydomain.com" rel="noreferrer" target="_blank">mycache.mydomain.com</a>, connecting to port 3128.<br>
<br>
# The Web proxy server used by this account<br>
http_proxy="<a href="http://mycache.mydomain.com:3128" rel="noreferrer" target="_blank">http://mycache.<wbr>mydomain.com:3128</a>"<br>
export http_proxy<br>
<br>
This is generally a good thing, but it can lead to formerly unexpected<br>
consequences.<br>
<br>
Case-to-point: The Lago job reposync failures of last Thursday (Dec 22nd, 2016).<br>
<br>
The root-cause behind the failures was that the<br>
"ovirt-web-ui-0.1.0-4.el7.<wbr>centos.x86_64.rpm" file was changed in the<br>
"ovirt-master-snapshot-static" repo. Updating an RPM file without<br>
changing the version or revision numbers breaks YUM`s rules and makes<br>
reposync choke. We already knew about this and actually had a<br>
work-around in the Lago code [2].<br>
<br>
We I came in Thursday morning, and saw reposync failing in all the<br>
Lago jobs, I just assumed that our work-around simply failed to work.<br>
My assumption was enforced by the fact that I was able to reproduce<br>
the issue by running 'reposync' manually on the Lago hosts, and also<br>
managed to rectify it by removing the offending from file the reposync<br>
cache. I spent the next few hours chasing down failing jobs and<br>
cleaning up the caches on the hosts they ran on. I took me a while to<br>
figure out that I was seeing the problem (Essentially, the older<br>
version of the package file) reappear on the same hosts over and over<br>
again!<br>
Wondering how could that be, and after ensuring the older package file<br>
was nowhere to be found on any of the repos the jobs were using, Me<br>
and Gal took a look at the Lago code to see if it could be causing the<br>
issue. Imagine our puzzlement when we realized the work-around code<br>
was doing _exactly_ what I was doing manually, and still somehow<br>
managed to make the very issue it was designed to solve reappear!<br>
Eventually the problem seemed to disappear on its own. Now, armed with<br>
the knowledge above I can provide a plausible explanation to what we<br>
were seeing.<br>
The difference between my manual executions of 'reposync' and the way<br>
Lago was running it was that Lago was running within Mock, where<br>
'http_proxy' was defined. What was probably happening is that reposync<br>
kept getting the old RPM file from the proxy while still getting a<br>
newer yum metadate file.<br>
<br>
Conclusion - The next time such an issue arises, we must make sure to<br>
clear the PHX proxy cache, there is actually no need to clear the<br>
cache on the Lago hosts themselves, because our work-around will<br>
resolve the issue there. Longer term we may configure the proxy to not<br>
cache files coming from <a href="http://resources.ovirt.org" rel="noreferrer" target="_blank">resources.ovirt.org</a>.<br>
<br>
<br>
[1]: <a href="https://www.centos.org/docs/5/html/yum/sn-yum-proxy-server.html" rel="noreferrer" target="_blank">https://www.centos.org/docs/5/<wbr>html/yum/sn-yum-proxy-server.<wbr>html</a><br>
[2]: <a href="https://github.com/lago-project/lago/blob/master/ovirtlago/reposetup.py#L141-L153" rel="noreferrer" target="_blank">https://github.com/lago-<wbr>project/lago/blob/master/<wbr>ovirtlago/reposetup.py#L141-<wbr>L153</a><br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
Barak Korren<br>
<a href="mailto:bkorren@redhat.com">bkorren@redhat.com</a><br>
RHCE, RHCi, RHV-DevOps Team<br>
<a href="https://ifireball.wordpress.com/" rel="noreferrer" target="_blank">https://ifireball.wordpress.<wbr>com/</a><br>
______________________________<wbr>_________________<br>
Infra mailing list<br>
<a href="mailto:Infra@ovirt.org">Infra@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/infra" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/infra</a><br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><span><font color="#888888"><div>Anton Marchukov<br>Senior Software Engineer - <span><span><font color="#888888"><span><span><font color="#888888"><span><span><font color="#888888">RHEV CI - </font></span></span></font></span></span>Red Hat</font></span></span><br><br></div></font></span></div></div></div></div>
</div>