Heads up! Influence of our recent Mock/Proxy changes on Lago jobs

[Anton Marchukov]

Hello Barak.

But why this should be handled on infra side? Was that infra code that
produced two RPMs with same name and version and different content? If not
then I would bug it against whoever code is creating such RPMs and it then
should be rebuild with at least rpm release incremented and hence does not
require cache invalidation.

Any reason we are not doing that?

Anton.

On Fri, Dec 23, 2016 at 7:54 PM, Barak Korren <bkorren at redhat.com> wrote:

> Hi infra team members!
>
> As you may know, we've recently changed out proxied Mock configuration
> so that the 'http_proxy' environment variable gets defined inside the
> Mock environment. This was in an effort to make 'pip', 'curl' and
> 'wget' commands go through our PHX proxy. As it turns out, this also
> have unforeseen influence on yum tools.
>
> Now, when it come to yum, as it is used inside the mock environmet, we
> long has the proxied configuration hard-wiring it to use the proxy by
> setting it in "yum.conf". However, so far, yum tools (Such as
> reposync) that brought their own configuration, essentially bypassed
> the "yum.conf" file and hence were not using the proxy.
>
> Well, now it turns out that 'yum' and the derived tools also respect
> the 'http_proxy' environment variable [1]:
>
>     10.2. Configuring Proxy Server Access for a Single User
>
>     To enable proxy access for a specific user, add the lines in the
> example box
>     below to the user's shell profile. For the default bash shell, the
> profile is
>     the file ~/.bash_profile. The settings below enable yum to use the
> proxy
>     server mycache.mydomain.com, connecting to port 3128.
>
>         # The Web proxy server used by this account
>         http_proxy="http://mycache.mydomain.com:3128"
>         export http_proxy
>
> This is generally a good thing, but it can lead to formerly unexpected
> consequences.
>
> Case-to-point: The Lago job reposync failures of last Thursday (Dec 22nd,
> 2016).
>
> The root-cause behind the failures was that the
> "ovirt-web-ui-0.1.0-4.el7.centos.x86_64.rpm" file was changed in the
> "ovirt-master-snapshot-static" repo. Updating an RPM file without
> changing the version or revision numbers breaks YUM`s rules and makes
> reposync choke. We already knew about this and actually had a
> work-around in the Lago code [2].
>
> We I came in Thursday morning, and saw reposync failing in all the
> Lago jobs, I just assumed that our work-around simply failed to work.
> My assumption was enforced by the fact that I was able to reproduce
> the issue by running 'reposync' manually on the Lago hosts, and also
> managed to rectify it by removing the offending from file the reposync
> cache. I spent the next few hours chasing down failing jobs and
> cleaning up the caches on the hosts they ran on. I took me a while to
> figure out that I was seeing the problem (Essentially, the older
> version of the package file) reappear on the same hosts over and over
> again!
> Wondering how could that be, and after ensuring the older package file
> was nowhere to be found on any of the repos the jobs were using, Me
> and Gal took a look at the Lago code to see if it could be causing the
> issue. Imagine our puzzlement when we realized the work-around code
> was doing _exactly_ what I was doing manually, and still somehow
> managed to make the very issue it was designed to solve reappear!
> Eventually the problem seemed to disappear on its own. Now, armed with
> the knowledge above I can provide a plausible explanation to what we
> were seeing.
> The difference between my manual executions of 'reposync' and the way
> Lago was running it was that Lago was running within Mock, where
> 'http_proxy' was defined. What was probably happening is that reposync
> kept getting the old RPM file from the proxy while still getting a
> newer yum metadate file.
>
> Conclusion - The next time such an issue arises, we must make sure to
> clear the PHX proxy cache, there is actually no need to clear the
> cache on the Lago hosts themselves, because our work-around will
> resolve the issue there. Longer term we may configure the proxy to not
> cache files coming from resources.ovirt.org.
>
>
> [1]: https://www.centos.org/docs/5/html/yum/sn-yum-proxy-server.html
> [2]: https://github.com/lago-project/lago/blob/master/
> ovirtlago/reposetup.py#L141-L153
>
> --
> Barak Korren
> bkorren at redhat.com
> RHCE, RHCi, RHV-DevOps Team
> https://ifireball.wordpress.com/
> _______________________________________________
> Infra mailing list
> Infra at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>



-- 
Anton Marchukov
Senior Software Engineer - RHEV CI - Red Hat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20161223/535d035b/attachment.html>