[JIRA] (OVIRT-869) Investigate proxy errors from proxy.phx.ovirt.org

[Barak Korren (oVirt JIRA)]

    [ https://ovirt-jira.atlassian.net/browse/OVIRT-869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=23301#comment-23301 ] 

Barak Korren edited comment on OVIRT-869 at 11/27/16 10:50 AM:
---------------------------------------------------------------

The error seems to be a DNS error:
{code}
DEBUG util.py:421:  http://proxy.phx.ovirt.org:5000/centos-base/7/x86_64/Packages/unzip-6.0-15.el7.x86_64.rpm: [Errno 14] curl#5 - "Could not resolve proxy: proxy.phx.ovirt.org"
{code}

Port 5000 is where the python script is running no? So the Squid had an issue resolving the name of the very host it is running on? We should probably just resolve this forever by placing the name in {{/etc/hosts}} on the proxy server....

{quote}
Either use original repos if proxy is down or add watchdog / alerts for the proxy service.
{quote}
We have this:
# {{mock_runner.sh}} tries to contact proxy and uses un-proxyed configuration if it fails (This is why it has both {{\*.cnf}} and {{*-proxy.cnf}} files)
# There is a cron job watchdog running on the proxy itself and restarting it
# AFAIK we also have Icinga monitoring it.

Typically the failures we're seeing are because of failures in the origin mirror we are actually proxying (Fedora and EPEL seem to be notoriously non-atomic when updating their mirrors). The way to solve this is only to used fully-fledged mirrors instead of a proxy (But this takes more maintenance and more storage). 



was (Author: bkorren at redhat.com):
The error seems to be a DNS error:
{code}
DEBUG util.py:421:  http://proxy.phx.ovirt.org:5000/centos-base/7/x86_64/Packages/unzip-6.0-15.el7.x86_64.rpm: [Errno 14] curl#5 - "Could not resolve proxy: proxy.phx.ovirt.org"
{code}

Port 5000 is where the python script is running no? So the Squid had an issue resolving the name of the very host it is running on? We should probably just resolve this forever by placing the name in {{/etc/hosts}} on the proxy server....

{code}
Either use original repos if proxy is down or add watchdog / alerts for the proxy service.
{code}
We have this:
# {{mock_runner.sh}} tries to contact proxy and uses un-proxyed configuration if it fails (This is why it has both {{\*.cnf}} and {{*-proxy.cnf}} files)
# There is a cron job watchdog running on the proxy itself and restarting it
# AFAIK we also have Icinga monitoring it.

Typically the failures we're seeing are because of failures in the origin mirror we are actually proxying (Fedora and EPEL seem to be notoriously non-atomic when updating their mirrors). The way to solve this is only to used fully-fledged mirrors instead of a proxy (But this takes more maintenance and more storage). 


> Investigate proxy errors from proxy.phx.ovirt.org
> -------------------------------------------------
>
>                 Key: OVIRT-869
>                 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-869
>             Project: oVirt - virtualization made easy
>          Issue Type: Bug
>            Reporter: eyal edri [Administrator]
>            Assignee: infra
>            Priority: Highest
>
> We need to make sure jobs don't fail on proxy like [1].
> Either use original repos if proxy is down or add watchdog / alerts for the proxy service.
> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/3693/artifact/exported-artifacts/basic_suite_master.sh-el7/logs/mocker-epel-7-x86_64.el7.init/root.log



--
This message was sent by Atlassian JIRA
(v1000.571.2#100021)