Build failed in Jenkins: ovirt_4.0_he-system-tests #560
by jenkins＠jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/560/changes>
Changes:
[Eyal Edri] adding missing python-simplejson to epel repo
------------------------------------------
[...truncated 669 lines...]
## took 356 seconds
## rc = 1
##########################################################
##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
##! Last 20 log entries: logs/mocker-epel-7-x86_64.el7.he_basic_suite_4.0.sh/he_basic_suite_4.0.sh.log
##!
+ env_cleanup
+ echo '#########################'
#########################
+ local res=0
+ local uuid
+ echo '======== Cleaning up'
======== Cleaning up
+ [[ -e <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-te...> ]]
+ echo '----------- Cleaning with lago'
----------- Cleaning with lago
+ lago --workdir <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-te...> destroy --yes --all-prefixes
+ echo '----------- Cleaning with lago done'
----------- Cleaning with lago done
+ [[ 0 != \0 ]]
+ echo '======== Cleanup done'
======== Cleanup done
+ exit 0
+ exit
Took 205 seconds
===================================
##!
##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
##!########################################################
##########################################################
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for :.* : True
Logical operation result is TRUE
Running script : #!/bin/bash -xe
echo 'shell_scripts/system_tests.collect_logs.sh'
#
# Required jjb vars:
# version
#
VERSION=4.0
SUITE_TYPE=
WORKSPACE="$PWD"
OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION"
TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts"
rm -rf "$WORKSPACE/exported-artifacts"
mkdir -p "$WORKSPACE/exported-artifacts"
if [[ -d "$TESTS_LOGS" ]]; then
mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/"
fi
[ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson5732644382445829429.sh
+ echo shell_scripts/system_tests.collect_logs.sh
shell_scripts/system_tests.collect_logs.sh
+ VERSION=4.0
+ SUITE_TYPE=
+ WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/>
+ OVIRT_SUITE=4.0
+ TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-te...>
+ rm -rf <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/560/artifact/expor...>
+ mkdir -p <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/560/artifact/expor...>
+ [[ -d <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-te...> ]]
+ mv <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-te...> <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/560/artifact/expor...>
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Match found for :.* : True
Logical operation result is TRUE
Running script : #!/bin/bash -xe
echo "shell-scripts/mock_cleanup.sh"
shopt -s nullglob
WORKSPACE="$PWD"
# Make clear this is the cleanup, helps reading the jenkins logs
cat <<EOC
_______________________________________________________________________
#######################################################################
# #
# CLEANUP #
# #
#######################################################################
EOC
# Archive the logs, we want them anyway
logs=(
./*log
./*/logs
)
if [[ "$logs" ]]; then
for log in "${logs[@]}"
do
echo "Copying ${log} to exported-artifacts"
mv $log exported-artifacts/
done
fi
# stop any processes running inside the chroot
failed=false
mock_confs=("$WORKSPACE"/*/mocker*)
# Clean current jobs mockroot if any
for mock_conf_file in "${mock_confs[@]}"; do
[[ "$mock_conf_file" ]] || continue
echo "Cleaning up mock $mock_conf"
mock_root="${mock_conf_file##*/}"
mock_root="${mock_root%.*}"
my_mock="/usr/bin/mock"
my_mock+=" --configdir=${mock_conf_file%/*}"
my_mock+=" --root=${mock_root}"
my_mock+=" --resultdir=$WORKSPACE"
#TODO: investigate why mock --clean fails to umount certain dirs sometimes,
#so we can use it instead of manually doing all this.
echo "Killing all mock orphan processes, if any."
$my_mock \
--orphanskill \
|| {
echo "ERROR: Failed to kill orphans on $chroot."
failed=true
}
mock_root="$(\
grep \
-Po "(?<=config_opts\['root'\] = ')[^']*" \
"$mock_conf_file" \
)" || :
[[ "$mock_root" ]] || continue
mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || :
if [[ "$mounts" ]]; then
echo "Found mounted dirs inside the chroot $chroot. Trying to umount."
fi
for mount in "${mounts[@]}"; do
sudo umount --lazy "$mount" \
|| {
echo "ERROR: Failed to umount $mount."
failed=true
}
done
done
# Clean any leftover chroot from other jobs
for mock_root in /var/lib/mock/*; do
this_chroot_failed=false
mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || :
if [[ "$mounts" ]]; then
echo "Found mounted dirs inside the chroot $mock_root." \
"Trying to umount."
fi
for mount in "${mounts[@]}"; do
sudo umount --lazy "$mount" \
|| {
echo "ERROR: Failed to umount $mount."
failed=true
this_chroot_failed=true
}
done
if ! $this_chroot_failed; then
sudo rm -rf "$mock_root"
fi
done
if $failed; then
echo "Aborting."
exit 1
fi
# remove mock system cache, we will setup proxies to do the caching and this
# takes lots of space between runs
shopt -u nullglob
sudo rm -Rf /var/cache/mock/*
# restore the permissions in the working dir, as sometimes it leaves files
# owned by root and then the 'cleanup workspace' from jenkins job fails to
# clean and breaks the jobs
sudo chown -R "$USER" "$WORKSPACE"
# Drop all left over libvirt domains
for UUID in $(virsh list --all --uuid); do
virsh destroy $UUID || :
sleep 2
virsh undefine --remove-all-storage --storage vda --snapshots-metadata $UUID || :
done
[ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson7475623662613716644.sh
+ echo shell-scripts/mock_cleanup.sh
shell-scripts/mock_cleanup.sh
+ shopt -s nullglob
+ WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/>
+ cat
_______________________________________________________________________
#######################################################################
# #
# CLEANUP #
# #
#######################################################################
+ logs=(./*log ./*/logs)
+ [[ -n ./ovirt-system-tests/logs ]]
+ for log in '"${logs[@]}"'
+ echo 'Copying ./ovirt-system-tests/logs to exported-artifacts'
Copying ./ovirt-system-tests/logs to exported-artifacts
+ mv ./ovirt-system-tests/logs exported-artifacts/
+ failed=false
+ mock_confs=("$WORKSPACE"/*/mocker*)
+ for mock_conf_file in '"${mock_confs[@]}"'
+ [[ -n <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-te...> ]]
+ echo 'Cleaning up mock '
Cleaning up mock
+ mock_root=mocker-epel-7-x86_64.el7.cfg
+ mock_root=mocker-epel-7-x86_64.el7
+ my_mock=/usr/bin/mock
+ my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests'>
+ my_mock+=' --root=mocker-epel-7-x86_64.el7'
+ my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/'>
+ echo 'Killing all mock orphan processes, if any.'
Killing all mock orphan processes, if any.
+ /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests> --root=mocker-epel-7-x86_64.el7 --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> --orphanskill
WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-te....> Using default...
INFO: mock.py version 1.2.21 starting (python version = 3.5.1)...
Start: init plugins
INFO: selinux enabled
Finish: init plugins
Start: run
Finish: run
++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-te...>
+ mock_root=epel-7-x86_64-0c362156a2fa4a935ea8b988eb73b2ad
+ [[ -n epel-7-x86_64-0c362156a2fa4a935ea8b988eb73b2ad ]]
+ mounts=($(mount | awk '{print $3}' | grep "$mock_root"))
++ mount
++ awk '{print $3}'
++ grep epel-7-x86_64-0c362156a2fa4a935ea8b988eb73b2ad
+ :
+ [[ -n '' ]]
+ false
+ shopt -u nullglob
+ sudo rm -Rf /var/cache/mock/epel-7-x86_64-0c362156a2fa4a935ea8b988eb73b2ad
+ sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/>
++ virsh list --all --uuid
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 1
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?
Archiving artifacts
7 years, 5 months
[JIRA] (OVIRT-867) Re: [oVirt Jenkins] repos_3.6_check-closure_el7_merged - Build # 92 - Still Failing!
by eyal edri [Administrator] (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-867?page=com.atlassian.jira... ]
eyal edri [Administrator] commented on OVIRT-867:
-------------------------------------------------
Evgheni,
Looks like we're hitting too much issues lately with external repos (
epel/centos/fedora ),
I think its time we'll consider having a mirror for what we need on
resources.ovirt.org to reduce the failures ( we need to ensure we have
enough space on it as well )
On Fri, Nov 25, 2016 at 5:13 PM, Sandro Bonazzola <sbonazzo(a)redhat.com>
wrote:
>
>
> On Fri, Nov 25, 2016 at 3:43 PM, Evgheni Dereveanchin <ederevea(a)redhat.com
> > wrote:
>
>> After more checks - indeed this looks like an inconsistency
>> on the mirror:
>>
>> http://mirror.switch.ch/ftp/mirror/epel/7/x86_64/repodata/
>>
>> 1ef1290202fb9e09f1aa15efccea3456913df8657cc0e23a1df61d9ec18
>> e5659-comps-epel7.xml.xz
>> ab638ffbf7ef7e7f41b8f630113ff6bd251612eefa570b8725b612500d8
>> c90fc-comps-epel7.xml
>> repomd.xml
>>
>> there is no primary.sqlite.xz or filelists.xml.gz in this directory
>> at the moment, hence the failures. As said, this mirror is hardcoded
>> in repo_closure_check.sh so changing that to a working mirror will
>> fix the issue, as well as waiting till mirror.switch.ch gets sync'ed.
>>
>> In fact, I don't know why we are using a Swiss mirror for tests that
>> are run on US servers.
>>
>>
> feel free to change mirror or use a local one
>
>
>
>> Regards,
>> Evgheni Dereveanchin
>>
>> ----- Original Message -----
>> From: "Evgheni Dereveanchin" <ederevea(a)redhat.com>
>> To: "Sandro Bonazzola" <sbonazzo(a)redhat.com>
>> Cc: "infra" <infra(a)ovirt.org>, infra-support(a)ovirt.org, "Yedidyah Bar
>> David" <didi(a)redhat.com>, "Eyal Edri" <eedri(a)redhat.com>
>> Sent: Friday, 25 November, 2016 3:10:47 PM
>> Subject: Re: [oVirt Jenkins] repos_3.6_check-closure_el7_merged - Build
>> # 92 - Still Failing!
>>
>> Hi Sandro,
>>
>> From first glance this may be caused by an upstream EPEL mirror,
>> especially given the 404 errors:
>>
>> 12:07:12 Added check-epel-el7 repo from http://mirror.switch.ch/ftp/mi
>> rror/epel/7/x86_64/
>> ...
>> 12:07:12 Reading in repository metadata - please wait....
>> 12:07:20 Can't download or revert repomd.xml for check-epel-el7
>> ...
>> 12:07:21 failure: repodata/9edce4d5e3c9437849fbb
>> bae7c2faa50a9b0326b968dd8faa66ece83984c23de-primary.sqlite.xz from
>> check-epel-el7: [Errno 256] No more mirrors to try.
>> 12:07:21 http://mirror.switch.ch/ftp/mirror/epel/7/x86_64/repodata/9e
>> dce4d5e3c9437849fbbbae7c2faa50a9b0326b968dd8faa66ece83984c23
>> de-primary.sqlite.xz: [Errno 14] HTTP Error 404 - Not Found
>>
>> This mirror is hardcoded in repo_closure_check.sh and looks
>> fine when accessed by browser. I will do some more investigation
>> to see if some stale metadata may be causing this, however I've
>> seen the failure on several different slaves.
>>
>> Regards,
>> Evgheni Dereveanchin
>>
>> ----- Original Message -----
>> From: "Sandro Bonazzola" <sbonazzo(a)redhat.com>
>> To: "infra" <infra(a)ovirt.org>, infra-support(a)ovirt.org, "Evgheni
>> Dereveanchin" <ederevea(a)redhat.com>
>> Cc: "Yedidyah Bar David" <didi(a)redhat.com>, "Eyal Edri" <eedri(a)redhat.com
>> >
>> Sent: Friday, 25 November, 2016 2:41:02 PM
>> Subject: Re: [oVirt Jenkins] repos_3.6_check-closure_el7_merged - Build
>> # 92 - Still Failing!
>>
>> *00:01:53.955* Can't download or revert repomd.xml for
>> check-epel-el7*00:01:53.955* Some dependencies may not be complete for
>> this repository*00:01:53.956* Run as root to get all dependencies or
>> use -t to enable a user temp cache
>>
>>
>> Can you check? looks like it's not possible to update local cache of
>> metadata.
>>
>>
>>
>> On Fri, Nov 25, 2016 at 1:07 PM, <jenkins(a)jenkins.phx.ovirt.org> wrote:
>>
>> > Project: http://jenkins.ovirt.org/job/repos_3.6_check-closure_el7_mer
>> ged/
>> > Build: http://jenkins.ovirt.org/job/repos_3.6_check-closure_el7_mer
>> ged/92/
>> > Build Number: 92
>> > Build Status: Still Failing
>> > Triggered By: Started by user Sandro Bonazzola
>> >
>> > -------------------------------------
>> > Changes Since Last Success:
>> > -------------------------------------
>> > Changes for Build #89
>> > [Eyal Edri] deploy ovirt-engine-cli 3.6 to 4.0 and master repos as well
>> >
>> > [Yedidyah Bar David] master_upgrade_from_master: Upgrade to self instead
>> > of snapshot
>> >
>> >
>> > Changes for Build #90
>> > No changes
>> >
>> > Changes for Build #91
>> > No changes
>> >
>> > Changes for Build #92
>> > No changes
>> >
>> >
>> >
>> > -----------------
>> > Failed Tests:
>> > -----------------
>> > No tests ran.
>> >
>> >
>> > _______________________________________________
>> > Infra mailing list
>> > Infra(a)ovirt.org
>> > http://lists.ovirt.org/mailman/listinfo/infra
>> >
>> >
>>
>>
>> --
>> Sandro Bonazzola
>> Better technology. Faster innovation. Powered by community collaboration.
>> See how it works at redhat.com
>>
>
>
>
> --
> Sandro Bonazzola
> Better technology. Faster innovation. Powered by community collaboration.
> See how it works at redhat.com
>
--
Eyal Edri
Associate Manager
RHV DevOps
EMEA ENG Virtualization R&D
Red Hat Israel
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
> Re: [oVirt Jenkins] repos_3.6_check-closure_el7_merged - Build # 92 - Still Failing!
> ------------------------------------------------------------------------------------
>
> Key: OVIRT-867
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-867
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: sbonazzo
> Assignee: infra
>
> *00:01:53.955* Can't download or revert repomd.xml for
> check-epel-el7*00:01:53.955* Some dependencies may not be complete for
> this repository*00:01:53.956* Run as root to get all dependencies or
> use -t to enable a user temp cache
> Can you check? looks like it's not possible to update local cache of metadata.
> On Fri, Nov 25, 2016 at 1:07 PM, <jenkins(a)jenkins.phx.ovirt.org> wrote:
> > Project: http://jenkins.ovirt.org/job/repos_3.6_check-closure_el7_merged/
> > Build: http://jenkins.ovirt.org/job/repos_3.6_check-closure_el7_merged/92/
> > Build Number: 92
> > Build Status: Still Failing
> > Triggered By: Started by user Sandro Bonazzola
> >
> > -------------------------------------
> > Changes Since Last Success:
> > -------------------------------------
> > Changes for Build #89
> > [Eyal Edri] deploy ovirt-engine-cli 3.6 to 4.0 and master repos as well
> >
> > [Yedidyah Bar David] master_upgrade_from_master: Upgrade to self instead
> > of snapshot
> >
> >
> > Changes for Build #90
> > No changes
> >
> > Changes for Build #91
> > No changes
> >
> > Changes for Build #92
> > No changes
> >
> >
> >
> > -----------------
> > Failed Tests:
> > -----------------
> > No tests ran.
> >
> >
> > _______________________________________________
> > Infra mailing list
> > Infra(a)ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/infra
> >
> >
> --
> Sandro Bonazzola
> Better technology. Faster innovation. Powered by community collaboration.
> See how it works at redhat.com
--
This message was sent by Atlassian JIRA
(v1000.571.2#100021)
7 years, 5 months
[JIRA] (OVIRT-869) Investigate proxy errors from proxy.phx.ovirt.org
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-869?page=com.atlassian.jira... ]
Barak Korren commented on OVIRT-869:
------------------------------------
{quote}
It currently has :127.0.0.1 proxy.phx.ovirt.org
Do you mean adding the public IP address as well?
{quote}
If it has this already then the error is strange indeed, all I can think of is that maybe the mock environment running the failed job was not configured correctly (Trying to use the proxy repo URLs without setting it up as an HTTP proxy _in addition to_ not having proper _DNS_ config. If it was justh the HTTP proxy setting I would expect to see a "_connection refused_" error rather then "_could not resolve_").
{quote}
By full fledged mirrors you mean have it on resources.ovirt.org ? will we need Katello to manage them or mirror with rsync will suffice?
{quote}
Not necessarily on resources.ovirt.org, we could (and maybe should) use a different server for that.
WRT the mirroring solution itself we have many options with varying degrees of automation, features and required effort:
# rsync (though not all upsreams may allow this kind of access)
# yum reposync
# repoman
# Pulp
# Artifactory
# Katello
The benefit of Katello would be in allowing us to control exactly which packages get tested by leveraging the work flow environments and the content views, but we could build comparable solutions with other tools.
> Investigate proxy errors from proxy.phx.ovirt.org
> -------------------------------------------------
>
> Key: OVIRT-869
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-869
> Project: oVirt - virtualization made easy
> Issue Type: Bug
> Reporter: eyal edri [Administrator]
> Assignee: infra
> Priority: Highest
>
> We need to make sure jobs don't fail on proxy like [1].
> Either use original repos if proxy is down or add watchdog / alerts for the proxy service.
> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/3693/art...
--
This message was sent by Atlassian JIRA
(v1000.571.2#100021)
7 years, 5 months
[JIRA] (OVIRT-869) Investigate proxy errors from proxy.phx.ovirt.org
by eyal edri [Administrator] (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-869?page=com.atlassian.jira... ]
eyal edri [Administrator] commented on OVIRT-869:
-------------------------------------------------
bq. Port 5000 is where the python script is running no? So the Squid had an issue resolving the name of the very host it is running on? We should probably just resolve this forever by placing the name in /etc/hosts on the proxy server....
It currently has :127.0.0.1 proxy.phx.ovirt.org
Do you mean adding the public IP address as well?
bq. Typically the failures we're seeing are because of failures in the origin mirror we are actually proxying (Fedora and EPEL seem to be notoriously non-atomic when updating their mirrors). The way to solve this is only to used fully-fledged mirrors instead of a proxy (But this takes more maintenance and more storage).
By full fledged mirrors you mean have it on resources.ovirt.org ? will we need Katello to manage them or mirror with rsync will suffice?
> Investigate proxy errors from proxy.phx.ovirt.org
> -------------------------------------------------
>
> Key: OVIRT-869
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-869
> Project: oVirt - virtualization made easy
> Issue Type: Bug
> Reporter: eyal edri [Administrator]
> Assignee: infra
> Priority: Highest
>
> We need to make sure jobs don't fail on proxy like [1].
> Either use original repos if proxy is down or add watchdog / alerts for the proxy service.
> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/3693/art...
--
This message was sent by Atlassian JIRA
(v1000.571.2#100021)
7 years, 5 months
[JIRA] (OVIRT-869) Investigate proxy errors from proxy.phx.ovirt.org
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-869?page=com.atlassian.jira... ]
Barak Korren edited comment on OVIRT-869 at 11/27/16 10:50 AM:
---------------------------------------------------------------
The error seems to be a DNS error:
{code}
DEBUG util.py:421: http://proxy.phx.ovirt.org:5000/centos-base/7/x86_64/Packages/unzip-6.0-1...: [Errno 14] curl#5 - "Could not resolve proxy: proxy.phx.ovirt.org"
{code}
Port 5000 is where the python script is running no? So the Squid had an issue resolving the name of the very host it is running on? We should probably just resolve this forever by placing the name in {{/etc/hosts}} on the proxy server....
{quote}
Either use original repos if proxy is down or add watchdog / alerts for the proxy service.
{quote}
We have this:
# {{mock_runner.sh}} tries to contact proxy and uses un-proxyed configuration if it fails (This is why it has both {{\*.cnf}} and {{*-proxy.cnf}} files)
# There is a cron job watchdog running on the proxy itself and restarting it
# AFAIK we also have Icinga monitoring it.
Typically the failures we're seeing are because of failures in the origin mirror we are actually proxying (Fedora and EPEL seem to be notoriously non-atomic when updating their mirrors). The way to solve this is only to used fully-fledged mirrors instead of a proxy (But this takes more maintenance and more storage).
was (Author: bkorren(a)redhat.com):
The error seems to be a DNS error:
{code}
DEBUG util.py:421: http://proxy.phx.ovirt.org:5000/centos-base/7/x86_64/Packages/unzip-6.0-1...: [Errno 14] curl#5 - "Could not resolve proxy: proxy.phx.ovirt.org"
{code}
Port 5000 is where the python script is running no? So the Squid had an issue resolving the name of the very host it is running on? We should probably just resolve this forever by placing the name in {{/etc/hosts}} on the proxy server....
{code}
Either use original repos if proxy is down or add watchdog / alerts for the proxy service.
{code}
We have this:
# {{mock_runner.sh}} tries to contact proxy and uses un-proxyed configuration if it fails (This is why it has both {{\*.cnf}} and {{*-proxy.cnf}} files)
# There is a cron job watchdog running on the proxy itself and restarting it
# AFAIK we also have Icinga monitoring it.
Typically the failures we're seeing are because of failures in the origin mirror we are actually proxying (Fedora and EPEL seem to be notoriously non-atomic when updating their mirrors). The way to solve this is only to used fully-fledged mirrors instead of a proxy (But this takes more maintenance and more storage).
> Investigate proxy errors from proxy.phx.ovirt.org
> -------------------------------------------------
>
> Key: OVIRT-869
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-869
> Project: oVirt - virtualization made easy
> Issue Type: Bug
> Reporter: eyal edri [Administrator]
> Assignee: infra
> Priority: Highest
>
> We need to make sure jobs don't fail on proxy like [1].
> Either use original repos if proxy is down or add watchdog / alerts for the proxy service.
> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/3693/art...
--
This message was sent by Atlassian JIRA
(v1000.571.2#100021)
7 years, 5 months
[JIRA] (OVIRT-869) Investigate proxy errors from proxy.phx.ovirt.org
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-869?page=com.atlassian.jira... ]
Barak Korren commented on OVIRT-869:
------------------------------------
The error seems to be a DNS error:
{code}
DEBUG util.py:421: http://proxy.phx.ovirt.org:5000/centos-base/7/x86_64/Packages/unzip-6.0-1...: [Errno 14] curl#5 - "Could not resolve proxy: proxy.phx.ovirt.org"
{code}
Port 5000 is where the python script is running no? So the Squid had an issue resolving the name of the very host it is running on? We should probably just resolve this forever by placing the name in {{/etc/hosts}} on the proxy server....
{code}
Either use original repos if proxy is down or add watchdog / alerts for the proxy service.
{code}
We have this:
# {{mock_runner.sh}} tries to contact proxy and uses un-proxyed configuration if it fails (This is why it has both {{\*.cnf}} and {{*-proxy.cnf}} files)
# There is a cron job watchdog running on the proxy itself and restarting it
# AFAIK we also have Icinga monitoring it.
Typically the failures we're seeing are because of failures in the origin mirror we are actually proxying (Fedora and EPEL seem to be notoriously non-atomic when updating their mirrors). The way to solve this is only to used fully-fledged mirrors instead of a proxy (But this takes more maintenance and more storage).
> Investigate proxy errors from proxy.phx.ovirt.org
> -------------------------------------------------
>
> Key: OVIRT-869
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-869
> Project: oVirt - virtualization made easy
> Issue Type: Bug
> Reporter: eyal edri [Administrator]
> Assignee: infra
> Priority: Highest
>
> We need to make sure jobs don't fail on proxy like [1].
> Either use original repos if proxy is down or add watchdog / alerts for the proxy service.
> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/3693/art...
--
This message was sent by Atlassian JIRA
(v1000.571.2#100021)
7 years, 5 months
