November 2016 - Infra - oVirt List Archives

Build failed in Jenkins: ovirt_4.0_he-system-tests #560
by jenkins＠jenkins.phx.ovirt.org 27 Nov '16

27 Nov '16

See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/560/changes> Changes: [Eyal Edri] adding missing python-simplejson to epel repo ------------------------------------------ [...truncated 669 lines...] ## took 356 seconds ## rc = 1 ########################################################## ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv ##! Last 20 log entries: logs/mocker-epel-7-x86_64.el7.he_basic_suite_4.0.sh/he_basic_suite_4.0.sh.log ##! + env_cleanup + echo '#########################' ######################### + local res=0 + local uuid + echo '======== Cleaning up' ======== Cleaning up + [[ -e <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-test…> ]] + echo '----------- Cleaning with lago' ----------- Cleaning with lago + lago --workdir <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-test…> destroy --yes --all-prefixes + echo '----------- Cleaning with lago done' ----------- Cleaning with lago done + [[ 0 != \0 ]] + echo '======== Cleanup done' ======== Cleanup done + exit 0 + exit Took 205 seconds =================================== ##! ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ##!######################################################## ########################################################## Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh' # # Required jjb vars: # version # VERSION=4.0 SUITE_TYPE= WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson5732644382445829429.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=4.0 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + OVIRT_SUITE=4.0 + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-test…> + rm -rf <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/560/artifact/exporte…> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/560/artifact/exporte…> + [[ -d <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-test…> ]] + mv <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-test…> <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/560/artifact/exporte…> POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo "shell-scripts/mock_cleanup.sh" shopt -s nullglob WORKSPACE="$PWD" # Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC # Archive the logs, we want them anyway logs=( ./*log ./*/logs ) if [[ "$logs" ]]; then for log in "${logs[@]}" do echo "Copying ${log} to exported-artifacts" mv $log exported-artifacts/ done fi # stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE" #TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true } mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true } done done # Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true this_chroot_failed=true } done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done if $failed; then echo "Aborting." exit 1 fi # remove mock system cache, we will setup proxies to do the caching and this # takes lots of space between runs shopt -u nullglob sudo rm -Rf /var/cache/mock/* # restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE" # Drop all left over libvirt domains for UUID in $(virsh list --all --uuid); do virsh destroy $UUID || : sleep 2 virsh undefine --remove-all-storage --storage vda --snapshots-metadata $UUID || : done [ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson7475623662613716644.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + logs=(./*log ./*/logs) + [[ -n ./ovirt-system-tests/logs ]] + for log in '"${logs[@]}"' + echo 'Copying ./ovirt-system-tests/logs to exported-artifacts' Copying ./ovirt-system-tests/logs to exported-artifacts + mv ./ovirt-system-tests/logs exported-artifacts/ + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + for mock_conf_file in '"${mock_confs[@]}"' + [[ -n <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-test…> ]] + echo 'Cleaning up mock ' Cleaning up mock + mock_root=mocker-epel-7-x86_64.el7.cfg + mock_root=mocker-epel-7-x86_64.el7 + my_mock=/usr/bin/mock + my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests'> + my_mock+=' --root=mocker-epel-7-x86_64.el7' + my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/'> + echo 'Killing all mock orphan processes, if any.' Killing all mock orphan processes, if any. + /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-tests> --root=mocker-epel-7-x86_64.el7 --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> --orphanskill WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-test….> Using default... INFO: mock.py version 1.2.21 starting (python version = 3.5.1)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Finish: run ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ovirt-system-test…> + mock_root=epel-7-x86_64-0c362156a2fa4a935ea8b988eb73b2ad + [[ -n epel-7-x86_64-0c362156a2fa4a935ea8b988eb73b2ad ]] + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep epel-7-x86_64-0c362156a2fa4a935ea8b988eb73b2ad + : + [[ -n '' ]] + false + shopt -u nullglob + sudo rm -Rf /var/cache/mock/epel-7-x86_64-0c362156a2fa4a935ea8b988eb73b2ad + sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> ++ virsh list --all --uuid POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 1 Recording test results ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Archiving artifacts

3 3

[JIRA] (OVIRT-867) Re: [oVirt Jenkins] repos_3.6_check-closure_el7_merged - Build # 92 - Still Failing!
by eyal edri [Administrator] (oVirt JIRA) 27 Nov '16

27 Nov '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-867?page=com.atlassian.jira.p… ] eyal edri [Administrator] commented on OVIRT-867: ------------------------------------------------- Evgheni, Looks like we're hitting too much issues lately with external repos ( epel/centos/fedora ), I think its time we'll consider having a mirror for what we need on resources.ovirt.org to reduce the failures ( we need to ensure we have enough space on it as well ) On Fri, Nov 25, 2016 at 5:13 PM, Sandro Bonazzola <sbonazzo(a)redhat.com> wrote: > > > On Fri, Nov 25, 2016 at 3:43 PM, Evgheni Dereveanchin <ederevea(a)redhat.com > > wrote: > >> After more checks - indeed this looks like an inconsistency >> on the mirror: >> >> http://mirror.switch.ch/ftp/mirror/epel/7/x86_64/repodata/ >> >> 1ef1290202fb9e09f1aa15efccea3456913df8657cc0e23a1df61d9ec18 >> e5659-comps-epel7.xml.xz >> ab638ffbf7ef7e7f41b8f630113ff6bd251612eefa570b8725b612500d8 >> c90fc-comps-epel7.xml >> repomd.xml >> >> there is no primary.sqlite.xz or filelists.xml.gz in this directory >> at the moment, hence the failures. As said, this mirror is hardcoded >> in repo_closure_check.sh so changing that to a working mirror will >> fix the issue, as well as waiting till mirror.switch.ch gets sync'ed. >> >> In fact, I don't know why we are using a Swiss mirror for tests that >> are run on US servers. >> >> > feel free to change mirror or use a local one > > > >> Regards, >> Evgheni Dereveanchin >> >> ----- Original Message ----- >> From: "Evgheni Dereveanchin" <ederevea(a)redhat.com> >> To: "Sandro Bonazzola" <sbonazzo(a)redhat.com> >> Cc: "infra" <infra(a)ovirt.org>, infra-support(a)ovirt.org, "Yedidyah Bar >> David" <didi(a)redhat.com>, "Eyal Edri" <eedri(a)redhat.com> >> Sent: Friday, 25 November, 2016 3:10:47 PM >> Subject: Re: [oVirt Jenkins] repos_3.6_check-closure_el7_merged - Build >> # 92 - Still Failing! >> >> Hi Sandro, >> >> From first glance this may be caused by an upstream EPEL mirror, >> especially given the 404 errors: >> >> 12:07:12 Added check-epel-el7 repo from http://mirror.switch.ch/ftp/mi >> rror/epel/7/x86_64/ >> ... >> 12:07:12 Reading in repository metadata - please wait.... >> 12:07:20 Can't download or revert repomd.xml for check-epel-el7 >> ... >> 12:07:21 failure: repodata/9edce4d5e3c9437849fbb >> bae7c2faa50a9b0326b968dd8faa66ece83984c23de-primary.sqlite.xz from >> check-epel-el7: [Errno 256] No more mirrors to try. >> 12:07:21 http://mirror.switch.ch/ftp/mirror/epel/7/x86_64/repodata/9e >> dce4d5e3c9437849fbbbae7c2faa50a9b0326b968dd8faa66ece83984c23 >> de-primary.sqlite.xz: [Errno 14] HTTP Error 404 - Not Found >> >> This mirror is hardcoded in repo_closure_check.sh and looks >> fine when accessed by browser. I will do some more investigation >> to see if some stale metadata may be causing this, however I've >> seen the failure on several different slaves. >> >> Regards, >> Evgheni Dereveanchin >> >> ----- Original Message ----- >> From: "Sandro Bonazzola" <sbonazzo(a)redhat.com> >> To: "infra" <infra(a)ovirt.org>, infra-support(a)ovirt.org, "Evgheni >> Dereveanchin" <ederevea(a)redhat.com> >> Cc: "Yedidyah Bar David" <didi(a)redhat.com>, "Eyal Edri" <eedri(a)redhat.com >> > >> Sent: Friday, 25 November, 2016 2:41:02 PM >> Subject: Re: [oVirt Jenkins] repos_3.6_check-closure_el7_merged - Build >> # 92 - Still Failing! >> >> *00:01:53.955* Can't download or revert repomd.xml for >> check-epel-el7*00:01:53.955* Some dependencies may not be complete for >> this repository*00:01:53.956* Run as root to get all dependencies or >> use -t to enable a user temp cache >> >> >> Can you check? looks like it's not possible to update local cache of >> metadata. >> >> >> >> On Fri, Nov 25, 2016 at 1:07 PM, <jenkins(a)jenkins.phx.ovirt.org> wrote: >> >> > Project: http://jenkins.ovirt.org/job/repos_3.6_check-closure_el7_mer >> ged/ >> > Build: http://jenkins.ovirt.org/job/repos_3.6_check-closure_el7_mer >> ged/92/ >> > Build Number: 92 >> > Build Status: Still Failing >> > Triggered By: Started by user Sandro Bonazzola >> > >> > ------------------------------------- >> > Changes Since Last Success: >> > ------------------------------------- >> > Changes for Build #89 >> > [Eyal Edri] deploy ovirt-engine-cli 3.6 to 4.0 and master repos as well >> > >> > [Yedidyah Bar David] master_upgrade_from_master: Upgrade to self instead >> > of snapshot >> > >> > >> > Changes for Build #90 >> > No changes >> > >> > Changes for Build #91 >> > No changes >> > >> > Changes for Build #92 >> > No changes >> > >> > >> > >> > ----------------- >> > Failed Tests: >> > ----------------- >> > No tests ran. >> > >> > >> > _______________________________________________ >> > Infra mailing list >> > Infra(a)ovirt.org >> > http://lists.ovirt.org/mailman/listinfo/infra >> > >> > >> >> >> -- >> Sandro Bonazzola >> Better technology. Faster innovation. Powered by community collaboration. >> See how it works at redhat.com >> > > > > -- > Sandro Bonazzola > Better technology. Faster innovation. Powered by community collaboration. > See how it works at redhat.com > -- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ) > Re: [oVirt Jenkins] repos_3.6_check-closure_el7_merged - Build # 92 - Still Failing! > ------------------------------------------------------------------------------------ > > Key: OVIRT-867 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-867 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: sbonazzo > Assignee: infra > > *00:01:53.955* Can't download or revert repomd.xml for > check-epel-el7*00:01:53.955* Some dependencies may not be complete for > this repository*00:01:53.956* Run as root to get all dependencies or > use -t to enable a user temp cache > Can you check? looks like it's not possible to update local cache of metadata. > On Fri, Nov 25, 2016 at 1:07 PM, <jenkins(a)jenkins.phx.ovirt.org> wrote: > > Project: http://jenkins.ovirt.org/job/repos_3.6_check-closure_el7_merged/ > > Build: http://jenkins.ovirt.org/job/repos_3.6_check-closure_el7_merged/92/ > > Build Number: 92 > > Build Status: Still Failing > > Triggered By: Started by user Sandro Bonazzola > > > > ------------------------------------- > > Changes Since Last Success: > > ------------------------------------- > > Changes for Build #89 > > [Eyal Edri] deploy ovirt-engine-cli 3.6 to 4.0 and master repos as well > > > > [Yedidyah Bar David] master_upgrade_from_master: Upgrade to self instead > > of snapshot > > > > > > Changes for Build #90 > > No changes > > > > Changes for Build #91 > > No changes > > > > Changes for Build #92 > > No changes > > > > > > > > ----------------- > > Failed Tests: > > ----------------- > > No tests ran. > > > > > > _______________________________________________ > > Infra mailing list > > Infra(a)ovirt.org > > http://lists.ovirt.org/mailman/listinfo/infra > > > > > -- > Sandro Bonazzola > Better technology. Faster innovation. Powered by community collaboration. > See how it works at redhat.com -- This message was sent by Atlassian JIRA (v1000.571.2#100021)

1 0

[oVirt Jenkins] repos_3.6_check-closure_el7_merged - Build # 89 - Failure!
by jenkins＠jenkins.phx.ovirt.org 27 Nov '16

27 Nov '16

------=_Part_927_1734224177.1480053856233 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Project: http://jenkins.ovirt.org/job/repos_3.6_check-closure_el7_merged/ Build: http://jenkins.ovirt.org/job/repos_3.6_check-closure_el7_merged/89/ Build Number: 89 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #89 [Eyal Edri] deploy ovirt-engine-cli 3.6 to 4.0 and master repos as well [Yedidyah Bar David] master_upgrade_from_master: Upgrade to self instead of snapshot ----------------- Failed Tests: ----------------- No tests ran. ------=_Part_927_1734224177.1480053856233--

4 9

[JIRA] (OVIRT-869) Investigate proxy errors from proxy.phx.ovirt.org
by Barak Korren (oVirt JIRA) 27 Nov '16

27 Nov '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-869?page=com.atlassian.jira.p… ] Barak Korren commented on OVIRT-869: ------------------------------------ {quote} It currently has :127.0.0.1 proxy.phx.ovirt.org Do you mean adding the public IP address as well? {quote} If it has this already then the error is strange indeed, all I can think of is that maybe the mock environment running the failed job was not configured correctly (Trying to use the proxy repo URLs without setting it up as an HTTP proxy _in addition to_ not having proper _DNS_ config. If it was justh the HTTP proxy setting I would expect to see a "_connection refused_" error rather then "_could not resolve_"). {quote} By full fledged mirrors you mean have it on resources.ovirt.org ? will we need Katello to manage them or mirror with rsync will suffice? {quote} Not necessarily on resources.ovirt.org, we could (and maybe should) use a different server for that. WRT the mirroring solution itself we have many options with varying degrees of automation, features and required effort: # rsync (though not all upsreams may allow this kind of access) # yum reposync # repoman # Pulp # Artifactory # Katello The benefit of Katello would be in allowing us to control exactly which packages get tested by leveraging the work flow environments and the content views, but we could build comparable solutions with other tools. > Investigate proxy errors from proxy.phx.ovirt.org > ------------------------------------------------- > > Key: OVIRT-869 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-869 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: eyal edri [Administrator] > Assignee: infra > Priority: Highest > > We need to make sure jobs don't fail on proxy like [1]. > Either use original repos if proxy is down or add watchdog / alerts for the proxy service. > http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/3693/artif… -- This message was sent by Atlassian JIRA (v1000.571.2#100021)

1 0

Re: Change in ovirt-engine[master]: core: improving vms_with_tags view performance
by Martin Perina 27 Nov '16

27 Nov '16

Could you please take a look at upgrade from 3.6 job? Following error seems to me like an CI infra issue: error: could not apply 8525684... Modified cleanup.sh in 'upgrade_to_4.0' Thanks Martin On Fri, Nov 25, 2016 at 6:54 PM, Code Review <gerrit(a)ovirt.org> wrote: > From Jenkins CI: > > Jenkins CI has posted comments on this change. > > Change subject: core: improving vms_with_tags view performance > ...................................................................... > > > Patch Set 4: > > Build Failed > > http://jenkins.ovirt.org/job/ovirt-engine_master_upgrade- > from-3.6_el7_merged/1039/ : FAILURE > > http://jenkins.ovirt.org/job/ovirt-engine_master_check- > merged-el7-x86_64/2650/ : SUCCESS > > http://jenkins.ovirt.org/job/ovirt-engine_master_upgrade- > from-4.0_el7_merged/2183/ : SUCCESS > > http://jenkins.ovirt.org/job/ovirt-engine_master_upgrade- > from-master_el7_merged/2657/ : SUCCESS > > http://jenkins.ovirt.org/job/ovirt-engine_master_check- > merged-fc24-x86_64/1753/ : SUCCESS > > -- > To view, visit https://gerrit.ovirt.org/66165 > To unsubscribe, visit https://gerrit.ovirt.org/settings > > Gerrit-MessageType: comment > Gerrit-Change-Id: Ib5a6637b7c890e445940c8d0a770c2bd7e11e72b > Gerrit-PatchSet: 4 > Gerrit-Project: ovirt-engine > Gerrit-Branch: master > Gerrit-Owner: Eli Mesika <emesika(a)redhat.com> > Gerrit-Reviewer: Arik Hadas <ahadas(a)redhat.com> > Gerrit-Reviewer: Eli Mesika <emesika(a)redhat.com> > Gerrit-Reviewer: Jenkins CI > Gerrit-Reviewer: Martin Peřina <mperina(a)redhat.com> > Gerrit-Reviewer: gerrit-hooks <automation(a)ovirt.org> > Gerrit-HasComments: No >

2 1

[JIRA] (OVIRT-869) Investigate proxy errors from proxy.phx.ovirt.org
by eyal edri [Administrator] (oVirt JIRA) 27 Nov '16

27 Nov '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-869?page=com.atlassian.jira.p… ] eyal edri [Administrator] commented on OVIRT-869: ------------------------------------------------- bq. Port 5000 is where the python script is running no? So the Squid had an issue resolving the name of the very host it is running on? We should probably just resolve this forever by placing the name in /etc/hosts on the proxy server.... It currently has :127.0.0.1 proxy.phx.ovirt.org Do you mean adding the public IP address as well? bq. Typically the failures we're seeing are because of failures in the origin mirror we are actually proxying (Fedora and EPEL seem to be notoriously non-atomic when updating their mirrors). The way to solve this is only to used fully-fledged mirrors instead of a proxy (But this takes more maintenance and more storage). By full fledged mirrors you mean have it on resources.ovirt.org ? will we need Katello to manage them or mirror with rsync will suffice? > Investigate proxy errors from proxy.phx.ovirt.org > ------------------------------------------------- > > Key: OVIRT-869 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-869 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: eyal edri [Administrator] > Assignee: infra > Priority: Highest > > We need to make sure jobs don't fail on proxy like [1]. > Either use original repos if proxy is down or add watchdog / alerts for the proxy service. > http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/3693/artif… -- This message was sent by Atlassian JIRA (v1000.571.2#100021)

1 0

[oVirt Jenkins] test-repo_ovirt_experimental_master - Build #3693 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 27 Nov '16

27 Nov '16

------=_Part_1004_2088548965.1480231380289 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Build: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/3693/, Build Number: 3693, Build Status: FAILURE ------=_Part_1004_2088548965.1480231380289--

1 1

[JIRA] (OVIRT-869) Investigate proxy errors from proxy.phx.ovirt.org
by Barak Korren (oVirt JIRA) 27 Nov '16

27 Nov '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-869?page=com.atlassian.jira.p… ] Barak Korren edited comment on OVIRT-869 at 11/27/16 10:50 AM: --------------------------------------------------------------- The error seems to be a DNS error: {code} DEBUG util.py:421: http://proxy.phx.ovirt.org:5000/centos-base/7/x86_64/Packages/unzip-6.0-15.…: [Errno 14] curl#5 - "Could not resolve proxy: proxy.phx.ovirt.org" {code} Port 5000 is where the python script is running no? So the Squid had an issue resolving the name of the very host it is running on? We should probably just resolve this forever by placing the name in {{/etc/hosts}} on the proxy server.... {quote} Either use original repos if proxy is down or add watchdog / alerts for the proxy service. {quote} We have this: # {{mock_runner.sh}} tries to contact proxy and uses un-proxyed configuration if it fails (This is why it has both {{\*.cnf}} and {{*-proxy.cnf}} files) # There is a cron job watchdog running on the proxy itself and restarting it # AFAIK we also have Icinga monitoring it. Typically the failures we're seeing are because of failures in the origin mirror we are actually proxying (Fedora and EPEL seem to be notoriously non-atomic when updating their mirrors). The way to solve this is only to used fully-fledged mirrors instead of a proxy (But this takes more maintenance and more storage). was (Author: bkorren(a)redhat.com) The error seems to be a DNS error: {code} DEBUG util.py:421: http://proxy.phx.ovirt.org:5000/centos-base/7/x86_64/Packages/unzip-6.0-15.…: [Errno 14] curl#5 - "Could not resolve proxy: proxy.phx.ovirt.org" {code} Port 5000 is where the python script is running no? So the Squid had an issue resolving the name of the very host it is running on? We should probably just resolve this forever by placing the name in {{/etc/hosts}} on the proxy server.... {code} Either use original repos if proxy is down or add watchdog / alerts for the proxy service. {code} We have this: # {{mock_runner.sh}} tries to contact proxy and uses un-proxyed configuration if it fails (This is why it has both {{\*.cnf}} and {{*-proxy.cnf}} files) # There is a cron job watchdog running on the proxy itself and restarting it # AFAIK we also have Icinga monitoring it. Typically the failures we're seeing are because of failures in the origin mirror we are actually proxying (Fedora and EPEL seem to be notoriously non-atomic when updating their mirrors). The way to solve this is only to used fully-fledged mirrors instead of a proxy (But this takes more maintenance and more storage). > Investigate proxy errors from proxy.phx.ovirt.org > ------------------------------------------------- > > Key: OVIRT-869 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-869 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: eyal edri [Administrator] > Assignee: infra > Priority: Highest > > We need to make sure jobs don't fail on proxy like [1]. > Either use original repos if proxy is down or add watchdog / alerts for the proxy service. > http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/3693/artif… -- This message was sent by Atlassian JIRA (v1000.571.2#100021)

1 0

[JIRA] (OVIRT-869) Investigate proxy errors from proxy.phx.ovirt.org
by Barak Korren (oVirt JIRA) 27 Nov '16

27 Nov '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-869?page=com.atlassian.jira.p… ] Barak Korren commented on OVIRT-869: ------------------------------------ The error seems to be a DNS error: {code} DEBUG util.py:421: http://proxy.phx.ovirt.org:5000/centos-base/7/x86_64/Packages/unzip-6.0-15.…: [Errno 14] curl#5 - "Could not resolve proxy: proxy.phx.ovirt.org" {code} Port 5000 is where the python script is running no? So the Squid had an issue resolving the name of the very host it is running on? We should probably just resolve this forever by placing the name in {{/etc/hosts}} on the proxy server.... {code} Either use original repos if proxy is down or add watchdog / alerts for the proxy service. {code} We have this: # {{mock_runner.sh}} tries to contact proxy and uses un-proxyed configuration if it fails (This is why it has both {{\*.cnf}} and {{*-proxy.cnf}} files) # There is a cron job watchdog running on the proxy itself and restarting it # AFAIK we also have Icinga monitoring it. Typically the failures we're seeing are because of failures in the origin mirror we are actually proxying (Fedora and EPEL seem to be notoriously non-atomic when updating their mirrors). The way to solve this is only to used fully-fledged mirrors instead of a proxy (But this takes more maintenance and more storage). > Investigate proxy errors from proxy.phx.ovirt.org > ------------------------------------------------- > > Key: OVIRT-869 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-869 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: eyal edri [Administrator] > Assignee: infra > Priority: Highest > > We need to make sure jobs don't fail on proxy like [1]. > Either use original repos if proxy is down or add watchdog / alerts for the proxy service. > http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/3693/artif… -- This message was sent by Atlassian JIRA (v1000.571.2#100021)

1 0

[JIRA] (OVIRT-869) Investigate proxy errors from proxy.phx.ovirt.org
by eyal edri [Administrator] (oVirt JIRA) 27 Nov '16

27 Nov '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-869?page=com.atlassian.jira.p… ] eyal edri [Administrator] commented on OVIRT-869: ------------------------------------------------- also, add any needed into to infra docs on troubleshooting errors > Investigate proxy errors from proxy.phx.ovirt.org > ------------------------------------------------- > > Key: OVIRT-869 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-869 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: eyal edri [Administrator] > Assignee: infra > Priority: Highest > > We need to make sure jobs don't fail on proxy like [1]. > Either use original repos if proxy is down or add watchdog / alerts for the proxy service. > http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/3693/artif… -- This message was sent by Atlassian JIRA (v1000.571.2#100021)

1 0