December 2018 - Infra - oVirt List Archives

[JIRA] (OVIRT-2586) Jenkins terribly slow and unresponsive
by Evgheni Dereveanchin (oVirt JIRA) 04 Dec '18

04 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2586?page=com.atlassian.jira.… ] Evgheni Dereveanchin commented on OVIRT-2586: --------------------------------------------- Another strange result I see is that there has been around a 1000 downloads of the 4.2 node-ng ISOs from ovirt-node-ng_ovirt-4.2_build-artifacts-el7-x86_64/lastSuccessfulBuild/artifact/exported-artifacts over the course of the last week. Not sure what's up with that but we may be advertising this Jenkins URL directly from the website and it should not be used for thousands of ISO downloads as these block UI worker threads for a long period of time and may become a contributing factor to UI responsiveness. > Jenkins terribly slow and unresponsive > -------------------------------------- > > Key: OVIRT-2586 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2586 > Project: oVirt - virtualization made easy > Issue Type: Outage > Reporter: sbonazzo > Assignee: Evgheni Dereveanchin > Priority: Highest > > Hi, > jenkins is terribly slow and becoming worse every day. > I tried to gain some speed by adding 4 cores to the VM through engine-phx. > It's a bit better but the real issue doesn't seem related to CPU power. > Can anybody investigate? > -- > SANDRO BONAZZOLA > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > Red Hat EMEA <https://www.redhat.com/> > sbonazzo(a)redhat.com > <https://red.ht/sig> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

[JIRA] (OVIRT-2613) add_host fails on master and 4.2 ovirt due to missing deps
by Dafna Ron (oVirt JIRA) 04 Dec '18

04 Dec '18

Dafna Ron created OVIRT-2613: -------------------------------- Summary: add_host fails on master and 4.2 ovirt due to missing deps Key: OVIRT-2613 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2613 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Dafna Ron Assignee: infra Priority: Highest Gal Ben Haim 9:28 AM (3 hours ago) to rhv-devops, RHV-CI We are going to rebase [1] which will solve the deps error. We need to replace the image otherwise add_host fails with a segmentation fault. An image from virt-builder probably won't be available today, so we will build our own el7.6 image. [1] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11873/ -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

[JIRA] (OVIRT-2586) Jenkins terribly slow and unresponsive
by Evgheni Dereveanchin (oVirt JIRA) 04 Dec '18

04 Dec '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2586?page=com.atlassian.jira.… ] Evgheni Dereveanchin commented on OVIRT-2586: --------------------------------------------- Had an issue now reported by [~eyonasi(a)redhat.com] that Jenkins was not responding. I was monitoring CPU and it was just 100% used (400-600% used normally and 800% during high load). So Jenkins was idling for a few minutes and logs seem to show long running requests finishing before it came back to normal: [04/Dec/2018:12:05:54 +0000] "POST /ajaxBuildQueue HTTP/1.1" 502 408 [04/Dec/2018:12:04:59 +0000] "GET /view/oVirt%20system%20tests/job/ovirt-system-tests_manual/build?delay=0sec HTTP/1.1" 502 497 [04/Dec/2018:12:05:45 +0000] "POST /ajaxExecutors HTTP/1.1" 502 406 [04/Dec/2018:12:05:49 +0000] "POST /ajaxExecutors HTTP/1.1" 502 406 [04/Dec/2018:12:05:03 +0000] "GET /job/system-sync_mirrors-centos-kvm-common-el7-x86_64/ HTTP/1.1" 502 485 [04/Dec/2018:12:05:44 +0000] "GET / HTTP/1.1" 502 379 [04/Dec/2018:12:05:44 +0000] "POST /ajaxBuildQueue HTTP/1.1" 502 408 [04/Dec/2018:12:03:47 +0000] "GET /sse-gateway/listen/jenkins-blueocean-core-js-1543923704114-zly0n HTTP/1.1" 200 139 [04/Dec/2018:12:04:07 +0000] "GET /sse-gateway/listen/jenkins-blueocean-core-js-1543923241074-hu9gfi HTTP/1.1" 200 112 [04/Dec/2018:12:04:07 +0000] "GET /sse-gateway/listen/jenkins-blueocean-core-js-1543917181943-ulooc HTTP/1.1" 200 111 [04/Dec/2018:12:03:40 +0000] "GET /sse-gateway/listen/jenkins-blueocean-core-js-1543915343510-2uggp HTTP/1.1" 200 112 [04/Dec/2018:12:04:08 +0000] "GET /sse-gateway/listen/jenkins-blueocean-core-js-1543844460885-fshsx HTTP/1.1" 200 112 [04/Dec/2018:12:03:45 +0000] "GET /sse-gateway/listen/jenkins-blueocean-core-js-1543916086881-v6zmif HTTP/1.1" 200 113 [04/Dec/2018:12:06:06 +0000] "GET /favicon.ico HTTP/1.1" 200 17542 As can be seen from this log, sse-gateway requests started around 12:03 terminated only after 12:06 which is just too long. As noted before, these requests are initiated from BlueOcean as it seems. [~bkorren(a)redhat.com] did we by chance recently add BlueOcean links to any report emails or something else that could increase popularity of BlueOcean and induce this behavior? There were no updates in the past month that could directly influence this so I'm trying to locate a possible root cause. I can note as well that there's a bunch of requests related to OpenShift yet I do not have any proof they are slowing doen the Master at this point. > Jenkins terribly slow and unresponsive > -------------------------------------- > > Key: OVIRT-2586 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2586 > Project: oVirt - virtualization made easy > Issue Type: Outage > Reporter: sbonazzo > Assignee: Evgheni Dereveanchin > Priority: Highest > > Hi, > jenkins is terribly slow and becoming worse every day. > I tried to gain some speed by adding 4 cores to the VM through engine-phx. > It's a bit better but the real issue doesn't seem related to CPU power. > Can anybody investigate? > -- > SANDRO BONAZZOLA > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > Red Hat EMEA <https://www.redhat.com/> > sbonazzo(a)redhat.com > <https://red.ht/sig> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100095)

1 0

Build failed in Jenkins: system-sync_mirrors-centos-kvm-common-el7-x86_64 #1968
by jenkins＠jenkins.phx.ovirt.org 04 Dec '18

04 Dec '18

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-kvm-common-el7-x86_…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-kvm-common-el7-x86_…> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision fe8d4c3fee68d33b3407bba8a1408937639442a4 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f fe8d4c3fee68d33b3407bba8a1408937639442a4 Commit message: "Remove from lago fc27-EOL" > git rev-list --no-walk fe8d4c3fee68d33b3407bba8a1408937639442a4 # timeout=10 [system-sync_mirrors-centos-kvm-common-el7-x86_64] $ /bin/bash -xe /tmp/jenkins1601595848307799396.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-kvm-common-el7 x86_64 jenkins/data/mirrors-reposync.conf Checking if mirror needs a resync Traceback (most recent call last): File "/usr/bin/reposync", line 343, in <module> main() File "/usr/bin/reposync", line 175, in main my.doRepoSetup() File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup return self._getRepos(thisrepo, True) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos self._repos.doSetup(thisrepo) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1479, in _commonLoadRepoXML result = self._getFileRepoXML(local, text) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1256, in _getFileRepoXML size=102400) # setting max size as 100K File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1039, in _getFile raise e yum.Errors.NoMoreMirrorsRepoError: failure: repodata/repomd.xml from glusterfs-3.10-el7: [Errno 256] No more mirrors to try. http://mirror.centos.org/centos/7/storage/x86_64/gluster-3.10/repodata/repo…: [Errno 14] HTTP Error 404 - Not Found Build step 'Execute shell' marked build as failure

1 2

[JENKINS] Failed to setup proejct kubevirt_kubevirt_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 04 Dec '18

04 Dec '18

Failed to run project_setup.sh for: #2762 kubevirt [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on openshift-qhvsl.

1 0

[JENKINS] Failed to setup proejct kubevirt_kubevirt_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 04 Dec '18

04 Dec '18

Failed to run project_setup.sh for: #2761 kubevirt [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on openshift-r441f.

1 0

[oVirt Jenkins] ovirt-node-ng-image_master_build-artifacts-el7-x86_64 - Build # 133 - Failure!
by jenkins＠jenkins.phx.ovirt.org 04 Dec '18

04 Dec '18

Project: http://jenkins.ovirt.org/job/ovirt-node-ng-image_master_build-artifacts-el7… Build: http://jenkins.ovirt.org/job/ovirt-node-ng-image_master_build-artifacts-el7… Build Number: 133 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #133 [Yuval Turgeman] stdci: move build-artifacts to a v1 job [Evgheni Dereveanchin] Add distribution-gpg-keys to global_setup ----------------- Failed Tests: ----------------- No tests ran.

1 1

[JENKINS] Failed to setup proejct kubevirt_kubevirt_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 04 Dec '18

04 Dec '18

Failed to run project_setup.sh for: #2760 kubevirt [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on openshift-rc8g7.

1 0

[CQ]: 95927,3 (ovirt-engine) failed "ovirt-master" system tests
by oVirt Jenkins 04 Dec '18

04 Dec '18

Change 95927,3 (ovirt-engine) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/95927/3 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11873/

1 0

[oVirt Jenkins] ovirt-appliance_master_build-artifacts-fc28-x86_64 - Build # 142 - Failure!
by jenkins＠jenkins.phx.ovirt.org 04 Dec '18

04 Dec '18

Project: http://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-fc28-x8… Build: http://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-fc28-x8… Build Number: 142 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #142 [Evgheni Dereveanchin] Add distribution-gpg-keys to global_setup [Yuval Turgeman] automation: remove build-artifacts* from stdci-v2 ----------------- Failed Tests: ----------------- No tests ran.

1 1