- Infra - oVirt List Archives

[JIRA] (OVIRT-730) OOM errors on foreman.phx.org
by Evgheni Dereveanchin (oVirt JIRA) 14 Sep '16

14 Sep '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-730?page=com.atlassian.jira.p… ] Evgheni Dereveanchin reassigned OVIRT-730: ------------------------------------------ Assignee: Evgheni Dereveanchin (was: infra) > OOM errors on foreman.phx.org > ----------------------------- > > Key: OVIRT-730 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-730 > Project: oVirt - virtualization made easy > Issue Type: Bug > Components: Foreman > Reporter: Evgheni Dereveanchin > Assignee: Evgheni Dereveanchin > > The foreman master regularly runs out of memory. This causes the kernel to kill processes and impacts productivity > {quote} > # dmesg | grep "Out of memory" > Out of memory: Kill process 77275 (ruby) score 494 or sacrifice child > Out of memory: Kill process 156121 (rake) score 260 or sacrifice child > Out of memory: Kill process 147139 (rake) score 284 or sacrifice child > Out of memory: Kill process 148300 (rake) score 185 or sacrifice child > Out of memory: Kill process 64891 (rake) score 355 or sacrifice child > Out of memory: Kill process 25571 (rake) score 350 or sacrifice child > Out of memory: Kill process 1420 (rake) score 327 or sacrifice child > Out of memory: Kill process 159304 (rake) score 327 or sacrifice child > Out of memory: Kill process 64165 (java) score 56 or sacrifice child > Out of memory: Kill process 45386 (rake) score 356 or sacrifice child > Out of memory: Kill process 46306 (rake) score 344 or sacrifice child > Out of memory: Kill process 73278 (rake) score 344 or sacrifice child > Out of memory: Kill process 29643 (java) score 55 or sacrifice child > Out of memory: Kill process 2176 (rake) score 362 or sacrifice child > Out of memory: Kill process 74941 (java) score 55 or sacrifice child > Out of memory: Kill process 2827 (rake) score 379 or sacrifice child > Out of memory: Kill process 42191 (rake) score 370 or sacrifice child > Out of memory: Kill process 51354 (rake) score 373 or sacrifice child > Out of memory: Kill process 90964 (rake) score 372 or sacrifice child > Out of memory: Kill process 101104 (rake) score 361 or sacrifice child > Out of memory: Kill process 96969 (rake) score 382 or sacrifice child > Out of memory: Kill process 38014 (rake) score 382 or sacrifice child > Out of memory: Kill process 16020 (rake) score 373 or sacrifice child > Out of memory: Kill process 140459 (rake) score 383 or sacrifice child > Out of memory: Kill process 146590 (rake) score 368 or sacrifice child > Out of memory: Kill process 125684 (rake) score 395 or sacrifice child > Out of memory: Kill process 107210 (rake) score 486 or sacrifice child > Out of memory: Kill process 96070 (rake) score 420 or sacrifice child > Out of memory: Kill process 81115 (rake) score 374 or sacrifice child > Out of memory: Kill process 72827 (rake) score 384 or sacrifice child > Out of memory: Kill process 11442 (java) score 56 or sacrifice child > # uptime > 14:07:04 up 408 days, 58 min, 1 user, load average: 1.65, 2.11, 2.52 > # free -m > total used free shared buffers cached > Mem: 5832 4203 1628 33 159 750 > -/+ buffers/cache: 3293 2538 > Swap: 1023 91 932 > {quote} -- This message was sent by Atlassian JIRA (v1000.319.1#100012)

1 0

[JIRA] (OVIRT-730) OOM errors on foreman.phx.org
by Evgheni Dereveanchin (oVirt JIRA) 14 Sep '16

14 Sep '16

Evgheni Dereveanchin created OVIRT-730: ------------------------------------------ Summary: OOM errors on foreman.phx.org Key: OVIRT-730 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-730 Project: oVirt - virtualization made easy Issue Type: Bug Components: Foreman Reporter: Evgheni Dereveanchin Assignee: infra The foreman master regularly runs out of memory. This causes the kernel to kill processes and impacts productivity {quote} # dmesg | grep "Out of memory" Out of memory: Kill process 77275 (ruby) score 494 or sacrifice child Out of memory: Kill process 156121 (rake) score 260 or sacrifice child Out of memory: Kill process 147139 (rake) score 284 or sacrifice child Out of memory: Kill process 148300 (rake) score 185 or sacrifice child Out of memory: Kill process 64891 (rake) score 355 or sacrifice child Out of memory: Kill process 25571 (rake) score 350 or sacrifice child Out of memory: Kill process 1420 (rake) score 327 or sacrifice child Out of memory: Kill process 159304 (rake) score 327 or sacrifice child Out of memory: Kill process 64165 (java) score 56 or sacrifice child Out of memory: Kill process 45386 (rake) score 356 or sacrifice child Out of memory: Kill process 46306 (rake) score 344 or sacrifice child Out of memory: Kill process 73278 (rake) score 344 or sacrifice child Out of memory: Kill process 29643 (java) score 55 or sacrifice child Out of memory: Kill process 2176 (rake) score 362 or sacrifice child Out of memory: Kill process 74941 (java) score 55 or sacrifice child Out of memory: Kill process 2827 (rake) score 379 or sacrifice child Out of memory: Kill process 42191 (rake) score 370 or sacrifice child Out of memory: Kill process 51354 (rake) score 373 or sacrifice child Out of memory: Kill process 90964 (rake) score 372 or sacrifice child Out of memory: Kill process 101104 (rake) score 361 or sacrifice child Out of memory: Kill process 96969 (rake) score 382 or sacrifice child Out of memory: Kill process 38014 (rake) score 382 or sacrifice child Out of memory: Kill process 16020 (rake) score 373 or sacrifice child Out of memory: Kill process 140459 (rake) score 383 or sacrifice child Out of memory: Kill process 146590 (rake) score 368 or sacrifice child Out of memory: Kill process 125684 (rake) score 395 or sacrifice child Out of memory: Kill process 107210 (rake) score 486 or sacrifice child Out of memory: Kill process 96070 (rake) score 420 or sacrifice child Out of memory: Kill process 81115 (rake) score 374 or sacrifice child Out of memory: Kill process 72827 (rake) score 384 or sacrifice child Out of memory: Kill process 11442 (java) score 56 or sacrifice child # uptime 14:07:04 up 408 days, 58 min, 1 user, load average: 1.65, 2.11, 2.52 # free -m total used free shared buffers cached Mem: 5832 4203 1628 33 159 750 -/+ buffers/cache: 3293 2538 Swap: 1023 91 932 {quote} -- This message was sent by Atlassian JIRA (v1000.319.1#100012)

1 0

[ACTION REQUIRED] oVirt 4.0.4 RC3 build starting
by Sandro Bonazzola 14 Sep '16

14 Sep '16

Fyi oVirt products maintainers, An oVirt build for an official release is going to start right now. If you're a maintainer for any of the projects included in oVirt distribution and you have changes in your package ready to be released please: - bump version and release to be GA ready - tag your release within git (implies a GitHub Release to be automatically created) - build your packages within jenkins / koji / copr / whatever - verify all bugs on MODIFIED have target release and target milestone set. - add your builds to releng-tools/releases/ovirt-4.0.4_rc3.conf withini releng-tools project -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com <https://www.redhat.com/it/about/events/red-hat-open-source-day-2016>

1 0

oVirt infra daily report - unstable production jobs - 76
by jenkins＠jenkins.phx.ovirt.org 14 Sep '16

14 Sep '16

------=_Part_241_990179947.1473807609865 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/76//artifact/exported-ar… Cheers, Jenkins ------=_Part_241_990179947.1473807609865 Content-Type: application/octet-stream; name=upstream_report.html Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=upstream_report.html Content-ID: <upstream_report.html> <!DOCTYPE html><head><style type="text/css"> table.gridtable { border-collapse: collapse; table-layout:fixed; width:1600px; font-family: monospace; font-size:13px; } .head { font-size:20px; font-family: arial; } .sub { font-size:18px; background-color:#e5e5e5; font-family: arial; } pre { font-family: monospace; display: inline; white-space: pre-wrap; white-space: -moz-pre-wrap !important; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word; } </style> </head> <body> <table class="gridtable" border=2> <tr><th colspan=2 class=head> RHEVM CI Jenkins Daily Report - 13/09/2016 </th></tr><tr><th colspan=2 class=sub> <font color="blue"><a href="http://jenkins.ovirt.org/">00 Unstable Jobs (Production)</a></font> </th></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/fabric-ovirt_master_check-merged-el7-x86_64/">fabric-ovirt_master_check-merged-el7-x86_64</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/fabric-ovirt_master_check-merged-fc23-x86_64/">fabric-ovirt_master_check-merged-fc23-x86_64</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/ovirt-node-ng_master_build-artifacts-el7-x86_6…">ovirt-node-ng_master_build-artifacts-el7-x86_64</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/ovirt-node-ng_ovirt-3.6_build-artifacts-el7-x8…">ovirt-node-ng_ovirt-3.6_build-artifacts-el7-x86_64</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/ovirt-release_master_build-artifacts-all-x86_6…">ovirt-release_master_build-artifacts-all-x86_64</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/repos_master_check-closure_fc24_merged/">repos_master_check-closure_fc24_merged</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> <tr><td> <a href="http://jenkins.ovirt.org/job/vdsm_4.0_check-merged-fc23-x86_64/">vdsm_4.0_check-merged-fc23-x86_64</a> </td><td> This job is automatically updated by jenkins job builder, any manual change will be lost in the next update. If you want to make permanent changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;hb=…"> jenkins</a> repo.  </td></tr> ------=_Part_241_990179947.1473807609865--

1 0

[JIRA] (OVIRT-726) increase max number of concurrent builds
by eyal edri [Administrator] (oVirt JIRA) 13 Sep '16

13 Sep '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-726?page=com.atlassian.jira.p… ] eyal edri [Administrator] commented on OVIRT-726: ------------------------------------------------- feel free to update gradually and lets see how it goes. > increase max number of concurrent builds > ---------------------------------------- > > Key: OVIRT-726 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-726 > Project: oVirt - virtualization made easy > Issue Type: Task > Components: Jenkins > Reporter: Evgheni Dereveanchin > Assignee: infra > > I saw the build queue rise to ~200 tasks today due to these two jobs configured for running only 6 builds in parallel: > ovirt-engine_master_upgrade-from-4.0_el7_created > ovirt-engine_master_upgrade-from-master_el7_created > I bumped the default limit to 10 in https://gerrit.ovirt.org/61319/ but engine-upgrade job override this setting 6: > https://gerrit.ovirt.org/gitweb?p=jenkins.git;a=blob;f=jobs/confs/yaml/temp… > there's more places where various limits are set - I think we should review them and even remove if possible as the number of slaves is going to increase. > git grep max-total > jobs/confs/projects/defaults.yaml: max-total: 3 > jobs/confs/projects/defaults.yaml: max-total: 3 > jobs/confs/projects/ovirt/publish-rpms-nightly-3.6.yaml: max-total: 3 > jobs/confs/projects/ovirt/publish-rpms-nightly-4.0.yaml: max-total: 3 > jobs/confs/projects/ovirt/publish-rpms-nightly-master.yaml: max-total: 3 > jobs/confs/yaml/templates/engine-upgrade.yaml: max-total: 6 > jobs/confs/yaml/templates/find-bugs.yaml: max-total: 3 > jobs/confs/yaml/templates/install-rpm-sanity.yaml: max-total: 3 > jobs/confs/yaml/templates/ovirt-node-job-templates.yaml: max-total: 3 > jobs/confs/yaml/templates/standard-stage.yaml: max-total: 10 > jobs/confs/yaml/templates/standard-stage.yaml: max-total: 1 -- This message was sent by Atlassian JIRA (v1000.319.1#100012)

1 0

Jenkins build became unstable: ovirt_3.6_he-system-tests #422
by jenkins＠jenkins.phx.ovirt.org 13 Sep '16

13 Sep '16

See <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/422/changes>

1 1

[JIRA] (OVIRT-729) Jenkins leaves behind connections in ESTABLISHED state
by Evgheni Dereveanchin (oVirt JIRA) 13 Sep '16

13 Sep '16

Evgheni Dereveanchin created OVIRT-729: ------------------------------------------ Summary: Jenkins leaves behind connections in ESTABLISHED state Key: OVIRT-729 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-729 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Evgheni Dereveanchin Assignee: infra Priority: Low Throughout the last days I noticed several times when a slave got non-responsive and after rebooting it never recovered until being removed/re-added to Jenkins. Further investigation on el7-vm06.phx.ovirt.org revealed that on Jenkins the connection is still marked as "ESTABLISHED" hours after the slave crashed and rebooted: {quote} tcp6 0 0 66.187.230.92:33374 66.187.230.77:22 ESTABLISHED 14625/java {quote} Due to this new SSH connections are not opened and Jenkins can't reconnect to the slave. I didn't see this behavior before, maybe it is related to the latest security upgrade of jenkins. -- This message was sent by Atlassian JIRA (v1000.319.1#100012)

1 0

[JIRA] (OVIRT-728) Jenkins data partition running out of space
by Evgheni Dereveanchin (oVirt JIRA) 13 Sep '16

13 Sep '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-728?page=com.atlassian.jira.p… ] Evgheni Dereveanchin reassigned OVIRT-728: ------------------------------------------ Assignee: Evgheni Dereveanchin (was: infra) > Jenkins data partition running out of space > ------------------------------------------- > > Key: OVIRT-728 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-728 > Project: oVirt - virtualization made easy > Issue Type: Task > Reporter: Evgheni Dereveanchin > Assignee: Evgheni Dereveanchin > Priority: High > Attachments: jenkins_disk_usage.png > > > jenkins.phx.ovirt.org has only 8% of free space left on /var/lib/data > Size Used Avail Use% Mounted on > 700G 638G 63G 92% /var/lib/data > We still have around 60G left but it's a bad idea to have less than 20% free space on a partition. Also, as job history was increased, disk space usage will continue to grow. The amount of data saved in a week is around 50G so we need to act fast -- This message was sent by Atlassian JIRA (v1000.319.1#100012)

1 0

[JIRA] (OVIRT-728) Jenkins data partition running out of space
by eyal edri [Administrator] (oVirt JIRA) 13 Sep '16

13 Sep '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-728?page=com.atlassian.jira.p… ] eyal edri [Administrator] commented on OVIRT-728: ------------------------------------------------- This might be due to increased history to 120 on all jobs, which is needed for debugging errors on jobs. Please extend the disk as needed, probably to 1TB for now and lets see how it goes. > Jenkins data partition running out of space > ------------------------------------------- > > Key: OVIRT-728 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-728 > Project: oVirt - virtualization made easy > Issue Type: Task > Reporter: Evgheni Dereveanchin > Assignee: infra > Priority: High > > jenkins.phx.ovirt.org has only 8% of free space left on /var/lib/data > Size Used Avail Use% Mounted on > 700G 638G 63G 92% /var/lib/data > We still have around 60G left but it's a bad idea to have less than 20% free space on a partition. Also, as job history was increased, disk space usage will continue to grow. The amount of data saved in a week is around 50G so we need to act fast -- This message was sent by Atlassian JIRA (v1000.319.1#100012)

1 0

[JIRA] (OVIRT-728) Jenkins data partition running out of space
by Evgheni Dereveanchin (oVirt JIRA) 13 Sep '16

13 Sep '16

Evgheni Dereveanchin created OVIRT-728: ------------------------------------------ Summary: Jenkins data partition running out of space Key: OVIRT-728 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-728 Project: oVirt - virtualization made easy Issue Type: Task Reporter: Evgheni Dereveanchin Assignee: infra Priority: High jenkins.phx.ovirt.org has only 8% of free space left on /var/lib/data Size Used Avail Use% Mounted on 700G 638G 63G 92% /var/lib/data We still have around 60G left but it's a bad idea to have less than 20% free space on a partition. Also, as job history was increased, disk space usage will continue to grow. The amount of data saved in a week is around 50G so we need to act fast -- This message was sent by Atlassian JIRA (v1000.319.1#100012)

1 0