- Infra - oVirt List Archives

[oVirt Jenkins] deploy-to-ovirt_experimental - Build #3795 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 25 Jul '17

25 Jul '17

------=_Part_2_1886544267.1500974426698 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Build: http://jenkins.ovirt.org/job/deploy-to-ovirt_experimental/3795/ Build Name: 3795 ovirt-engine-api-model (4.1) Build Status: FAILURE Gerrit change: https://gerrit.ovirt.org/79787 - title: Clarify that the CloudInit type is deprecated - project: ovirt-engine-api-model - branch: model_4.1 - author: Juan Hernandez <juan.hernandez(a)redhat.com> ------=_Part_2_1886544267.1500974426698--

1 1

[oVirt Jenkins] deploy-to-ovirt_experimental - Build #3794 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 25 Jul '17

25 Jul '17

------=_Part_4_1571992509.1500974475756 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Build: http://jenkins.ovirt.org/job/deploy-to-ovirt_experimental/3794/ Build Name: 3794 ovirt-engine-api-model (master) Build Status: FAILURE Gerrit change: https://gerrit.ovirt.org/79545 - title: Clarify that the CloudInit type is deprecated - project: ovirt-engine-api-model - branch: master - author: Juan Hernandez <juan.hernandez(a)redhat.com> ------=_Part_4_1571992509.1500974475756--

1 0

'change-queue' is now in full production for the 'master' flow
by Barak Korren 25 Jul '17

25 Jul '17

Yesterday we've applied a patch with JJB that made the 'master' change queue push updates into 'tested' instead of the 'experimental' flow. This essentially means that the change queue is now the main production flow for 'master'. A quick refresher on how to find and handle issues with CQ jobs: 1. You don't really need to monitor the jobs, when the CQ suspects it found a real code issue it will send infra(a)ovirt.org an email with the suspected patch. 2. The *-tester job is what actually runs the tests. Don't worry if it fails occasionally, the CQ will run bisection and tell you exactly which change caused it. 3. The output of the *-tester job is very similar to the old 'experimental' job. 4. When you get an email, you need to check the output of the job linked to it to see if this is a race-condition kind of failure, a build failure or a real regression. Unless its a build failure, you probably need to report it and document in the table. 5. If any other CQ job besides the *-tester job fails, this is a real infra issue. When you see such an issue you need to investigate it and fix the infra. Real infra issues typically affect many jobs, so once you are done fixing it, you probably need to go back and re-trigger Gerrit merge events to get packages rebuilt and submitted to the queue. 6. One very simple kind of an infra issue, is trying to submit to a queue that does not exist. You should nag maintainers to remove jobs for versions of oVirt that we don't have queues for. Good luck! -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

1 0

Jenkins build is back to normal : system-sync_mirrors-centos-updates-el6-x86_64 #574
by jenkins＠jenkins.phx.ovirt.org 25 Jul '17

25 Jul '17

See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-el6-x86_64/…>

1 0

[JIRA] (OVIRT-1551) Add slaves to staging Jenkins instance
by Barak Korren (oVirt JIRA) 24 Jul '17

24 Jul '17

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1551?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1551: -------------------------------- Epic Link: OVIRT-403 > Add slaves to staging Jenkins instance > -------------------------------------- > > Key: OVIRT-1551 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1551 > Project: oVirt - virtualization made easy > Issue Type: New Feature > Components: oVirt Infra > Reporter: Barak Korren > Assignee: infra > Labels: jenkins, slaves > > We currently have just one slave attached to the staging Jenkins instance. > We need some more, at least one of each slave platform we support so that we can: > # Test jobs running in paralllel > # Test jobs running on different slave platforms. -- This message was sent by Atlassian JIRA (v1000.1139.0#100055)

1 0

[JIRA] (OVIRT-1551) Add slaves to staging Jenkins instance
by Barak Korren (oVirt JIRA) 24 Jul '17

24 Jul '17

Barak Korren created OVIRT-1551: ----------------------------------- Summary: Add slaves to staging Jenkins instance Key: OVIRT-1551 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1551 Project: oVirt - virtualization made easy Issue Type: New Feature Components: oVirt Infra Reporter: Barak Korren Assignee: infra We currently have just one slave attached to the staging Jenkins instance. We need some more, at least one of each slave platform we support so that we can: # Test jobs running in paralllel # Test jobs running on different slave platforms. -- This message was sent by Atlassian JIRA (v1000.1139.0#100055)

1 0

Jenkins error: java.io.FileNotFoundException: ... (Too many open files)
by Nir Soffer 24 Jul '17

24 Jul '17

Jenkins needs restart? Building remotely on ovirt-srv21.phx.ovirt.org <http://jenkins.ovirt.org/computer/ovirt-srv21.phx.ovirt.org> (fc24 phx integ-tests physical) in workspace /home/jenkins/workspace/ovirt-system-tests_manual java.io.FileNotFoundException: /var/lib/data/jenkins/jobs/ovirt-system-tests_manual/builds/856/changelog.xml (Too many open files) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:162) at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:119) at hudson.scm.SCM.checkout(SCM.java:495) at hudson.model.AbstractProject.checkout(AbstractProject.java:1276) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:560) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:485) at hudson.model.Run.execute(Run.java:1735) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:405) ERROR: Step ?Post build task? aborted due to exception:

2 2

[JIRA] (OVIRT-1550) Configure and monitor nofiles on Jenkins
by Barak Korren (oVirt JIRA) 24 Jul '17

24 Jul '17

Barak Korren created OVIRT-1550: ----------------------------------- Summary: Configure and monitor nofiles on Jenkins Key: OVIRT-1550 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1550 Project: oVirt - virtualization made easy Issue Type: Improvement Components: oVirt CI Reporter: Barak Korren Assignee: infra Priority: High We had an issue today (Jul 25th, 2017) of Jenkins running out of file descriptors. Things already reached the point where Jenkins was barely functioning so to get things back to working order I've: * Manually added a line in '{{/etc/security/limits.conf}}' to set 'nofiles' to 16384 for the 'jenkins' user (Previously it was just not configured and seems to have defaulted to 4096). * Restarted the 'jenkins' service. We now need to: * Place this configuration in Puppet/Ansible. * Add monitoring so we'll know in advance the next time Jenkins goes near to the nofiles limit. -- This message was sent by Atlassian JIRA (v1000.1139.0#100055)

1 0

[JIRA] (OVIRT-1550) Configure and monitor nofiles on Jenkins
by Barak Korren (oVirt JIRA) 24 Jul '17

24 Jul '17

[ https://ovirt-jira.atlassian.net/browse/OVIRT-1550?page=com.atlassian.jira.… ] Barak Korren updated OVIRT-1550: -------------------------------- Epic Link: OVIRT-403 > Configure and monitor nofiles on Jenkins > ---------------------------------------- > > Key: OVIRT-1550 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1550 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Components: oVirt CI > Reporter: Barak Korren > Assignee: infra > Priority: High > Labels: infra, jenkins > > We had an issue today (Jul 25th, 2017) of Jenkins running out of file descriptors. > Things already reached the point where Jenkins was barely functioning so to get things back to working order I've: > * Manually added a line in '{{/etc/security/limits.conf}}' to set 'nofiles' to 16384 for the 'jenkins' user (Previously it was just not configured and seems to have defaulted to 4096). > * Restarted the 'jenkins' service. > We now need to: > * Place this configuration in Puppet/Ansible. > * Add monitoring so we'll know in advance the next time Jenkins goes near to the nofiles limit. -- This message was sent by Atlassian JIRA (v1000.1139.0#100055)

1 0

Jenkins was restarted
by Barak Korren 24 Jul '17

24 Jul '17

Jenkins ran out of file descriptors so we had to restart it to bump the number. Jobs should be running jobs already but the UI will probably take 30-40 minutes to become available. Issues seem to have started around 15:00 UTC yesterday (18:00 IST) so you may need to re-trigger jobs for patches submitted since then. -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

1 0