[JIRA] (OVIRT-919) Fwd: CI slaves extremely slow - overloaded slaves?
by Barak Korren (oVirt JIRA)
Barak Korren created OVIRT-919:
----------------------------------
Summary: Fwd: CI slaves extremely slow - overloaded slaves?
Key: OVIRT-919
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-919
Project: oVirt - virtualization made easy
Issue Type: By-EMAIL
Reporter: Barak Korren
Assignee: infra
More discussion and tracking needed, moving to Jira (please put any
further discussion there)
---------- Forwarded message ----------
From: Nir Soffer <nsoffer(a)redhat.com>
Date: 7 December 2016 at 21:33
Subject: CI slaves extremely slow - overloaded slaves?
To: infra <infra(a)ovirt.org>, Eyal Edri <eedri(a)redhat.com>, Dan
Kenigsberg <danken(a)redhat.com>
Hi all,
In the last weeks we see more and more test failures due to timeouts in the CI.
For example:
17:19:49 ======================================================================
17:19:49 FAIL: test_scale (storage_filesd_test.GetAllVolumesTests)
17:19:49 ----------------------------------------------------------------------
17:19:49 Traceback (most recent call last):
17:19:49 File
"/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/tests/storage_filesd_test.py",
line 165, in test_scale
17:19:49 self.assertTrue(elapsed < 1.0, "Elapsed time: %f seconds"
% elapsed)
17:19:49 AssertionError: Elapsed time: 1.105877 seconds
17:19:49 -------------------- >> begin captured stdout << ---------------------
17:19:49 1.105877 seconds
This test runs in 0.048 seconds on my laptop:
$ ./run_tests_local.sh storage_filesd_test.py -s
nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
storage_filesd_test.GetAllVolumesTests
test_no_templates OK
test_no_volumes OK
test_scale 0.047932 seconds
OK
test_with_template OK
----------------------------------------------------------------------
Ran 4 tests in 0.189s
It seems that we are overloading the CI slaves. We should not use nested kvm
for the CI, such vms are much slower then regular vms, and we probably run
too many vms per cpu.
We can disable such tests in the CI, but we do want to know when there is
a regression in this code. Before it was fixed, the same test took 9 seconds
on my laptop. We need fast machines in the CI for this.
Nir
_______________________________________________
Infra mailing list
Infra(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
--
Barak Korren
bkorren(a)redhat.com
RHCE, RHCi, RHV-DevOps Team
https://ifireball.wordpress.com/
--
This message was sent by Atlassian JIRA
(v1000.620.0#100023)
7 years, 11 months
oVirt infra daily report - unstable production jobs - 161
by jenkins＠jenkins.phx.ovirt.org
------=_Part_41_1154478995.1481151609825
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Good morning!
Attached is the HTML page with the jenkins status report. You can see it also here:
- http://jenkins.ovirt.org/job/system_jenkins-report/161//artifact/exported...
Cheers,
Jenkins
------=_Part_41_1154478995.1481151609825
Content-Type: text/html; charset=us-ascii; name=upstream_report.html
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename=upstream_report.html
Content-ID: <upstream_report.html>
<!DOCTYPE html><head><style type="text/css">
table.gridtable {
border-collapse: collapse;
table-layout:fixed;
width:1600px;
font-family: monospace;
font-size:13px;
}
.head {
font-size:20px;
font-family: arial;
}
.sub {
font-size:18px;
background-color:#e5e5e5;
font-family: arial;
}
pre {
font-family: monospace;
display: inline;
white-space: pre-wrap;
white-space: -moz-pre-wrap !important;
white-space: -pre-wrap;
white-space: -o-pre-wrap;
word-wrap: break-word;
}
</style>
</head>
<body>
<table class="gridtable" border=2>
<tr><th colspan=2 class=head>
RHEVM CI Jenkins Daily Report - 07/12/2016
</th></tr><tr><th colspan=2 class=sub>
<font color="blue"><a href="http://jenkins.ovirt.org/">00 Unstable Jobs (Production)</a></font>
</th></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/cockpit-ovirt_ovirt-4.1_build-artifacts-fc23...">cockpit-ovirt_ovirt-4.1_build-artifacts-fc23-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/fabric-ovirt_master_check-merged-el7-x86_64/">fabric-ovirt_master_check-merged-el7-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt-engine-api-explorer_master_build-artif...">ovirt-engine-api-explorer_master_build-artifacts-fc23-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt-guest-agent_master_build-artifacts-el6...">ovirt-guest-agent_master_build-artifacts-el6-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt-guest-agent_master_build-artifacts-fc2...">ovirt-guest-agent_master_build-artifacts-fc23-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt-hosted-engine-ha_master_build-artifact...">ovirt-hosted-engine-ha_master_build-artifacts-fc24-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt-live_master_experimental_create-iso-el...">ovirt-live_master_experimental_create-iso-el7-x86_64</a>
</td><td>
<h3>This job generates a nightly iso of ovirt-live using experimental repos</h3>
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt-node-ng_master_build-artifacts-el7-x86...">ovirt-node-ng_master_build-artifacts-el7-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt-node-ng_ovirt-3.6_build-artifacts-el7-...">ovirt-node-ng_ovirt-3.6_build-artifacts-el7-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt-node-ng_ovirt-4.0_build-artifacts-el7-...">ovirt-node-ng_ovirt-4.0_build-artifacts-el7-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt-node-ng_ovirt-4.1-pre_build-artifacts-...">ovirt-node-ng_ovirt-4.1-pre_build-artifacts-el7-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt-node-ng_ovirt-master-experimental_buil...">ovirt-node-ng_ovirt-master-experimental_build-artifacts-el7-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt-node_ovirt-3.6_create-iso-el7_merged/">ovirt-node_ovirt-3.6_create-iso-el7_merged</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt-optimizer_4.0_build-artifacts-el6-x86_64/">ovirt-optimizer_4.0_build-artifacts-el6-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/">ovirt_3.6_he-system-tests</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/repos_master_check-closure_fc24_merged/">repos_master_check-closure_fc24_merged</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/vdsm_4.0_check-merged-fc23-x86_64/">vdsm_4.0_check-merged-fc23-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
------=_Part_41_1154478995.1481151609825--
7 years, 11 months
Vdsm tests are 4X times faster on travis
by Nir Soffer
HI all,
Watching vdsm travis builds in the last weeks, it is clear that vdsm tests
on travis are about 4X times faster compared with jenkins builds.
Here is a typical build:
ovirt ci: http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/5101/con...
travis ci: https://travis-ci.org/nirs/vdsm/builds/179056079
The build took 4:34 on travis, and 19:34 on ovirt ci.
This has a huge impact on vdsm maintainers. Having to wait 20 minutes
for each patch
means that we must ignore the ci and merge and hope that previous tests without
rebasing on master were good enough.
The builds are mostly the same, expect:
- In travis we don't check if the build system was changed and
packages should be built
takes 9:18 minutes in ovirt ci.
- In travis we don't clean or install anything before the test, we use
a container with all the
available packages, pulled from dockerhub.
takes about 3:52 minutes in ovirt ci
- In travis we don't enable coverage. Running the tests with coverage
may slow down the tests
takes 5:04 minutes in ovirt ci
creating the coverage report takes only 15 seconds, not interesting
- In travis we don't cleanup anything after the test
this takes 34 seconds in ovirt ci
The biggest problem is the build system check taking 9:18 minutes.
fixing it will cut the build time in half.
This is how time is spent in ovirt ci:
1. Starting (1:28)
00:00:00.001 Triggered by Gerrit: https://gerrit.ovirt.org/67338
00:00:00.039 [EnvInject] - Loading node environment variables.
00:00:00.056 Building remotely on fc24-vm15.phx.ovirt.org (phx nested
local_disk fc24) in workspace
/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64
2. Installing packages (2:24)
00:01:28.338 INFO: installing package(s): autoconf automake gdb git
libguestfs-tools-c libselinux-python3 libvirt-python3 m2crypto make
mom openvswitch policycoreutils-python PyYAML python-blivet
python-coverage python2-decorator python-devel python-inotify
python-ioprocess python-mock python-netaddr python-pthreading
python-setuptools python-six python-requests python3-decorator
python3-netaddr python3-nose python3-six python3-yaml rpm-build
sanlock-python sudo yum yum-utils
3. Setup in check-patch.sh (00:04)
00:03:52.838 + export VDSM_AUTOMATION=1
4. Running the actual tests
00:03:56.670 + ./autogen.sh --system --enable-hooks --enable-vhostmd
5. Installing python-debugifo (00:19)
00:04:15.385 Yum-utils package has been deprecated, use dnf instead.
6. Running make check (05:04)
00:04:30.948 + TIMEOUT=600
00:04:30.948 + make check NOSE_WITH_COVERAGE=1
NOSE_COVER_PACKAGE=/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/vdsm,/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/lib
...
00:09:33.981 tests: commands succeeded
00:09:33.981 congratulations :)
7. Creating coverage report (00:15)
00:09:34.017 + coverage html -d
/home/jenkins/workspace/vdsm_master_check-patch-fc24-x86_64/vdsm/exported-artifacts/htmlcov
8. Finding modified files (09:18)
00:09:49.213 + git diff-tree --no-commit-id --name-only -r HEAD
00:09:49.213 + egrep --quiet 'vdsm.spec.in|Makefile.am'
9. Cleaing up (00:27)
00:19:07.994 Took 915 seconds
00:19:34.973 Finished: SUCCESS
7 years, 11 months