9:48 a.m.
This is a multi-part message in MIME format.
--------------5185435FD38913B821CED967
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Hi,
We had a failure in upgrade suite for 002_bootstrap.add_hosts. I am not
seeing any error that can suggest on an issue in engine.
I can see in the host messages host that we have stopped writing to the
log for 15 minutes and it may suggest that there is something that is
keeping the host from starting which causes us to fail the test on timeout.
However, i can use some help in determining the cause for this failure
and weather its connected to the bootstrap_add_host test in upgrade.
**
*Link to suspected patches: As I said,**I do not think its related, but
this is the patch that was reported. *
*
https://gerrit.ovirt.org/#/c/83854/
Link to Job:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3795/*
*
*
*Link to all logs:*
*http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3795/artifact/*
*
*
*(Relevant) error snippet from the log: *
*
<error>
**
_*Test error: *_**
**
*
Error Message
False != True after 900 seconds
Stacktrace
Traceback (most recent call last):
File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
testMethod()
File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in
wrapped_test
test()
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in
wrapper
return func(get_test_prefix(), *args, **kwargs)
File
"/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/upgrade-from-release-suite-master/test-scenarios-after-upgrade/002_bootstrap.py",
line 187, in add_hosts
testlib.assert_true_within(_host_is_up_4, timeout=15*60)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 263, in
assert_true_within
assert_equals_within(func, True, timeout, allowed_exceptions)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 237, in
assert_equals_within
'%s != %s after %s seconds' % (res, value, timeout)
AssertionError: False != True after 900 seconds
**
**
**
**_lago log: _*
*
*2017-11-13
15:31:23,212::log_utils.py::__enter__::600::lago.prefix::INFO::ESC[0mESC[0m
2017-11-13
15:31:23,213::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:0468ed2f-b174-4d94-bc66-2b6e08087a86:Get ssh client for
lago-upgrade-from-release-suite-master-host0:
2017-11-13
15:31:23,213::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:91372413-f7fd-4b72-85b9-9f5216ca7ae9:Get ssh client for
lago-upgrade-from-release-suite-master-engine:
2017-11-13
15:31:23,213::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1
tries for lago-upgrade-from-release-suite-master-host0
2017-11-13
15:31:23,213::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1
tries for lago-upgrade-from-release-suite-master-engine
2017-11-13
15:31:26,220::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error
connecting to lago-upgrade-from-release-suite-master-engine: [Errno
None] Unable to connect to port 22 on 192.168.200.3
2017-11-13
15:31:26,221::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error
connecting to lago-upgrade-from-release-suite-master-host0: [Errno None]
Unable to connect to port 22 on 192.168.200.2
2017-11-13
15:31:27,222::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:91372413-f7fd-4b72-85b9-9f5216ca7ae9:Get ssh client for
lago-upgrade-from-release-suite-master-engine:
2017-11-13
15:31:27,222::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:0468ed2f-b174-4d94-bc66-2b6e08087a86:Get ssh client for
lago-upgrade-from-release-suite-master-host0:
2017-11-13 15:31:27,222::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got
exception while sshing to lago-upgrade-from-release-suite-master-engine:
Timed out (in 4 s) trying to ssh to
lago-upgrade-from-release-suite-master-engine
2017-11-13 15:31:27,222::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got
exception while sshing to lago-upgrade-from-release-suite-master-host0:
Timed out (in 4 s) trying to ssh to
lago-upgrade-from-release-suite-master-host0
2017-11-13
15:31:28,224::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:fc88f641-b012-4636-a471-9ccaaf361a53:Get ssh client for
lago-upgrade-from-release-suite-master-engine:
2017-11-13
15:31:28,224::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:afb01a46-6338-407f-b7c9-9d5c6b91404d:Get ssh client for
lago-upgrade-from-release-suite-master-host0:
2017-11-13
15:31:28,224::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1
tries for lago-upgrade-from-release-suite-master-engine
2017-11-13
15:31:28,225::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1
tries for lago-upgrade-from-release-suite-master-host0
2017-11-13
15:31:28,226::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error
connecting to lago-upgrade-from-release-suite-master-host0: [Errno None]
Unable to connect to port 22 on 192.168.200.2
2017-11-13
15:31:29,228::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:afb01a46-6338-407f-b7c9-9d5c6b91404d:Get ssh client for
lago-upgrade-from-release-suite-master-host0:
2017-11-13 15:31:29,228::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got
exception while sshing to lago-upgrade-from-release-suite-master-host0:
Timed out (in 1 s) trying to ssh to
lago-upgrade-from-release-suite-master-host0
2017-11-13
15:31:29,229::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error
connecting to lago-upgrade-from-release-suite-master-engine: [Errno
None] Unable to connect to port 22 on 192.168.200.3
2017-11-13
15:31:30,229::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:fc88f641-b012-4636-a471-9ccaaf361a53:Get ssh client for
lago-upgrade-from-release-suite-master-engine:
2017-11-13
15:31:30,230::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:9df5ddca-bbc4-485a-9f3a-b0b9a5bb5990:Get ssh client for
lago-upgrade-from-release-suite-master-host0:
2017-11-13 15:31:30,230::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got
exception while sshing to lago-upgrade-from-release-suite-master-engine:
Timed out (in 2 s) trying to ssh to
lago-upgrade-from-release-suite-master-engine
2017-11-13
15:31:30,230::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1
tries for lago-upgrade-from-release-suite-master-host0
2017-11-13
15:31:30,231::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error
connecting to lago-upgrade-from-release-suite-master-host0: [Errno None]
Unable to connect to port 22 on 192.168.200.2
2017-11-13
15:31:31,231::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:53fe67da-a632-49fe-b697-e8a2c4cb7d23:Get ssh client for
lago-upgrade-from-release-suite-master-engine:
2017-11-13
15:31:31,232::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1
tries for lago-upgrade-from-release-suite-master-engine
2017-11-13
15:31:31,232::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error
connecting to lago-upgrade-from-release-suite-master-engine: [Errno
None] Unable to connect to port 22 on 192.168.200.3
2017-11-13
15:31:31,232::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:9df5ddca-bbc4-485a-9f3a-b0b9a5bb5990:Get ssh client for
lago-upgrade-from-release-suite-master-host0:
2017-11-13 15:31:31,233::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got
exception while sshing to lago-upgrade-from-release-suite-master-host0:
Timed out (in 1 s) trying to ssh to
lago-upgrade-from-release-suite-master-host0
2017-11-13
15:31:32,234::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:53fe67da-a632-49fe-b697-e8a2c4cb7d23:Get ssh client for
lago-upgrade-from-release-suite-master-engine:
2017-11-13 15:31:32,234::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got
exception while sshing to lago-upgrade-from-release-suite-master-engine:
Timed out (in 1 s) trying to ssh to
lago-upgrade-from-release-suite-master-engine
2017-11-13
15:31:32,234::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:1969bf5b-4e66-43d7-91c6-a28949e98fe8:Get ssh client for
lago-upgrade-from-release-suite-master-host0:
2017-11-13
15:31:32,234::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1
tries for lago-upgrade-from-release-suite-master-host0
2017-11-13
15:31:32,235::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error
connecting to lago-upgrade-from-release-suite-master-host0: [Errno None]
Unable to connect to port 22 on 192.168.200.2
2017-11-13
15:31:33,235::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:5fd0e3c3-d83c-46f9-9c88-bc739aa9c430:Get ssh client for
lago-upgrade-from-release-suite-master-engine:
2017-11-13
15:31:33,235::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1
tries for lago-upgrade-from-release-suite-master-engine
2017-11-13
15:31:33,236::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:1969bf5b-4e66-43d7-91c6-a28949e98fe8:Get ssh client for
lago-upgrade-from-release-suite-master-host0:
2017-11-13
15:31:33,236::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error
connecting to lago-upgrade-from-release-suite-master-engine: [Errno
None] Unable to connect to port 22 on 192.168.200.3
2017-11-13 15:31:33,237::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got
exception while sshing to lago-upgrade-from-release-suite-master-host0:
Timed out (in 1 s) trying to ssh to
lago-upgrade-from-release-suite-master-host0
2017-11-13
15:31:34,238::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:5fd0e3c3-d83c-46f9-9c88-bc739aa9c430:Get ssh client for
lago-upgrade-from-release-suite-master-engine:
2017-11-13 15:31:34,238::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got
exception while sshing to lago-upgrade-from-release-suite-master-engine:
Timed out (in 1 s) trying to ssh to
lago-upgrade-from-release-suite-master-engine
2017-11-13
15:31:34,238::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:22550d6e-c397-43dc-9691-c8d9debfe416:Get ssh client for
lago-upgrade-from-release-suite-master-host0:
2017-11-13
15:31:34,239::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got 1
tries for lago-upgrade-from-release-suite-master-host0
2017-11-13
15:31:34,407::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:22550d6e-c397-43dc-9691-c8d9debfe416:Get ssh client for
lago-upgrade-from-release-suite-master-host0:
2017-11-13 15:31:34,639::ssh.py::ssh::58::lago.ssh::DEBUG::Running
bb49adfa on lago-upgrade-from-release-suite-master-host0: true
2017-11-13 15:31:34,665::ssh.py::ssh::81::lago.ssh::DEBUG::Command
bb49adfa on lago-upgrade-from-release-suite-master-host0 returned with 0
2017-11-13
15:31:34,665::ssh.py::wait_for_ssh::153::lago.ssh::DEBUG::Wait succeeded
for ssh to lago-upgrade-from-release-suite-master-host0
2017-11-13
15:31:34,665::log_utils.py::__exit__::611::lago.prefix::INFO::ESC[32mSuccessESC[0m
(in 0:00:11)
2017-11-13
15:31:34,666::log_utils.py::__enter__::600::lago.prefix::INFO::ESC[0mESC[0m
2017-11-13
15:31:34,666::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:4e2a7421-28f7-4679-955f-7261fe377939:Get ssh client for
lago-upgrade-from-release-suite-master-host0:
2017-11-13
15:31:34,667::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still got
100 tries for lago-upgrade-from-release-suite-master-host0
2017-11-13
15:31:34,805::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:4e2a7421-28f7-4679-955f-7261fe377939:Get ssh client for
lago-upgrade-from-release-suite-master-host0:
2017-11-13 15:31:34,906::ssh.py::ssh::58::lago.ssh::DEBUG::Running
bb724bf2 on lago-upgrade-from-release-suite-master-host0: bash -s < "set -xe
DIST=$(uname -r | sed -r 's/^.*\.([^\.]+)\.[^\.]+$/\1/')
*
*
*
**_from host: _*
*
ions. Proceeding anyway.
Nov 13 10:31:34 lago-upgrade-from-release-suite-master-host0 systemd:
Configuration file /usr/lib/systemd/system/ebtables.service is marked
executable. Please remove executable permission bits. Proceeding anyway.
Nov 13 10:31:34 lago-upgrade-from-release-suite-master-host0 systemd:
Configuration file /usr/lib/systemd/system/wpa_supplicant.service is
marked executable. Please remove executable permission bits. Proceeding
anyway.
Nov 13 10:31:55 lago-upgrade-from-release-suite-master-host0 yum[1294]:
Updated: iptables-1.4.21-18.2.el7_4.x86_64
Nov 13 10:31:56 lago-upgrade-from-release-suite-master-host0
systemd-logind: Removed session 3.
Nov 13 10:31:56 lago-upgrade-from-release-suite-master-host0 systemd:
Removed slice user-0.slice.
Nov 13 10:31:56 lago-upgrade-from-release-suite-master-host0 systemd:
Stopping user-0.slice.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd:
Created slice user-0.slice.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd:
Starting user-0.slice.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd:
Started Session 4 of user root.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd:
Starting Session 4 of user root.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0
systemd-logind: New session 4 of user root.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0
systemd-logind: Removed session 4.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd:
Removed slice user-0.slice.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd:
Stopping user-0.slice.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd:
Created slice user-0.slice.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd:
Starting user-0.slice.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd:
Started Session 5 of user root.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd:
Starting Session 5 of user root.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0
systemd-logind: New session 5 of user root.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0
systemd-logind: Removed session 5.
Nov 13 10:44:01 lago-upgrade-from-release-suite-master-host0 systemd:
Removed slice user-0.slice.
Nov 13 10:44:01 lago-upgrade-from-release-suite-master-host0 systemd:
Stopping user-0.slice.
Nov 13 10:46:27 lago-upgrade-from-release-suite-master-host0 systemd:
Starting Cleanup of Temporary Directories...
Nov 13 10:46:27 lago-upgrade-from-release-suite-master-host0 systemd:
Started Cleanup of Temporary Directories.
Nov 13 10:54:36 lago-upgrade-from-release-suite-master-host0
dhclient[906]: DHCPREQUEST on eth0 to 192.168.200.1 port 67 (xid=0x564629cb)
Nov 13 10:54:36 lago-upgrade-from-release-suite-master-host0
dhclient[906]: DHCPACK from 192.168.200.1 (xid=0x564629cb)**
*
*
*</error>*
**
--------------5185435FD38913B821CED967
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="content-type" content="text/html;
charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi, <br>
</p>
<p>We had a failure in upgrade suite for 002_bootstrap.add_hosts. I
am not seeing any error that can suggest on an issue in engine. <br>
</p>
<p>I can see in the host messages host that we have stopped writing
to the log for 15 minutes and it may suggest that there is
something that is keeping the host from starting which causes us
to fail the test on timeout. <br>
</p>
<p>However, i can use some help in determining the cause for this
failure and weather its connected to the bootstrap_add_host test
in upgrade.<br>
</p>
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-ba60-843c-cfdd-7366571bcb57">
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link
to suspected patches: As I said,<b> </b>I do not think its related, but this
is the patch that was reported.
</span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">
</span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><a
class="moz-txt-link-freetext"
href="https://gerrit.ovirt.org/#/c/83854/">https://gerrit.ov...
<br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link
to Job:</span></p>
<a class="moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-ba60-843c-cfdd-7366571bcb57"><br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link
to all logs:</span></p>
<a class="moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-ba60-843c-cfdd-7366571bcb57"><br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant)
error snippet from the log: </span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error></span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">
</span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">
</span></p>
</b><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-ba60-843c-cfdd-7366571bcb57">
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><u><font
size="+1"><b>Test error: </b></font></u><b>
</b></span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><b>
</b></span></p>
</b></p>
<h3 style="box-sizing: border-box; color: rgb(51, 51, 51);
font-family: Helvetica, Arial, sans-serif; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
letter-spacing: normal; orphans: 2; text-align: start;
text-indent: 0px; text-transform: none; white-space: normal;
widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration-style: initial; text-decoration-color: initial;">Error
Message</h3>
<pre style="box-sizing: border-box; white-space: pre-wrap; word-wrap:
break-word; margin: 0px; color: rgb(51, 51, 51); font-size: 14px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal;
letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform:
none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style:
initial; text-decoration-color: initial;">False != True after 900
seconds</pre>
<h3 style="box-sizing: border-box; color: rgb(51, 51, 51);
font-family: Helvetica, Arial, sans-serif; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
letter-spacing: normal; orphans: 2; text-align: start;
text-indent: 0px; text-transform: none; white-space: normal;
widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration-style: initial; text-decoration-color:
initial;">Stacktrace</h3>
<pre style="box-sizing: border-box; white-space: pre-wrap; word-wrap:
break-word; margin: 0px; color: rgb(51, 51, 51); font-size: 14px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal;
letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform:
none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style:
initial; text-decoration-color: initial;">Traceback (most recent call last):
File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
testMethod()
File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in
wrapped_test
test()
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in
wrapper
return func(get_test_prefix(), *args, **kwargs)
File
"/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/upgrade-from-release-suite-master/test-scenarios-after-upgrade/002_bootstrap.py",
line 187, in add_hosts
testlib.assert_true_within(_host_is_up_4, timeout=15*60)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 263, in
assert_true_within
assert_equals_within(func, True, timeout, allowed_exceptions)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 237, in
assert_equals_within
'%s != %s after %s seconds' % (res, value, timeout)
AssertionError: False != True after 900 seconds</pre>
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-ba60-843c-cfdd-7366571bcb57">
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></span></p>
</b></p>
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-ba60-843c-cfdd-7366571bcb57"><b><u><font
size="+1">lago log:
</font></u></b><br>
</b></p>
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-ba60-843c-cfdd-7366571bcb57">2017-11-13
15:31:23,212::log_utils.py::__enter__::600::lago.prefix::<a
class="moz-txt-link-freetext"
href="INFO::ESC">INFO::ESC</a>[0mESC[0m<br>
2017-11-13
15:31:23,213::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:0468ed2f-b174-4d94-bc66-2b6e08087a86:Get ssh client for
lago-upgrade-from-release-suite-master-host0:<br>
2017-11-13
15:31:23,213::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:91372413-f7fd-4b72-85b9-9f5216ca7ae9:Get ssh client for
lago-upgrade-from-release-suite-master-engine:<br>
2017-11-13
15:31:23,213::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still
got 1 tries for lago-upgrade-from-release-suite-master-host0<br>
2017-11-13
15:31:23,213::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still
got 1 tries for lago-upgrade-from-release-suite-master-engine<br>
2017-11-13
15:31:26,220::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket
error connecting to
lago-upgrade-from-release-suite-master-engine: [Errno None]
Unable to connect to port 22 on 192.168.200.3<br>
2017-11-13
15:31:26,221::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket
error connecting to
lago-upgrade-from-release-suite-master-host0: [Errno None]
Unable to connect to port 22 on 192.168.200.2<br>
2017-11-13
15:31:27,222::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:91372413-f7fd-4b72-85b9-9f5216ca7ae9:Get ssh client for
lago-upgrade-from-release-suite-master-engine:<br>
2017-11-13
15:31:27,222::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:0468ed2f-b174-4d94-bc66-2b6e08087a86:Get ssh client for
lago-upgrade-from-release-suite-master-host0:<br>
2017-11-13
15:31:27,222::ssh.py::wait_for_<a class="moz-txt-link-freetext"
href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a>
exception while sshing to
lago-upgrade-from-release-suite-master-engine: Timed out (in 4
s) trying to ssh to
lago-upgrade-from-release-suite-master-engine<br>
2017-11-13
15:31:27,222::ssh.py::wait_for_<a class="moz-txt-link-freetext"
href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a>
exception while sshing to
lago-upgrade-from-release-suite-master-host0: Timed out (in 4 s)
trying to ssh to lago-upgrade-from-release-suite-master-host0<br>
2017-11-13
15:31:28,224::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:fc88f641-b012-4636-a471-9ccaaf361a53:Get ssh client for
lago-upgrade-from-release-suite-master-engine:<br>
2017-11-13
15:31:28,224::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:afb01a46-6338-407f-b7c9-9d5c6b91404d:Get ssh client for
lago-upgrade-from-release-suite-master-host0:<br>
2017-11-13
15:31:28,224::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still
got 1 tries for lago-upgrade-from-release-suite-master-engine<br>
2017-11-13
15:31:28,225::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still
got 1 tries for lago-upgrade-from-release-suite-master-host0<br>
2017-11-13
15:31:28,226::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket
error connecting to
lago-upgrade-from-release-suite-master-host0: [Errno None]
Unable to connect to port 22 on 192.168.200.2<br>
2017-11-13
15:31:29,228::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:afb01a46-6338-407f-b7c9-9d5c6b91404d:Get ssh client for
lago-upgrade-from-release-suite-master-host0:<br>
2017-11-13
15:31:29,228::ssh.py::wait_for_<a class="moz-txt-link-freetext"
href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a>
exception while sshing to
lago-upgrade-from-release-suite-master-host0: Timed out (in 1 s)
trying to ssh to lago-upgrade-from-release-suite-master-host0<br>
2017-11-13
15:31:29,229::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket
error connecting to
lago-upgrade-from-release-suite-master-engine: [Errno None]
Unable to connect to port 22 on 192.168.200.3<br>
2017-11-13
15:31:30,229::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:fc88f641-b012-4636-a471-9ccaaf361a53:Get ssh client for
lago-upgrade-from-release-suite-master-engine:<br>
2017-11-13
15:31:30,230::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:9df5ddca-bbc4-485a-9f3a-b0b9a5bb5990:Get ssh client for
lago-upgrade-from-release-suite-master-host0:<br>
2017-11-13
15:31:30,230::ssh.py::wait_for_<a class="moz-txt-link-freetext"
href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a>
exception while sshing to
lago-upgrade-from-release-suite-master-engine: Timed out (in 2
s) trying to ssh to
lago-upgrade-from-release-suite-master-engine<br>
2017-11-13
15:31:30,230::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still
got 1 tries for lago-upgrade-from-release-suite-master-host0<br>
2017-11-13
15:31:30,231::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket
error connecting to
lago-upgrade-from-release-suite-master-host0: [Errno None]
Unable to connect to port 22 on 192.168.200.2<br>
2017-11-13
15:31:31,231::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:53fe67da-a632-49fe-b697-e8a2c4cb7d23:Get ssh client for
lago-upgrade-from-release-suite-master-engine:<br>
2017-11-13
15:31:31,232::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still
got 1 tries for lago-upgrade-from-release-suite-master-engine<br>
2017-11-13
15:31:31,232::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket
error connecting to
lago-upgrade-from-release-suite-master-engine: [Errno None]
Unable to connect to port 22 on 192.168.200.3<br>
2017-11-13
15:31:31,232::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:9df5ddca-bbc4-485a-9f3a-b0b9a5bb5990:Get ssh client for
lago-upgrade-from-release-suite-master-host0:<br>
2017-11-13
15:31:31,233::ssh.py::wait_for_<a class="moz-txt-link-freetext"
href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a>
exception while sshing to
lago-upgrade-from-release-suite-master-host0: Timed out (in 1 s)
trying to ssh to lago-upgrade-from-release-suite-master-host0<br>
2017-11-13
15:31:32,234::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:53fe67da-a632-49fe-b697-e8a2c4cb7d23:Get ssh client for
lago-upgrade-from-release-suite-master-engine:<br>
2017-11-13
15:31:32,234::ssh.py::wait_for_<a class="moz-txt-link-freetext"
href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a>
exception while sshing to
lago-upgrade-from-release-suite-master-engine: Timed out (in 1
s) trying to ssh to
lago-upgrade-from-release-suite-master-engine<br>
2017-11-13
15:31:32,234::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:1969bf5b-4e66-43d7-91c6-a28949e98fe8:Get ssh client for
lago-upgrade-from-release-suite-master-host0:<br>
2017-11-13
15:31:32,234::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still
got 1 tries for lago-upgrade-from-release-suite-master-host0<br>
2017-11-13
15:31:32,235::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket
error connecting to
lago-upgrade-from-release-suite-master-host0: [Errno None]
Unable to connect to port 22 on 192.168.200.2<br>
2017-11-13
15:31:33,235::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:5fd0e3c3-d83c-46f9-9c88-bc739aa9c430:Get ssh client for
lago-upgrade-from-release-suite-master-engine:<br>
2017-11-13
15:31:33,235::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still
got 1 tries for lago-upgrade-from-release-suite-master-engine<br>
2017-11-13
15:31:33,236::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:1969bf5b-4e66-43d7-91c6-a28949e98fe8:Get ssh client for
lago-upgrade-from-release-suite-master-host0:<br>
2017-11-13
15:31:33,236::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket
error connecting to
lago-upgrade-from-release-suite-master-engine: [Errno None]
Unable to connect to port 22 on 192.168.200.3<br>
2017-11-13
15:31:33,237::ssh.py::wait_for_<a class="moz-txt-link-freetext"
href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a>
exception while sshing to
lago-upgrade-from-release-suite-master-host0: Timed out (in 1 s)
trying to ssh to lago-upgrade-from-release-suite-master-host0<br>
2017-11-13
15:31:34,238::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:5fd0e3c3-d83c-46f9-9c88-bc739aa9c430:Get ssh client for
lago-upgrade-from-release-suite-master-engine:<br>
2017-11-13
15:31:34,238::ssh.py::wait_for_<a class="moz-txt-link-freetext"
href="ssh::129::lago.ssh::DEBUG::Got">ssh::129::lago.ssh::DEBUG::Got</a>
exception while sshing to
lago-upgrade-from-release-suite-master-engine: Timed out (in 1
s) trying to ssh to
lago-upgrade-from-release-suite-master-engine<br>
2017-11-13
15:31:34,238::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:22550d6e-c397-43dc-9691-c8d9debfe416:Get ssh client for
lago-upgrade-from-release-suite-master-host0:<br>
2017-11-13
15:31:34,239::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still
got 1 tries for lago-upgrade-from-release-suite-master-host0<br>
2017-11-13
15:31:34,407::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:22550d6e-c397-43dc-9691-c8d9debfe416:Get ssh client for
lago-upgrade-from-release-suite-master-host0:<br>
2017-11-13
15:31:34,639::ssh.py::<a class="moz-txt-link-freetext"
href="ssh::58::lago.ssh::DEBUG::Running">ssh::58::lago.ssh::DEBUG::Running</a>
bb49adfa
on lago-upgrade-from-release-suite-master-host0: true<br>
2017-11-13
15:31:34,665::ssh.py::<a class="moz-txt-link-freetext"
href="ssh::81::lago.ssh::DEBUG::Command">ssh::81::lago.ssh::DEBUG::Command</a>
bb49adfa
on lago-upgrade-from-release-suite-master-host0 returned with 0<br>
2017-11-13
15:31:34,665::ssh.py::wait_for_<a class="moz-txt-link-freetext"
href="ssh::153::lago.ssh::DEBUG::Wait">ssh::153::lago.ssh::DEBUG::Wait</a>
succeeded for ssh to
lago-upgrade-from-release-suite-master-host0<br>
2017-11-13
15:31:34,665::log_utils.py::__exit__::611::lago.prefix::<a
class="moz-txt-link-freetext"
href="INFO::ESC">INFO::ESC</a>[32mSuccessESC[0m
(in 0:00:11)<br>
2017-11-13
15:31:34,666::log_utils.py::__enter__::600::lago.prefix::<a
class="moz-txt-link-freetext"
href="INFO::ESC">INFO::ESC</a>[0mESC[0m<br>
2017-11-13
15:31:34,666::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:4e2a7421-28f7-4679-955f-7261fe377939:Get ssh client for
lago-upgrade-from-release-suite-master-host0:<br>
2017-11-13
15:31:34,667::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still
got 100 tries for lago-upgrade-from-release-suite-master-host0<br>
2017-11-13
15:31:34,805::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:4e2a7421-28f7-4679-955f-7261fe377939:Get ssh client for
lago-upgrade-from-release-suite-master-host0:<br>
2017-11-13
15:31:34,906::ssh.py::<a class="moz-txt-link-freetext"
href="ssh::58::lago.ssh::DEBUG::Running">ssh::58::lago.ssh::DEBUG::Running</a>
bb724bf2
on lago-upgrade-from-release-suite-master-host0: bash -s <
"set -xe<br>
DIST=$(uname -r | sed -r 's/^.*\.([^\.]+)\.[^\.]+$/\1/')<br>
<br>
</b></p>
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-ba60-843c-cfdd-7366571bcb57"><br>
</b></p>
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-ba60-843c-cfdd-7366571bcb57"><b><u><font
size="+1">from host:
</font></u></b><br>
</b></p>
<p><span style="font-weight:normal;">ions. Proceeding
anyway.<br>
Nov 13 10:31:34 lago-upgrade-from-release-suite-master-host0
systemd: Configuration file
/usr/lib/systemd/system/ebtables.service is marked executable.
Please remove executable permission bits. Proceeding anyway.<br>
Nov 13 10:31:34 lago-upgrade-from-release-suite-master-host0
systemd: Configuration file
/usr/lib/systemd/system/wpa_supplicant.service is marked
executable. Please remove executable permission bits. Proceeding
anyway.<br>
Nov 13 10:31:55 lago-upgrade-from-release-suite-master-host0
yum[1294]: Updated: iptables-1.4.21-18.2.el7_4.x86_64<br>
Nov 13 10:31:56 lago-upgrade-from-release-suite-master-host0
systemd-logind: Removed session 3.<br>
Nov 13 10:31:56 lago-upgrade-from-release-suite-master-host0
systemd: Removed slice user-0.slice.<br>
Nov 13 10:31:56 lago-upgrade-from-release-suite-master-host0
systemd: Stopping user-0.slice.<br>
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0
systemd: Created slice user-0.slice.<br>
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0
systemd: Starting user-0.slice.<br>
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0
systemd: Started Session 4 of user root.<br>
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0
systemd: Starting Session 4 of user root.<br>
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0
systemd-logind: New session 4 of user root.<br>
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0
systemd-logind: Removed session 4.<br>
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0
systemd: Removed slice user-0.slice.<br>
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0
systemd: Stopping user-0.slice.<br>
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0
systemd: Created slice user-0.slice.<br>
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0
systemd: Starting user-0.slice.<br>
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0
systemd: Started Session 5 of user root.<br>
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0
systemd: Starting Session 5 of user root.<br>
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0
systemd-logind: New session 5 of user root.<br>
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0
systemd-logind: Removed session 5.<br>
Nov 13 10:44:01 lago-upgrade-from-release-suite-master-host0
systemd: Removed slice user-0.slice.<br>
Nov 13 10:44:01 lago-upgrade-from-release-suite-master-host0
systemd: Stopping user-0.slice.<br>
Nov 13 10:46:27 lago-upgrade-from-release-suite-master-host0
systemd: Starting Cleanup of Temporary Directories...<br>
Nov 13 10:46:27 lago-upgrade-from-release-suite-master-host0
systemd: Started Cleanup of Temporary Directories.<br>
Nov 13 10:54:36 lago-upgrade-from-release-suite-master-host0
dhclient[906]: DHCPREQUEST on eth0 to 192.168.200.1 port 67
(xid=0x564629cb)<br>
Nov 13 10:54:36 lago-upgrade-from-release-suite-master-host0
dhclient[906]: DHCPACK from 192.168.200.1 (xid=0x564629cb)</span><b
style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-ba60-843c-cfdd-7366571bcb57"><b><br>
</b><br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error></span></p>
</b><br class="Apple-interchange-newline">
</p>
</body>
</html>
--------------5185435FD38913B821CED967--