11:01 a.m.
On Tue, Nov 14, 2017 at 5:48 PM, Dafna Ron <dron(a)redhat.com> wrote:
Hi,
We had a failure in upgrade suite for 002_bootstrap.add_hosts. I am not
seeing any error that can suggest on an issue in engine.
I can see in the host messages host that we have stopped writing to the log
for 15 minutes and it may suggest that there is something that is keeping
the host from starting which causes us to fail the test on timeout.
However, i can use some help in determining the cause for this failure and
weather its connected to the bootstrap_add_host test in upgrade.
Link to suspected patches: As I said, I do not think its related, but this
is the patch that was reported.
https://gerrit.ovirt.org/#/c/83854/
Link to Job:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3795/
Link to all logs:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3795/artifact/
(Relevant) error snippet from the log:
<error>
Test error:
Error Message
False != True after 900 seconds
Stacktrace
Traceback (most recent call last):
File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
testMethod()
File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in
wrapped_test
test()
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in
wrapper
return func(get_test_prefix(), *args, **kwargs)
File
"/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/upgrade-from-release-suite-master/test-scenarios-after-upgrade/002_bootstrap.py",
line 187, in add_hosts
testlib.assert_true_within(_host_is_up_4, timeout=15*60)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 263, in
assert_true_within
assert_equals_within(func, True, timeout, allowed_exceptions)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 237, in
assert_equals_within
'%s != %s after %s seconds' % (res, value, timeout)
AssertionError: False != True after 900 seconds
Hi,
It took me way too long to find the log file that includes the above
stack trace. There is a known bug that pressing '(all files in zip)' does
not get all of them, only some. If we can't fix it, please add something
that will create such a zip/tar/whatever as part of the job, so that we
do not rely on jenkins.
Something causes things to be very slow, no idea what in particular,
but could not find a real problem.
Since we timed out and killed the engine in the middle of host-deploy,
we do not have full logs of it. These are copied to the engine from the
host only after it finishes. Before that, the log is written on the host
in /tmp. We might want to make ost collect that as well, to help debug
similar cases.
What we can see in engine log is that we start installing the host [1]:
2017-11-13 11:00:12,331-05 INFO
[org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [45082420] Before
Installation host 83090668-908c-4f49-8690-5348ec12f931,
lago-upgrade-from-release-suite-master-host0
2017-11-13 11:00:12,354-05 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [45082420] START,
SetVdsStatusVDSCommand(HostName =
lago-upgrade-from-release-suite-master-host0,
SetVdsStatusVDSCommandParameters:{hostId='83090668-908c-4f49-8690-5348ec12f931',
status='Installing', nonOperationalReason='NONE',
stopSpmFailureLogged='false', maintenanceReason='null'}), log id:
22a831f7
First line from host-deploy is:
2017-11-13 11:00:13,335-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [45082420] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509),
Installing Host lago-upgrade-from-release-suite-master-host0. Stage:
Initializing.
which is one second later, ok.
...
2017-11-13 11:00:13,459-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [45082420] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509),
Installing Host lago-upgrade-from-release-suite-master-host0. Stage:
Environment packages setup.
It goes install (immediately) packages that it needs for itself,
in this case it updates the package 'dmidecode', from 2.12-9
to 3.0-5. I guess our image is a bit old.
...
2017-11-13 11:00:19,516-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [45082420] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509),
Installing Host lago-upgrade-from-release-suite-master-host0. Yum
Verify: 2/2: dmidecode.x86_64 1:2.12-9.el7 - ud.
So it took (the equivalent of 'yum update dmidecode') 6 seconds.
Reasonable.
...
Later on, it starts installing/updating the actual packages it
should install/update. Starts with:
2017-11-13 11:00:24,982-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [45082420] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509),
Installing Host lago-upgrade-from-release-suite-master-host0. Yum
Status: Downloading Packages.
Then there are many lines about each package download. First is:
2017-11-13 11:00:25,049-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [45082420] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509),
Installing Host lago-upgrade-from-release-suite-master-host0. Yum
Download/Verify: GeoIP-1.5.0-11.el7.x86_64.
Last is:
2017-11-13 11:00:43,790-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [45082420] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509),
Installing Host lago-upgrade-from-release-suite-master-host0. Yum
Download/Verify: zlib-1.2.7-17.el7.x86_64.
Later we see it updates 570 packages. So downloading 570 packages
took ~ 20 seconds, good too. Then it installs them. First is:
2017-11-13 11:00:46,114-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [45082420] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509),
Installing Host lago-upgrade-from-release-suite-master-host0. Yum
update: 1/570: libgcc-4.8.5-16.el7.x86_64.
And then many similar ones. I only skimmed through them, trying to
find large gaps - didn't write a script - and a large one I find is:
2017-11-13 11:10:04,522-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [45082420] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509),
Installing Host lago-upgrade-from-release-suite-master-host0. Yum
updated: 506/570: audit.
2017-11-13 11:13:06,483-05 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(VdsDeploy) [45082420] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509),
Installing Host lago-upgrade-from-release-suite-master-host0. Yum
updated: 507/570: openssl-libs.
In between these two lines, there are a few unrelated ones, from some
other engine thread. So something caused it to need 3 minutes to install
openssl-libs.
A wild guess: Perhaps we have low entropy, and openssl does something
that needs entropy?
Checking engine-setup log there [2], I see that the following command
took almost 1.5 minutes to run:
2017-11-13 10:58:19,459-0500 DEBUG
otopi.plugins.ovirt_engine_setup.ovirt_engine.config.sso
plugin.executeRaw:813 execute:
('/usr/share/ovirt-engine/bin/ovirt-engine-crypto-tool.sh',
'pbe-encode', '--password=env:pass'), executable='None',
cwd='None',
env={'pass': '**FILTERED**', 'LESSOPEN':
'||/usr/bin/lesspipe.sh %s',
'SSH_CLIENT': '192.168.200.1 52372 22',
'SELINUX_USE_CURRENT_RANGE':
'', 'LOGNAME': 'root', 'USER': 'root',
'OVIRT_ENGINE_JAVA_HOME':
u'/usr/lib/jvm/jre', 'PATH':
'/opt/rh/rh-postgresql95/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin',
'HOME': '/root', 'OVIRT_JBOSS_HOME':
'/usr/share/ovirt-engine-wildfly', 'LD_LIBRARY_PATH':
'/opt/rh/rh-postgresql95/root/usr/lib64', 'LANG': 'en_US.UTF-8',
'SHELL': '/bin/bash', 'LIBRARY_PATH':
'/opt/rh/rh-postgresql95/root/usr/lib64', 'SHLVL': '4',
'POSTGRESQLENV':
'COMMAND/pg_dump=str:/opt/rh/rh-postgresql95/root/usr/bin/pg_dump
COMMAND/psql=str:/opt/rh/rh-postgresql95/root/usr/bin/psql
COMMAND/pg_restore=str:/opt/rh/rh-postgresql95/root/usr/bin/pg_restore
COMMAND/postgresql-setup=str:/opt/rh/rh-postgresql95/root/usr/bin/postgresql-setup
OVESETUP_PROVISIONING/postgresService=str:rh-postgresql95-postgresql
OVESETUP_PROVISIONING/postgresConf=str:/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf
OVESETUP_PROVISIONING/postgresPgHba=str:/var/opt/rh/rh-postgresql95/lib/pgsql/data/pg_hba.conf
OVESETUP_PROVISIONING/postgresPgVersion=str:/var/opt/rh/rh-postgresql95/lib/pgsql/data/PG_VERSION',
'MANPATH': '/opt/rh/rh-postgresql95/root/usr/share/man:',
'X_SCLS':
'rh-postgresql95 ', 'XDG_RUNTIME_DIR': '/run/user/0',
'OVIRT_ENGINE_JAVA_HOME_FORCE': '1', 'PYTHONPATH':
'/usr/share/ovirt-engine/setup/bin/..::', 'SELINUX_ROLE_REQUESTED':
'', 'MAIL': '/var/mail/root', 'PKG_CONFIG_PATH':
'/opt/rh/rh-postgresql95/root/usr/lib64/pkgconfig', 'XDG_SESSION_ID':
'14', 'sclenv': 'rh-postgresql95', 'XDG_CONFIG_DIRS':
'/etc/opt/rh/rh-postgresql95/xdg:/etc/xdg', 'JAVACONFDIRS':
'/etc/opt/rh/rh-postgresql95/java:/etc/java',
'SELINUX_LEVEL_REQUESTED': '', 'XDG_DATA_DIRS':
'/opt/rh/rh-postgresql95/root/usr/share', 'PWD': '/root',
'CPATH':
'/opt/rh/rh-postgresql95/root/usr/include', 'OTOPI_LOGFILE':
'/var/log/ovirt-engine/setup/ovirt-engine-setup-20171113105548-qrc7zo.log',
'SSH_CONNECTION': '192.168.200.1 52372 192.168.200.3 22',
'OTOPI_EXECDIR': '/root'}
2017-11-13 10:59:43,545-0500 DEBUG
otopi.plugins.ovirt_engine_setup.ovirt_engine.config.sso
plugin.executeRaw:863 execute-result:
('/usr/share/ovirt-engine/bin/ovirt-engine-crypto-tool.sh',
'pbe-encode', '--password=env:pass'), rc=0
2017-11-13 10:59:43,546-0500 DEBUG
otopi.plugins.ovirt_engine_setup.ovirt_engine.config.sso
plugin.execute:921 execute-output:
('/usr/share/ovirt-engine/bin/ovirt-engine-crypto-tool.sh',
'pbe-encode', '--password=env:pass') stdout:
eyJhcnRpZmFjdCI6IkVudmVsb3BlUEJFIiwic2FsdCI6ImNjSmhFbHRnUEJxeUlNTUJSaU1OdFRYL3M0RGRJT1hOSWJjV2F1NFZGT0U9Iiwic2VjcmV0IjoiK2ZlTzVyZm9kNGlsVmZLRENaRjdseVVQZHZnWnBTWUF0cnBYUWVpQnJaTT0iLCJ2ZXJzaW9uIjoiMSIsIml0ZXJhdGlvbnMiOiI0MDAwIiwiYWxnb3JpdGhtIjoiUEJLREYyV2l0aEhtYWNTSEExIn0=
2017-11-13 10:59:43,546-0500 DEBUG
otopi.plugins.ovirt_engine_setup.ovirt_engine.config.sso
plugin.execute:926 execute-output:
('/usr/share/ovirt-engine/bin/ovirt-engine-crypto-tool.sh',
'pbe-encode', '--password=env:pass') stderr:
This is almost always due to not enough entropy.
So please check if everything - from physical machines to
lago/ost/libvirt/etc., makes sure to supply all VMs enough
entropy. We usually do this using virtio-rng.
[1]
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3795/artifa...
[2]
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/3795/artifa...
lago log:
2017-11-13
15:31:23,212::log_utils.py::__enter__::600::lago.prefix::INFO::ESC[0mESC[0m
2017-11-13
15:31:23,213::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:0468ed2f-b174-4d94-bc66-2b6e08087a86:Get ssh client for
lago-upgrade-from-release-suite-master-host0:
2017-11-13
15:31:23,213::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:91372413-f7fd-4b72-85b9-9f5216ca7ae9:Get ssh client for
lago-upgrade-from-release-suite-master-engine:
2017-11-13 15:31:23,213::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still
got 1 tries for lago-upgrade-from-release-suite-master-host0
2017-11-13 15:31:23,213::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still
got 1 tries for lago-upgrade-from-release-suite-master-engine
2017-11-13
15:31:26,220::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error
connecting to lago-upgrade-from-release-suite-master-engine: [Errno None]
Unable to connect to port 22 on 192.168.200.3
2017-11-13
15:31:26,221::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error
connecting to lago-upgrade-from-release-suite-master-host0: [Errno None]
Unable to connect to port 22 on 192.168.200.2
2017-11-13 15:31:27,222::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:91372413-f7fd-4b72-85b9-9f5216ca7ae9:Get ssh client for
lago-upgrade-from-release-suite-master-engine:
2017-11-13 15:31:27,222::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:0468ed2f-b174-4d94-bc66-2b6e08087a86:Get ssh client for
lago-upgrade-from-release-suite-master-host0:
2017-11-13 15:31:27,222::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got
exception while sshing to lago-upgrade-from-release-suite-master-engine:
Timed out (in 4 s) trying to ssh to
lago-upgrade-from-release-suite-master-engine
2017-11-13 15:31:27,222::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got
exception while sshing to lago-upgrade-from-release-suite-master-host0:
Timed out (in 4 s) trying to ssh to
lago-upgrade-from-release-suite-master-host0
2017-11-13
15:31:28,224::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:fc88f641-b012-4636-a471-9ccaaf361a53:Get ssh client for
lago-upgrade-from-release-suite-master-engine:
2017-11-13
15:31:28,224::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:afb01a46-6338-407f-b7c9-9d5c6b91404d:Get ssh client for
lago-upgrade-from-release-suite-master-host0:
2017-11-13 15:31:28,224::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still
got 1 tries for lago-upgrade-from-release-suite-master-engine
2017-11-13 15:31:28,225::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still
got 1 tries for lago-upgrade-from-release-suite-master-host0
2017-11-13
15:31:28,226::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error
connecting to lago-upgrade-from-release-suite-master-host0: [Errno None]
Unable to connect to port 22 on 192.168.200.2
2017-11-13 15:31:29,228::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:afb01a46-6338-407f-b7c9-9d5c6b91404d:Get ssh client for
lago-upgrade-from-release-suite-master-host0:
2017-11-13 15:31:29,228::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got
exception while sshing to lago-upgrade-from-release-suite-master-host0:
Timed out (in 1 s) trying to ssh to
lago-upgrade-from-release-suite-master-host0
2017-11-13
15:31:29,229::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error
connecting to lago-upgrade-from-release-suite-master-engine: [Errno None]
Unable to connect to port 22 on 192.168.200.3
2017-11-13 15:31:30,229::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:fc88f641-b012-4636-a471-9ccaaf361a53:Get ssh client for
lago-upgrade-from-release-suite-master-engine:
2017-11-13
15:31:30,230::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:9df5ddca-bbc4-485a-9f3a-b0b9a5bb5990:Get ssh client for
lago-upgrade-from-release-suite-master-host0:
2017-11-13 15:31:30,230::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got
exception while sshing to lago-upgrade-from-release-suite-master-engine:
Timed out (in 2 s) trying to ssh to
lago-upgrade-from-release-suite-master-engine
2017-11-13 15:31:30,230::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still
got 1 tries for lago-upgrade-from-release-suite-master-host0
2017-11-13
15:31:30,231::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error
connecting to lago-upgrade-from-release-suite-master-host0: [Errno None]
Unable to connect to port 22 on 192.168.200.2
2017-11-13
15:31:31,231::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:53fe67da-a632-49fe-b697-e8a2c4cb7d23:Get ssh client for
lago-upgrade-from-release-suite-master-engine:
2017-11-13 15:31:31,232::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still
got 1 tries for lago-upgrade-from-release-suite-master-engine
2017-11-13
15:31:31,232::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error
connecting to lago-upgrade-from-release-suite-master-engine: [Errno None]
Unable to connect to port 22 on 192.168.200.3
2017-11-13 15:31:31,232::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:9df5ddca-bbc4-485a-9f3a-b0b9a5bb5990:Get ssh client for
lago-upgrade-from-release-suite-master-host0:
2017-11-13 15:31:31,233::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got
exception while sshing to lago-upgrade-from-release-suite-master-host0:
Timed out (in 1 s) trying to ssh to
lago-upgrade-from-release-suite-master-host0
2017-11-13 15:31:32,234::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:53fe67da-a632-49fe-b697-e8a2c4cb7d23:Get ssh client for
lago-upgrade-from-release-suite-master-engine:
2017-11-13 15:31:32,234::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got
exception while sshing to lago-upgrade-from-release-suite-master-engine:
Timed out (in 1 s) trying to ssh to
lago-upgrade-from-release-suite-master-engine
2017-11-13
15:31:32,234::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:1969bf5b-4e66-43d7-91c6-a28949e98fe8:Get ssh client for
lago-upgrade-from-release-suite-master-host0:
2017-11-13 15:31:32,234::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still
got 1 tries for lago-upgrade-from-release-suite-master-host0
2017-11-13
15:31:32,235::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error
connecting to lago-upgrade-from-release-suite-master-host0: [Errno None]
Unable to connect to port 22 on 192.168.200.2
2017-11-13
15:31:33,235::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:5fd0e3c3-d83c-46f9-9c88-bc739aa9c430:Get ssh client for
lago-upgrade-from-release-suite-master-engine:
2017-11-13 15:31:33,235::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still
got 1 tries for lago-upgrade-from-release-suite-master-engine
2017-11-13 15:31:33,236::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:1969bf5b-4e66-43d7-91c6-a28949e98fe8:Get ssh client for
lago-upgrade-from-release-suite-master-host0:
2017-11-13
15:31:33,236::ssh.py::get_ssh_client::354::lago.ssh::DEBUG::Socket error
connecting to lago-upgrade-from-release-suite-master-engine: [Errno None]
Unable to connect to port 22 on 192.168.200.3
2017-11-13 15:31:33,237::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got
exception while sshing to lago-upgrade-from-release-suite-master-host0:
Timed out (in 1 s) trying to ssh to
lago-upgrade-from-release-suite-master-host0
2017-11-13 15:31:34,238::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:5fd0e3c3-d83c-46f9-9c88-bc739aa9c430:Get ssh client for
lago-upgrade-from-release-suite-master-engine:
2017-11-13 15:31:34,238::ssh.py::wait_for_ssh::129::lago.ssh::DEBUG::Got
exception while sshing to lago-upgrade-from-release-suite-master-engine:
Timed out (in 1 s) trying to ssh to
lago-upgrade-from-release-suite-master-engine
2017-11-13
15:31:34,238::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:22550d6e-c397-43dc-9691-c8d9debfe416:Get ssh client for
lago-upgrade-from-release-suite-master-host0:
2017-11-13 15:31:34,239::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still
got 1 tries for lago-upgrade-from-release-suite-master-host0
2017-11-13 15:31:34,407::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:22550d6e-c397-43dc-9691-c8d9debfe416:Get ssh client for
lago-upgrade-from-release-suite-master-host0:
2017-11-13 15:31:34,639::ssh.py::ssh::58::lago.ssh::DEBUG::Running bb49adfa
on lago-upgrade-from-release-suite-master-host0: true
2017-11-13 15:31:34,665::ssh.py::ssh::81::lago.ssh::DEBUG::Command bb49adfa
on lago-upgrade-from-release-suite-master-host0 returned with 0
2017-11-13 15:31:34,665::ssh.py::wait_for_ssh::153::lago.ssh::DEBUG::Wait
succeeded for ssh to lago-upgrade-from-release-suite-master-host0
2017-11-13
15:31:34,665::log_utils.py::__exit__::611::lago.prefix::INFO::ESC[32mSuccessESC[0m
(in 0:00:11)
2017-11-13
15:31:34,666::log_utils.py::__enter__::600::lago.prefix::INFO::ESC[0mESC[0m
2017-11-13
15:31:34,666::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:4e2a7421-28f7-4679-955f-7261fe377939:Get ssh client for
lago-upgrade-from-release-suite-master-host0:
2017-11-13 15:31:34,667::ssh.py::get_ssh_client::339::lago.ssh::DEBUG::Still
got 100 tries for lago-upgrade-from-release-suite-master-host0
2017-11-13 15:31:34,805::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:4e2a7421-28f7-4679-955f-7261fe377939:Get ssh client for
lago-upgrade-from-release-suite-master-host0:
2017-11-13 15:31:34,906::ssh.py::ssh::58::lago.ssh::DEBUG::Running bb724bf2
on lago-upgrade-from-release-suite-master-host0: bash -s < "set -xe
DIST=$(uname -r | sed -r 's/^.*\.([^\.]+)\.[^\.]+$/\1/')
from host:
ions. Proceeding anyway.
Nov 13 10:31:34 lago-upgrade-from-release-suite-master-host0 systemd:
Configuration file /usr/lib/systemd/system/ebtables.service is marked
executable. Please remove executable permission bits. Proceeding anyway.
Nov 13 10:31:34 lago-upgrade-from-release-suite-master-host0 systemd:
Configuration file /usr/lib/systemd/system/wpa_supplicant.service is marked
executable. Please remove executable permission bits. Proceeding anyway.
Nov 13 10:31:55 lago-upgrade-from-release-suite-master-host0 yum[1294]:
Updated: iptables-1.4.21-18.2.el7_4.x86_64
Nov 13 10:31:56 lago-upgrade-from-release-suite-master-host0 systemd-logind:
Removed session 3.
Nov 13 10:31:56 lago-upgrade-from-release-suite-master-host0 systemd:
Removed slice user-0.slice.
Nov 13 10:31:56 lago-upgrade-from-release-suite-master-host0 systemd:
Stopping user-0.slice.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd:
Created slice user-0.slice.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd:
Starting user-0.slice.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd:
Started Session 4 of user root.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd:
Starting Session 4 of user root.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind:
New session 4 of user root.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind:
Removed session 4.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd:
Removed slice user-0.slice.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd:
Stopping user-0.slice.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd:
Created slice user-0.slice.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd:
Starting user-0.slice.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd:
Started Session 5 of user root.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd:
Starting Session 5 of user root.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind:
New session 5 of user root.
Nov 13 10:44:00 lago-upgrade-from-release-suite-master-host0 systemd-logind:
Removed session 5.
Nov 13 10:44:01 lago-upgrade-from-release-suite-master-host0 systemd:
Removed slice user-0.slice.
Nov 13 10:44:01 lago-upgrade-from-release-suite-master-host0 systemd:
Stopping user-0.slice.
Nov 13 10:46:27 lago-upgrade-from-release-suite-master-host0 systemd:
Starting Cleanup of Temporary Directories...
Nov 13 10:46:27 lago-upgrade-from-release-suite-master-host0 systemd:
Started Cleanup of Temporary Directories.
Nov 13 10:54:36 lago-upgrade-from-release-suite-master-host0 dhclient[906]:
DHCPREQUEST on eth0 to 192.168.200.1 port 67 (xid=0x564629cb)
Nov 13 10:54:36 lago-upgrade-from-release-suite-master-host0 dhclient[906]:
DHCPACK from 192.168.200.1 (xid=0x564629cb)
</error>
--
Didi