[JIRA] (OVIRT-2539) master ovirt-engine fails on test
004_basic_sanity.run_vms
by Dafna Ron (oVirt JIRA)
Dafna Ron created OVIRT-2539:
--------------------------------
Summary: master ovirt-engine fails on test 004_basic_sanity.run_vms
Key: OVIRT-2539
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2539
Project: oVirt - virtualization made easy
Issue Type: Bug
Reporter: Dafna Ron
Assignee: infra
Priority: Highest
patch:
https://gerrit.ovirt.org/#/c/94864/ - core: allow HA VM with a lease on a non-active storage domain to run.
logs:
https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10598/arti...
'error:
>>
2018-10-14 11:45:18,556-04 DEBUG [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall] (default task-1) [56d8786f-62cf-4053-a3c7-92dfcbd9f37e] Compiled stored procedure. Call string
is [{call getvdsinterfacesbynetworkid(?)}]
2018-10-14 11:45:18,556-04 DEBUG [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall] (default task-1) [56d8786f-62cf-4053-a3c7-92dfcbd9f37e] SqlCall for procedure [GetVdsInterface
sByNetworkId] compiled
2018-10-14 11:45:18,565-04 ERROR [org.ovirt.engine.core.bll.RunVmOnceCommand] (default task-1) [56d8786f-62cf-4053-a3c7-92dfcbd9f37e] Error during ValidateFailure.: java.lang.NullPointerException'
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100094)
6 years, 2 months
[JIRA] (OVIRT-2538) ovirt-engine master failure on test
008_basic_ui_sanity.initialize_chrome
by Dafna Ron (oVirt JIRA)
Dafna Ron created OVIRT-2538:
--------------------------------
Summary: ovirt-engine master failure on test 008_basic_ui_sanity.initialize_chrome
Key: OVIRT-2538
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2538
Project: oVirt - virtualization made easy
Issue Type: Bug
Reporter: Dafna Ron
Assignee: infra
Priority: High
I have seen random failures of test 008_basic_ui_sanity.initialize_chrome which are not related to the changes.
Here is the latest failure which I confirmed with didi is not a real failure and is not related to the change
https://gerrit.ovirt.org/#/c/89572/
https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10601/
I cannot see anything in the http logs and in engine logs.
I can see a warn on ssl and some issues which seems to be resolved later on domain names:
'[Mon Oct 15 02:56:26.041204 2018] [ssl:warn] [pid 18661] AH01909: RSA certificate configured for 192.168.201.4:443 does NOT include an ID which matches the server name'
I think the issue is the test since the only errors I can see are in the lago logs.
I saved logs to allow debugging of the test failure.
[~gbenhaim(a)redhat.com]
[~gshereme(a)redhat.com]
Can you please check this issue?
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100094)
6 years, 2 months
[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-master - Build #
783 - Failure!
by jenkins@jenkins.phx.ovirt.org
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/
Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/783/
Build Number: 783
Build Status: Failure
Triggered By: Started by timer
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #783
[Eitan Raviv] networking: connect NFS to engine over IPv6
-----------------
Failed Tests:
-----------------
1 tests failed.
FAILED: 002_bootstrap.add_hosts
Error Message:
Host lago-hc-basic-suite-master-host-1 is in non operational state
-------------------- >> begin captured logging << --------------------
ovirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7fb2e4ac8230>
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within
res = func()
File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 258, in _host_is_up_4
raise RuntimeError('Host %s is in non operational state' % api_host.name)
RuntimeError: Host lago-hc-basic-suite-master-host-1 is in non operational state
--------------------- >> end captured logging << ---------------------
Stack Trace:
File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
testMethod()
File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test
test()
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper
return func(get_test_prefix(), *args, **kwargs)
File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 228, in add_hosts
add_hosts_4(prefix)
File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 275, in add_hosts_4
testlib.assert_true_within(_host_is_up_4, timeout=15*60)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 278, in assert_true_within
assert_equals_within(func, True, timeout, allowed_exceptions)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within
res = func()
File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 258, in _host_is_up_4
raise RuntimeError('Host %s is in non operational state' % api_host.name)
'Host lago-hc-basic-suite-master-host-1 is in non operational state\n-------------------- >> begin captured logging << --------------------\novirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7fb2e4ac8230>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 258, in _host_is_up_4\n raise RuntimeError(\'Host %s is in non operational state\' % api_host.name)\nRuntimeError: Host lago-hc-basic-suite-master-host-1 is in non operational state\n--------------------- >> end captured logging << ---------------------'
6 years, 2 months
[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-4.2 - Build # 563
- Failure!
by jenkins@jenkins.phx.ovirt.org
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/
Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/563/
Build Number: 563
Build Status: Failure
Triggered By: Started by timer
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #563
[Eitan Raviv] networking: connect NFS to engine over IPv6
-----------------
Failed Tests:
-----------------
1 tests failed.
FAILED: 002_bootstrap.add_hosts
Error Message:
Failed to read response: [(<pycurl.Curl object at 0x7f9985ee0050>, 7, 'TCP connection reset by peer')]
-------------------- >> begin captured logging << --------------------
ovirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7f9985e83578>
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within
res = func()
File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 254, in _host_is_up_4
host_obj = host_service.get()
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 37093, in get
return self._internal_get(headers, query, wait)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get
return future.wait() if wait else future
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 54, in wait
response = self._connection.wait(self._context)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 496, in wait
return self.__wait(context, failed_auth)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 510, in __wait
raise Error("Failed to read response: {}".format(err_list))
Error: Failed to read response: [(<pycurl.Curl object at 0x7f9985ee0050>, 7, 'TCP connection reset by peer')]
--------------------- >> end captured logging << ---------------------
Stack Trace:
File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
testMethod()
File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test
test()
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper
return func(get_test_prefix(), *args, **kwargs)
File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 229, in add_hosts
add_hosts_4(prefix)
File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 273, in add_hosts_4
testlib.assert_true_within(_host_is_up_4, timeout=15*60)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 278, in assert_true_within
assert_equals_within(func, True, timeout, allowed_exceptions)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within
res = func()
File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 254, in _host_is_up_4
host_obj = host_service.get()
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 37093, in get
return self._internal_get(headers, query, wait)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get
return future.wait() if wait else future
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 54, in wait
response = self._connection.wait(self._context)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 496, in wait
return self.__wait(context, failed_auth)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 510, in __wait
raise Error("Failed to read response: {}".format(err_list))
'Failed to read response: [(<pycurl.Curl object at 0x7f9985ee0050>, 7, \'TCP connection reset by peer\')]\n-------------------- >> begin captured logging << --------------------\novirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7f9985e83578>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 254, in _host_is_up_4\n host_obj = host_service.get()\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 37093, in get\n return self._internal_get(headers, query, wait)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get\n return future.wait() if wait else future\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 54, in wait\n response = self._connection.wait(self._context)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 496, in wait\n return self.__wait(context, failed_auth)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 510, in __wait\n raise Error("Failed to read response: {}".format(err_list))\nError: Failed to read response: [(<pycurl.Curl object at 0x7f9985ee0050>, 7, \'TCP connection reset by peer\')]\n--------------------- >> end captured logging << ---------------------'
6 years, 2 months
[oVirt Jenkins] ovirt-system-tests_he-node-ng-suite-4.2 - Build #
347 - Failure!
by jenkins@jenkins.phx.ovirt.org
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/
Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/347/
Build Number: 347
Build Status: Failure
Triggered By: Started by timer
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #347
[Dafna Ron] ovirt-4.2: block external links from running in ost
-----------------
Failed Tests:
-----------------
1 tests failed.
FAILED: 010_local_maintenance_cli.local_maintenance
Error Message:
143
-------------------- >> begin captured logging << --------------------
root: INFO: * Waiting For System Stability...
lago.ssh: DEBUG: start task:90555ac6-a749-4c6d-97d0-ca20f9ee7e56:Get ssh client for lago-he-node-ng-suite-4-2-host-1:
lago.ssh: DEBUG: end task:90555ac6-a749-4c6d-97d0-ca20f9ee7e56:Get ssh client for lago-he-node-ng-suite-4-2-host-1:
lago.ssh: DEBUG: Running 4d4a6eae on lago-he-node-ng-suite-4-2-host-1: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 4d4a6eae on lago-he-node-ng-suite-4-2-host-1 returned with 0
lago.ssh: DEBUG: Command 4d4a6eae on lago-he-node-ng-suite-4-2-host-1 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4241 (Wed Sep 26 17:40:36 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=4241 (Wed Sep 26 17:40:36 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "49ba6ea7", "local_conf_timestamp": 4241, "host-ts": 4241}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4247 (Wed Sep 26 17:40:41 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4247 (Wed Sep 26 17:40:41 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e8286b5a", "local_conf_timestamp": 4247, "host-ts": 4247}, "global_maintenance": false}
lago.ssh: DEBUG: start task:cdd8bc8a-3f38-4e62-9699-7ef8e4864f9a:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: end task:cdd8bc8a-3f38-4e62-9699-7ef8e4864f9a:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: Running 4e65c50e on lago-he-node-ng-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local
lago.ssh: DEBUG: Command 4e65c50e on lago-he-node-ng-suite-4-2-host-0 returned with 0
root: INFO: * Waiting for engine to migrate...
lago.ssh: DEBUG: start task:cdb61607-0d9d-4531-931e-e2a2c02f6b15:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: end task:cdb61607-0d9d-4531-931e-e2a2c02f6b15:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: Running 4fac57de on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 4fac57de on lago-he-node-ng-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command 4fac57de on lago-he-node-ng-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4251 (Wed Sep 26 17:40:46 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=4251 (Wed Sep 26 17:40:46 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "95dd2e98", "local_conf_timestamp": 4251, "host-ts": 4251}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4247 (Wed Sep 26 17:40:41 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4247 (Wed Sep 26 17:40:41 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e8286b5a", "local_conf_timestamp": 4247, "host-ts": 4247}, "global_maintenance": false}
lago.ssh: DEBUG: start task:1fc0139b-3d03-4e0a-8168-69c54559563d:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: end task:1fc0139b-3d03-4e0a-8168-69c54559563d:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: Running 56ec62dc on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 56ec62dc on lago-he-node-ng-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command 56ec62dc on lago-he-node-ng-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4262 (Wed Sep 26 17:40:56 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4262 (Wed Sep 26 17:40:56 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "01cda987", "local_conf_timestamp": 4262, "host-ts": 4262}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4257 (Wed Sep 26 17:40:51 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4257 (Wed Sep 26 17:40:52 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ec6e945f", "local_conf_timestamp": 4257, "host-ts": 4257}, "global_maintenance": false}
lago.ssh: DEBUG: start task:12189e9a-68f8-42d0-ba7a-126282d28b98:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: end task:12189e9a-68f8-42d0-ba7a-126282d28b98:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: Running 5aec01a8 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 5aec01a8 on lago-he-node-ng-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command 5aec01a8 on lago-he-node-ng-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4272 (Wed Sep 26 17:41:07 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4272 (Wed Sep 26 17:41:07 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "03cc6058", "local_conf_timestamp": 4272, "host-ts": 4272}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4267 (Wed Sep 26 17:41:02 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4267 (Wed Sep 26 17:41:02 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e0a59550", "local_conf_timestamp": 4267, "host-ts": 4267}, "global_maintenance": false}
lago.ssh: DEBUG: start task:6e63e48d-6a63-41da-92db-73b872e79e0a:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: end task:6e63e48d-6a63-41da-92db-73b872e79e0a:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: Running 5e4ddc0e on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 5e4ddc0e on lago-he-node-ng-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command 5e4ddc0e on lago-he-node-ng-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4272 (Wed Sep 26 17:41:07 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4272 (Wed Sep 26 17:41:07 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "03cc6058", "local_conf_timestamp": 4272, "host-ts": 4272}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4267 (Wed Sep 26 17:41:02 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4267 (Wed Sep 26 17:41:02 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e0a59550", "local_conf_timestamp": 4267, "host-ts": 4267}, "global_maintenance": false}
lago.ssh: DEBUG: start task:b30d6158-ccd3-4752-8b2f-cad0c236e420:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: end task:b30d6158-ccd3-4752-8b2f-cad0c236e420:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: Running 6147daf4 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 6147daf4 on lago-he-node-ng-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command 6147daf4 on lago-he-node-ng-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4282 (Wed Sep 26 17:41:17 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4283 (Wed Sep 26 17:41:17 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f97e01d5", "local_conf_timestamp": 4283, "host-ts": 4282}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4277 (Wed Sep 26 17:41:12 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4277 (Wed Sep 26 17:41:12 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "93de5fcd", "local_conf_timestamp": 4277, "host-ts": 4277}, "global_maintenance": false}
lago.ssh: DEBUG: start task:634c9638-320f-4848-ad7f-30ba5f612b08:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: end task:634c9638-320f-4848-ad7f-30ba5f612b08:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: Running 6443f238 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 6443f238 on lago-he-node-ng-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command 6443f238 on lago-he-node-ng-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4292 (Wed Sep 26 17:41:26 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4292 (Wed Sep 26 17:41:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "204a0f81", "local_conf_timestamp": 4292, "host-ts": 4292}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4287 (Wed Sep 26 17:41:21 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4287 (Wed Sep 26 17:41:21 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ac395afe", "local_conf_timestamp": 4287, "host-ts": 4287}, "global_maintenance": false}
lago.ssh: DEBUG: start task:e6c602be-6983-4556-98c4-4b57f63c7006:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: end task:e6c602be-6983-4556-98c4-4b57f63c7006:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: Running 674f8cc6 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 674f8cc6 on lago-he-node-ng-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command 674f8cc6 on lago-he-node-ng-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4292 (Wed Sep 26 17:41:26 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4292 (Wed Sep 26 17:41:27 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "204a0f81", "local_conf_timestamp": 4292, "host-ts": 4292}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4287 (Wed Sep 26 17:41:21 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4287 (Wed Sep 26 17:41:21 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ac395afe", "local_conf_timestamp": 4287, "host-ts": 4287}, "global_maintenance": false}
lago.ssh: DEBUG: start task:4beec84d-e882-43d6-9fab-667e4e81cbda:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: end task:4beec84d-e882-43d6-9fab-667e4e81cbda:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: Running 6a680c80 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 6a680c80 on lago-he-node-ng-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command 6a680c80 on lago-he-node-ng-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4302 (Wed Sep 26 17:41:37 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4302 (Wed Sep 26 17:41:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "8a734436", "local_conf_timestamp": 4302, "host-ts": 4302}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4297 (Wed Sep 26 17:41:31 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4297 (Wed Sep 26 17:41:31 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a87fa5fb", "local_conf_timestamp": 4297, "host-ts": 4297}, "global_maintenance": false}
lago.ssh: DEBUG: start task:7cd1cc45-58d8-4c69-92e3-62a2d08a6904:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: end task:7cd1cc45-58d8-4c69-92e3-62a2d08a6904:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: Running 6d5c6b66 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 6d5c6b66 on lago-he-node-ng-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command 6d5c6b66 on lago-he-node-ng-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4302 (Wed Sep 26 17:41:37 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4302 (Wed Sep 26 17:41:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "8a734436", "local_conf_timestamp": 4302, "host-ts": 4302}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4297 (Wed Sep 26 17:41:31 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4297 (Wed Sep 26 17:41:31 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a87fa5fb", "local_conf_timestamp": 4297, "host-ts": 4297}, "global_maintenance": false}
lago.ssh: DEBUG: start task:a81b0359-b525-41fa-9906-7f21f0f27dd6:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: end task:a81b0359-b525-41fa-9906-7f21f0f27dd6:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: Running 704b805a on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 704b805a on lago-he-node-ng-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command 704b805a on lago-he-node-ng-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4302 (Wed Sep 26 17:41:37 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4302 (Wed Sep 26 17:41:37 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "8a734436", "local_conf_timestamp": 4302, "host-ts": 4302}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4307 (Wed Sep 26 17:41:42 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4307 (Wed Sep 26 17:41:42 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "4d44c828", "local_conf_timestamp": 4307, "host-ts": 4307}, "global_maintenance": false}
lago.ssh: DEBUG: start task:e51f9ed4-1717-44c9-b0ee-37ea5e05676a:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: end task:e51f9ed4-1717-44c9-b0ee-37ea5e05676a:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: Running 7349afde on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 7349afde on lago-he-node-ng-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command 7349afde on lago-he-node-ng-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4312 (Wed Sep 26 17:41:47 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4313 (Wed Sep 26 17:41:47 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "53474a62", "local_conf_timestamp": 4313, "host-ts": 4312}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4307 (Wed Sep 26 17:41:42 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4307 (Wed Sep 26 17:41:42 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "4d44c828", "local_conf_timestamp": 4307, "host-ts": 4307}, "global_maintenance": false}
lago.ssh: DEBUG: start task:243d2486-a3ca-4ca5-8fff-878978118a3d:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: end task:243d2486-a3ca-4ca5-8fff-878978118a3d:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: Running 76684004 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 76684004 on lago-he-node-ng-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command 76684004 on lago-he-node-ng-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4322 (Wed Sep 26 17:41:56 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4322 (Wed Sep 26 17:41:57 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "0d643ff3", "local_conf_timestamp": 4322, "host-ts": 4322}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4317 (Wed Sep 26 17:41:52 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4317 (Wed Sep 26 17:41:52 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "4902372d", "local_conf_timestamp": 4317, "host-ts": 4317}, "global_maintenance": false}
lago.ssh: DEBUG: start task:c1b8be35-3faa-4959-8a52-56991f4fa241:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: end task:c1b8be35-3faa-4959-8a52-56991f4fa241:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: Running 7999ef0c on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 7999ef0c on lago-he-node-ng-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command 7999ef0c on lago-he-node-ng-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4322 (Wed Sep 26 17:41:56 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4322 (Wed Sep 26 17:41:57 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "0d643ff3", "local_conf_timestamp": 4322, "host-ts": 4322}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4317 (Wed Sep 26 17:41:52 2018)\nhost-id=2\nscore=3000\nvm_conf_refresh_time=4317 (Wed Sep 26 17:41:52 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "4902372d", "local_conf_timestamp": 4317, "host-ts": 4317}, "global_maintenance": false}
lago.ssh: DEBUG: start task:c6947cc4-e42c-4461-a2ca-e144eed75f88:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: end task:c6947cc4-e42c-4461-a2ca-e144eed75f88:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: Running 7c7e5744 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 7c7e5744 on lago-he-node-ng-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command 7c7e5744 on lago-he-node-ng-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4332 (Wed Sep 26 17:42:07 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4332 (Wed Sep 26 17:42:07 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "VM migrated away successfully", "health": "bad", "vm": "down", "detail": "Down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "c23b8814", "local_conf_timestamp": 4332, "host-ts": 4332}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4327 (Wed Sep 26 17:42:01 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4327 (Wed Sep 26 17:42:01 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "4488c592", "local_conf_timestamp": 4327, "host-ts": 4327}, "global_maintenance": false}
lago.ssh: DEBUG: start task:25677b7b-0a3f-4018-9a2d-399570c4f390:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: end task:25677b7b-0a3f-4018-9a2d-399570c4f390:Get ssh client for lago-he-node-ng-suite-4-2-host-0:
lago.ssh: DEBUG: Running 7d97c89a on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json
lago.ssh: DEBUG: Command 7d97c89a on lago-he-node-ng-suite-4-2-host-0 returned with 0
lago.ssh: DEBUG: Command 7d97c89a on lago-he-node-ng-suite-4-2-host-0 output:
{"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4332 (Wed Sep 26 17:42:07 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=4332 (Wed Sep 26 17:42:07 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "VM migrated away successfully", "health": "bad", "vm": "down", "detail": "Down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "c23b8814", "local_conf_timestamp": 4332, "host-ts": 4332}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4327 (Wed Sep 26 17:42:01 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4327 (Wed Sep 26 17:42:01 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "4488c592", "local_conf_timestamp": 4327, "host-ts": 4327}, "global_maintenance": false}
root: INFO: * Engine has migrated.
root: INFO: * Waiting For System Stability...
cli: DEBUG: signal 15 was caught
--------------------- >> end captured logging << ---------------------
Stack Trace:
File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
testMethod()
File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test
test()
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper
return func(get_test_prefix(), *args, **kwargs)
File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 125, in local_maintenance
_wait_for_engine_migration(host, he_index, "bad", "Migration Destination")
File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 45, in _wait_for_engine_migration
time.sleep(wait_value)
File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 922, in exit_handler
sys.exit(128 + signum)
'143\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\nlago.ssh: DEBUG: start task:90555ac6-a749-4c6d-97d0-ca20f9ee7e56:Get ssh client for lago-he-node-ng-suite-4-2-host-1:\nlago.ssh: DEBUG: end task:90555ac6-a749-4c6d-97d0-ca20f9ee7e56:Get ssh client for lago-he-node-ng-suite-4-2-host-1:\nlago.ssh: DEBUG: Running 4d4a6eae on lago-he-node-ng-suite-4-2-host-1: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 4d4a6eae on lago-he-node-ng-suite-4-2-host-1 returned with 0\nlago.ssh: DEBUG: Command 4d4a6eae on lago-he-node-ng-suite-4-2-host-1 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4241 (Wed Sep 26 17:40:36 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=4241 (Wed Sep 26 17:40:36 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "49ba6ea7", "local_conf_timestamp": 4241, "host-ts": 4241}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4247 (Wed Sep 26 17:40:41 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4247 (Wed Sep 26 17:40:41 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e8286b5a", "local_conf_timestamp": 4247, "host-ts": 4247}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:cdd8bc8a-3f38-4e62-9699-7ef8e4864f9a:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:cdd8bc8a-3f38-4e62-9699-7ef8e4864f9a:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 4e65c50e on lago-he-node-ng-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local\nlago.ssh: DEBUG: Command 4e65c50e on lago-he-node-ng-suite-4-2-host-0 returned with 0\nroot: INFO: * Waiting for engine to migrate...\nlago.ssh: DEBUG: start task:cdb61607-0d9d-4531-931e-e2a2c02f6b15:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:cdb61607-0d9d-4531-931e-e2a2c02f6b15:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 4fac57de on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 4fac57de on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 4fac57de on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4251 (Wed Sep 26 17:40:46 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=4251 (Wed Sep 26 17:40:46 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "95dd2e98", "local_conf_timestamp": 4251, "host-ts": 4251}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4247 (Wed Sep 26 17:40:41 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4247 (Wed Sep 26 17:40:41 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e8286b5a", "local_conf_timestamp": 4247, "host-ts": 4247}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:1fc0139b-3d03-4e0a-8168-69c54559563d:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:1fc0139b-3d03-4e0a-8168-69c54559563d:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 56ec62dc on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 56ec62dc on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 56ec62dc on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4262 (Wed Sep 26 17:40:56 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4262 (Wed Sep 26 17:40:56 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "01cda987", "local_conf_timestamp": 4262, "host-ts": 4262}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4257 (Wed Sep 26 17:40:51 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4257 (Wed Sep 26 17:40:52 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ec6e945f", "local_conf_timestamp": 4257, "host-ts": 4257}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:12189e9a-68f8-42d0-ba7a-126282d28b98:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:12189e9a-68f8-42d0-ba7a-126282d28b98:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 5aec01a8 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 5aec01a8 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 5aec01a8 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4272 (Wed Sep 26 17:41:07 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4272 (Wed Sep 26 17:41:07 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "03cc6058", "local_conf_timestamp": 4272, "host-ts": 4272}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4267 (Wed Sep 26 17:41:02 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4267 (Wed Sep 26 17:41:02 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e0a59550", "local_conf_timestamp": 4267, "host-ts": 4267}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:6e63e48d-6a63-41da-92db-73b872e79e0a:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:6e63e48d-6a63-41da-92db-73b872e79e0a:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 5e4ddc0e on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 5e4ddc0e on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 5e4ddc0e on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4272 (Wed Sep 26 17:41:07 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4272 (Wed Sep 26 17:41:07 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "03cc6058", "local_conf_timestamp": 4272, "host-ts": 4272}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4267 (Wed Sep 26 17:41:02 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4267 (Wed Sep 26 17:41:02 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "e0a59550", "local_conf_timestamp": 4267, "host-ts": 4267}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:b30d6158-ccd3-4752-8b2f-cad0c236e420:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:b30d6158-ccd3-4752-8b2f-cad0c236e420:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 6147daf4 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 6147daf4 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 6147daf4 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4282 (Wed Sep 26 17:41:17 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4283 (Wed Sep 26 17:41:17 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "f97e01d5", "local_conf_timestamp": 4283, "host-ts": 4282}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4277 (Wed Sep 26 17:41:12 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4277 (Wed Sep 26 17:41:12 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "93de5fcd", "local_conf_timestamp": 4277, "host-ts": 4277}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:634c9638-320f-4848-ad7f-30ba5f612b08:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:634c9638-320f-4848-ad7f-30ba5f612b08:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 6443f238 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 6443f238 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 6443f238 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4292 (Wed Sep 26 17:41:26 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4292 (Wed Sep 26 17:41:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "204a0f81", "local_conf_timestamp": 4292, "host-ts": 4292}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4287 (Wed Sep 26 17:41:21 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4287 (Wed Sep 26 17:41:21 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ac395afe", "local_conf_timestamp": 4287, "host-ts": 4287}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:e6c602be-6983-4556-98c4-4b57f63c7006:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:e6c602be-6983-4556-98c4-4b57f63c7006:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 674f8cc6 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 674f8cc6 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 674f8cc6 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4292 (Wed Sep 26 17:41:26 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4292 (Wed Sep 26 17:41:27 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "204a0f81", "local_conf_timestamp": 4292, "host-ts": 4292}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4287 (Wed Sep 26 17:41:21 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4287 (Wed Sep 26 17:41:21 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "ac395afe", "local_conf_timestamp": 4287, "host-ts": 4287}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:4beec84d-e882-43d6-9fab-667e4e81cbda:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:4beec84d-e882-43d6-9fab-667e4e81cbda:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 6a680c80 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 6a680c80 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 6a680c80 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4302 (Wed Sep 26 17:41:37 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4302 (Wed Sep 26 17:41:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "8a734436", "local_conf_timestamp": 4302, "host-ts": 4302}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4297 (Wed Sep 26 17:41:31 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4297 (Wed Sep 26 17:41:31 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a87fa5fb", "local_conf_timestamp": 4297, "host-ts": 4297}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:7cd1cc45-58d8-4c69-92e3-62a2d08a6904:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:7cd1cc45-58d8-4c69-92e3-62a2d08a6904:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 6d5c6b66 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 6d5c6b66 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 6d5c6b66 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4302 (Wed Sep 26 17:41:37 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4302 (Wed Sep 26 17:41:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "8a734436", "local_conf_timestamp": 4302, "host-ts": 4302}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4297 (Wed Sep 26 17:41:31 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4297 (Wed Sep 26 17:41:31 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "a87fa5fb", "local_conf_timestamp": 4297, "host-ts": 4297}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:a81b0359-b525-41fa-9906-7f21f0f27dd6:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:a81b0359-b525-41fa-9906-7f21f0f27dd6:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 704b805a on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 704b805a on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 704b805a on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4302 (Wed Sep 26 17:41:37 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4302 (Wed Sep 26 17:41:37 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "8a734436", "local_conf_timestamp": 4302, "host-ts": 4302}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4307 (Wed Sep 26 17:41:42 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4307 (Wed Sep 26 17:41:42 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "4d44c828", "local_conf_timestamp": 4307, "host-ts": 4307}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:e51f9ed4-1717-44c9-b0ee-37ea5e05676a:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:e51f9ed4-1717-44c9-b0ee-37ea5e05676a:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 7349afde on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 7349afde on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 7349afde on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4312 (Wed Sep 26 17:41:47 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4313 (Wed Sep 26 17:41:47 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "53474a62", "local_conf_timestamp": 4313, "host-ts": 4312}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4307 (Wed Sep 26 17:41:42 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4307 (Wed Sep 26 17:41:42 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "4d44c828", "local_conf_timestamp": 4307, "host-ts": 4307}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:243d2486-a3ca-4ca5-8fff-878978118a3d:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:243d2486-a3ca-4ca5-8fff-878978118a3d:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 76684004 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 76684004 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 76684004 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4322 (Wed Sep 26 17:41:56 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4322 (Wed Sep 26 17:41:57 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "0d643ff3", "local_conf_timestamp": 4322, "host-ts": 4322}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4317 (Wed Sep 26 17:41:52 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4317 (Wed Sep 26 17:41:52 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "4902372d", "local_conf_timestamp": 4317, "host-ts": 4317}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:c1b8be35-3faa-4959-8a52-56991f4fa241:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:c1b8be35-3faa-4959-8a52-56991f4fa241:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 7999ef0c on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 7999ef0c on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 7999ef0c on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4322 (Wed Sep 26 17:41:56 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4322 (Wed Sep 26 17:41:57 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "0d643ff3", "local_conf_timestamp": 4322, "host-ts": 4322}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4317 (Wed Sep 26 17:41:52 2018)\\nhost-id=2\\nscore=3000\\nvm_conf_refresh_time=4317 (Wed Sep 26 17:41:52 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "4902372d", "local_conf_timestamp": 4317, "host-ts": 4317}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:c6947cc4-e42c-4461-a2ca-e144eed75f88:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:c6947cc4-e42c-4461-a2ca-e144eed75f88:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 7c7e5744 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 7c7e5744 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 7c7e5744 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4332 (Wed Sep 26 17:42:07 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4332 (Wed Sep 26 17:42:07 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "VM migrated away successfully", "health": "bad", "vm": "down", "detail": "Down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "c23b8814", "local_conf_timestamp": 4332, "host-ts": 4332}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4327 (Wed Sep 26 17:42:01 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4327 (Wed Sep 26 17:42:01 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "4488c592", "local_conf_timestamp": 4327, "host-ts": 4327}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:25677b7b-0a3f-4018-9a2d-399570c4f390:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:25677b7b-0a3f-4018-9a2d-399570c4f390:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 7d97c89a on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 7d97c89a on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 7d97c89a on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4332 (Wed Sep 26 17:42:07 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=4332 (Wed Sep 26 17:42:07 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"reason": "VM migrated away successfully", "health": "bad", "vm": "down", "detail": "Down"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "c23b8814", "local_conf_timestamp": 4332, "host-ts": 4332}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4327 (Wed Sep 26 17:42:01 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4327 (Wed Sep 26 17:42:01 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "4488c592", "local_conf_timestamp": 4327, "host-ts": 4327}, "global_maintenance": false}\n\nroot: INFO: * Engine has migrated.\nroot: INFO: * Waiting For System Stability...\ncli: DEBUG: signal 15 was caught\n--------------------- >> end captured logging << ---------------------'
6 years, 2 months
[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-4.2 - Build # 561
- Failure!
by jenkins@jenkins.phx.ovirt.org
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/
Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/561/
Build Number: 561
Build Status: Failure
Triggered By: Started by timer
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #561
[Eitan Raviv] networking: connect NFS to engine over IPv6
-----------------
Failed Tests:
-----------------
1 tests failed.
FAILED: 002_bootstrap.add_hosts
Error Message:
Failed to read response: [(<pycurl.Curl object at 0x7f7dc1d10050>, 7, 'TCP connection reset by peer')]
-------------------- >> begin captured logging << --------------------
ovirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7f7dc1cb2578>
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within
res = func()
File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 254, in _host_is_up_4
host_obj = host_service.get()
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 37093, in get
return self._internal_get(headers, query, wait)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get
return future.wait() if wait else future
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 54, in wait
response = self._connection.wait(self._context)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 496, in wait
return self.__wait(context, failed_auth)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 510, in __wait
raise Error("Failed to read response: {}".format(err_list))
Error: Failed to read response: [(<pycurl.Curl object at 0x7f7dc1d10050>, 7, 'TCP connection reset by peer')]
--------------------- >> end captured logging << ---------------------
Stack Trace:
File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
testMethod()
File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test
test()
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper
return func(get_test_prefix(), *args, **kwargs)
File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 229, in add_hosts
add_hosts_4(prefix)
File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 273, in add_hosts_4
testlib.assert_true_within(_host_is_up_4, timeout=15*60)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 278, in assert_true_within
assert_equals_within(func, True, timeout, allowed_exceptions)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within
res = func()
File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 254, in _host_is_up_4
host_obj = host_service.get()
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 37093, in get
return self._internal_get(headers, query, wait)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get
return future.wait() if wait else future
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 54, in wait
response = self._connection.wait(self._context)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 496, in wait
return self.__wait(context, failed_auth)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 510, in __wait
raise Error("Failed to read response: {}".format(err_list))
'Failed to read response: [(<pycurl.Curl object at 0x7f7dc1d10050>, 7, \'TCP connection reset by peer\')]\n-------------------- >> begin captured logging << --------------------\novirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7f7dc1cb2578>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 254, in _host_is_up_4\n host_obj = host_service.get()\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 37093, in get\n return self._internal_get(headers, query, wait)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get\n return future.wait() if wait else future\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 54, in wait\n response = self._connection.wait(self._context)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 496, in wait\n return self.__wait(context, failed_auth)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 510, in __wait\n raise Error("Failed to read response: {}".format(err_list))\nError: Failed to read response: [(<pycurl.Curl object at 0x7f7dc1d10050>, 7, \'TCP connection reset by peer\')]\n--------------------- >> end captured logging << ---------------------'
6 years, 2 months