- Infra - oVirt List Archives

oVirt infra daily report - unstable production jobs - 622
by jenkins＠jenkins.phx.ovirt.org 23 Oct '18

23 Oct '18

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/622//artifact/exported-a… Cheers, Jenkins

1 0

[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-master - Build # 804 - Failure!
by jenkins＠jenkins.phx.ovirt.org 23 Oct '18

23 Oct '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/804/ Build Number: 804 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #804 [Galit] Fix stdci.yaml to be able to run poll upstream sources ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 002_bootstrap.add_hosts Error Message: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML -------------------- >> begin captured logging << -------------------- ovirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7fde909da230> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 253, in _host_is_up_4 host_obj = host_service.get() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 38104, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault body = self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 850, in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 890, in _check_content_type raise Error(msg) Error: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 228, in add_hosts add_hosts_4(prefix) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 275, in add_hosts_4 testlib.assert_true_within(_host_is_up_4, timeout=15*60) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 278, in assert_true_within assert_equals_within(func, True, timeout, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 253, in _host_is_up_4 host_obj = host_service.get() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 38104, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault body = self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 850, in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 890, in _check_content_type raise Error(msg) 'The response content type \'text/html; charset=iso-8859-1\' isn\'t the expected XML\n-------------------- >> begin captured logging << --------------------\novirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7fde909da230>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 253, in _host_is_up_4\n host_obj = host_service.get()\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 38104, in get\n return self._internal_get(headers, query, wait)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get\n return future.wait() if wait else future\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait\n return self._code(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback\n self._check_fault(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault\n body = self._internal_read_body(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body\n self._connection.check_xml_content_type(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 850, in check_xml_content_type\n response.headers\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 890, in _check_content_type\n raise Error(msg)\nError: The response content type \'text/html; charset=iso-8859-1\' isn\'t the expected XML\n--------------------- >> end captured logging << ---------------------'

1 8

[oVirt Jenkins] ovirt-system-tests_he-node-ng-suite-4.2 - Build # 411 - Failure!
by jenkins＠jenkins.phx.ovirt.org 23 Oct '18

23 Oct '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/411/ Build Number: 411 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #411 [Galit] Update the config for poll u/s sources ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 010_local_maintenance_cli.local_maintenance Error Message: 143 -------------------- >> begin captured logging << -------------------- root: INFO: * Waiting For System Stability... lago.ssh: DEBUG: start task:461f166d-b3cb-4766-af33-4319bd3b5227:Get ssh client for lago-he-node-ng-suite-4-2-host-1: lago.ssh: DEBUG: end task:461f166d-b3cb-4766-af33-4319bd3b5227:Get ssh client for lago-he-node-ng-suite-4-2-host-1: lago.ssh: DEBUG: Running 5e792150 on lago-he-node-ng-suite-4-2-host-1: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 5e792150 on lago-he-node-ng-suite-4-2-host-1 returned with 0 lago.ssh: DEBUG: Command 5e792150 on lago-he-node-ng-suite-4-2-host-1 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3933 (Thu Oct 18 01:44:27 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=3933 (Thu Oct 18 01:44:28 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f7c0dd5c", "local_conf_timestamp": 3933, "host-ts": 3933}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3935 (Thu Oct 18 01:44:29 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3935 (Thu Oct 18 01:44:29 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "da25a075", "local_conf_timestamp": 3935, "host-ts": 3935}, "global_maintenance": false} lago.ssh: DEBUG: start task:a64ae178-4ca1-4b6a-b5a6-e4250fe83398:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:a64ae178-4ca1-4b6a-b5a6-e4250fe83398:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 5f815572 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local lago.ssh: DEBUG: Command 5f815572 on lago-he-node-ng-suite-4-2-host-0 returned with 0 root: INFO: * Waiting for engine to migrate... lago.ssh: DEBUG: start task:23a6c556-e234-43d1-8313-4612bd35fdf0:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:23a6c556-e234-43d1-8313-4612bd35fdf0:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 60b78506 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 60b78506 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 60b78506 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3943 (Thu Oct 18 01:44:38 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=3943 (Thu Oct 18 01:44:38 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8f671722", "local_conf_timestamp": 3943, "host-ts": 3943}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3935 (Thu Oct 18 01:44:29 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3935 (Thu Oct 18 01:44:29 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "da25a075", "local_conf_timestamp": 3935, "host-ts": 3935}, "global_maintenance": false} lago.ssh: DEBUG: start task:080e3a23-a075-43dd-895c-00855470de8a:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:080e3a23-a075-43dd-895c-00855470de8a:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 67d368b4 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 67d368b4 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 67d368b4 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3953 (Thu Oct 18 01:44:47 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3953 (Thu Oct 18 01:44:47 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "706f0031", "local_conf_timestamp": 3953, "host-ts": 3953}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3945 (Thu Oct 18 01:44:39 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3945 (Thu Oct 18 01:44:39 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c7f55d6e", "local_conf_timestamp": 3945, "host-ts": 3945}, "global_maintenance": false} lago.ssh: DEBUG: start task:592ee562-ac13-4dee-9eeb-8e72bbe78ced:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:592ee562-ac13-4dee-9eeb-8e72bbe78ced:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 6bf19f06 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 6bf19f06 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 6bf19f06 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3963 (Thu Oct 18 01:44:58 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3963 (Thu Oct 18 01:44:58 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "6263fdaa", "local_conf_timestamp": 3963, "host-ts": 3963}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3955 (Thu Oct 18 01:44:50 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3956 (Thu Oct 18 01:44:50 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "5abaf3d1", "local_conf_timestamp": 3956, "host-ts": 3955}, "global_maintenance": false} lago.ssh: DEBUG: start task:3aac038b-901e-4bca-8616-7dae3ce335b0:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:3aac038b-901e-4bca-8616-7dae3ce335b0:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 6ef7539e on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 6ef7539e on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 6ef7539e on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3963 (Thu Oct 18 01:44:58 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3963 (Thu Oct 18 01:44:58 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "6263fdaa", "local_conf_timestamp": 3963, "host-ts": 3963}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3955 (Thu Oct 18 01:44:50 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3956 (Thu Oct 18 01:44:50 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "5abaf3d1", "local_conf_timestamp": 3956, "host-ts": 3955}, "global_maintenance": false} lago.ssh: DEBUG: start task:a8248fe7-4e2c-455d-b5ec-b1954bf41a8e:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:a8248fe7-4e2c-455d-b5ec-b1954bf41a8e:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 71ed0dfa on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 71ed0dfa on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 71ed0dfa on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3973 (Thu Oct 18 01:45:08 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3974 (Thu Oct 18 01:45:08 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "523456cb", "local_conf_timestamp": 3974, "host-ts": 3973}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3965 (Thu Oct 18 01:44:59 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3965 (Thu Oct 18 01:44:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b84596fc", "local_conf_timestamp": 3965, "host-ts": 3965}, "global_maintenance": false} lago.ssh: DEBUG: start task:65b6000b-8e80-4dee-af83-a6a74c034238:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:65b6000b-8e80-4dee-af83-a6a74c034238:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 74df0478 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 74df0478 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 74df0478 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3973 (Thu Oct 18 01:45:08 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3974 (Thu Oct 18 01:45:08 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "523456cb", "local_conf_timestamp": 3974, "host-ts": 3973}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3965 (Thu Oct 18 01:44:59 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3965 (Thu Oct 18 01:44:59 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b84596fc", "local_conf_timestamp": 3965, "host-ts": 3965}, "global_maintenance": false} lago.ssh: DEBUG: start task:847d9012-2456-4e12-b2f9-c7e4da9c4ff7:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:847d9012-2456-4e12-b2f9-c7e4da9c4ff7:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 77d8a918 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 77d8a918 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 77d8a918 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3983 (Thu Oct 18 01:45:17 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3983 (Thu Oct 18 01:45:17 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "41e59273", "local_conf_timestamp": 3983, "host-ts": 3983}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3975 (Thu Oct 18 01:45:09 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3975 (Thu Oct 18 01:45:09 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "bc0369f9", "local_conf_timestamp": 3975, "host-ts": 3975}, "global_maintenance": false} lago.ssh: DEBUG: start task:dd6b5da4-d22b-4284-85d5-dfe9a51ee00c:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:dd6b5da4-d22b-4284-85d5-dfe9a51ee00c:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 7ac56292 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 7ac56292 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 7ac56292 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3983 (Thu Oct 18 01:45:17 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3983 (Thu Oct 18 01:45:17 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "41e59273", "local_conf_timestamp": 3983, "host-ts": 3983}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3975 (Thu Oct 18 01:45:09 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3975 (Thu Oct 18 01:45:09 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "bc0369f9", "local_conf_timestamp": 3975, "host-ts": 3975}, "global_maintenance": false} lago.ssh: DEBUG: start task:315c897e-fc06-484c-a036-617859dbb2ec:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:315c897e-fc06-484c-a036-617859dbb2ec:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 7db577e4 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 7db577e4 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 7db577e4 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3993 (Thu Oct 18 01:45:28 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3994 (Thu Oct 18 01:45:28 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "71b23912", "local_conf_timestamp": 3994, "host-ts": 3993}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3985 (Thu Oct 18 01:45:19 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3985 (Thu Oct 18 01:45:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "83e46cca", "local_conf_timestamp": 3985, "host-ts": 3985}, "global_maintenance": false} lago.ssh: DEBUG: start task:c5f00620-37e4-4d82-b99e-e9ac42e3e145:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:c5f00620-37e4-4d82-b99e-e9ac42e3e145:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running 81550626 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command 81550626 on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command 81550626 on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3993 (Thu Oct 18 01:45:28 2018)\nhost-id=1\nscore=3000\nvm_conf_refresh_time=3994 (Thu Oct 18 01:45:28 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineMigratingAway\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "71b23912", "local_conf_timestamp": 3994, "host-ts": 3993}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3985 (Thu Oct 18 01:45:19 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3985 (Thu Oct 18 01:45:19 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "83e46cca", "local_conf_timestamp": 3985, "host-ts": 3985}, "global_maintenance": false} cli: DEBUG: signal 15 was caught --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 125, in local_maintenance _wait_for_engine_migration(host, he_index, "bad", "Migration Destination") File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 36, in _wait_for_engine_migration testlib.assert_true_within_long(lambda: _get_he_status(host) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 246, in assert_equals_within time.sleep(3) File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 922, in exit_handler sys.exit(128 + signum) '143\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\nlago.ssh: DEBUG: start task:461f166d-b3cb-4766-af33-4319bd3b5227:Get ssh client for lago-he-node-ng-suite-4-2-host-1:\nlago.ssh: DEBUG: end task:461f166d-b3cb-4766-af33-4319bd3b5227:Get ssh client for lago-he-node-ng-suite-4-2-host-1:\nlago.ssh: DEBUG: Running 5e792150 on lago-he-node-ng-suite-4-2-host-1: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 5e792150 on lago-he-node-ng-suite-4-2-host-1 returned with 0\nlago.ssh: DEBUG: Command 5e792150 on lago-he-node-ng-suite-4-2-host-1 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3933 (Thu Oct 18 01:44:27 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=3933 (Thu Oct 18 01:44:28 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "f7c0dd5c", "local_conf_timestamp": 3933, "host-ts": 3933}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3935 (Thu Oct 18 01:44:29 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3935 (Thu Oct 18 01:44:29 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "da25a075", "local_conf_timestamp": 3935, "host-ts": 3935}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:a64ae178-4ca1-4b6a-b5a6-e4250fe83398:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:a64ae178-4ca1-4b6a-b5a6-e4250fe83398:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 5f815572 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local\nlago.ssh: DEBUG: Command 5f815572 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nroot: INFO: * Waiting for engine to migrate...\nlago.ssh: DEBUG: start task:23a6c556-e234-43d1-8313-4612bd35fdf0:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:23a6c556-e234-43d1-8313-4612bd35fdf0:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 60b78506 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 60b78506 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 60b78506 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3943 (Thu Oct 18 01:44:38 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=3943 (Thu Oct 18 01:44:38 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "8f671722", "local_conf_timestamp": 3943, "host-ts": 3943}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3935 (Thu Oct 18 01:44:29 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3935 (Thu Oct 18 01:44:29 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "da25a075", "local_conf_timestamp": 3935, "host-ts": 3935}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:080e3a23-a075-43dd-895c-00855470de8a:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:080e3a23-a075-43dd-895c-00855470de8a:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 67d368b4 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 67d368b4 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 67d368b4 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3953 (Thu Oct 18 01:44:47 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3953 (Thu Oct 18 01:44:47 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "706f0031", "local_conf_timestamp": 3953, "host-ts": 3953}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3945 (Thu Oct 18 01:44:39 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3945 (Thu Oct 18 01:44:39 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "c7f55d6e", "local_conf_timestamp": 3945, "host-ts": 3945}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:592ee562-ac13-4dee-9eeb-8e72bbe78ced:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:592ee562-ac13-4dee-9eeb-8e72bbe78ced:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 6bf19f06 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 6bf19f06 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 6bf19f06 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3963 (Thu Oct 18 01:44:58 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3963 (Thu Oct 18 01:44:58 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "6263fdaa", "local_conf_timestamp": 3963, "host-ts": 3963}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3955 (Thu Oct 18 01:44:50 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3956 (Thu Oct 18 01:44:50 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "5abaf3d1", "local_conf_timestamp": 3956, "host-ts": 3955}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:3aac038b-901e-4bca-8616-7dae3ce335b0:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:3aac038b-901e-4bca-8616-7dae3ce335b0:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 6ef7539e on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 6ef7539e on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 6ef7539e on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3963 (Thu Oct 18 01:44:58 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3963 (Thu Oct 18 01:44:58 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "6263fdaa", "local_conf_timestamp": 3963, "host-ts": 3963}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3955 (Thu Oct 18 01:44:50 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3956 (Thu Oct 18 01:44:50 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "5abaf3d1", "local_conf_timestamp": 3956, "host-ts": 3955}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:a8248fe7-4e2c-455d-b5ec-b1954bf41a8e:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:a8248fe7-4e2c-455d-b5ec-b1954bf41a8e:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 71ed0dfa on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 71ed0dfa on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 71ed0dfa on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3973 (Thu Oct 18 01:45:08 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3974 (Thu Oct 18 01:45:08 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "523456cb", "local_conf_timestamp": 3974, "host-ts": 3973}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3965 (Thu Oct 18 01:44:59 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3965 (Thu Oct 18 01:44:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b84596fc", "local_conf_timestamp": 3965, "host-ts": 3965}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:65b6000b-8e80-4dee-af83-a6a74c034238:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:65b6000b-8e80-4dee-af83-a6a74c034238:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 74df0478 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 74df0478 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 74df0478 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3973 (Thu Oct 18 01:45:08 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3974 (Thu Oct 18 01:45:08 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "523456cb", "local_conf_timestamp": 3974, "host-ts": 3973}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3965 (Thu Oct 18 01:44:59 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3965 (Thu Oct 18 01:44:59 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "b84596fc", "local_conf_timestamp": 3965, "host-ts": 3965}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:847d9012-2456-4e12-b2f9-c7e4da9c4ff7:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:847d9012-2456-4e12-b2f9-c7e4da9c4ff7:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 77d8a918 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 77d8a918 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 77d8a918 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3983 (Thu Oct 18 01:45:17 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3983 (Thu Oct 18 01:45:17 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "41e59273", "local_conf_timestamp": 3983, "host-ts": 3983}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3975 (Thu Oct 18 01:45:09 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3975 (Thu Oct 18 01:45:09 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "bc0369f9", "local_conf_timestamp": 3975, "host-ts": 3975}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:dd6b5da4-d22b-4284-85d5-dfe9a51ee00c:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:dd6b5da4-d22b-4284-85d5-dfe9a51ee00c:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 7ac56292 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 7ac56292 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 7ac56292 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3983 (Thu Oct 18 01:45:17 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3983 (Thu Oct 18 01:45:17 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "41e59273", "local_conf_timestamp": 3983, "host-ts": 3983}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3975 (Thu Oct 18 01:45:09 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3975 (Thu Oct 18 01:45:09 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "bc0369f9", "local_conf_timestamp": 3975, "host-ts": 3975}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:315c897e-fc06-484c-a036-617859dbb2ec:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:315c897e-fc06-484c-a036-617859dbb2ec:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 7db577e4 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 7db577e4 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 7db577e4 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3993 (Thu Oct 18 01:45:28 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3994 (Thu Oct 18 01:45:28 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "71b23912", "local_conf_timestamp": 3994, "host-ts": 3993}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3985 (Thu Oct 18 01:45:19 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3985 (Thu Oct 18 01:45:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "83e46cca", "local_conf_timestamp": 3985, "host-ts": 3985}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:c5f00620-37e4-4d82-b99e-e9ac42e3e145:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:c5f00620-37e4-4d82-b99e-e9ac42e3e145:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running 81550626 on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command 81550626 on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command 81550626 on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3993 (Thu Oct 18 01:45:28 2018)\\nhost-id=1\\nscore=3000\\nvm_conf_refresh_time=3994 (Thu Oct 18 01:45:28 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineMigratingAway\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Migration Source"}, "score": 3000, "stopped": false, "maintenance": false, "crc32": "71b23912", "local_conf_timestamp": 3994, "host-ts": 3993}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=3985 (Thu Oct 18 01:45:19 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=3985 (Thu Oct 18 01:45:19 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "Migration Destination"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "83e46cca", "local_conf_timestamp": 3985, "host-ts": 3985}, "global_maintenance": false}\n\ncli: DEBUG: signal 15 was caught\n--------------------- >> end captured logging << ---------------------'

1 17

[JIRA] (OVIRT-2558) containerize more bare metals
by Evgheni Dereveanchin (oVirt JIRA) 23 Oct '18

23 Oct '18

Evgheni Dereveanchin created OVIRT-2558: ------------------------------------------- Summary: containerize more bare metals Key: OVIRT-2558 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2558 Project: oVirt - virtualization made easy Issue Type: Task Reporter: Evgheni Dereveanchin Assignee: infra The build queue on the Jenkins is growing and we still have two larger bare metals under-utilized as they run just a single Lago instance at a time: ovirt-srv07 and ovirt-srv08 I've offlined them so that they can be containerized and run up to 7 CI suites at the same time, same as we have on ovirt-srv04-06 at the moment. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100094)

1 0

[CQ]: 95011, 1 (otopi) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 23 Oct '18

23 Oct '18

A system test invoked by the "ovirt-master" change queue including change 95011,1 (otopi) failed. However, this change seems not to be the root cause for this failure. Change 91669,5 (otopi) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 91669,5 (otopi) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/95011/1 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/91669/5 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/10910/

2 1

[oVirt Jenkins] ovirt-node-ng-image_master_build-artifacts-fc28-x86_64 - Build # 92 - Failure!
by jenkins＠jenkins.phx.ovirt.org 23 Oct '18

23 Oct '18

Project: http://jenkins.ovirt.org/job/ovirt-node-ng-image_master_build-artifacts-fc2… Build: http://jenkins.ovirt.org/job/ovirt-node-ng-image_master_build-artifacts-fc2… Build Number: 92 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #92 [Yuval Turgeman] stdci: move build-artifacts to a v1 job ----------------- Failed Tests: ----------------- No tests ran.

1 1

[JIRA] (OVIRT-2557) upgrade jenkins.ovirt.org to 2.138.2 LTS
by Evgheni Dereveanchin (oVirt JIRA) 23 Oct '18

23 Oct '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2557?page=com.atlassian.jira.… ] Evgheni Dereveanchin reassigned OVIRT-2557: ------------------------------------------- Assignee: Evgheni Dereveanchin (was: infra) > upgrade jenkins.ovirt.org to 2.138.2 LTS > ---------------------------------------- > > Key: OVIRT-2557 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2557 > Project: oVirt - virtualization made easy > Issue Type: Task > Reporter: Evgheni Dereveanchin > Assignee: Evgheni Dereveanchin > > Jenkins 2.138.2 LTS is out, our instance needs to be updated. > https://jenkins.io/changelog-stable/ > Plugin security updates were also released so we should bump versions, too -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100094)

1 0

[JIRA] (OVIRT-2557) upgrade jenkins.ovirt.org to 2.138.2 LTS
by Evgheni Dereveanchin (oVirt JIRA) 23 Oct '18

23 Oct '18

Evgheni Dereveanchin created OVIRT-2557: ------------------------------------------- Summary: upgrade jenkins.ovirt.org to 2.138.2 LTS Key: OVIRT-2557 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2557 Project: oVirt - virtualization made easy Issue Type: Task Reporter: Evgheni Dereveanchin Assignee: infra Jenkins 2.138.2 LTS is out, our instance needs to be updated. https://jenkins.io/changelog-stable/ Plugin security updates were also released so we should bump versions, too -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100094)

1 0

[JIRA] (OVIRT-2556) kubevirt test runs and nested docker issue
by Gal Ben Haim (oVirt JIRA) 23 Oct '18

23 Oct '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2556?page=com.atlassian.jira.… ] Gal Ben Haim commented on OVIRT-2556: ------------------------------------- [~lbednar] FYI > kubevirt test runs and nested docker issue > ------------------------------------------ > > Key: OVIRT-2556 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2556 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Roman Mohr > Assignee: infra > > Hi, > we see a lot of tests which fail with during the setup [1] : > ``` > /usr/bin/docker-current: Error response from daemon: grpc: the > connection is unavailable. > time="2018-10-23T07:17:55Z" level=error msg="error getting events from > daemon: context canceled" > ``` > It looks like the docker daemon may not run properly. > Best Regards, > Roman > [1] https://jenkins.ovirt.org/job/kubevirt_kubevirt_standard-check-pr/2344//art… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100094)

1 0

[JIRA] (OVIRT-2556) kubevirt test runs and nested docker issue
by Gal Ben Haim (oVirt JIRA) 23 Oct '18

23 Oct '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2556?page=com.atlassian.jira.… ] Gal Ben Haim commented on OVIRT-2556: ------------------------------------- I've killed the old STDCI container and spawned a new one instead. > kubevirt test runs and nested docker issue > ------------------------------------------ > > Key: OVIRT-2556 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2556 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Roman Mohr > Assignee: infra > > Hi, > we see a lot of tests which fail with during the setup [1] : > ``` > /usr/bin/docker-current: Error response from daemon: grpc: the > connection is unavailable. > time="2018-10-23T07:17:55Z" level=error msg="error getting events from > daemon: context canceled" > ``` > It looks like the docker daemon may not run properly. > Best Regards, > Roman > [1] https://jenkins.ovirt.org/job/kubevirt_kubevirt_standard-check-pr/2344//art… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100094)

1 0