Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.2/306/ Build Number: 306 Build Status: Still Failing Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #299 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file [Sandro Bonazzola] ovirt-iso-uploader: drop 4.1 jobs Changes for Build #300 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #301 [Ehud Yonasi] ovirt-master.repo: Added new packages to repo file Changes for Build #302 [Miguel Duarte Barroso] Repos: add python2-ovsdbapp to the reposync config [Sandro Bonazzola] ovirt-iso-uploader: branched for 4.2 Changes for Build #303 [Miguel Duarte Barroso] Repos: add python2-ovsdbapp to the reposync config Changes for Build #304 [Miguel Duarte Barroso] Repos: add python2-ovsdbapp to the reposync config Changes for Build #305 [Miguel Duarte Barroso] Repos: add python2-ovsdbapp to the reposync config [Gal Ben Haim] Add jobs for ovs-cni Changes for Build #306 [Ales Musil] network: syncutil: Improve sync utility with kwargs ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 010_local_maintenance_cli.local_maintenance Error Message: 143 -------------------- >> begin captured logging << -------------------- root: INFO: * Waiting For System Stability... lago.ssh: DEBUG: start task:4a0efca8-3ca5-47cf-b00d-88857a5a8d3a:Get ssh client for lago-he-node-ng-suite-4-2-host-1: lago.ssh: DEBUG: end task:4a0efca8-3ca5-47cf-b00d-88857a5a8d3a:Get ssh client for lago-he-node-ng-suite-4-2-host-1: lago.ssh: DEBUG: Running b40b2e48 on lago-he-node-ng-suite-4-2-host-1: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b40b2e48 on lago-he-node-ng-suite-4-2-host-1 returned with 0 lago.ssh: DEBUG: Command b40b2e48 on lago-he-node-ng-suite-4-2-host-1 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4214 (Thu Sep 13 01:45:28 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=4215 (Thu Sep 13 01:45:29 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "3c4c2825", "local_conf_timestamp": 4215, "host-ts": 4214}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4218 (Thu Sep 13 01:45:32 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4218 (Thu Sep 13 01:45:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "af7686ed", "local_conf_timestamp": 4218, "host-ts": 4218}, "global_maintenance": false} lago.ssh: DEBUG: start task:7a44342b-111e-4ac1-800b-901237657d7d:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:7a44342b-111e-4ac1-800b-901237657d7d:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running b518cdea on lago-he-node-ng-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local lago.ssh: DEBUG: Command b518cdea on lago-he-node-ng-suite-4-2-host-0 returned with 0 root: INFO: * Waiting for engine to migrate... lago.ssh: DEBUG: start task:1aaae337-32fb-4c17-89c6-8818ebefdc10:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: end task:1aaae337-32fb-4c17-89c6-8818ebefdc10:Get ssh client for lago-he-node-ng-suite-4-2-host-0: lago.ssh: DEBUG: Running b62e953e on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json lago.ssh: DEBUG: Command b62e953e on lago-he-node-ng-suite-4-2-host-0 returned with 0 lago.ssh: DEBUG: Command b62e953e on lago-he-node-ng-suite-4-2-host-0 output: {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4214 (Thu Sep 13 01:45:28 2018)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=4215 (Thu Sep 13 01:45:29 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "3c4c2825", "local_conf_timestamp": 4215, "host-ts": 4214}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4218 (Thu Sep 13 01:45:32 2018)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4218 (Thu Sep 13 01:45:32 2018)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "af7686ed", "local_conf_timestamp": 4218, "host-ts": 4218}, "global_maintenance": false} cli: DEBUG: signal 15 was caught --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 125, in local_maintenance _wait_for_engine_migration(host, he_index, "bad", "Migration Destination") File "/home/jenkins/workspace/ovirt-system-tests_he-node-ng-suite-4.2/ovirt-system-tests/he-node-ng-suite-4.2/test-scenarios/010_local_maintenance_cli.py", line 36, in _wait_for_engine_migration testlib.assert_true_within_long(lambda: _get_he_status(host) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 248, in assert_equals_within time.sleep(initial_wait) File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 922, in exit_handler sys.exit(128 + signum) '143\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\nlago.ssh: DEBUG: start task:4a0efca8-3ca5-47cf-b00d-88857a5a8d3a:Get ssh client for lago-he-node-ng-suite-4-2-host-1:\nlago.ssh: DEBUG: end task:4a0efca8-3ca5-47cf-b00d-88857a5a8d3a:Get ssh client for lago-he-node-ng-suite-4-2-host-1:\nlago.ssh: DEBUG: Running b40b2e48 on lago-he-node-ng-suite-4-2-host-1: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b40b2e48 on lago-he-node-ng-suite-4-2-host-1 returned with 0\nlago.ssh: DEBUG: Command b40b2e48 on lago-he-node-ng-suite-4-2-host-1 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4214 (Thu Sep 13 01:45:28 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=4215 (Thu Sep 13 01:45:29 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "3c4c2825", "local_conf_timestamp": 4215, "host-ts": 4214}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4218 (Thu Sep 13 01:45:32 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4218 (Thu Sep 13 01:45:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "af7686ed", "local_conf_timestamp": 4218, "host-ts": 4218}, "global_maintenance": false}\n\nlago.ssh: DEBUG: start task:7a44342b-111e-4ac1-800b-901237657d7d:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:7a44342b-111e-4ac1-800b-901237657d7d:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running b518cdea on lago-he-node-ng-suite-4-2-host-0: hosted-engine --set-maintenance --mode=local\nlago.ssh: DEBUG: Command b518cdea on lago-he-node-ng-suite-4-2-host-0 returned with 0\nroot: INFO: * Waiting for engine to migrate...\nlago.ssh: DEBUG: start task:1aaae337-32fb-4c17-89c6-8818ebefdc10:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: end task:1aaae337-32fb-4c17-89c6-8818ebefdc10:Get ssh client for lago-he-node-ng-suite-4-2-host-0:\nlago.ssh: DEBUG: Running b62e953e on lago-he-node-ng-suite-4-2-host-0: hosted-engine --vm-status --json\nlago.ssh: DEBUG: Command b62e953e on lago-he-node-ng-suite-4-2-host-0 returned with 0\nlago.ssh: DEBUG: Command b62e953e on lago-he-node-ng-suite-4-2-host-0 output:\n {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4214 (Thu Sep 13 01:45:28 2018)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=4215 (Thu Sep 13 01:45:29 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineUp\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-0", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "3c4c2825", "local_conf_timestamp": 4215, "host-ts": 4214}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=4218 (Thu Sep 13 01:45:32 2018)\\nhost-id=2\\nscore=3400\\nvm_conf_refresh_time=4218 (Thu Sep 13 01:45:32 2018)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineDown\\nstopped=False\\n", "hostname": "lago-he-node-ng-suite-4-2-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "af7686ed", "local_conf_timestamp": 4218, "host-ts": 4218}, "global_maintenance": false}\n\ncli: DEBUG: signal 15 was caught\n--------------------- >> end captured logging << ---------------------'