[JIRA] (OVIRT-2586) Memory leak in Jenkins SSE Gateway plugin
by Evgheni Dereveanchin (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-2586?page=com.atlassian.jir... ]
Evgheni Dereveanchin updated OVIRT-2586:
----------------------------------------
Summary: Memory leak in Jenkins SSE Gateway plugin (was: Jenkins terribly slow and unresponsive)
> Memory leak in Jenkins SSE Gateway plugin
> -----------------------------------------
>
> Key: OVIRT-2586
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2586
> Project: oVirt - virtualization made easy
> Issue Type: Outage
> Reporter: sbonazzo
> Assignee: Evgheni Dereveanchin
> Attachments: usedMemory_year.png
>
>
> Hi,
> jenkins is terribly slow and becoming worse every day.
> I tried to gain some speed by adding 4 cores to the VM through engine-phx.
> It's a bit better but the real issue doesn't seem related to CPU power.
> Can anybody investigate?
> --
> SANDRO BONAZZOLA
> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
> Red Hat EMEA <https://www.redhat.com/>
> sbonazzo(a)redhat.com
> <https://red.ht/sig>
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100099)
5 years, 1 month
[oVirt Jenkins] ovirt-system-tests_compat-4.3-suite-master - Build
# 17 - Failure!
by jenkins＠jenkins.phx.ovirt.org
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_compat-4.3-suite-master/
Build: http://jenkins.ovirt.org/job/ovirt-system-tests_compat-4.3-suite-master/17/
Build Number: 17
Build Status: Failure
Triggered By: Started by timer
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #17
[Milan Zamazal] Add hc-basic-suite-4.3
-----------------
Failed Tests:
-----------------
2 tests failed.
FAILED: 002_bootstrap.add_master_storage_domain_43
Error Message:
Could not find hosts that are up in DC test-dc
-------------------- >> begin captured logging << --------------------
lago.ssh: DEBUG: start task:eac93cd6-f7d5-4709-b8d4-6abf4ede9ba8:Get ssh client for lago-compat-4-3-suite-master-engine:
lago.ssh: DEBUG: end task:eac93cd6-f7d5-4709-b8d4-6abf4ede9ba8:Get ssh client for lago-compat-4-3-suite-master-engine:
lago.ssh: DEBUG: Running c6f4be5c on lago-compat-4-3-suite-master-engine: cat /root/multipath.txt
lago.ssh: DEBUG: Command c6f4be5c on lago-compat-4-3-suite-master-engine returned with 0
lago.ssh: DEBUG: Command c6f4be5c on lago-compat-4-3-suite-master-engine output:
360014052ab13391ff944443894c2b0f7
360014052caa297d60e9440592a1f3d04
36001405bc22d1395bd6476299a975c10
36001405edd5271bd714406584bfb9d26
36001405f0546ac01e2c444ab58bb99d3
--------------------- >> end captured logging << ---------------------
Stack Trace:
File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
testMethod()
File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test
test()
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper
return func(get_test_prefix(), *args, **kwargs)
File "/home/jenkins/workspace/ovirt-system-tests_compat-4.3-suite-master/ovirt-system-tests/compat-4.3-suite-master/test-scenarios/002_bootstrap.py", line 417, in add_master_storage_domain
add_iscsi_storage_domain(prefix)
File "/home/jenkins/workspace/ovirt-system-tests_compat-4.3-suite-master/ovirt-system-tests/compat-4.3-suite-master/test-scenarios/002_bootstrap.py", line 561, in add_iscsi_storage_domain
host=_random_host_from_dc(api, DC_NAME),
File "/home/jenkins/workspace/ovirt-system-tests_compat-4.3-suite-master/ovirt-system-tests/compat-4.3-suite-master/test-scenarios/002_bootstrap.py", line 122, in _random_host_from_dc
return _hosts_in_dc(api, dc_name, True)
File "/home/jenkins/workspace/ovirt-system-tests_compat-4.3-suite-master/ovirt-system-tests/compat-4.3-suite-master/test-scenarios/002_bootstrap.py", line 119, in _hosts_in_dc
raise RuntimeError('Could not find hosts that are up in DC %s' % dc_name)
'Could not find hosts that are up in DC test-dc\n-------------------- >> begin captured logging << --------------------\nlago.ssh: DEBUG: start task:eac93cd6-f7d5-4709-b8d4-6abf4ede9ba8:Get ssh client for lago-compat-4-3-suite-master-engine:\nlago.ssh: DEBUG: end task:eac93cd6-f7d5-4709-b8d4-6abf4ede9ba8:Get ssh client for lago-compat-4-3-suite-master-engine:\nlago.ssh: DEBUG: Running c6f4be5c on lago-compat-4-3-suite-master-engine: cat /root/multipath.txt\nlago.ssh: DEBUG: Command c6f4be5c on lago-compat-4-3-suite-master-engine returned with 0\nlago.ssh: DEBUG: Command c6f4be5c on lago-compat-4-3-suite-master-engine output:\n 360014052ab13391ff944443894c2b0f7\n360014052caa297d60e9440592a1f3d04\n36001405bc22d1395bd6476299a975c10\n36001405edd5271bd714406584bfb9d26\n36001405f0546ac01e2c444ab58bb99d3\n\n--------------------- >> end captured logging << ---------------------'
FAILED: 002_bootstrap.add_master_storage_domain_43
Error Message:
Failed to copy files/directory from /var/log to /dev/shm/ost/deployment-compat-4.3-suite-master/default/test_logs/002_bootstrap.add_master_storage_domain_43-20190323005122/lago-compat-4-3-suite-master-host-1/_var_log
Unable to get valid context for root
-------------------- >> begin captured logging << --------------------
lago.ssh: DEBUG: start task:eac93cd6-f7d5-4709-b8d4-6abf4ede9ba8:Get ssh client for lago-compat-4-3-suite-master-engine:
lago.ssh: DEBUG: end task:eac93cd6-f7d5-4709-b8d4-6abf4ede9ba8:Get ssh client for lago-compat-4-3-suite-master-engine:
lago.ssh: DEBUG: Running c6f4be5c on lago-compat-4-3-suite-master-engine: cat /root/multipath.txt
lago.ssh: DEBUG: Command c6f4be5c on lago-compat-4-3-suite-master-engine returned with 0
lago.ssh: DEBUG: Command c6f4be5c on lago-compat-4-3-suite-master-engine output:
360014052ab13391ff944443894c2b0f7
360014052caa297d60e9440592a1f3d04
36001405bc22d1395bd6476299a975c10
36001405edd5271bd714406584bfb9d26
36001405f0546ac01e2c444ab58bb99d3
lago.prefix: INFO: * Collect artifacts: ?[0m?[0m
lago.prefix: INFO: ?[0m?[0m
lago.prefix: INFO: ?[0m?[0m
lago.prefix: INFO: ?[0m?[0m
lago.plugins.vm: DEBUG: Extracting scp://lago-compat-4-3-suite-master-host-0:/var/log to /dev/shm/ost/deployment-compat-4.3-suite-master/default/test_logs/002_bootstrap.add_master_storage_domain_43-20190323005122/lago-compat-4-3-suite-master-host-0/_var_log
lago.plugins.vm: INFO: ?[0m?[0m
lago.plugins.vm: DEBUG: Extracting scp://lago-compat-4-3-suite-master-host-1:/var/log to /dev/shm/ost/deployment-compat-4.3-suite-master/default/test_logs/002_bootstrap.add_master_storage_domain_43-20190323005122/lago-compat-4-3-suite-master-host-1/_var_log
lago.ssh: DEBUG: start task:8f203123-42fa-4a8f-9c1b-7d4dfbd394aa:Get ssh client for lago-compat-4-3-suite-master-host-0:
lago.plugins.vm: DEBUG: Extracting scp://lago-compat-4-3-suite-master-engine:/tmp/otopi* to /dev/shm/ost/deployment-compat-4.3-suite-master/default/test_logs/002_bootstrap.add_master_storage_domain_43-20190323005122/lago-compat-4-3-suite-master-engine/_tmp_otopi*
lago.plugins.vm: INFO: ?[0m?[0m
lago.plugins.vm: INFO: ?[0m?[0m
lago.ssh: DEBUG: start task:9f190a0b-bcd0-46c2-93ea-713e1a42be0e:Get ssh client for lago-compat-4-3-suite-master-host-1:
lago.ssh: DEBUG: start task:6a194b08-2d10-4d75-9af6-f813f6113ea6:Get ssh client for lago-compat-4-3-suite-master-engine:
lago.ssh: DEBUG: end task:6a194b08-2d10-4d75-9af6-f813f6113ea6:Get ssh client for lago-compat-4-3-suite-master-engine:
lago.ssh: DEBUG: end task:8f203123-42fa-4a8f-9c1b-7d4dfbd394aa:Get ssh client for lago-compat-4-3-suite-master-host-0:
lago.ssh: DEBUG: end task:9f190a0b-bcd0-46c2-93ea-713e1a42be0e:Get ssh client for lago-compat-4-3-suite-master-host-1:
root: ERROR: ~ [Thread-561] Copy from lago-compat-4-3-suite-master-host-1:/var/log to /dev/shm/ost/deployment-compat-4.3-suite-master/default/test_logs/002_bootstrap.add_master_storage_domain_43-20190323005122/lago-compat-4-3-suite-master-host-1/_var_log: ?[31mERROR?[0m (in 0:00:00)
lago.plugins.vm: DEBUG: File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 445, in copy_from
remote_path, local_path, err.args[0]
root: ERROR: - [Thread-561] lago-compat-4-3-suite-master-host-1: ?[31mERROR?[0m (in 0:00:00)
lago.prefix: DEBUG: File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in _collect_artifacts
vm.collect_artifacts(path, ignore_nopath)
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 659, in collect_artifacts
ignore_nopath=ignore_nopath
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 379, in extract_paths
return self.provider.extract_paths(paths, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 398, in extract_paths
ignore_nopath=ignore_nopath,
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 249, in extract_paths
self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath)
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 99, in wrapper
return func(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 269, in _extract_paths_scp
propagate_fail=not ignore_nopath
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 445, in copy_from
remote_path, local_path, err.args[0]
lago.utils: DEBUG: Error while running thread Thread-561
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue
queue.put({'return': func()})
File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in _collect_artifacts
vm.collect_artifacts(path, ignore_nopath)
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 659, in collect_artifacts
ignore_nopath=ignore_nopath
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 379, in extract_paths
return self.provider.extract_paths(paths, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 398, in extract_paths
ignore_nopath=ignore_nopath,
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 249, in extract_paths
self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath)
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 99, in wrapper
return func(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 269, in _extract_paths_scp
propagate_fail=not ignore_nopath
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 445, in copy_from
remote_path, local_path, err.args[0]
LagoCopyFilesFromVMError: Failed to copy files/directory from /var/log to /dev/shm/ost/deployment-compat-4.3-suite-master/default/test_logs/002_bootstrap.add_master_storage_domain_43-20190323005122/lago-compat-4-3-suite-master-host-1/_var_log
Unable to get valid context for root
root: ERROR: ~ [Thread-562] Copy from lago-compat-4-3-suite-master-engine:/tmp/otopi* to /dev/shm/ost/deployment-compat-4.3-suite-master/default/test_logs/002_bootstrap.add_master_storage_domain_43-20190323005122/lago-compat-4-3-suite-master-engine/_tmp_otopi*: ?[31mERROR?[0m (in 0:00:00)
lago.plugins.vm: DEBUG: File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 442, in copy_from
raise ExtractPathNoPathError(err.args[0])
root: ERROR: - [Thread-562] lago-compat-4-3-suite-master-engine: ?[31mERROR?[0m (in 0:00:00)
lago.prefix: DEBUG: File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in _collect_artifacts
vm.collect_artifacts(path, ignore_nopath)
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 659, in collect_artifacts
ignore_nopath=ignore_nopath
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 379, in extract_paths
return self.provider.extract_paths(paths, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 398, in extract_paths
ignore_nopath=ignore_nopath,
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 249, in extract_paths
self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath)
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 99, in wrapper
return func(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 269, in _extract_paths_scp
propagate_fail=not ignore_nopath
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 442, in copy_from
raise ExtractPathNoPathError(err.args[0])
lago.utils: DEBUG: Error while running thread Thread-562
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue
queue.put({'return': func()})
File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in _collect_artifacts
vm.collect_artifacts(path, ignore_nopath)
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 659, in collect_artifacts
ignore_nopath=ignore_nopath
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 379, in extract_paths
return self.provider.extract_paths(paths, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 398, in extract_paths
ignore_nopath=ignore_nopath,
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 249, in extract_paths
self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath)
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 99, in wrapper
return func(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 269, in _extract_paths_scp
propagate_fail=not ignore_nopath
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 442, in copy_from
raise ExtractPathNoPathError(err.args[0])
ExtractPathNoPathError: Failed to extract files: ?scp: /tmp/otopi*: No such file or directory
lago.plugins.vm: INFO: ?[32mSuccess?[0m (in 0:00:01)
lago.plugins.vm: DEBUG: Extracting scp://lago-compat-4-3-suite-master-host-0:/etc/resolv.conf to /dev/shm/ost/deployment-compat-4.3-suite-master/default/test_logs/002_bootstrap.add_master_storage_domain_43-20190323005122/lago-compat-4-3-suite-master-host-0/_etc_resolv.conf
lago.plugins.vm: INFO: ?[0m?[0m
lago.ssh: DEBUG: start task:4f64c62d-26aa-4e98-bd49-e0b46d4a10c3:Get ssh client for lago-compat-4-3-suite-master-host-0:
lago.ssh: DEBUG: end task:4f64c62d-26aa-4e98-bd49-e0b46d4a10c3:Get ssh client for lago-compat-4-3-suite-master-host-0:
lago.plugins.vm: INFO: ?[32mSuccess?[0m (in 0:00:00)
lago.prefix: INFO: ?[31mERROR?[0m (in 0:00:01)
root: ERROR: * Collect artifacts: ?[31mERROR?[0m (in 0:00:01)
lago.prefix: DEBUG: File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636, in wrapper
return func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1530, in collect_artifacts
self.virt_env.get_vms().values(),
File "/usr/lib/python2.7/site-packages/lago/utils.py", line 103, in invoke_in_parallel
return vt.join_all()
File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue
queue.put({'return': func()})
File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in _collect_artifacts
vm.collect_artifacts(path, ignore_nopath)
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 659, in collect_artifacts
ignore_nopath=ignore_nopath
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 379, in extract_paths
return self.provider.extract_paths(paths, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 398, in extract_paths
ignore_nopath=ignore_nopath,
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 249, in extract_paths
self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath)
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 99, in wrapper
return func(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 269, in _extract_paths_scp
propagate_fail=not ignore_nopath
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 445, in copy_from
remote_path, local_path, err.args[0]
--------------------- >> end captured logging << ---------------------
Stack Trace:
File "/usr/lib/python2.7/site-packages/nose/case.py", line 133, in run
self.runTest(result)
File "/usr/lib/python2.7/site-packages/nose/case.py", line 151, in runTest
test(result)
File "/usr/lib64/python2.7/unittest/case.py", line 433, in __call__
return self.run(*args, **kwds)
File "/usr/lib64/python2.7/unittest/case.py", line 393, in run
result.addError(self, sys.exc_info())
File "/usr/lib/python2.7/site-packages/nose/proxy.py", line 131, in addError
plugins.addError(self.test, err)
File "/usr/lib/python2.7/site-packages/nose/plugins/manager.py", line 99, in __call__
return self.call(*arg, **kw)
File "/usr/lib/python2.7/site-packages/nose/plugins/manager.py", line 167, in simple
result = meth(*arg, **kw)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 163, in addError
self._addFault(test, err)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 174, in _addFault
self._prefix.paths.test_logs(test_name), False
File "/usr/lib/python2.7/site-packages/lago/sdk_utils.py", line 50, in wrapped
return func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636, in wrapper
return func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1530, in collect_artifacts
self.virt_env.get_vms().values(),
File "/usr/lib/python2.7/site-packages/lago/utils.py", line 103, in invoke_in_parallel
return vt.join_all()
File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue
queue.put({'return': func()})
File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in _collect_artifacts
vm.collect_artifacts(path, ignore_nopath)
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 659, in collect_artifacts
ignore_nopath=ignore_nopath
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 379, in extract_paths
return self.provider.extract_paths(paths, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 398, in extract_paths
ignore_nopath=ignore_nopath,
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 249, in extract_paths
self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath)
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 99, in wrapper
return func(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 269, in _extract_paths_scp
propagate_fail=not ignore_nopath
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 445, in copy_from
remote_path, local_path, err.args[0]
'Failed to copy files/directory from /var/log to /dev/shm/ost/deployment-compat-4.3-suite-master/default/test_logs/002_bootstrap.add_master_storage_domain_43-20190323005122/lago-compat-4-3-suite-master-host-1/_var_log\nUnable to get valid context for root\n-------------------- >> begin captured logging << --------------------\nlago.ssh: DEBUG: start task:eac93cd6-f7d5-4709-b8d4-6abf4ede9ba8:Get ssh client for lago-compat-4-3-suite-master-engine:\nlago.ssh: DEBUG: end task:eac93cd6-f7d5-4709-b8d4-6abf4ede9ba8:Get ssh client for lago-compat-4-3-suite-master-engine:\nlago.ssh: DEBUG: Running c6f4be5c on lago-compat-4-3-suite-master-engine: cat /root/multipath.txt\nlago.ssh: DEBUG: Command c6f4be5c on lago-compat-4-3-suite-master-engine returned with 0\nlago.ssh: DEBUG: Command c6f4be5c on lago-compat-4-3-suite-master-engine output:\n 360014052ab13391ff944443894c2b0f7\n360014052caa297d60e9440592a1f3d04\n36001405bc22d1395bd6476299a975c10\n36001405edd5271bd714406584bfb9d26\n36001405f0546ac01e2c444ab58bb99d3\n\nlago.prefix: INFO: * Collect artifacts: \x1b[0m\x1b[0m\nlago.prefix: INFO: \x1b[0m\x1b[0m\nlago.prefix: INFO: \x1b[0m\x1b[0m\nlago.prefix: INFO: \x1b[0m\x1b[0m\nlago.plugins.vm: DEBUG: Extracting scp://lago-compat-4-3-suite-master-host-0:/var/log to /dev/shm/ost/deployment-compat-4.3-suite-master/default/test_logs/002_bootstrap.add_master_storage_domain_43-20190323005122/lago-compat-4-3-suite-master-host-0/_var_log\nlago.plugins.vm: INFO: \x1b[0m\x1b[0m\nlago.plugins.vm: DEBUG: Extracting scp://lago-compat-4-3-suite-master-host-1:/var/log to /dev/shm/ost/deployment-compat-4.3-suite-master/default/test_logs/002_bootstrap.add_master_storage_domain_43-20190323005122/lago-compat-4-3-suite-master-host-1/_var_log\nlago.ssh: DEBUG: start task:8f203123-42fa-4a8f-9c1b-7d4dfbd394aa:Get ssh client for lago-compat-4-3-suite-master-host-0:\nlago.plugins.vm: DEBUG: Extracting scp://lago-compat-4-3-suite-master-engine:/tmp/otopi* to /dev/shm/ost/deployment-compat-4.3-suite-master/default/test_logs/002_bootstrap.add_master_storage_domain_43-20190323005122/lago-compat-4-3-suite-master-engine/_tmp_otopi*\nlago.plugins.vm: INFO: \x1b[0m\x1b[0m\nlago.plugins.vm: INFO: \x1b[0m\x1b[0m\nlago.ssh: DEBUG: start task:9f190a0b-bcd0-46c2-93ea-713e1a42be0e:Get ssh client for lago-compat-4-3-suite-master-host-1:\nlago.ssh: DEBUG: start task:6a194b08-2d10-4d75-9af6-f813f6113ea6:Get ssh client for lago-compat-4-3-suite-master-engine:\nlago.ssh: DEBUG: end task:6a194b08-2d10-4d75-9af6-f813f6113ea6:Get ssh client for lago-compat-4-3-suite-master-engine:\nlago.ssh: DEBUG: end task:8f203123-42fa-4a8f-9c1b-7d4dfbd394aa:Get ssh client for lago-compat-4-3-suite-master-host-0:\nlago.ssh: DEBUG: end task:9f190a0b-bcd0-46c2-93ea-713e1a42be0e:Get ssh client for lago-compat-4-3-suite-master-host-1:\nroot: ERROR: ~ [Thread-561] Copy from lago-compat-4-3-suite-master-host-1:/var/log to /dev/shm/ost/deployment-compat-4.3-suite-master/default/test_logs/002_bootstrap.add_master_storage_domain_43-20190323005122/lago-compat-4-3-suite-master-host-1/_var_log: \x1b[31mERROR\x1b[0m (in 0:00:00)\nlago.plugins.vm: DEBUG: File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 445, in copy_from\n remote_path, local_path, err.args[0]\n\nroot: ERROR: - [Thread-561] lago-compat-4-3-suite-master-host-1: \x1b[31mERROR\x1b[0m (in 0:00:00)\nlago.prefix: DEBUG: File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in _collect_artifacts\n vm.collect_artifacts(path, ignore_nopath)\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 659, in collect_artifacts\n ignore_nopath=ignore_nopath\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 379, in extract_paths\n return self.provider.extract_paths(paths, *args, **kwargs)\n File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 398, in extract_paths\n ignore_nopath=ignore_nopath,\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 249, in extract_paths\n self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath)\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 99, in wrapper\n return func(self, *args, **kwargs)\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 269, in _extract_paths_scp\n propagate_fail=not ignore_nopath\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 445, in copy_from\n remote_path, local_path, err.args[0]\n\nlago.utils: DEBUG: Error while running thread Thread-561\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue\n queue.put({\'return\': func()})\n File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in _collect_artifacts\n vm.collect_artifacts(path, ignore_nopath)\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 659, in collect_artifacts\n ignore_nopath=ignore_nopath\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 379, in extract_paths\n return self.provider.extract_paths(paths, *args, **kwargs)\n File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 398, in extract_paths\n ignore_nopath=ignore_nopath,\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 249, in extract_paths\n self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath)\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 99, in wrapper\n return func(self, *args, **kwargs)\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 269, in _extract_paths_scp\n propagate_fail=not ignore_nopath\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 445, in copy_from\n remote_path, local_path, err.args[0]\nLagoCopyFilesFromVMError: Failed to copy files/directory from /var/log to /dev/shm/ost/deployment-compat-4.3-suite-master/default/test_logs/002_bootstrap.add_master_storage_domain_43-20190323005122/lago-compat-4-3-suite-master-host-1/_var_log\nUnable to get valid context for root\nroot: ERROR: ~ [Thread-562] Copy from lago-compat-4-3-suite-master-engine:/tmp/otopi* to /dev/shm/ost/deployment-compat-4.3-suite-master/default/test_logs/002_bootstrap.add_master_storage_domain_43-20190323005122/lago-compat-4-3-suite-master-engine/_tmp_otopi*: \x1b[31mERROR\x1b[0m (in 0:00:00)\nlago.plugins.vm: DEBUG: File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 442, in copy_from\n raise ExtractPathNoPathError(err.args[0])\n\nroot: ERROR: - [Thread-562] lago-compat-4-3-suite-master-engine: \x1b[31mERROR\x1b[0m (in 0:00:00)\nlago.prefix: DEBUG: File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in _collect_artifacts\n vm.collect_artifacts(path, ignore_nopath)\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 659, in collect_artifacts\n ignore_nopath=ignore_nopath\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 379, in extract_paths\n return self.provider.extract_paths(paths, *args, **kwargs)\n File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 398, in extract_paths\n ignore_nopath=ignore_nopath,\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 249, in extract_paths\n self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath)\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 99, in wrapper\n return func(self, *args, **kwargs)\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 269, in _extract_paths_scp\n propagate_fail=not ignore_nopath\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 442, in copy_from\n raise ExtractPathNoPathError(err.args[0])\n\nlago.utils: DEBUG: Error while running thread Thread-562\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue\n queue.put({\'return\': func()})\n File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in _collect_artifacts\n vm.collect_artifacts(path, ignore_nopath)\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 659, in collect_artifacts\n ignore_nopath=ignore_nopath\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 379, in extract_paths\n return self.provider.extract_paths(paths, *args, **kwargs)\n File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 398, in extract_paths\n ignore_nopath=ignore_nopath,\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 249, in extract_paths\n self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath)\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 99, in wrapper\n return func(self, *args, **kwargs)\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 269, in _extract_paths_scp\n propagate_fail=not ignore_nopath\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 442, in copy_from\n raise ExtractPathNoPathError(err.args[0])\nExtractPathNoPathError: Failed to extract files: \x01scp: /tmp/otopi*: No such file or directory\nlago.plugins.vm: INFO: \x1b[32mSuccess\x1b[0m (in 0:00:01)\nlago.plugins.vm: DEBUG: Extracting scp://lago-compat-4-3-suite-master-host-0:/etc/resolv.conf to /dev/shm/ost/deployment-compat-4.3-suite-master/default/test_logs/002_bootstrap.add_master_storage_domain_43-20190323005122/lago-compat-4-3-suite-master-host-0/_etc_resolv.conf\nlago.plugins.vm: INFO: \x1b[0m\x1b[0m\nlago.ssh: DEBUG: start task:4f64c62d-26aa-4e98-bd49-e0b46d4a10c3:Get ssh client for lago-compat-4-3-suite-master-host-0:\nlago.ssh: DEBUG: end task:4f64c62d-26aa-4e98-bd49-e0b46d4a10c3:Get ssh client for lago-compat-4-3-suite-master-host-0:\nlago.plugins.vm: INFO: \x1b[32mSuccess\x1b[0m (in 0:00:00)\nlago.prefix: INFO: \x1b[31mERROR\x1b[0m (in 0:00:01)\nroot: ERROR: * Collect artifacts: \x1b[31mERROR\x1b[0m (in 0:00:01)\nlago.prefix: DEBUG: File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 636, in wrapper\n return func(*args, **kwargs)\n File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1530, in collect_artifacts\n self.virt_env.get_vms().values(),\n File "/usr/lib/python2.7/site-packages/lago/utils.py", line 103, in invoke_in_parallel\n return vt.join_all()\n File "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in _ret_via_queue\n queue.put({\'return\': func()})\n File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in _collect_artifacts\n vm.collect_artifacts(path, ignore_nopath)\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 659, in collect_artifacts\n ignore_nopath=ignore_nopath\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 379, in extract_paths\n return self.provider.extract_paths(paths, *args, **kwargs)\n File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 398, in extract_paths\n ignore_nopath=ignore_nopath,\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 249, in extract_paths\n self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath)\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 99, in wrapper\n return func(self, *args, **kwargs)\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 269, in _extract_paths_scp\n propagate_fail=not ignore_nopath\n File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 445, in copy_from\n remote_path, local_path, err.args[0]\n\n--------------------- >> end captured logging << ---------------------'
5 years, 1 month
[JIRA] (OVIRT-2706) how do we cache docker images in OST?
by Greg Sheremeta (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-2706?page=com.atlassian.jir... ]
Greg Sheremeta commented on OVIRT-2706:
---------------------------------------
HUB_CONTAINER_IMAGE = 'selenium/hub:3.9.1-actinium'
FIREFOX_CONTAINER_IMAGE = 'selenium/node-firefox-debug:3.9.1-actinium'
CHROME_CONTAINER_IMAGE = 'selenium/node-chrome-debug:3.9.1-actinium'
> how do we cache docker images in OST?
> -------------------------------------
>
> Key: OVIRT-2706
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2706
> Project: oVirt - virtualization made easy
> Issue Type: Improvement
> Components: docker_cleanup.py
> Reporter: Greg Sheremeta
> Assignee: infra
>
> Currently we download the selenium grid containers on every run of 008
> basic ui sanity, and the containers are huge. Can we cache them somewhere
> so OST doesn't need to download them every time? Please advise how and what
> we need to do to enable this. Thanks!
> Best wishes,
> Greg
> --
> GREG SHEREMETA
> SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX
> Red Hat NA
> <https://www.redhat.com/>
> gshereme(a)redhat.com IRC: gshereme
> <https://red.ht/sig>
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100099)
5 years, 1 month
