[ovirt-devel] Re: OST fails for collecting artifacts

Hi, I thought it's been removed already, but it seems it's not. I'm the author of the currently used and newer implementation of artifact collection in lago, but from my experiments, I've learned that the extraction of wildcard paths never worked. Here's an email I wrote to Galit about it some time ago: ================================= Hi Galit, aaah yes - wildcard collection doesn't work - it never worked, even before my changes. TL; DR - we just need to remove wildcard stuff from "LagoInitFile.in" files ("/tmp/otopi*", "/tmp/ovirt*").

If you're curious what really happens... :) The old algorithm uses "SCPClient" from "scp" library to copy files. "scp.get" function accepts, among others, two arguments - "remote_path" and "local_path". What we do, is we change slashes in "remote_path" to underscores and pass it as "local_path": https://github.com/lago-project/lago/blob/7024fb7cabbd4ebc87f3ad12e35e0f5... The effect is, we command "scp.get" to retrieve "/tmp/otopi*" and save it as "_tmp_otopi*"... which of course makes no sense at all and doesn't work... The new implementation *could* work with wildcards because the collection is divided into two stages: https://github.com/lago-project/lago/blob/9803eeacd41b3f91cd6661a110aa028... First we do the "tar -> copy tar with ssh -> untar to tmpdir" thing and *only then* we use "shutil.move" to rename the files to the underscored version. We could use "glob" module to try to iterate over stuff like "/tmp/otopi*" and rename the files appropriately. However, we maintain two parallel implementations of artifacts collection - the old one being a "plan B" in case there's no "tar" or "gzip" on the target machine. This is the reason we have to keep both implementations identical in behavior to avoid confusion. BTW the new implementation could drop the underscore-renaming process completely - I think the only reason we do the renaming in the old algorithm is because "scp" won't create intermediary directories for you... untarring stuff handles that case well, but that's a backwards-compatibility-breaking change :) ================================= I will post a patch that removes this. Regards, Marcin On 11/18/19 2:45 PM, Amit Bawer wrote: Happens for several runs, full log can be seen at http://jenkins.ovirt.org/job/ovirt-system-tests_manual/6057/artifact/expo... 2019-11-18 12:28:12,710::log_utils.py::end_log_task::670::root::ERROR:: - [Thread-42] lago-basic-suite-master-engine: [31mERROR [0m (in 0:00:08) 2019-11-18 12:28:12,731::log_utils.py::__exit__::607::lago.prefix::DEBUG:: File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in _collect_artifacts vm.collect_artifacts(path, ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 748, in collect_artifacts ignore_nopath=ignore_nopath File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 468, in extract_paths return self.provider.extract_paths(paths, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 398, in extract_paths ignore_nopath=ignore_nopath, File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 253, in extract_paths self._extract_paths_tar_gz(paths, ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 102, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 341, in _extract_paths_tar_gz raise ExtractPathNoPathError(remote_path) 2019-11-18 12:28:12,731::utils.py::_ret_via_queue::63::lago.utils::DEBUG::Error while running thread Thread-42 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in _collect_artifacts vm.collect_artifacts(path, ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 748, in collect_artifacts ignore_nopath=ignore_nopath File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 468, in extract_paths return self.provider.extract_paths(paths, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 398, in extract_paths ignore_nopath=ignore_nopath, File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 253, in extract_paths self._extract_paths_tar_gz(paths, ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 102, in wrapper return func(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 341, in _extract_paths_tar_gz raise ExtractPathNoPathError(remote_path) ExtractPathNoPathError: Failed to extract files: /tmp/otopi* _______________________________________________ Devel mailing list -- devel(a)ovirt.org To unsubscribe send an email to devel-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/PCS27XT3ODF... _______________________________________________ Devel mailing list -- devel(a)ovirt.org To unsubscribe send an email to devel-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/5BSU44MUSBK...