Hi,

I thought it's been removed already, but it seems it's not.
I'm the author of the currently used and newer implementation of artifact collection in lago,
but from my experiments, I've learned that the extraction of wildcard paths never worked.
Here's an email I wrote to Galit about it some time ago:

=================================

Hi Galit,

aaah yes - wildcard collection doesn't work - it never worked, even before my changes.

TL; DR - we just need to remove wildcard stuff from "LagoInitFile.in" files ("/tmp/otopi*", "/tmp/ovirt*").


If you're curious what really happens... :)

The old algorithm uses "SCPClient" from "scp" library to copy files.
"scp.get" function accepts, among others, two arguments - "remote_path" and "local_path".
What we do, is we change slashes in "remote_path" to underscores and pass it as "local_path":

https://github.com/lago-project/lago/blob/7024fb7cabbd4ebc87f3ad12e35e0f5832af7d56/lago/plugins/vm.py#L651

The effect is, we command "scp.get" to retrieve "/tmp/otopi*" and save it as "_tmp_otopi*"... which of course makes no sense at all and doesn't work...


The new implementation *could* work with wildcards because the collection is divided into two stages:

https://github.com/lago-project/lago/blob/9803eeacd41b3f91cd6661a110aa0285aaf4b957/lago/plugins/vm.py#L313

First we do the "tar -> copy tar with ssh -> untar to tmpdir" thing and *only then* we use "shutil.move" to rename the files to the underscored version.
We could use "glob" module to try to iterate over stuff like "/tmp/otopi*" and rename the files appropriately.
However, we maintain two parallel implementations of artifacts collection - the old one being a "plan B" in case there's no "tar" or "gzip" on the target machine.
This is the reason we have to keep both implementations identical in behavior to avoid confusion. BTW the new implementation could drop the underscore-renaming process completely - I think the only reason we do the renaming in the old algorithm is because "scp" won't create intermediary directories for you... untarring stuff handles that case well, but that's a backwards-compatibility-breaking change :)

=================================

I will post a patch that removes this.

Regards, Marcin

On 11/18/19 2:45 PM, Amit Bawer wrote:
Happens for several runs, full log can be seen at
http://jenkins.ovirt.org/job/ovirt-system-tests_manual/6057/artifact/exported-artifacts/test_logs/basic-suite-master/post-002_bootstrap.py/lago_logs/lago.log
2019-11-18 12:28:12,710::log_utils.py::end_log_task::670::root::ERROR::      - [Thread-42] lago-basic-suite-master-engine:  [31mERROR [0m (in 0:00:08)
2019-11-18 12:28:12,731::log_utils.py::__exit__::607::lago.prefix::DEBUG::  File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in _collect_artifacts
    vm.collect_artifacts(path, ignore_nopath)
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 748, in collect_artifacts
    ignore_nopath=ignore_nopath
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 468, in extract_paths
    return self.provider.extract_paths(paths, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 398, in extract_paths
    ignore_nopath=ignore_nopath,
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 253, in extract_paths
    self._extract_paths_tar_gz(paths, ignore_nopath)
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 102, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 341, in _extract_paths_tar_gz
    raise ExtractPathNoPathError(remote_path)

2019-11-18 12:28:12,731::utils.py::_ret_via_queue::63::lago.utils::DEBUG::Error while running thread Thread-42
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue
    queue.put({'return': func()})
  File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in _collect_artifacts
    vm.collect_artifacts(path, ignore_nopath)
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 748, in collect_artifacts
    ignore_nopath=ignore_nopath
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 468, in extract_paths
    return self.provider.extract_paths(paths, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 398, in extract_paths
    ignore_nopath=ignore_nopath,
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 253, in extract_paths
    self._extract_paths_tar_gz(paths, ignore_nopath)
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 102, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 341, in _extract_paths_tar_gz
    raise ExtractPathNoPathError(remote_path)
ExtractPathNoPathError: Failed to extract files: /tmp/otopi*

_______________________________________________
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/PCS27XT3ODFL5ZRUPQMB5BFOW6BQXDGS/