[ovirt-devel] Re: OST fails for collecting artifacts

On Mon, Nov 18, 2019 at 4:38 PM Marcin Sobczyk <msobczyk(a)redhat.com&gt; wrote: > > Hi, > > I thought it's been removed already, but it seems it's not. > I'm the author of the currently used and newer implementation of artifact collection in lago, > but from my experiments, I've learned that the extraction of wildcard paths never worked. > Here's an email I wrote to Galit about it some time ago: > > ================================= > > Hi Galit, > > aaah yes - wildcard collection doesn't work - it never worked, even before my changes. > > TL; DR - we just need to remove wildcard stuff from > "LagoInitFile.in" files ("/tmp/otopi*", "/tmp/ovirt*"). I'd like to clarify why these were added. When host-deploy runs, during "Add host", it writes its log file locally. It defaults to /tmp/otopi*.log, and we do not change that, so it uses the default. If/when it finishes successfully, the engine copies this log to /var/log/ovirt-engine/host-deploy on the engine machine, and host-deploy removes its log from /tmp. If it fails in the middle, it will (likely) leave its log in /tmp. We had failures in the past that were hard to diagnose, and therefore added /tmp/otopi*, to help debug future similar cases. We can instead configure otopi/host-deploy on the host to keep its logs by default elsewhere. E.g. by doing this on the _host_ before adding it: mkdir -p /var/log/otopi mkdir -p /etc/otopi.env.d echo 'export OTOPI_LOGDIR=/var/log/otopi' > /etc/otopi.env.d/99-logdir.env and then collect /var/log/otopi. If this solves the problem, fine with me to do this and remove /tmp/otopi*. That said, I think collecting wildcards can be useful, so perhaps we should fix this anyway. Also, /tmp/otopi* should normally do not exist, so it should almost always fail to collect that. Is this really a problem? Or just annoying to have to skip over when reading lago/ost logs?

> If you're curious what really happens... :) > > The old algorithm uses "SCPClient" from "scp" library to copy files. > "scp.get" function accepts, among others, two arguments - "remote_path" and "local_path". > What we do, is we change slashes in "remote_path" to underscores and pass it as "local_path": > > https://github.com/lago-project/lago/blob/7024fb7cabbd4ebc87f3ad12e35e0f5... > > The effect is, we command "scp.get" to retrieve "/tmp/otopi*" and save it as "_tmp_otopi*"... which of course makes no sense at all and doesn't work... > > > The new implementation *could* work with wildcards because the collection is divided into two stages: > > https://github.com/lago-project/lago/blob/9803eeacd41b3f91cd6661a110aa028... > > First we do the "tar -> copy tar with ssh -> untar to tmpdir" thing and *only then* we use "shutil.move" to rename the files to the underscored version. > We could use "glob" module to try to iterate over stuff like "/tmp/otopi*" and rename the files appropriately. > However, we maintain two parallel implementations of artifacts collection - the old one being a "plan B" in case there's no "tar" or "gzip" on the target machine. > This is the reason we have to keep both implementations identical in behavior to avoid confusion. BTW the new implementation could drop the underscore-renaming process completely - I think the only reason we do the renaming in the old algorithm is because "scp" won't create intermediary directories for you... untarring stuff handles that case well, but that's a backwards-compatibility-breaking change :) > > ================================= > > I will post a patch that removes this. > > Regards, Marcin > > On 11/18/19 2:45 PM, Amit Bawer wrote: > > Happens for several runs, full log can be seen at > http://jenkins.ovirt.org/job/ovirt-system-tests_manual/6057/artifact/expo... > > 2019-11-18 12:28:12,710::log_utils.py::end_log_task::670::root::ERROR:: - [Thread-42] lago-basic-suite-master-engine: [31mERROR [0m (in 0:00:08) > 2019-11-18 12:28:12,731::log_utils.py::__exit__::607::lago.prefix::DEBUG:: File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in _collect_artifacts > vm.collect_artifacts(path, ignore_nopath) > File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 748, in collect_artifacts > ignore_nopath=ignore_nopath > File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 468, in extract_paths > return self.provider.extract_paths(paths, *args, **kwargs) > File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 398, in extract_paths > ignore_nopath=ignore_nopath, > File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 253, in extract_paths > self._extract_paths_tar_gz(paths, ignore_nopath) > File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 102, in wrapper > return func(self, *args, **kwargs) > File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 341, in _extract_paths_tar_gz > raise ExtractPathNoPathError(remote_path) > > 2019-11-18 12:28:12,731::utils.py::_ret_via_queue::63::lago.utils::DEBUG::Error while running thread Thread-42 > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue > queue.put({'return': func()}) > File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in _collect_artifacts > vm.collect_artifacts(path, ignore_nopath) > File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 748, in collect_artifacts > ignore_nopath=ignore_nopath > File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 468, in extract_paths > return self.provider.extract_paths(paths, *args, **kwargs) > File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 398, in extract_paths > ignore_nopath=ignore_nopath, > File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 253, in extract_paths > self._extract_paths_tar_gz(paths, ignore_nopath) > File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 102, in wrapper > return func(self, *args, **kwargs) > File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 341, in _extract_paths_tar_gz > raise ExtractPathNoPathError(remote_path) > ExtractPathNoPathError: Failed to extract files: /tmp/otopi* > > > _______________________________________________ > Devel mailing list -- devel(a)ovirt.org > To unsubscribe send an email to devel-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/PCS27XT3ODF... > > > _______________________________________________ > Devel mailing list -- devel(a)ovirt.org > To unsubscribe send an email to devel-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/5BSU44MUSBK...