[ovirt-devel] Re: OST fails for collecting artifacts

Yedidyah Bar David <didi(a)redhat.com&gt; writes: > On Mon, Nov 18, 2019 at 4:38 PM Marcin Sobczyk <msobczyk(a)redhat.com&gt; wrote: >> >> Hi, > >> >> I thought it's been removed already, but it seems it's not. >> I'm the author of the currently used and newer implementation of artifact collection in lago, >> but from my experiments, I've learned that the extraction of wildcard paths never worked. >> Here's an email I wrote to Galit about it some time ago: >> >> ================================= >> >> Hi Galit, >> >> aaah yes - wildcard collection doesn't work - it never worked, even before my changes. >> >> TL; DR - we just need to remove wildcard stuff from >> "LagoInitFile.in" files ("/tmp/otopi*", "/tmp/ovirt*"). > > I'd like to clarify why these were added. > > When host-deploy runs, during "Add host", it writes its log file > locally. It defaults to /tmp/otopi*.log, and we do not change that, so > it uses the default. > > If/when it finishes successfully, the engine copies this log to > /var/log/ovirt-engine/host-deploy on the engine machine, and > host-deploy removes its log from /tmp. > > If it fails in the middle, it will (likely) leave its log in /tmp. > > We had failures in the past that were hard to diagnose, and therefore > added /tmp/otopi*, to help debug future similar cases. > > We can instead configure otopi/host-deploy on the host to keep its > logs by default elsewhere. E.g. by doing this on the _host_ before > adding it: > > mkdir -p /var/log/otopi > mkdir -p /etc/otopi.env.d > echo 'export OTOPI_LOGDIR=/var/log/otopi' > /etc/otopi.env.d/99-logdir.env > > and then collect /var/log/otopi. > > If this solves the problem, fine with me to do this and remove /tmp/otopi*. > > That said, I think collecting wildcards can be useful, so perhaps we > should fix this anyway. > > Also, /tmp/otopi* should normally do not exist, so it should almost > always fail to collect that. Is this really a problem? Or just > annoying to have to skip over when reading lago/ost logs? It is at least confusing and annoying to see these kind of errors in OST logs. If it is expected to fail in some cases, it shouldn't be reported as error and should be silently ignored when the file doesn't exist.

Thanks, Milan >> If you're curious what really happens... :) >> >> The old algorithm uses "SCPClient" from "scp" library to copy files. >> "scp.get" function accepts, among others, two arguments - "remote_path" and "local_path". >> What we do, is we change slashes in "remote_path" to underscores and pass it as "local_path": >> >> https://github.com/lago-project/lago/blob/7024fb7cabbd4ebc87f3ad12e35e0f5... >> >> The effect is, we command "scp.get" to retrieve "/tmp/otopi*" and save it as "_tmp_otopi*"... which of course makes no sense at all and doesn't work... >> >> >> The new implementation *could* work with wildcards because the collection is divided into two stages: >> >> https://github.com/lago-project/lago/blob/9803eeacd41b3f91cd6661a110aa028... >> >> First we do the "tar -> copy tar with ssh -> untar to tmpdir" thing and *only then* we use "shutil.move" to rename the files to the underscored version. >> We could use "glob" module to try to iterate over stuff like "/tmp/otopi*" and rename the files appropriately. >> However, we maintain two parallel implementations of artifacts collection - the old one being a "plan B" in case there's no "tar" or "gzip" on the target machine. >> This is the reason we have to keep both implementations identical in behavior to avoid confusion. BTW the new implementation could drop the underscore-renaming process completely - I think the only reason we do the renaming in the old algorithm is because "scp" won't create intermediary directories for you... untarring stuff handles that case well, but that's a backwards-compatibility-breaking change :) >> >> ================================= >> >> I will post a patch that removes this. >> >> Regards, Marcin >> >> On 11/18/19 2:45 PM, Amit Bawer wrote: >> >> Happens for several runs, full log can be seen at >> http://jenkins.ovirt.org/job/ovirt-system-tests_manual/6057/artifact/expo... >> >> 2019-11-18 12:28:12,710::log_utils.py::end_log_task::670::root::ERROR:: - [Thread-42] lago-basic-suite-master-engine: [31mERROR [0m (in 0:00:08) >> 2019-11-18 12:28:12,731::log_utils.py::__exit__::607::lago.prefix::DEBUG:: File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in _collect_artifacts >> vm.collect_artifacts(path, ignore_nopath) >> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 748, in collect_artifacts >> ignore_nopath=ignore_nopath >> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 468, in extract_paths >> return self.provider.extract_paths(paths, *args, **kwargs) >> File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 398, in extract_paths >> ignore_nopath=ignore_nopath, >> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 253, in extract_paths >> self._extract_paths_tar_gz(paths, ignore_nopath) >> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 102, in wrapper >> return func(self, *args, **kwargs) >> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 341, in _extract_paths_tar_gz >> raise ExtractPathNoPathError(remote_path) >> >> 2019-11-18 12:28:12,731::utils.py::_ret_via_queue::63::lago.utils::DEBUG::Error while running thread Thread-42 >> Traceback (most recent call last): >> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue >> queue.put({'return': func()}) >> File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in _collect_artifacts >> vm.collect_artifacts(path, ignore_nopath) >> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 748, in collect_artifacts >> ignore_nopath=ignore_nopath >> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 468, in extract_paths >> return self.provider.extract_paths(paths, *args, **kwargs) >> File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 398, in extract_paths >> ignore_nopath=ignore_nopath, >> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 253, in extract_paths >> self._extract_paths_tar_gz(paths, ignore_nopath) >> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 102, in wrapper >> return func(self, *args, **kwargs) >> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 341, in _extract_paths_tar_gz >> raise ExtractPathNoPathError(remote_path) >> ExtractPathNoPathError: Failed to extract files: /tmp/otopi* >> >> >> _______________________________________________ >> Devel mailing list -- devel(a)ovirt.org >> To unsubscribe send an email to devel-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/PCS27XT3ODF... >> >> >> _______________________________________________ >> Devel mailing list -- devel(a)ovirt.org >> To unsubscribe send an email to devel-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/5BSU44MUSBK...