<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Wed, Jul 5, 2017 at 9:46 AM, Nadav Goldin <span dir="ltr"><<a href="mailto:ngoldin@redhat.com" target="_blank">ngoldin@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
Did it happen more than once?<br></blockquote><div><br></div><div>No </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Looking at the logs what happened was:<br>
1. Lago checked the engine was SSH reachable - this was true.<br>
2. Then it tried connecting via SSH and collect the logs and timed out.<br>
<br>
On (1) we have retries and guards, on (2) we don't, as we assume (1)<br>
just passed. I guess in some conditions that logic can be flawed. Can<br>
you open an issue[1]?<br></blockquote><div><br></div><div>Will do </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I'll try to fix it as soon as possible.<br>
<br>
Thanks,<br>
<br>
<br>
[1] <a href="https://github.com/lago-project/lago/issues" rel="noreferrer" target="_blank">https://github.com/lago-<wbr>project/lago/issues</a><br>
<span class="HOEnZb"><font color="#888888">Nadav.<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
On Wed, Jul 5, 2017 at 9:07 AM, Gil Shinar <<a href="mailto:gshinar@redhat.com">gshinar@redhat.com</a>> wrote:<br>
> Hi Nadav/Gal,<br>
><br>
> I see the folowing exceptions in lago log:<br>
><br>
> 2017-07-04 14:24:39,254::log_utils.py::__<wbr>exit__::606::lago.prefix::<wbr>DEBUG::<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/prefix.py", line 1476, in<br>
> _collect_artifacts<br>
> vm.collect_artifacts(path, ignore_nopath)<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/plugins/vm.py", line 624, in<br>
> collect_artifacts<br>
> ignore_nopath=ignore_nopath<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/plugins/vm.py", line 381, in<br>
> extract_paths<br>
> return self.provider.extract_paths(<wbr>paths, *args, **kwargs)<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/providers/<wbr>libvirt/vm.py", line<br>
> 297, in extract_paths<br>
> ignore_nopath=ignore_nopath,<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/plugins/vm.py", line 247, in<br>
> extract_paths<br>
> self._extract_paths_scp(paths=<wbr>paths, ignore_nopath=ignore_nopath)<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/plugins/vm.py", line 266, in<br>
> _extract_paths_scp<br>
> propagate_fail=False<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/plugins/vm.py", line 425, in<br>
> copy_from<br>
> local_path=local_path,<br>
> File "/usr/lib/python2.7/site-<wbr>packages/scp.py", line 125, in get<br>
> self._recv_all()<br>
> File "/usr/lib/python2.7/site-<wbr>packages/scp.py", line 250, in _recv_all<br>
> msg = self.channel.recv(1024)<br>
> File "/usr/lib/python2.7/site-<wbr>packages/paramiko/channel.py", line 615, in<br>
> recv<br>
> raise socket.timeout()<br>
><br>
> 2017-07-04<br>
> 14:24:39,254::utils.py::_ret_<wbr>via_queue::60::lago.utils::<wbr>ERROR::Error while<br>
> running thread<br>
> Traceback (most recent call last):<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/utils.py", line 58, in<br>
> _ret_via_queue<br>
> queue.put({'return': func()})<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/prefix.py", line 1476, in<br>
> _collect_artifacts<br>
> vm.collect_artifacts(path, ignore_nopath)<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/plugins/vm.py", line 624, in<br>
> collect_artifacts<br>
> ignore_nopath=ignore_nopath<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/plugins/vm.py", line 381, in<br>
> extract_paths<br>
> return self.provider.extract_paths(<wbr>paths, *args, **kwargs)<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/providers/<wbr>libvirt/vm.py", line<br>
> 297, in extract_paths<br>
> ignore_nopath=ignore_nopath,<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/plugins/vm.py", line 247, in<br>
> extract_paths<br>
> self._extract_paths_scp(paths=<wbr>paths, ignore_nopath=ignore_nopath)<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/plugins/vm.py", line 266, in<br>
> _extract_paths_scp<br>
> propagate_fail=False<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/plugins/vm.py", line 425, in<br>
> copy_from<br>
> local_path=local_path,<br>
> File "/usr/lib/python2.7/site-<wbr>packages/scp.py", line 125, in get<br>
> self._recv_all()<br>
> File "/usr/lib/python2.7/site-<wbr>packages/scp.py", line 250, in _recv_all<br>
> msg = self.channel.recv(1024)<br>
> File "/usr/lib/python2.7/site-<wbr>packages/paramiko/channel.py", line 615, in<br>
> recv<br>
> raise socket.timeout()<br>
> timeout<br>
> 2017-07-04 14:24:39,255::log_utils.py::<wbr>end_log_task::669::root::<wbr>ERROR::@<br>
> Collect artifacts: [31mERROR [0m (in 0:00:05)<br>
> 2017-07-04 14:24:39,256::log_utils.py::__<wbr>exit__::606::lago.prefix::<wbr>DEBUG::<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/log_utils.py", line 635, in<br>
> wrapper<br>
> return func(*args, **kwargs)<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/prefix.py", line 1480, in<br>
> collect_artifacts<br>
> self.virt_env.get_vms().<wbr>values(),<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/utils.py", line 100, in<br>
> invoke_in_parallel<br>
> return vt.join_all()<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/utils.py", line 58, in<br>
> _ret_via_queue<br>
> queue.put({'return': func()})<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/prefix.py", line 1476, in<br>
> _collect_artifacts<br>
> vm.collect_artifacts(path, ignore_nopath)<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/plugins/vm.py", line 624, in<br>
> collect_artifacts<br>
> ignore_nopath=ignore_nopath<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/plugins/vm.py", line 381, in<br>
> extract_paths<br>
> return self.provider.extract_paths(<wbr>paths, *args, **kwargs)<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/providers/<wbr>libvirt/vm.py", line<br>
> 297, in extract_paths<br>
> ignore_nopath=ignore_nopath,<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/plugins/vm.py", line 247, in<br>
> extract_paths<br>
> self._extract_paths_scp(paths=<wbr>paths, ignore_nopath=ignore_nopath)<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/plugins/vm.py", line 266, in<br>
> _extract_paths_scp<br>
> propagate_fail=False<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/plugins/vm.py", line 425, in<br>
> copy_from<br>
> local_path=local_path,<br>
> File "/usr/lib/python2.7/site-<wbr>packages/scp.py", line 125, in get<br>
> self._recv_all()<br>
> File "/usr/lib/python2.7/site-<wbr>packages/scp.py", line 250, in _recv_all<br>
> msg = self.channel.recv(1024)<br>
> File "/usr/lib/python2.7/site-<wbr>packages/paramiko/channel.py", line 615, in<br>
> recv<br>
> raise socket.timeout()<br>
><br>
> 2017-07-04 14:24:39,256::cmd.py::main::<wbr>960::cli::ERROR::Error occured,<br>
> aborting<br>
> Traceback (most recent call last):<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/cmd.py", line 954, in main<br>
> cli_plugins[args.verb].do_run(<wbr>args)<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/plugins/cli.py", line 184, in<br>
> do_run<br>
> self._do_run(**vars(args))<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/utils.py", line 501, in<br>
> wrapper<br>
> return func(*args, **kwargs)<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/utils.py", line 512, in<br>
> wrapper<br>
> return func(*args, prefix=prefix, **kwargs)<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/cmd.py", line 751, in<br>
> do_collect<br>
> prefix.collect_artifacts(<wbr>output, ignore_nopath=not no_skip)<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/sdk_utils.py", line 50, in<br>
> wrapped<br>
> return func(*args, **kwargs)<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/log_utils.py", line 635, in<br>
> wrapper<br>
> return func(*args, **kwargs)<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/prefix.py", line 1480, in<br>
> collect_artifacts<br>
> self.virt_env.get_vms().<wbr>values(),<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/utils.py", line 100, in<br>
> invoke_in_parallel<br>
> return vt.join_all()<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/utils.py", line 58, in<br>
> _ret_via_queue<br>
> queue.put({'return': func()})<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/prefix.py", line 1476, in<br>
> _collect_artifacts<br>
> vm.collect_artifacts(path, ignore_nopath)<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/plugins/vm.py", line 624, in<br>
> collect_artifacts<br>
> ignore_nopath=ignore_nopath<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/plugins/vm.py", line 381, in<br>
> extract_paths<br>
> return self.provider.extract_paths(<wbr>paths, *args, **kwargs)<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/providers/<wbr>libvirt/vm.py", line<br>
> 297, in extract_paths<br>
> ignore_nopath=ignore_nopath,<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/plugins/vm.py", line 247, in<br>
> extract_paths<br>
> self._extract_paths_scp(paths=<wbr>paths, ignore_nopath=ignore_nopath)<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/plugins/vm.py", line 266, in<br>
> _extract_paths_scp<br>
> propagate_fail=False<br>
> File "/usr/lib/python2.7/site-<wbr>packages/lago/plugins/vm.py", line 425, in<br>
> copy_from<br>
> local_path=local_path,<br>
> File "/usr/lib/python2.7/site-<wbr>packages/scp.py", line 125, in get<br>
> self._recv_all()<br>
> File "/usr/lib/python2.7/site-<wbr>packages/scp.py", line 250, in _recv_all<br>
> msg = self.channel.recv(1024)<br>
> File "/usr/lib/python2.7/site-<wbr>packages/paramiko/channel.py", line 615, in<br>
> recv<br>
> raise socket.timeout()<br>
> timeout<br>
><br>
><br>
> Link to build:<br>
> <a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/7459" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>test-repo_ovirt_experimental_<wbr>master/7459</a><br>
><br>
> Thanks<br>
> Gil<br>
</div></div></blockquote></div><br></div></div>