Re: ssh connection fails after restarting engine VM

Hi, What do you mean by "the email to infra-support didn't work"? Email was rejected? Issue was not created? Something else?

Thanks. On Wed, Feb 19, 2020 at 11:21 AM Yedidyah Bar David <didi(a)redhat.com&gt; wrote: > > Forwarding to infra, as the email to infra-support didn't work. > > ---------- Forwarded message --------- > From: Yedidyah Bar David <didi(a)redhat.com&gt; > Date: Wed, Feb 19, 2020 at 10:37 AM > Subject: ssh connection fails after restarting engine VM > To: <infra-support(a)ovirt.org&gt; > > > Hi all, > > Apparently my jira account was stuck in some state and all my emails > to infra-support from last years were dropped to /dev/null. Now saw > another case of a ticket I tried to open a month ago, and considered > updating it, so spent the time to actually login and see that I can't > find it anywhere. So copy/pasting below my original email. > > The new case is: > > https://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.3/341/ > > lago.log has: > > 2020-02-19 05:10:50,890::log_utils.py::__exit__::611::lago.prefix::INFO:: > # [Thread-2] lago-he-node-ng-suite-4-3-host-1: [32mSuccess [0m (in > 0:00:08) > 2020-02-19 05:10:52,116::ssh.py::get_ssh_client::373::lago.ssh::DEBUG::SSH > error connecting to lago-he-node-ng-suite-4-3-engine: No existing > session > 2020-02-19 05:10:52,116::ssh.py::get_ssh_client::381::lago.ssh::DEBUG::Still > got 0 tries for lago-he-node-ng-suite-4-3-engine > 2020-02-19 05:10:53,117::log_utils.py::__exit__::611::lago.ssh::DEBUG::end > task:e0b52607-6e65-4583-bf43-b615aa901cc7:Get ssh client for > lago-he-node-ng-suite-4-3-engine: > 2020-02-19 05:10:53,232::log_utils.py::end_log_task::670::root::ERROR:: > # [Thread-3] lago-he-node-ng-suite-4-3-engine: [31mERROR [0m (in > 0:00:11) > 2020-02-19 05:10:53,245::log_utils.py::__exit__::607::lago.prefix::DEBUG:: > File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in > _collect_artifacts > vm.collect_artifacts(path, ignore_nopath) > File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line > 748, in collect_artifacts > ignore_nopath=ignore_nopath > File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line > 468, in extract_paths > return self.provider.extract_paths(paths, *args, **kwargs) > File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line > 259, in extract_paths > format(self.vm.name()) > > 2020-02-19 05:10:53,245::utils.py::_ret_via_queue::63::lago.utils::DEBUG::Error > while running thread Thread-3 > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in > _ret_via_queue > queue.put({'return': func()}) > File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, > in _collect_artifacts > vm.collect_artifacts(path, ignore_nopath) > File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line > 748, in collect_artifacts > ignore_nopath=ignore_nopath > File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line > 468, in extract_paths > return self.provider.extract_paths(paths, *args, **kwargs) > File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line > 259, in extract_paths > format(self.vm.name()) > ExtractPathError: Unable to extract paths from > lago-he-node-ng-suite-4-3-engine: unreachable with SSH > > Original email I wrote (and was never received) follows: > > Hi all, > > See e.g. [1]. > In lago.log [2]: > > 2020-01-19 05:38:06,052::log_utils.py::__exit__::611::ovirtlago.prefix::INFO::@ > Run test: 008_restart_he_vm.py: [32mSuccess [0m (in 0:23:27) > ... > > 2020-01-19 05:38:07,680::log_utils.py::__enter__::600::lago.prefix::INFO:: > # [Thread-3] lago-he-basic-suite-4-3-engine: [0m [0m > ... > > 2020-01-19 05:38:07,686::log_utils.py::__enter__::600::lago.ssh::DEBUG::start > task:170b4eaa-fbf5-48ca-b81a-4ddf0c9a3bd5:Get ssh client for > lago-he-basic-suite-4-3-engine: > ... > > 2020-01-19 05:38:12,415::log_utils.py::__exit__::611::lago.prefix::INFO:: > # [Thread-1] lago-he-basic-suite-4-3-host-0: [32mSuccess [0m (in > 0:00:04) > 2020-01-19 05:38:17,729::ssh.py::get_ssh_client::373::lago.ssh::DEBUG::SSH > error connecting to lago-he-basic-suite-4-3-engine: No existing > session > 2020-01-19 05:38:17,730::ssh.py::get_ssh_client::381::lago.ssh::DEBUG::Still > got 0 tries for lago-he-basic-suite-4-3-engine > 2020-01-19 05:38:18,731::log_utils.py::__exit__::611::lago.ssh::DEBUG::end > task:170b4eaa-fbf5-48ca-b81a-4ddf0c9a3bd5:Get ssh client for > lago-he-basic-suite-4-3-engine: > 2020-01-19 05:38:18,739::log_utils.py::end_log_task::670::root::ERROR:: > # [Thread-3] lago-he-basic-suite-4-3-engine: [31mERROR [0m (in > 0:00:11) > ... > ExtractPathError: Unable to extract paths from > lago-he-basic-suite-4-3-engine: unreachable with SSH > > I wonder what might "No existing session" mean. Perhaps paramiko > caches the connection, and after the engine VM is restarted it does > not try to connect again? Or something similar? > > Searching the net, [3] looks similar, although I am definitely not > sure we do not already do this (or similar) in lago. > > Anyway, instead of (perhaps) spending time on this, it might be > better/possible to explicitly expose some "closeconnection" function > in lago, to be used by OST after such an engine vm restart (or similar > cases). > > [1] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/324/ > [2] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/324/a... > [3] https://stackoverflow.com/questions/57508919/paramiko-ssh-exception-sshex... > > Thanks, > -- > Didi > > > -- > Didi > _______________________________________________ > Infra mailing list -- infra(a)ovirt.org > To unsubscribe send an email to infra-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/I2FEUC5J2SM... -- Emil Natan RHV/CNV DevOps