Forwarding to infra, as the email to infra-support didn't work. ---------- Forwarded message --------- From: Yedidyah Bar David <didi(a)redhat.com&gt; Date: Wed, Feb 19, 2020 at 10:37 AM Subject: ssh connection fails after restarting engine VM To: <infra-support(a)ovirt.org&gt; Hi all, Apparently my jira account was stuck in some state and all my emails to infra-support from last years were dropped to /dev/null. Now saw another case of a ticket I tried to open a month ago, and considered updating it, so spent the time to actually login and see that I can't find it anywhere. So copy/pasting below my original email. The new case is: https://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.3/341/ lago.log has: 2020-02-19 05:10:50,890::log_utils.py::__exit__::611::lago.prefix::INFO:: # [Thread-2] lago-he-node-ng-suite-4-3-host-1: [32mSuccess [0m (in 0:00:08) 2020-02-19 05:10:52,116::ssh.py::get_ssh_client::373::lago.ssh::DEBUG::SSH error connecting to lago-he-node-ng-suite-4-3-engine: No existing session 2020-02-19 05:10:52,116::ssh.py::get_ssh_client::381::lago.ssh::DEBUG::Still got 0 tries for lago-he-node-ng-suite-4-3-engine 2020-02-19 05:10:53,117::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:e0b52607-6e65-4583-bf43-b615aa901cc7:Get ssh client for lago-he-node-ng-suite-4-3-engine: 2020-02-19 05:10:53,232::log_utils.py::end_log_task::670::root::ERROR:: # [Thread-3] lago-he-node-ng-suite-4-3-engine: [31mERROR [0m (in 0:00:11) 2020-02-19 05:10:53,245::log_utils.py::__exit__::607::lago.prefix::DEBUG:: File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in _collect_artifacts vm.collect_artifacts(path, ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 748, in collect_artifacts ignore_nopath=ignore_nopath File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 468, in extract_paths return self.provider.extract_paths(paths, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 259, in extract_paths format(self.vm.name()) 2020-02-19 05:10:53,245::utils.py::_ret_via_queue::63::lago.utils::DEBUG::Error while running thread Thread-3 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in _collect_artifacts vm.collect_artifacts(path, ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 748, in collect_artifacts ignore_nopath=ignore_nopath File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 468, in extract_paths return self.provider.extract_paths(paths, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 259, in extract_paths format(self.vm.name()) ExtractPathError: Unable to extract paths from lago-he-node-ng-suite-4-3-engine: unreachable with SSH Original email I wrote (and was never received) follows: Hi all, See e.g. [1]. In lago.log [2]: 2020-01-19 05:38:06,052::log_utils.py::__exit__::611::ovirtlago.prefix::INFO::@ Run test: 008_restart_he_vm.py: [32mSuccess [0m (in 0:23:27) ... 2020-01-19 05:38:07,680::log_utils.py::__enter__::600::lago.prefix::INFO:: # [Thread-3] lago-he-basic-suite-4-3-engine: [0m [0m ... 2020-01-19 05:38:07,686::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:170b4eaa-fbf5-48ca-b81a-4ddf0c9a3bd5:Get ssh client for lago-he-basic-suite-4-3-engine: ... 2020-01-19 05:38:12,415::log_utils.py::__exit__::611::lago.prefix::INFO:: # [Thread-1] lago-he-basic-suite-4-3-host-0: [32mSuccess [0m (in 0:00:04) 2020-01-19 05:38:17,729::ssh.py::get_ssh_client::373::lago.ssh::DEBUG::SSH error connecting to lago-he-basic-suite-4-3-engine: No existing session 2020-01-19 05:38:17,730::ssh.py::get_ssh_client::381::lago.ssh::DEBUG::Still got 0 tries for lago-he-basic-suite-4-3-engine 2020-01-19 05:38:18,731::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:170b4eaa-fbf5-48ca-b81a-4ddf0c9a3bd5:Get ssh client for lago-he-basic-suite-4-3-engine: 2020-01-19 05:38:18,739::log_utils.py::end_log_task::670::root::ERROR:: # [Thread-3] lago-he-basic-suite-4-3-engine: [31mERROR [0m (in 0:00:11) ... ExtractPathError: Unable to extract paths from lago-he-basic-suite-4-3-engine: unreachable with SSH I wonder what might "No existing session" mean. Perhaps paramiko caches the connection, and after the engine VM is restarted it does not try to connect again? Or something similar? Searching the net, [3] looks similar, although I am definitely not sure we do not already do this (or similar) in lago. Anyway, instead of (perhaps) spending time on this, it might be better/possible to explicitly expose some "closeconnection" function in lago, to be used by OST after such an engine vm restart (or similar cases). [1] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/324/ [2] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/324/a... [3] https://stackoverflow.com/questions/57508919/paramiko-ssh-exception-sshex... Thanks, -- Didi -- Didi _______________________________________________ Infra mailing list -- infra(a)ovirt.org To unsubscribe send an email to infra-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/I2FEUC5J2SM...

On Wed, Feb 19, 2020 at 11:38 AM Emil Natan <ena(a)redhat.com&gt; wrote: > > Hi, > > What do you mean by "the email to infra-support didn't work"? Email was rejected? Issue was not created? Something else? Please see my other email "Jira account revoked". Thanks.

> Thanks. > > On Wed, Feb 19, 2020 at 11:21 AM Yedidyah Bar David <didi(a)redhat.com&gt; wrote: >> >> Forwarding to infra, as the email to infra-support didn't work. >> >> ---------- Forwarded message --------- >> From: Yedidyah Bar David <didi(a)redhat.com&gt; >> Date: Wed, Feb 19, 2020 at 10:37 AM >> Subject: ssh connection fails after restarting engine VM >> To: <infra-support(a)ovirt.org&gt; >> >> >> Hi all, >> >> Apparently my jira account was stuck in some state and all my emails >> to infra-support from last years were dropped to /dev/null. Now saw >> another case of a ticket I tried to open a month ago, and considered >> updating it, so spent the time to actually login and see that I can't >> find it anywhere. So copy/pasting below my original email. >> >> The new case is: >> >> https://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.3/341/ >> >> lago.log has: >> >> 2020-02-19 05:10:50,890::log_utils.py::__exit__::611::lago.prefix::INFO:: >> # [Thread-2] lago-he-node-ng-suite-4-3-host-1: [32mSuccess [0m (in >> 0:00:08) >> 2020-02-19 05:10:52,116::ssh.py::get_ssh_client::373::lago.ssh::DEBUG::SSH >> error connecting to lago-he-node-ng-suite-4-3-engine: No existing >> session >> 2020-02-19 05:10:52,116::ssh.py::get_ssh_client::381::lago.ssh::DEBUG::Still >> got 0 tries for lago-he-node-ng-suite-4-3-engine >> 2020-02-19 05:10:53,117::log_utils.py::__exit__::611::lago.ssh::DEBUG::end >> task:e0b52607-6e65-4583-bf43-b615aa901cc7:Get ssh client for >> lago-he-node-ng-suite-4-3-engine: >> 2020-02-19 05:10:53,232::log_utils.py::end_log_task::670::root::ERROR:: >> # [Thread-3] lago-he-node-ng-suite-4-3-engine: [31mERROR [0m (in >> 0:00:11) >> 2020-02-19 05:10:53,245::log_utils.py::__exit__::607::lago.prefix::DEBUG:: >> File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in >> _collect_artifacts >> vm.collect_artifacts(path, ignore_nopath) >> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line >> 748, in collect_artifacts >> ignore_nopath=ignore_nopath >> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line >> 468, in extract_paths >> return self.provider.extract_paths(paths, *args, **kwargs) >> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line >> 259, in extract_paths >> format(self.vm.name()) >> >> 2020-02-19 05:10:53,245::utils.py::_ret_via_queue::63::lago.utils::DEBUG::Error >> while running thread Thread-3 >> Traceback (most recent call last): >> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in >> _ret_via_queue >> queue.put({'return': func()}) >> File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, >> in _collect_artifacts >> vm.collect_artifacts(path, ignore_nopath) >> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line >> 748, in collect_artifacts >> ignore_nopath=ignore_nopath >> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line >> 468, in extract_paths >> return self.provider.extract_paths(paths, *args, **kwargs) >> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line >> 259, in extract_paths >> format(self.vm.name()) >> ExtractPathError: Unable to extract paths from >> lago-he-node-ng-suite-4-3-engine: unreachable with SSH >> >> Original email I wrote (and was never received) follows: >> >> Hi all, >> >> See e.g. [1]. >> In lago.log [2]: >> >> 2020-01-19 05:38:06,052::log_utils.py::__exit__::611::ovirtlago.prefix::INFO::@ >> Run test: 008_restart_he_vm.py: [32mSuccess [0m (in 0:23:27) >> ... >> >> 2020-01-19 05:38:07,680::log_utils.py::__enter__::600::lago.prefix::INFO:: >> # [Thread-3] lago-he-basic-suite-4-3-engine: [0m [0m >> ... >> >> 2020-01-19 05:38:07,686::log_utils.py::__enter__::600::lago.ssh::DEBUG::start >> task:170b4eaa-fbf5-48ca-b81a-4ddf0c9a3bd5:Get ssh client for >> lago-he-basic-suite-4-3-engine: >> ... >> >> 2020-01-19 05:38:12,415::log_utils.py::__exit__::611::lago.prefix::INFO:: >> # [Thread-1] lago-he-basic-suite-4-3-host-0: [32mSuccess [0m (in >> 0:00:04) >> 2020-01-19 05:38:17,729::ssh.py::get_ssh_client::373::lago.ssh::DEBUG::SSH >> error connecting to lago-he-basic-suite-4-3-engine: No existing >> session >> 2020-01-19 05:38:17,730::ssh.py::get_ssh_client::381::lago.ssh::DEBUG::Still >> got 0 tries for lago-he-basic-suite-4-3-engine >> 2020-01-19 05:38:18,731::log_utils.py::__exit__::611::lago.ssh::DEBUG::end >> task:170b4eaa-fbf5-48ca-b81a-4ddf0c9a3bd5:Get ssh client for >> lago-he-basic-suite-4-3-engine: >> 2020-01-19 05:38:18,739::log_utils.py::end_log_task::670::root::ERROR:: >> # [Thread-3] lago-he-basic-suite-4-3-engine: [31mERROR [0m (in >> 0:00:11) >> ... >> ExtractPathError: Unable to extract paths from >> lago-he-basic-suite-4-3-engine: unreachable with SSH >> >> I wonder what might "No existing session" mean. Perhaps paramiko >> caches the connection, and after the engine VM is restarted it does >> not try to connect again? Or something similar? >> >> Searching the net, [3] looks similar, although I am definitely not >> sure we do not already do this (or similar) in lago. >> >> Anyway, instead of (perhaps) spending time on this, it might be >> better/possible to explicitly expose some "closeconnection" function >> in lago, to be used by OST after such an engine vm restart (or similar >> cases). >> >> [1] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/324/ >> [2] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/324/a... >> [3] https://stackoverflow.com/questions/57508919/paramiko-ssh-exception-sshex... >> >> Thanks, >> -- >> Didi >> >> >> -- >> Didi >> _______________________________________________ >> Infra mailing list -- infra(a)ovirt.org >> To unsubscribe send an email to infra-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/I2FEUC5J2SM... > > > > -- > Emil Natan > RHV/CNV DevOps -- Didi _______________________________________________ Infra mailing list -- infra(a)ovirt.org To unsubscribe send an email to infra-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/Q44AQC3U5KP...