[ovirt-users] Non-responsive host, VM's are still running - how to resolve?

Tue Nov 14 17:49:13 UTC 2017

Try restarting vdsmd from the shell, “systemctl restart vdsmd”.

> From: Artem Tambovskiy <artem.tambovskiy at gmail.com>
> Subject: [ovirt-users] Non-responsive host, VM's are still running - how to resolve?
> Date: November 14, 2017 at 11:23:32 AM CST
> To: users
> 
> Apparently, i lost the host which was running hosted-engine and another 4 VM's exactly during migration of second host from bare-metal to second host in the cluster. For some reason first host entered the "Non reponsive" state. The interesting thing is that hosted-engine and all other VM's up and running, so its like a communication problem between hosted-engine and host. 
> 
> The engine.log at hosted-engine is full of following messages:
> 
> 2017-11-14 17:06:43,158Z INFO  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 <http://80.239.162.106/>
> 2017-11-14 17:06:43,159Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler9) [50938c3] Command 'GetAllVmStatsVDSCommand(HostName = ovirt2.telia.ru <http://ovirt2.telia.ru/>, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243'})' execution failed: java.net.NoRouteToHostException: No route to host
> 2017-11-14 17:06:43,159Z INFO  [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler9) [50938c3] Failed to fetch vms info for host 'ovirt2.telia.ru <http://ovirt2.telia.ru/>' - skipping VMs monitoring.
> 2017-11-14 17:06:45,929Z INFO  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 <http://80.239.162.106/>
> 2017-11-14 17:06:45,930Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [6080f1cc] Command 'GetCapabilitiesVDSCommand(HostName = ovirt2.telia.ru <http://ovirt2.telia.ru/>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', vds='Host[ovirt2.telia.ru <http://ovirt2.telia.ru/>,3970247c-69eb-4bd8-b263-9100703a8243]'})' execution failed: java.net.NoRouteToHostException: No route to host
> 2017-11-14 17:06:45,930Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [6080f1cc] Failure to refresh host 'ovirt2.telia.ru <http://ovirt2.telia.ru/>' runtime info: java.net.NoRouteToHostException: No route to host
> 2017-11-14 17:06:48,933Z INFO  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 <http://80.239.162.106/>
> 2017-11-14 17:06:48,934Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [1a64dfea] Command 'GetCapabilitiesVDSCommand(HostName = ovirt2.telia.ru <http://ovirt2.telia.ru/>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', vds='Host[ovirt2.telia.ru <http://ovirt2.telia.ru/>,3970247c-69eb-4bd8-b263-9100703a8243]'})' execution failed: java.net.NoRouteToHostException: No route to host
> 2017-11-14 17:06:48,934Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [1a64dfea] Failure to refresh host 'ovirt2.telia.ru <http://ovirt2.telia.ru/>' runtime info: java.net.NoRouteToHostException: No route to host
> 2017-11-14 17:06:50,931Z INFO  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 <http://80.239.162.106/>
> 2017-11-14 17:06:50,932Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler4) [6b19d168] Command 'SpmStatusVDSCommand(HostName = ovirt2.telia.ru <http://ovirt2.telia.ru/>, SpmStatusVDSCommandParameters:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', storagePoolId='5a044257-02ec-0382-0243-0000000001f2'})' execution failed: java.net.NoRouteToHostException: No route to host
> 2017-11-14 17:06:50,939Z INFO  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 <http://80.239.162.106/>
> 2017-11-14 17:06:50,940Z ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [6b19d168] IrsBroker::Failed::GetStoragePoolInfoVDS
> 2017-11-14 17:06:50,940Z ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.GetStoragePoolInfoVDSCommand] (DefaultQuartzScheduler4) [6b19d168] Command 'GetStoragePoolInfoVDSCommand( GetStoragePoolInfoVDSCommandParameters:{runAsync='true', storagePoolId='5a044257-02ec-0382-0243-0000000001f2', ignoreFailoverLimit='true'})' execution failed: IRSProtocolException: 
> 2017-11-14 17:06:51,937Z INFO  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 <http://80.239.162.106/>
> 2017-11-14 17:06:51,938Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [7f23a3bd] Command 'GetCapabilitiesVDSCommand(HostName = ovirt2.telia.ru <http://ovirt2.telia.ru/>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', vds='Host[ovirt2.telia.ru <http://ovirt2.telia.ru/>,3970247c-69eb-4bd8-b263-9100703a8243]'})' execution failed: java.net.NoRouteToHostException: No route to host
> 2017-11-14 17:06:51,938Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [7f23a3bd] Failure to refresh host 'ovirt2.telia.ru <http://ovirt2.telia.ru/>' runtime info: java.net.NoRouteToHostException: No route to host
> 2017-11-14 17:06:54,941Z INFO  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 <http://80.239.162.106/>
> 2017-11-14 17:06:54,942Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [7a769f6c] Command 'GetCapabilitiesVDSCommand(HostName = ovirt2.telia.ru <http://ovirt2.telia.ru/>, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', vds='Host[ovirt2.telia.ru <http://ovirt2.telia.ru/>,3970247c-69eb-4bd8-b263-9100703a8243]'})' execution failed: java.net.NoRouteToHostException: No route to host
> 2017-11-14 17:06:54,942Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [7a769f6c] Failure to refresh host 'ovirt2.telia.ru <http://ovirt2.telia.ru/>' runtime info: java.net.NoRouteToHostException: No route to host
> 
> Its a bit weird, since I can ping and login via ssh to the host from hosted-engine with no problem. I have added second host to the cluster, but it not running hosted-engine. Any suggestion for the further steps? Just reboot the host and hope for the best?
> 
> Regards,
> Artem
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20171114/56fcde6b/attachment.html>