Re: [ovirt-users] Non-responsive host, VM's are still running - how to resolve?

--Apple-Mail=_5C6B714A-9630-480A-A42C-A8B11D48E3FB Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Try restarting vdsmd from the shell, =E2=80=9Csystemctl restart = vdsmd=E2=80=9D. how to resolve? another 4 VM's exactly during migration of second host from bare-metal = to second host in the cluster. For some reason first host entered the = "Non reponsive" state. The interesting thing is that hosted-engine and = all other VM's up and running, so its like a communication problem = between hosted-engine and host.=20 [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to ovirt2/80.239.162.106 <http://80.239.162.106/> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] = (DefaultQuartzScheduler9) [50938c3] Command = 'GetAllVmStatsVDSCommand(HostName =3D ovirt2.telia.ru = <http://ovirt2.telia.ru/>;, = VdsIdVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'3970247c-69eb-4bd8-b263-9100703a8243'})' execution failed: = java.net.NoRouteToHostException: No route to host [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] = (DefaultQuartzScheduler9) [50938c3] Failed to fetch vms info for host = 'ovirt2.telia.ru <http://ovirt2.telia.ru/>' - skipping VMs monitoring. [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to ovirt2/80.239.162.106 <http://80.239.162.106/> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] = (DefaultQuartzScheduler2) [6080f1cc] Command = 'GetCapabilitiesVDSCommand(HostName =3D ovirt2.telia.ru = <http://ovirt2.telia.ru/>;, = VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'3970247c-69eb-4bd8-b263-9100703a8243', = vds=3D'Host[ovirt2.telia.ru = <http://ovirt2.telia.ru/>,3970247c-69eb-4bd8-b263-9100703a8243]'})&... = execution failed: java.net.NoRouteToHostException: No route to host [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] = (DefaultQuartzScheduler2) [6080f1cc] Failure to refresh host = 'ovirt2.telia.ru <http://ovirt2.telia.ru/>' runtime info: = java.net.NoRouteToHostException: No route to host [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to ovirt2/80.239.162.106 <http://80.239.162.106/> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] = (DefaultQuartzScheduler6) [1a64dfea] Command = 'GetCapabilitiesVDSCommand(HostName =3D ovirt2.telia.ru = <http://ovirt2.telia.ru/>;, = VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'3970247c-69eb-4bd8-b263-9100703a8243', = vds=3D'Host[ovirt2.telia.ru = <http://ovirt2.telia.ru/>,3970247c-69eb-4bd8-b263-9100703a8243]'})&... = execution failed: java.net.NoRouteToHostException: No route to host [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] = (DefaultQuartzScheduler6) [1a64dfea] Failure to refresh host = 'ovirt2.telia.ru <http://ovirt2.telia.ru/>' runtime info: = java.net.NoRouteToHostException: No route to host [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to ovirt2/80.239.162.106 <http://80.239.162.106/> [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler4) [6b19d168] Command = 'SpmStatusVDSCommand(HostName =3D ovirt2.telia.ru = <http://ovirt2.telia.ru/>;, = SpmStatusVDSCommandParameters:{runAsync=3D'true', = hostId=3D'3970247c-69eb-4bd8-b263-9100703a8243', = storagePoolId=3D'5a044257-02ec-0382-0243-0000000001f2'})' execution = failed: java.net.NoRouteToHostException: No route to host [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to ovirt2/80.239.162.106 <http://80.239.162.106/> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] = (DefaultQuartzScheduler4) [6b19d168] = IrsBroker::Failed::GetStoragePoolInfoVDS [org.ovirt.engine.core.vdsbroker.irsbroker.GetStoragePoolInfoVDSCommand] = (DefaultQuartzScheduler4) [6b19d168] Command = 'GetStoragePoolInfoVDSCommand( = GetStoragePoolInfoVDSCommandParameters:{runAsync=3D'true', = storagePoolId=3D'5a044257-02ec-0382-0243-0000000001f2', = ignoreFailoverLimit=3D'true'})' execution failed: IRSProtocolException:=20= [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to ovirt2/80.239.162.106 <http://80.239.162.106/> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] = (DefaultQuartzScheduler7) [7f23a3bd] Command = 'GetCapabilitiesVDSCommand(HostName =3D ovirt2.telia.ru = <http://ovirt2.telia.ru/>;, = VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'3970247c-69eb-4bd8-b263-9100703a8243', = vds=3D'Host[ovirt2.telia.ru = <http://ovirt2.telia.ru/>,3970247c-69eb-4bd8-b263-9100703a8243]'})&... = execution failed: java.net.NoRouteToHostException: No route to host [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] = (DefaultQuartzScheduler7) [7f23a3bd] Failure to refresh host = 'ovirt2.telia.ru <http://ovirt2.telia.ru/>' runtime info: = java.net.NoRouteToHostException: No route to host [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to ovirt2/80.239.162.106 <http://80.239.162.106/> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] = (DefaultQuartzScheduler2) [7a769f6c] Command = 'GetCapabilitiesVDSCommand(HostName =3D ovirt2.telia.ru = <http://ovirt2.telia.ru/>;, = VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'3970247c-69eb-4bd8-b263-9100703a8243', = vds=3D'Host[ovirt2.telia.ru = <http://ovirt2.telia.ru/>,3970247c-69eb-4bd8-b263-9100703a8243]'})&... = execution failed: java.net.NoRouteToHostException: No route to host [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] = (DefaultQuartzScheduler2) [7a769f6c] Failure to refresh host = 'ovirt2.telia.ru <http://ovirt2.telia.ru/>' runtime info: = java.net.NoRouteToHostException: No route to host hosted-engine with no problem. I have added second host to the cluster, = but it not running hosted-engine. Any suggestion for the further steps? = Just reboot the host and hope for the best? --Apple-Mail=_5C6B714A-9630-480A-A42C-A8B11D48E3FB Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" class=3D"">Try = restarting vdsmd from the shell, =E2=80=9Csystemctl restart = vdsmd=E2=80=9D.<div class=3D""><br class=3D""></div><div class=3D""><br = class=3D""><div><blockquote type=3D"cite" class=3D""><hr = style=3D"border:none;border-top:solid #B5C4DF 1.0pt;padding:0 0 0 = 0;margin:10px 0 5px 0;" class=3D""><span style=3D"margin: -1.3px 0.0px = 0.0px 0.0px" id=3D"RwhHeaderAttributes" class=3D""><font = face=3D"Helvetica" size=3D"4" color=3D"#000000" style=3D"font: 13.0px = Helvetica; color: #000000" class=3D""><b class=3D"">From:</b> Artem = Tambovskiy &lt;<a href=3D"mailto:artem.tambovskiy@gmail.com" = class=3D&quot;&quot;&gt;artem.tambovskiy(a)gmail.com&lt;/a&gt;&amp;gt;&lt;/font&gt;&lt;/span&gt;&lt;br class=3D"">= <span style=3D"margin: -1.3px 0.0px 0.0px 0.0px" class=3D""><font = face=3D"Helvetica" size=3D"4" color=3D"#000000" style=3D"font: 13.0px = Helvetica; color: #000000" class=3D""><b class=3D"">Subject:</b> = [ovirt-users] Non-responsive host, VM's are still running - how to = resolve?</font></span><br class=3D""> <span style=3D"margin: -1.3px 0.0px 0.0px 0.0px" class=3D""><font = face=3D"Helvetica" size=3D"4" color=3D"#000000" style=3D"font: 13.0px = Helvetica; color: #000000" class=3D""><b class=3D"">Date:</b> November = 14, 2017 at 11:23:32 AM CST</font></span><br class=3D""> <span style=3D"margin: -1.3px 0.0px 0.0px 0.0px" class=3D""><font = face=3D"Helvetica" size=3D"4" color=3D"#000000" style=3D"font: 13.0px = Helvetica; color: #000000" class=3D""><b class=3D"">To:</b> = users</font></span><br class=3D""> <br class=3D"Apple-interchange-newline"><div class=3D""><div dir=3D"ltr" = class=3D"">Apparently, i lost the host which was running hosted-engine = and another 4 VM's exactly during migration of second host from = bare-metal to second host in the cluster. For some reason first host = entered the "Non reponsive" state. The interesting thing is that = hosted-engine and all other VM's up and running, so its like a = communication problem between hosted-engine and host.&nbsp;<div = class=3D""><br class=3D""></div><div class=3D"">The engine.log at = hosted-engine is full of following messages:</div><div class=3D""><br = class=3D""></div><div class=3D""><div class=3D"">2017-11-14 = 17:06:43,158Z INFO&nbsp; = [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to ovirt2/<a href=3D"http://80.239.162.106/" = class=3D"">80.239.162.106</a></div><div class=3D"">2017-11-14 = 17:06:43,159Z ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] = (DefaultQuartzScheduler9) [50938c3] Command = 'GetAllVmStatsVDSCommand(HostName =3D <a href=3D"http://ovirt2.telia.ru/" = class=3D"">ovirt2.telia.ru</a>, = VdsIdVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'3970247c-69eb-4bd8-b263-9100703a8243'})' execution failed: = java.net.NoRouteToHostException: No route to host</div><div = class=3D"">2017-11-14 17:06:43,159Z INFO&nbsp; = [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] = (DefaultQuartzScheduler9) [50938c3] Failed to fetch vms info for host = '<a href=3D"http://ovirt2.telia.ru/" class=3D"">ovirt2.telia.ru</a>' - = skipping VMs monitoring.</div><div class=3D"">2017-11-14 17:06:45,929Z = INFO&nbsp; [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL = Stomp Reactor) [] Connecting to ovirt2/<a href=3D"http://80.239.162.106/" = class=3D"">80.239.162.106</a></div><div class=3D"">2017-11-14 = 17:06:45,930Z ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] = (DefaultQuartzScheduler2) [6080f1cc] Command = 'GetCapabilitiesVDSCommand(HostName =3D <a = href=3D"http://ovirt2.telia.ru/" class=3D"">ovirt2.telia.ru</a>, = VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'3970247c-69eb-4bd8-b263-9100703a8243', vds=3D'Host[<a = href=3D"http://ovirt2.telia.ru/" = class=3D"">ovirt2.telia.ru</a>,3970247c-69eb-4bd8-b263-9100703a8243]'})' = execution failed: java.net.NoRouteToHostException: No route to = host</div><div class=3D"">2017-11-14 17:06:45,930Z ERROR = [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] = (DefaultQuartzScheduler2) [6080f1cc] Failure to refresh host '<a = href=3D"http://ovirt2.telia.ru/" class=3D"">ovirt2.telia.ru</a>' runtime = info: java.net.NoRouteToHostException: No route to host</div><div = class=3D"">2017-11-14 17:06:48,933Z INFO&nbsp; = [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to ovirt2/<a href=3D"http://80.239.162.106/" = class=3D"">80.239.162.106</a></div><div class=3D"">2017-11-14 = 17:06:48,934Z ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] = (DefaultQuartzScheduler6) [1a64dfea] Command = 'GetCapabilitiesVDSCommand(HostName =3D <a = href=3D"http://ovirt2.telia.ru/" class=3D"">ovirt2.telia.ru</a>, = VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'3970247c-69eb-4bd8-b263-9100703a8243', vds=3D'Host[<a = href=3D"http://ovirt2.telia.ru/" = class=3D"">ovirt2.telia.ru</a>,3970247c-69eb-4bd8-b263-9100703a8243]'})' = execution failed: java.net.NoRouteToHostException: No route to = host</div><div class=3D"">2017-11-14 17:06:48,934Z ERROR = [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] = (DefaultQuartzScheduler6) [1a64dfea] Failure to refresh host '<a = href=3D"http://ovirt2.telia.ru/" class=3D"">ovirt2.telia.ru</a>' runtime = info: java.net.NoRouteToHostException: No route to host</div><div = class=3D"">2017-11-14 17:06:50,931Z INFO&nbsp; = [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to ovirt2/<a href=3D"http://80.239.162.106/" = class=3D"">80.239.162.106</a></div><div class=3D"">2017-11-14 = 17:06:50,932Z ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler4) [6b19d168] Command = 'SpmStatusVDSCommand(HostName =3D <a href=3D"http://ovirt2.telia.ru/" = class=3D"">ovirt2.telia.ru</a>, = SpmStatusVDSCommandParameters:{runAsync=3D'true', = hostId=3D'3970247c-69eb-4bd8-b263-9100703a8243', = storagePoolId=3D'5a044257-02ec-0382-0243-0000000001f2'})' execution = failed: java.net.NoRouteToHostException: No route to host</div><div = class=3D"">2017-11-14 17:06:50,939Z INFO&nbsp; = [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to ovirt2/<a href=3D"http://80.239.162.106/" = class=3D"">80.239.162.106</a></div><div class=3D"">2017-11-14 = 17:06:50,940Z ERROR = [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] = (DefaultQuartzScheduler4) [6b19d168] = IrsBroker::Failed::GetStoragePoolInfoVDS</div><div class=3D"">2017-11-14 = 17:06:50,940Z ERROR = [org.ovirt.engine.core.vdsbroker.irsbroker.GetStoragePoolInfoVDSCommand] = (DefaultQuartzScheduler4) [6b19d168] Command = 'GetStoragePoolInfoVDSCommand( = GetStoragePoolInfoVDSCommandParameters:{runAsync=3D'true', = storagePoolId=3D'5a044257-02ec-0382-0243-0000000001f2', = ignoreFailoverLimit=3D'true'})' execution failed: = IRSProtocolException:&nbsp;</div><div class=3D"">2017-11-14 = 17:06:51,937Z INFO&nbsp; = [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to ovirt2/<a href=3D"http://80.239.162.106/" = class=3D"">80.239.162.106</a></div><div class=3D"">2017-11-14 = 17:06:51,938Z ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] = (DefaultQuartzScheduler7) [7f23a3bd] Command = 'GetCapabilitiesVDSCommand(HostName =3D <a = href=3D"http://ovirt2.telia.ru/" class=3D"">ovirt2.telia.ru</a>, = VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'3970247c-69eb-4bd8-b263-9100703a8243', vds=3D'Host[<a = href=3D"http://ovirt2.telia.ru/" = class=3D"">ovirt2.telia.ru</a>,3970247c-69eb-4bd8-b263-9100703a8243]'})' = execution failed: java.net.NoRouteToHostException: No route to = host</div><div class=3D"">2017-11-14 17:06:51,938Z ERROR = [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] = (DefaultQuartzScheduler7) [7f23a3bd] Failure to refresh host '<a = href=3D"http://ovirt2.telia.ru/" class=3D"">ovirt2.telia.ru</a>' runtime = info: java.net.NoRouteToHostException: No route to host</div><div = class=3D"">2017-11-14 17:06:54,941Z INFO&nbsp; = [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to ovirt2/<a href=3D"http://80.239.162.106/" = class=3D"">80.239.162.106</a></div><div class=3D"">2017-11-14 = 17:06:54,942Z ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] = (DefaultQuartzScheduler2) [7a769f6c] Command = 'GetCapabilitiesVDSCommand(HostName =3D <a = href=3D"http://ovirt2.telia.ru/" class=3D"">ovirt2.telia.ru</a>, = VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'3970247c-69eb-4bd8-b263-9100703a8243', vds=3D'Host[<a = href=3D"http://ovirt2.telia.ru/" = class=3D"">ovirt2.telia.ru</a>,3970247c-69eb-4bd8-b263-9100703a8243]'})' = execution failed: java.net.NoRouteToHostException: No route to = host</div><div class=3D"">2017-11-14 17:06:54,942Z ERROR = [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] = (DefaultQuartzScheduler2) [7a769f6c] Failure to refresh host '<a = href=3D"http://ovirt2.telia.ru/" class=3D"">ovirt2.telia.ru</a>' runtime = info: java.net.NoRouteToHostException: No route to host</div></div><div = class=3D""><br class=3D""></div><div class=3D"">Its a bit weird, since I = can ping and login via ssh to the host from hosted-engine with no = problem. I have added second host to the cluster, but it not running = hosted-engine. Any suggestion for the further steps? Just reboot the = host and hope for the best?</div><div class=3D""><br class=3D""></div><div= class=3D"">Regards,</div><div class=3D"">Artem</div></div> _______________________________________________<br class=3D"">Users = mailing list<br class=3D""><a href=3D"mailto:Users@ovirt.org" = class=3D&quot;&quot;&gt;Users(a)ovirt.org&lt;/a&gt;&lt;br = class=3D"">http://lists.ovirt.org/mailman/listinfo/users<br = class=3D""></div></blockquote></div><br class=3D""></div></body></html>= --Apple-Mail=_5C6B714A-9630-480A-A42C-A8B11D48E3FB--