Non-responsive host, VM's are still running - how to resolve?
Apparently, i lost the host which was running hosted-engine and another 4 VM's exactly during migration of second host from bare-metal to second host in the cluster. For some reason first host entered the "Non reponsive" state. The interesting thing is that hosted-engine and all other VM's up and running, so its like a communication problem between hosted-engine and host. The engine.log at hosted-engine is full of following messages: 2017-11-14 17:06:43,158Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:43,159Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler9) [50938c3] Command 'GetAllVmStatsVDSCommand(HostName = ovirt2.telia.ru, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:43,159Z INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler9) [50938c3] Failed to fetch vms info for host ' ovirt2.telia.ru' - skipping VMs monitoring. 2017-11-14 17:06:45,929Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:45,930Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [6080f1cc] Command 'GetCapabilitiesVDSCommand(HostName = ovirt2.telia.ru, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', vds='Host[ovirt2.telia.ru,3970247c-69eb-4bd8-b263-9100703a8243]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:45,930Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [6080f1cc] Failure to refresh host ' ovirt2.telia.ru' runtime info: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:48,933Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:48,934Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [1a64dfea] Command 'GetCapabilitiesVDSCommand(HostName = ovirt2.telia.ru, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', vds='Host[ovirt2.telia.ru,3970247c-69eb-4bd8-b263-9100703a8243]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:48,934Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [1a64dfea] Failure to refresh host ' ovirt2.telia.ru' runtime info: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:50,931Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:50,932Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler4) [6b19d168] Command 'SpmStatusVDSCommand(HostName = ovirt2.telia.ru, SpmStatusVDSCommandParameters:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', storagePoolId='5a044257-02ec-0382-0243-0000000001f2'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:50,939Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:50,940Z ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [6b19d168] IrsBroker::Failed::GetStoragePoolInfoVDS 2017-11-14 17:06:50,940Z ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.GetStoragePoolInfoVDSCommand] (DefaultQuartzScheduler4) [6b19d168] Command 'GetStoragePoolInfoVDSCommand( GetStoragePoolInfoVDSCommandParameters:{runAsync='true', storagePoolId='5a044257-02ec-0382-0243-0000000001f2', ignoreFailoverLimit='true'})' execution failed: IRSProtocolException: 2017-11-14 17:06:51,937Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:51,938Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [7f23a3bd] Command 'GetCapabilitiesVDSCommand(HostName = ovirt2.telia.ru, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', vds='Host[ovirt2.telia.ru,3970247c-69eb-4bd8-b263-9100703a8243]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:51,938Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [7f23a3bd] Failure to refresh host ' ovirt2.telia.ru' runtime info: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:54,941Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:54,942Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [7a769f6c] Command 'GetCapabilitiesVDSCommand(HostName = ovirt2.telia.ru, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', vds='Host[ovirt2.telia.ru,3970247c-69eb-4bd8-b263-9100703a8243]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:54,942Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [7a769f6c] Failure to refresh host ' ovirt2.telia.ru' runtime info: java.net.NoRouteToHostException: No route to host Its a bit weird, since I can ping and login via ssh to the host from hosted-engine with no problem. I have added second host to the cluster, but it not running hosted-engine. Any suggestion for the further steps? Just reboot the host and hope for the best? Regards, Artem
--Apple-Mail=_5C6B714A-9630-480A-A42C-A8B11D48E3FB Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Try restarting vdsmd from the shell, =E2=80=9Csystemctl restart = vdsmd=E2=80=9D.
From: Artem Tambovskiy <artem.tambovskiy@gmail.com> Subject: [ovirt-users] Non-responsive host, VM's are still running - = how to resolve? Date: November 14, 2017 at 11:23:32 AM CST To: users =20 Apparently, i lost the host which was running hosted-engine and = another 4 VM's exactly during migration of second host from bare-metal = to second host in the cluster. For some reason first host entered the = "Non reponsive" state. The interesting thing is that hosted-engine and = all other VM's up and running, so its like a communication problem = between hosted-engine and host.=20 =20 The engine.log at hosted-engine is full of following messages: =20 2017-11-14 17:06:43,158Z INFO = [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to ovirt2/80.239.162.106 <http://80.239.162.106/> 2017-11-14 17:06:43,159Z ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] = (DefaultQuartzScheduler9) [50938c3] Command = 'GetAllVmStatsVDSCommand(HostName =3D ovirt2.telia.ru = <http://ovirt2.telia.ru/>, = VdsIdVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'3970247c-69eb-4bd8-b263-9100703a8243'})' execution failed: = java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:43,159Z INFO = [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] = (DefaultQuartzScheduler9) [50938c3] Failed to fetch vms info for host = 'ovirt2.telia.ru <http://ovirt2.telia.ru/>' - skipping VMs monitoring. 2017-11-14 17:06:45,929Z INFO = [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to ovirt2/80.239.162.106 <http://80.239.162.106/> 2017-11-14 17:06:45,930Z ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] = (DefaultQuartzScheduler2) [6080f1cc] Command = 'GetCapabilitiesVDSCommand(HostName =3D ovirt2.telia.ru = <http://ovirt2.telia.ru/>, = VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'3970247c-69eb-4bd8-b263-9100703a8243', = vds=3D'Host[ovirt2.telia.ru = <http://ovirt2.telia.ru/>,3970247c-69eb-4bd8-b263-9100703a8243]'})' = execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:45,930Z ERROR = [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] = (DefaultQuartzScheduler2) [6080f1cc] Failure to refresh host = 'ovirt2.telia.ru <http://ovirt2.telia.ru/>' runtime info: = java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:48,933Z INFO = [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to ovirt2/80.239.162.106 <http://80.239.162.106/> 2017-11-14 17:06:48,934Z ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] = (DefaultQuartzScheduler6) [1a64dfea] Command = 'GetCapabilitiesVDSCommand(HostName =3D ovirt2.telia.ru = <http://ovirt2.telia.ru/>, = VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'3970247c-69eb-4bd8-b263-9100703a8243', = vds=3D'Host[ovirt2.telia.ru = <http://ovirt2.telia.ru/>,3970247c-69eb-4bd8-b263-9100703a8243]'})' = execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:48,934Z ERROR = [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] = (DefaultQuartzScheduler6) [1a64dfea] Failure to refresh host = 'ovirt2.telia.ru <http://ovirt2.telia.ru/>' runtime info: = java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:50,931Z INFO = [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to ovirt2/80.239.162.106 <http://80.239.162.106/> 2017-11-14 17:06:50,932Z ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler4) [6b19d168] Command = 'SpmStatusVDSCommand(HostName =3D ovirt2.telia.ru = <http://ovirt2.telia.ru/>, = SpmStatusVDSCommandParameters:{runAsync=3D'true', = hostId=3D'3970247c-69eb-4bd8-b263-9100703a8243', = storagePoolId=3D'5a044257-02ec-0382-0243-0000000001f2'})' execution = failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:50,939Z INFO = [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to ovirt2/80.239.162.106 <http://80.239.162.106/> 2017-11-14 17:06:50,940Z ERROR = [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] = (DefaultQuartzScheduler4) [6b19d168] = IrsBroker::Failed::GetStoragePoolInfoVDS 2017-11-14 17:06:50,940Z ERROR = [org.ovirt.engine.core.vdsbroker.irsbroker.GetStoragePoolInfoVDSCommand] = (DefaultQuartzScheduler4) [6b19d168] Command = 'GetStoragePoolInfoVDSCommand( = GetStoragePoolInfoVDSCommandParameters:{runAsync=3D'true', = storagePoolId=3D'5a044257-02ec-0382-0243-0000000001f2', = ignoreFailoverLimit=3D'true'})' execution failed: IRSProtocolException:=20=
2017-11-14 17:06:51,937Z INFO = [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to ovirt2/80.239.162.106 <http://80.239.162.106/> 2017-11-14 17:06:51,938Z ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] = (DefaultQuartzScheduler7) [7f23a3bd] Command = 'GetCapabilitiesVDSCommand(HostName =3D ovirt2.telia.ru = <http://ovirt2.telia.ru/>, = VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'3970247c-69eb-4bd8-b263-9100703a8243', = vds=3D'Host[ovirt2.telia.ru = <http://ovirt2.telia.ru/>,3970247c-69eb-4bd8-b263-9100703a8243]'})' = execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:51,938Z ERROR = [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] = (DefaultQuartzScheduler7) [7f23a3bd] Failure to refresh host = 'ovirt2.telia.ru <http://ovirt2.telia.ru/>' runtime info: = java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:54,941Z INFO = [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to ovirt2/80.239.162.106 <http://80.239.162.106/> 2017-11-14 17:06:54,942Z ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] = (DefaultQuartzScheduler2) [7a769f6c] Command = 'GetCapabilitiesVDSCommand(HostName =3D ovirt2.telia.ru = <http://ovirt2.telia.ru/>, = VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'3970247c-69eb-4bd8-b263-9100703a8243', = vds=3D'Host[ovirt2.telia.ru = <http://ovirt2.telia.ru/>,3970247c-69eb-4bd8-b263-9100703a8243]'})' = execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:54,942Z ERROR = [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] = (DefaultQuartzScheduler2) [7a769f6c] Failure to refresh host = 'ovirt2.telia.ru <http://ovirt2.telia.ru/>' runtime info: = java.net.NoRouteToHostException: No route to host =20 Its a bit weird, since I can ping and login via ssh to the host from = hosted-engine with no problem. I have added second host to the cluster, = but it not running hosted-engine. Any suggestion for the further steps? = Just reboot the host and hope for the best? =20 Regards, Artem _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--Apple-Mail=_5C6B714A-9630-480A-A42C-A8B11D48E3FB Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" class=3D"">Try = restarting vdsmd from the shell, =E2=80=9Csystemctl restart = vdsmd=E2=80=9D.<div class=3D""><br class=3D""></div><div class=3D""><br = class=3D""><div><blockquote type=3D"cite" class=3D""><hr = style=3D"border:none;border-top:solid #B5C4DF 1.0pt;padding:0 0 0 = 0;margin:10px 0 5px 0;" class=3D""><span style=3D"margin: -1.3px 0.0px = 0.0px 0.0px" id=3D"RwhHeaderAttributes" class=3D""><font = face=3D"Helvetica" size=3D"4" color=3D"#000000" style=3D"font: 13.0px = Helvetica; color: #000000" class=3D""><b class=3D"">From:</b> Artem = Tambovskiy <<a href=3D"mailto:artem.tambovskiy@gmail.com" = class=3D"">artem.tambovskiy@gmail.com</a>></font></span><br class=3D"">= <span style=3D"margin: -1.3px 0.0px 0.0px 0.0px" class=3D""><font = face=3D"Helvetica" size=3D"4" color=3D"#000000" style=3D"font: 13.0px = Helvetica; color: #000000" class=3D""><b class=3D"">Subject:</b> = [ovirt-users] Non-responsive host, VM's are still running - how to = resolve?</font></span><br class=3D""> <span style=3D"margin: -1.3px 0.0px 0.0px 0.0px" class=3D""><font = face=3D"Helvetica" size=3D"4" color=3D"#000000" style=3D"font: 13.0px = Helvetica; color: #000000" class=3D""><b class=3D"">Date:</b> November = 14, 2017 at 11:23:32 AM CST</font></span><br class=3D""> <span style=3D"margin: -1.3px 0.0px 0.0px 0.0px" class=3D""><font = face=3D"Helvetica" size=3D"4" color=3D"#000000" style=3D"font: 13.0px = Helvetica; color: #000000" class=3D""><b class=3D"">To:</b> = users</font></span><br class=3D""> <br class=3D"Apple-interchange-newline"><div class=3D""><div dir=3D"ltr" = class=3D"">Apparently, i lost the host which was running hosted-engine = and another 4 VM's exactly during migration of second host from = bare-metal to second host in the cluster. For some reason first host = entered the "Non reponsive" state. The interesting thing is that = hosted-engine and all other VM's up and running, so its like a = communication problem between hosted-engine and host. <div = class=3D""><br class=3D""></div><div class=3D"">The engine.log at = hosted-engine is full of following messages:</div><div class=3D""><br = class=3D""></div><div class=3D""><div class=3D"">2017-11-14 = 17:06:43,158Z INFO = [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to ovirt2/<a href=3D"http://80.239.162.106/" = class=3D"">80.239.162.106</a></div><div class=3D"">2017-11-14 = 17:06:43,159Z ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] = (DefaultQuartzScheduler9) [50938c3] Command = 'GetAllVmStatsVDSCommand(HostName =3D <a href=3D"http://ovirt2.telia.ru/" = class=3D"">ovirt2.telia.ru</a>, = VdsIdVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'3970247c-69eb-4bd8-b263-9100703a8243'})' execution failed: = java.net.NoRouteToHostException: No route to host</div><div = class=3D"">2017-11-14 17:06:43,159Z INFO = [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] = (DefaultQuartzScheduler9) [50938c3] Failed to fetch vms info for host = '<a href=3D"http://ovirt2.telia.ru/" class=3D"">ovirt2.telia.ru</a>' - = skipping VMs monitoring.</div><div class=3D"">2017-11-14 17:06:45,929Z = INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL = Stomp Reactor) [] Connecting to ovirt2/<a href=3D"http://80.239.162.106/" = class=3D"">80.239.162.106</a></div><div class=3D"">2017-11-14 = 17:06:45,930Z ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] = (DefaultQuartzScheduler2) [6080f1cc] Command = 'GetCapabilitiesVDSCommand(HostName =3D <a = href=3D"http://ovirt2.telia.ru/" class=3D"">ovirt2.telia.ru</a>, = VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'3970247c-69eb-4bd8-b263-9100703a8243', vds=3D'Host[<a = href=3D"http://ovirt2.telia.ru/" = class=3D"">ovirt2.telia.ru</a>,3970247c-69eb-4bd8-b263-9100703a8243]'})' = execution failed: java.net.NoRouteToHostException: No route to = host</div><div class=3D"">2017-11-14 17:06:45,930Z ERROR = [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] = (DefaultQuartzScheduler2) [6080f1cc] Failure to refresh host '<a = href=3D"http://ovirt2.telia.ru/" class=3D"">ovirt2.telia.ru</a>' runtime = info: java.net.NoRouteToHostException: No route to host</div><div = class=3D"">2017-11-14 17:06:48,933Z INFO = [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to ovirt2/<a href=3D"http://80.239.162.106/" = class=3D"">80.239.162.106</a></div><div class=3D"">2017-11-14 = 17:06:48,934Z ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] = (DefaultQuartzScheduler6) [1a64dfea] Command = 'GetCapabilitiesVDSCommand(HostName =3D <a = href=3D"http://ovirt2.telia.ru/" class=3D"">ovirt2.telia.ru</a>, = VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'3970247c-69eb-4bd8-b263-9100703a8243', vds=3D'Host[<a = href=3D"http://ovirt2.telia.ru/" = class=3D"">ovirt2.telia.ru</a>,3970247c-69eb-4bd8-b263-9100703a8243]'})' = execution failed: java.net.NoRouteToHostException: No route to = host</div><div class=3D"">2017-11-14 17:06:48,934Z ERROR = [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] = (DefaultQuartzScheduler6) [1a64dfea] Failure to refresh host '<a = href=3D"http://ovirt2.telia.ru/" class=3D"">ovirt2.telia.ru</a>' runtime = info: java.net.NoRouteToHostException: No route to host</div><div = class=3D"">2017-11-14 17:06:50,931Z INFO = [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to ovirt2/<a href=3D"http://80.239.162.106/" = class=3D"">80.239.162.106</a></div><div class=3D"">2017-11-14 = 17:06:50,932Z ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] = (DefaultQuartzScheduler4) [6b19d168] Command = 'SpmStatusVDSCommand(HostName =3D <a href=3D"http://ovirt2.telia.ru/" = class=3D"">ovirt2.telia.ru</a>, = SpmStatusVDSCommandParameters:{runAsync=3D'true', = hostId=3D'3970247c-69eb-4bd8-b263-9100703a8243', = storagePoolId=3D'5a044257-02ec-0382-0243-0000000001f2'})' execution = failed: java.net.NoRouteToHostException: No route to host</div><div = class=3D"">2017-11-14 17:06:50,939Z INFO = [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to ovirt2/<a href=3D"http://80.239.162.106/" = class=3D"">80.239.162.106</a></div><div class=3D"">2017-11-14 = 17:06:50,940Z ERROR = [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] = (DefaultQuartzScheduler4) [6b19d168] = IrsBroker::Failed::GetStoragePoolInfoVDS</div><div class=3D"">2017-11-14 = 17:06:50,940Z ERROR = [org.ovirt.engine.core.vdsbroker.irsbroker.GetStoragePoolInfoVDSCommand] = (DefaultQuartzScheduler4) [6b19d168] Command = 'GetStoragePoolInfoVDSCommand( = GetStoragePoolInfoVDSCommandParameters:{runAsync=3D'true', = storagePoolId=3D'5a044257-02ec-0382-0243-0000000001f2', = ignoreFailoverLimit=3D'true'})' execution failed: = IRSProtocolException: </div><div class=3D"">2017-11-14 = 17:06:51,937Z INFO = [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to ovirt2/<a href=3D"http://80.239.162.106/" = class=3D"">80.239.162.106</a></div><div class=3D"">2017-11-14 = 17:06:51,938Z ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] = (DefaultQuartzScheduler7) [7f23a3bd] Command = 'GetCapabilitiesVDSCommand(HostName =3D <a = href=3D"http://ovirt2.telia.ru/" class=3D"">ovirt2.telia.ru</a>, = VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'3970247c-69eb-4bd8-b263-9100703a8243', vds=3D'Host[<a = href=3D"http://ovirt2.telia.ru/" = class=3D"">ovirt2.telia.ru</a>,3970247c-69eb-4bd8-b263-9100703a8243]'})' = execution failed: java.net.NoRouteToHostException: No route to = host</div><div class=3D"">2017-11-14 17:06:51,938Z ERROR = [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] = (DefaultQuartzScheduler7) [7f23a3bd] Failure to refresh host '<a = href=3D"http://ovirt2.telia.ru/" class=3D"">ovirt2.telia.ru</a>' runtime = info: java.net.NoRouteToHostException: No route to host</div><div = class=3D"">2017-11-14 17:06:54,941Z INFO = [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to ovirt2/<a href=3D"http://80.239.162.106/" = class=3D"">80.239.162.106</a></div><div class=3D"">2017-11-14 = 17:06:54,942Z ERROR = [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] = (DefaultQuartzScheduler2) [7a769f6c] Command = 'GetCapabilitiesVDSCommand(HostName =3D <a = href=3D"http://ovirt2.telia.ru/" class=3D"">ovirt2.telia.ru</a>, = VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', = hostId=3D'3970247c-69eb-4bd8-b263-9100703a8243', vds=3D'Host[<a = href=3D"http://ovirt2.telia.ru/" = class=3D"">ovirt2.telia.ru</a>,3970247c-69eb-4bd8-b263-9100703a8243]'})' = execution failed: java.net.NoRouteToHostException: No route to = host</div><div class=3D"">2017-11-14 17:06:54,942Z ERROR = [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] = (DefaultQuartzScheduler2) [7a769f6c] Failure to refresh host '<a = href=3D"http://ovirt2.telia.ru/" class=3D"">ovirt2.telia.ru</a>' runtime = info: java.net.NoRouteToHostException: No route to host</div></div><div = class=3D""><br class=3D""></div><div class=3D"">Its a bit weird, since I = can ping and login via ssh to the host from hosted-engine with no = problem. I have added second host to the cluster, but it not running = hosted-engine. Any suggestion for the further steps? Just reboot the = host and hope for the best?</div><div class=3D""><br class=3D""></div><div= class=3D"">Regards,</div><div class=3D"">Artem</div></div> _______________________________________________<br class=3D"">Users = mailing list<br class=3D""><a href=3D"mailto:Users@ovirt.org" = class=3D"">Users@ovirt.org</a><br = class=3D"">http://lists.ovirt.org/mailman/listinfo/users<br = class=3D""></div></blockquote></div><br class=3D""></div></body></html>= --Apple-Mail=_5C6B714A-9630-480A-A42C-A8B11D48E3FB--
Thanks, Darrell! Restarted vdsmd but it didn't helped. systemctl status vdsmd -l showing following: ● vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2017-11-14 21:01:31 MSK; 4min 53s ago Process: 54674 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh --post-stop (code=exited, status=0/SUCCESS) Process: 54677 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS) Main PID: 54971 (vdsm) CGroup: /system.slice/vdsmd.service ├─54971 /usr/bin/python2 /usr/share/vdsm/vdsm └─55099 /usr/libexec/ioprocess --read-pipe-fd 84 --write-pipe-fd 83 --max-threads 10 --max-queued-requests 10 Nov 14 21:01:33 ovirt2.telia.ru vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|e0970bbf-11d8-4517-acff-0f8dccbb10a9' args={u'e0970bbf-11d8-4517-acff-0f8dccbb10a9': {'status': 'Up', 'displayInfo': [{'tlsPort': '5901', 'ipAddress': '80.239.162.106', 'type': u'spice', 'port': '-1'}], 'hash': '-6982259661244130819', 'displayIp': '80.239.162.106', 'displayPort': '-1', 'displaySecurePort': '5901', 'timeOffset': u'0', 'pauseCode': 'NOERR', 'vcpuQuota': '-1', 'cpuUser': '0.00', 'monitorResponse': '0', 'elapsedTime': '370019', 'displayType': 'qxl', 'cpuSys': '0.00', 'clientIp': '172.16.11.6', 'vcpuPeriod': 100000L}} Nov 14 21:01:33 ovirt2.telia.ru vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|b366e466-b0ea-4a09-866b-d0248d7523a6' args={u'b366e466-b0ea-4a09-866b-d0248d7523a6': {'status': 'Up', 'displayInfo': [{'tlsPort': '5900', 'ipAddress': '0', 'type': u'spice', 'port': '-1'}], 'hash': '1858968312777883492', 'displayIp': '0', 'displayPort': '-1', 'displaySecurePort': '5900', 'timeOffset': '0', 'pauseCode': 'NOERR', 'vcpuQuota': '-1', 'cpuUser': '0.00', 'monitorResponse': '0', 'elapsedTime': '453444', 'displayType': 'qxl', 'cpuSys': '0.00', 'clientIp': '', 'vcpuPeriod': 100000L}} Nov 14 21:01:33 ovirt2.telia.ru vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|ca2815c5-f815-469d-869d-a8fe1cb8c2e7' args={u'ca2815c5-f815-469d-869d-a8fe1cb8c2e7': {'status': 'Up', 'displayInfo': [{'tlsPort': '5904', 'ipAddress': '80.239.162.106', 'type': u'spice', 'port': '-1'}], 'hash': '1149212890076264321', 'displayIp': '80.239.162.106', 'displayPort': '-1', 'displaySecurePort': '5904', 'timeOffset': u'0', 'pauseCode': 'NOERR', 'vcpuQuota': '-1', 'cpuUser': '0.00', 'monitorResponse': '0', 'elapsedTime': '105160', 'displayType': 'qxl', 'cpuSys': '0.00', 'clientIp': '172.16.11.6', 'vcpuPeriod': 100000L}} Nov 14 21:01:33 ovirt2.telia.ru vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|a083da47-3e39-458c-8822-459af3d2d93a' args={u'a083da47-3e39-458c-8822-459af3d2d93a': {'status': 'Up', 'displayInfo': [{'tlsPort': '5902', 'ipAddress': '80.239.162.106', 'type': u'spice', 'port': '-1'}], 'hash': '5529949835126538749', 'displayIp': '80.239.162.106', 'displayPort': '-1', 'displaySecurePort': '5902', 'timeOffset': u'0', 'pauseCode': 'NOERR', 'vcpuQuota': '-1', 'cpuUser': '0.00', 'monitorResponse': '0', 'elapsedTime': '365326', 'displayType': 'qxl', 'cpuSys': '0.00', 'clientIp': '', 'vcpuPeriod': 100000L}} Nov 14 21:01:33 ovirt2.telia.ru vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|0b7d02df-0286-4e0e-a50b-1d02915ba81c' args={u'0b7d02df-0286-4e0e-a50b-1d02915ba81c': {'status': 'Up', 'displayInfo': [{'tlsPort': '5903', 'ipAddress': '80.239.162.106', 'type': u'spice', 'port': '-1'}], 'hash': '3267121054607612619', 'displayIp': '80.239.162.106', 'displayPort': '-1', 'displaySecurePort': '5903', 'timeOffset': '-1', 'pauseCode': 'NOERR', 'vcpuQuota': '-1', 'cpuUser': '0.00', 'monitorResponse': '0', 'elapsedTime': '275708', 'displayType': 'qxl', 'cpuSys': '0.00', 'clientIp': '', 'vcpuPeriod': 100000L}} Nov 14 21:01:33 ovirt2.telia.ru vdsm[54971]: vdsm throttled WARN MOM not available. Nov 14 21:01:33 ovirt2.telia.ru vdsm[54971]: vdsm throttled WARN MOM not available, KSM stats will be missing. Nov 14 21:01:34 ovirt2.telia.ru vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|0b7d02df-0286-4e0e-a50b-1d02915ba81c' args={u'0b7d02df-0286-4e0e-a50b-1d02915ba81c': {'status': 'Up', 'username': 'Unknown', 'memUsage': '36', 'guestFQDN': '', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'swap_usage': '0', u'mem_cached': '548192', u'mem_free': '2679664', u'mem_buffers': '231016', u'swap_in': '0', u'swap_total': '786428', u'pageflt': '4346', u'mem_total': '3922564', u'mem_unused': '1900456'}, 'session': 'Unknown', 'netIfaces': [], 'guestCPUCount': -1, 'appsList': (), 'guestIPs': '', 'disksUsage': []}} Nov 14 21:01:34 ovirt2.telia.ru vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|a083da47-3e39-458c-8822-459af3d2d93a' args={u'a083da47-3e39-458c-8822-459af3d2d93a': {'status': 'Up', 'username': 'Unknown', 'memUsage': '49', 'guestFQDN': '', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'swap_usage': '0', u'mem_cached': '549844', u'mem_free': '1054040', u'mem_buffers': '2080', u'swap_in': '0', u'swap_total': '4064252', u'pageflt': '148', u'mem_total': '1815524', u'mem_unused': '502116'}, 'session': 'Unknown', 'netIfaces': [], 'guestCPUCount': -1, 'appsList': (), 'guestIPs': '', 'disksUsage': []}} Nov 14 21:01:34 ovirt2.telia.ru vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|ca2815c5-f815-469d-869d-a8fe1cb8c2e7' args={u'ca2815c5-f815-469d-869d-a8fe1cb8c2e7': {'status': 'Up', 'username': 'Unknown', 'memUsage': '14', 'guestFQDN': '', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'swap_usage': '0', u'mem_cached': '497136', u'mem_free': '1801440', u'mem_buffers': '102108', u'swap_in': '0', u'swap_total': '1046524', u'pageflt': '64', u'mem_total': '2046116', u'mem_unused': '1202196'}, 'session': 'Unknown', 'netIfaces': [], 'guestCPUCount': -1, 'appsList': (), 'guestIPs': '', 'disksUsage': []}} On Tue, Nov 14, 2017 at 8:49 PM, Darrell Budic <budic@onholyground.com> wrote:
Try restarting vdsmd from the shell, “systemctl restart vdsmd”.
------------------------------ *From:* Artem Tambovskiy <artem.tambovskiy@gmail.com> *Subject:* [ovirt-users] Non-responsive host, VM's are still running - how to resolve? *Date:* November 14, 2017 at 11:23:32 AM CST *To:* users
Apparently, i lost the host which was running hosted-engine and another 4 VM's exactly during migration of second host from bare-metal to second host in the cluster. For some reason first host entered the "Non reponsive" state. The interesting thing is that hosted-engine and all other VM's up and running, so its like a communication problem between hosted-engine and host.
The engine.log at hosted-engine is full of following messages:
2017-11-14 17:06:43,158Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:43,159Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler9) [50938c3] Command 'GetAllVmStatsVDSCommand(HostName = ovirt2.telia.ru, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:43,159Z INFO [org.ovirt.engine.core. vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler9) [50938c3] Failed to fetch vms info for host 'ovirt2.telia.ru' - skipping VMs monitoring. 2017-11-14 17:06:45,929Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:45,930Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [6080f1cc] Command 'GetCapabilitiesVDSCommand(HostName = ovirt2.telia.ru, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', vds='Host[ovirt2.telia.ru, 3970247c-69eb-4bd8-b263-9100703a8243]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:45,930Z ERROR [org.ovirt.engine.core. vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [6080f1cc] Failure to refresh host 'ovirt2.telia.ru' runtime info: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:48,933Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:48,934Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [1a64dfea] Command 'GetCapabilitiesVDSCommand(HostName = ovirt2.telia.ru, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', vds='Host[ovirt2.telia.ru, 3970247c-69eb-4bd8-b263-9100703a8243]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:48,934Z ERROR [org.ovirt.engine.core. vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [1a64dfea] Failure to refresh host 'ovirt2.telia.ru' runtime info: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:50,931Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:50,932Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler4) [6b19d168] Command 'SpmStatusVDSCommand(HostName = ovirt2.telia.ru, SpmStatusVDSCommandParameters:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', storagePoolId='5a044257-02ec-0382-0243-0000000001f2'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:50,939Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:50,940Z ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [6b19d168] IrsBroker::Failed:: GetStoragePoolInfoVDS 2017-11-14 17:06:50,940Z ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.GetStoragePoolInfoVDSCommand] (DefaultQuartzScheduler4) [6b19d168] Command 'GetStoragePoolInfoVDSCommand( GetStoragePoolInfoVDSCommandParameters:{runAsync='true', storagePoolId='5a044257-02ec-0382-0243-0000000001f2', ignoreFailoverLimit='true'})' execution failed: IRSProtocolException: 2017-11-14 17:06:51,937Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:51,938Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [7f23a3bd] Command 'GetCapabilitiesVDSCommand(HostName = ovirt2.telia.ru, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', vds='Host[ovirt2.telia.ru, 3970247c-69eb-4bd8-b263-9100703a8243]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:51,938Z ERROR [org.ovirt.engine.core. vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [7f23a3bd] Failure to refresh host 'ovirt2.telia.ru' runtime info: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:54,941Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:54,942Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [7a769f6c] Command 'GetCapabilitiesVDSCommand(HostName = ovirt2.telia.ru, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', vds='Host[ovirt2.telia.ru, 3970247c-69eb-4bd8-b263-9100703a8243]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:54,942Z ERROR [org.ovirt.engine.core. vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [7a769f6c] Failure to refresh host 'ovirt2.telia.ru' runtime info: java.net.NoRouteToHostException: No route to host
Its a bit weird, since I can ping and login via ssh to the host from hosted-engine with no problem. I have added second host to the cluster, but it not running hosted-engine. Any suggestion for the further steps? Just reboot the host and hope for the best?
Regards, Artem _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Tue, Nov 14, 2017 at 7:09 PM, Artem Tambovskiy <artem.tambovskiy@gmail.com> wrote:
Thanks, Darrell!
Restarted vdsmd but it didn't helped. systemctl status vdsmd -l showing following:
● vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2017-11-14 21:01:31 MSK; 4min 53s ago Process: 54674 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh --post-stop (code=exited, status=0/SUCCESS) Process: 54677 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS) Main PID: 54971 (vdsm) CGroup: /system.slice/vdsmd.service ├─54971 /usr/bin/python2 /usr/share/vdsm/vdsm └─55099 /usr/libexec/ioprocess --read-pipe-fd 84 --write-pipe-fd 83 --max-threads 10 --max-queued-requests 10
Nov 14 21:01:33 ovirt2.telia.ru vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|e0970bbf-11d8-4517-acff-0f8dccbb10a9' args={u'e0970bbf-11d8-4517-acff-0f8dccbb10a9': {'status': 'Up', 'displayInfo': [{'tlsPort': '5901', 'ipAddress': '80.239.162.106', 'type': u'spice', 'port': '-1'}], 'hash': '-6982259661244130819', 'displayIp': '80.239.162.106', 'displayPort': '-1', 'displaySecurePort': '5901', 'timeOffset': u'0', 'pauseCode': 'NOERR', 'vcpuQuota': '-1', 'cpuUser': '0.00', 'monitorResponse': '0', 'elapsedTime': '370019', 'displayType': 'qxl', 'cpuSys': '0.00', 'clientIp': '172.16.11.6', 'vcpuPeriod': 100000L}} Nov 14 21:01:33 ovirt2.telia.ru vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|b366e466-b0ea-4a09-866b-d0248d7523a6' args={u'b366e466-b0ea-4a09-866b-d0248d7523a6': {'status': 'Up', 'displayInfo': [{'tlsPort': '5900', 'ipAddress': '0', 'type': u'spice', 'port': '-1'}], 'hash': '1858968312777883492', 'displayIp': '0', 'displayPort': '-1', 'displaySecurePort': '5900', 'timeOffset': '0', 'pauseCode': 'NOERR', 'vcpuQuota': '-1', 'cpuUser': '0.00', 'monitorResponse': '0', 'elapsedTime': '453444', 'displayType': 'qxl', 'cpuSys': '0.00', 'clientIp': '', 'vcpuPeriod': 100000L}} Nov 14 21:01:33 ovirt2.telia.ru vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|ca2815c5-f815-469d-869d-a8fe1cb8c2e7' args={u'ca2815c5-f815-469d-869d-a8fe1cb8c2e7': {'status': 'Up', 'displayInfo': [{'tlsPort': '5904', 'ipAddress': '80.239.162.106', 'type': u'spice', 'port': '-1'}], 'hash': '1149212890076264321', 'displayIp': '80.239.162.106', 'displayPort': '-1', 'displaySecurePort': '5904', 'timeOffset': u'0', 'pauseCode': 'NOERR', 'vcpuQuota': '-1', 'cpuUser': '0.00', 'monitorResponse': '0', 'elapsedTime': '105160', 'displayType': 'qxl', 'cpuSys': '0.00', 'clientIp': '172.16.11.6', 'vcpuPeriod': 100000L}} Nov 14 21:01:33 ovirt2.telia.ru vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|a083da47-3e39-458c-8822-459af3d2d93a' args={u'a083da47-3e39-458c-8822-459af3d2d93a': {'status': 'Up', 'displayInfo': [{'tlsPort': '5902', 'ipAddress': '80.239.162.106', 'type': u'spice', 'port': '-1'}], 'hash': '5529949835126538749', 'displayIp': '80.239.162.106', 'displayPort': '-1', 'displaySecurePort': '5902', 'timeOffset': u'0', 'pauseCode': 'NOERR', 'vcpuQuota': '-1', 'cpuUser': '0.00', 'monitorResponse': '0', 'elapsedTime': '365326', 'displayType': 'qxl', 'cpuSys': '0.00', 'clientIp': '', 'vcpuPeriod': 100000L}} Nov 14 21:01:33 ovirt2.telia.ru vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|0b7d02df-0286-4e0e-a50b-1d02915ba81c' args={u'0b7d02df-0286-4e0e-a50b-1d02915ba81c': {'status': 'Up', 'displayInfo': [{'tlsPort': '5903', 'ipAddress': '80.239.162.106', 'type': u'spice', 'port': '-1'}], 'hash': '3267121054607612619', 'displayIp': '80.239.162.106', 'displayPort': '-1', 'displaySecurePort': '5903', 'timeOffset': '-1', 'pauseCode': 'NOERR', 'vcpuQuota': '-1', 'cpuUser': '0.00', 'monitorResponse': '0', 'elapsedTime': '275708', 'displayType': 'qxl', 'cpuSys': '0.00', 'clientIp': '', 'vcpuPeriod': 100000L}} Nov 14 21:01:33 ovirt2.telia.ru vdsm[54971]: vdsm throttled WARN MOM not available. Nov 14 21:01:33 ovirt2.telia.ru vdsm[54971]: vdsm throttled WARN MOM not available, KSM stats will be missing. Nov 14 21:01:34 ovirt2.telia.ru vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|0b7d02df-0286-4e0e-a50b-1d02915ba81c' args={u'0b7d02df-0286-4e0e-a50b-1d02915ba81c': {'status': 'Up', 'username': 'Unknown', 'memUsage': '36', 'guestFQDN': '', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'swap_usage': '0', u'mem_cached': '548192', u'mem_free': '2679664', u'mem_buffers': '231016', u'swap_in': '0', u'swap_total': '786428', u'pageflt': '4346', u'mem_total': '3922564', u'mem_unused': '1900456'}, 'session': 'Unknown', 'netIfaces': [], 'guestCPUCount': -1, 'appsList': (), 'guestIPs': '', 'disksUsage': []}} Nov 14 21:01:34 ovirt2.telia.ru vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|a083da47-3e39-458c-8822-459af3d2d93a' args={u'a083da47-3e39-458c-8822-459af3d2d93a': {'status': 'Up', 'username': 'Unknown', 'memUsage': '49', 'guestFQDN': '', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'swap_usage': '0', u'mem_cached': '549844', u'mem_free': '1054040', u'mem_buffers': '2080', u'swap_in': '0', u'swap_total': '4064252', u'pageflt': '148', u'mem_total': '1815524', u'mem_unused': '502116'}, 'session': 'Unknown', 'netIfaces': [], 'guestCPUCount': -1, 'appsList': (), 'guestIPs': '', 'disksUsage': []}} Nov 14 21:01:34 ovirt2.telia.ru vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|ca2815c5-f815-469d-869d-a8fe1cb8c2e7' args={u'ca2815c5-f815-469d-869d-a8fe1cb8c2e7': {'status': 'Up', 'username': 'Unknown', 'memUsage': '14', 'guestFQDN': '', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'swap_usage': '0', u'mem_cached': '497136', u'mem_free': '1801440', u'mem_buffers': '102108', u'swap_in': '0', u'swap_total': '1046524', u'pageflt': '64', u'mem_total': '2046116', u'mem_unused': '1202196'}, 'session': 'Unknown', 'netIfaces': [], 'guestCPUCount': -1, 'appsList': (), 'guestIPs': '', 'disksUsage': []}}
Above logs say that there is no connection from the engine so events won't be sent. Can you share engine logs?
On Tue, Nov 14, 2017 at 8:49 PM, Darrell Budic <budic@onholyground.com> wrote:
Try restarting vdsmd from the shell, “systemctl restart vdsmd”.
________________________________ From: Artem Tambovskiy <artem.tambovskiy@gmail.com> Subject: [ovirt-users] Non-responsive host, VM's are still running - how to resolve? Date: November 14, 2017 at 11:23:32 AM CST To: users
Apparently, i lost the host which was running hosted-engine and another 4 VM's exactly during migration of second host from bare-metal to second host in the cluster. For some reason first host entered the "Non reponsive" state. The interesting thing is that hosted-engine and all other VM's up and running, so its like a communication problem between hosted-engine and host.
The engine.log at hosted-engine is full of following messages:
2017-11-14 17:06:43,158Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:43,159Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler9) [50938c3] Command 'GetAllVmStatsVDSCommand(HostName = ovirt2.telia.ru, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:43,159Z INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler9) [50938c3] Failed to fetch vms info for host 'ovirt2.telia.ru' - skipping VMs monitoring. 2017-11-14 17:06:45,929Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:45,930Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [6080f1cc] Command 'GetCapabilitiesVDSCommand(HostName = ovirt2.telia.ru, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', vds='Host[ovirt2.telia.ru,3970247c-69eb-4bd8-b263-9100703a8243]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:45,930Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [6080f1cc] Failure to refresh host 'ovirt2.telia.ru' runtime info: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:48,933Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:48,934Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [1a64dfea] Command 'GetCapabilitiesVDSCommand(HostName = ovirt2.telia.ru, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', vds='Host[ovirt2.telia.ru,3970247c-69eb-4bd8-b263-9100703a8243]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:48,934Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [1a64dfea] Failure to refresh host 'ovirt2.telia.ru' runtime info: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:50,931Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:50,932Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler4) [6b19d168] Command 'SpmStatusVDSCommand(HostName = ovirt2.telia.ru, SpmStatusVDSCommandParameters:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', storagePoolId='5a044257-02ec-0382-0243-0000000001f2'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:50,939Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:50,940Z ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [6b19d168] IrsBroker::Failed::GetStoragePoolInfoVDS 2017-11-14 17:06:50,940Z ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.GetStoragePoolInfoVDSCommand] (DefaultQuartzScheduler4) [6b19d168] Command 'GetStoragePoolInfoVDSCommand( GetStoragePoolInfoVDSCommandParameters:{runAsync='true', storagePoolId='5a044257-02ec-0382-0243-0000000001f2', ignoreFailoverLimit='true'})' execution failed: IRSProtocolException: 2017-11-14 17:06:51,937Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:51,938Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [7f23a3bd] Command 'GetCapabilitiesVDSCommand(HostName = ovirt2.telia.ru, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', vds='Host[ovirt2.telia.ru,3970247c-69eb-4bd8-b263-9100703a8243]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:51,938Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [7f23a3bd] Failure to refresh host 'ovirt2.telia.ru' runtime info: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:54,941Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:54,942Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [7a769f6c] Command 'GetCapabilitiesVDSCommand(HostName = ovirt2.telia.ru, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', vds='Host[ovirt2.telia.ru,3970247c-69eb-4bd8-b263-9100703a8243]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:54,942Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [7a769f6c] Failure to refresh host 'ovirt2.telia.ru' runtime info: java.net.NoRouteToHostException: No route to host
Its a bit weird, since I can ping and login via ssh to the host from hosted-engine with no problem. I have added second host to the cluster, but it not running hosted-engine. Any suggestion for the further steps? Just reboot the host and hope for the best?
Regards, Artem _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Piotr, full engine log attached. On Tue, Nov 14, 2017 at 10:19 PM, Piotr Kliczewski < piotr.kliczewski@gmail.com> wrote:
On Tue, Nov 14, 2017 at 7:09 PM, Artem Tambovskiy <artem.tambovskiy@gmail.com> wrote:
Thanks, Darrell!
Restarted vdsmd but it didn't helped. systemctl status vdsmd -l showing following:
● vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2017-11-14 21:01:31 MSK; 4min 53s ago Process: 54674 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh --post-stop (code=exited, status=0/SUCCESS) Process: 54677 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS) Main PID: 54971 (vdsm) CGroup: /system.slice/vdsmd.service ├─54971 /usr/bin/python2 /usr/share/vdsm/vdsm └─55099 /usr/libexec/ioprocess --read-pipe-fd 84 --write-pipe-fd 83 --max-threads 10 --max-queued-requests 10
Nov 14 21:01:33 ovirt2.telia.ru vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|e0970bbf-11d8-4517-acff-0f8dccbb10a9' args={u'e0970bbf-11d8-4517-acff-0f8dccbb10a9': {'status': 'Up', 'displayInfo': [{'tlsPort': '5901', 'ipAddress': '80.239.162.106', 'type': u'spice', 'port': '-1'}], 'hash': '-6982259661244130819', 'displayIp': '80.239.162.106', 'displayPort': '-1', 'displaySecurePort': '5901', 'timeOffset': u'0', 'pauseCode': 'NOERR', 'vcpuQuota': '-1', 'cpuUser': '0.00', 'monitorResponse': '0', 'elapsedTime': '370019', 'displayType': 'qxl', 'cpuSys': '0.00', 'clientIp': '172.16.11.6', 'vcpuPeriod': 100000L}} Nov 14 21:01:33 ovirt2.telia.ru vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|b366e466-b0ea-4a09-866b-d0248d7523a6' args={u'b366e466-b0ea-4a09-866b-d0248d7523a6': {'status': 'Up', 'displayInfo': [{'tlsPort': '5900', 'ipAddress': '0', 'type': u'spice', 'port': '-1'}], 'hash': '1858968312777883492', 'displayIp': '0', 'displayPort': '-1', 'displaySecurePort': '5900', 'timeOffset': '0', 'pauseCode': 'NOERR', 'vcpuQuota': '-1', 'cpuUser': '0.00', 'monitorResponse': '0', 'elapsedTime': '453444', 'displayType': 'qxl', 'cpuSys': '0.00', 'clientIp': '', 'vcpuPeriod': 100000L}} Nov 14 21:01:33 ovirt2.telia.ru vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|ca2815c5-f815-469d-869d-a8fe1cb8c2e7' args={u'ca2815c5-f815-469d-869d-a8fe1cb8c2e7': {'status': 'Up', 'displayInfo': [{'tlsPort': '5904', 'ipAddress': '80.239.162.106', 'type': u'spice', 'port': '-1'}], 'hash': '1149212890076264321', 'displayIp': '80.239.162.106', 'displayPort': '-1', 'displaySecurePort': '5904', 'timeOffset': u'0', 'pauseCode': 'NOERR', 'vcpuQuota': '-1', 'cpuUser': '0.00', 'monitorResponse': '0', 'elapsedTime': '105160', 'displayType': 'qxl', 'cpuSys': '0.00', 'clientIp': '172.16.11.6', 'vcpuPeriod': 100000L}} Nov 14 21:01:33 ovirt2.telia.ru vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|a083da47-3e39-458c-8822-459af3d2d93a' args={u'a083da47-3e39-458c-8822-459af3d2d93a': {'status': 'Up', 'displayInfo': [{'tlsPort': '5902', 'ipAddress': '80.239.162.106', 'type': u'spice', 'port': '-1'}], 'hash': '5529949835126538749', 'displayIp': '80.239.162.106', 'displayPort': '-1', 'displaySecurePort': '5902', 'timeOffset': u'0', 'pauseCode': 'NOERR', 'vcpuQuota': '-1', 'cpuUser': '0.00', 'monitorResponse': '0', 'elapsedTime': '365326', 'displayType': 'qxl', 'cpuSys': '0.00', 'clientIp': '', 'vcpuPeriod': 100000L}} Nov 14 21:01:33 ovirt2.telia.ru vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|0b7d02df-0286-4e0e-a50b-1d02915ba81c' args={u'0b7d02df-0286-4e0e-a50b-1d02915ba81c': {'status': 'Up', 'displayInfo': [{'tlsPort': '5903', 'ipAddress': '80.239.162.106', 'type': u'spice', 'port': '-1'}], 'hash': '3267121054607612619', 'displayIp': '80.239.162.106', 'displayPort': '-1', 'displaySecurePort': '5903', 'timeOffset': '-1', 'pauseCode': 'NOERR', 'vcpuQuota': '-1', 'cpuUser': '0.00', 'monitorResponse': '0', 'elapsedTime': '275708', 'displayType': 'qxl', 'cpuSys': '0.00', 'clientIp': '', 'vcpuPeriod': 100000L}} Nov 14 21:01:33 ovirt2.telia.ru vdsm[54971]: vdsm throttled WARN MOM not available. Nov 14 21:01:33 ovirt2.telia.ru vdsm[54971]: vdsm throttled WARN MOM not available, KSM stats will be missing. Nov 14 21:01:34 ovirt2.telia.ru vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|0b7d02df-0286-4e0e-a50b-1d02915ba81c' args={u'0b7d02df-0286-4e0e-a50b-1d02915ba81c': {'status': 'Up', 'username': 'Unknown', 'memUsage': '36', 'guestFQDN': '', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'swap_usage': '0', u'mem_cached': '548192', u'mem_free': '2679664', u'mem_buffers': '231016', u'swap_in': '0', u'swap_total': '786428', u'pageflt': '4346', u'mem_total': '3922564', u'mem_unused': '1900456'}, 'session': 'Unknown', 'netIfaces': [], 'guestCPUCount': -1, 'appsList': (), 'guestIPs': '', 'disksUsage': []}} Nov 14 21:01:34 ovirt2.telia.ru vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|a083da47-3e39-458c-8822-459af3d2d93a' args={u'a083da47-3e39-458c-8822-459af3d2d93a': {'status': 'Up', 'username': 'Unknown', 'memUsage': '49', 'guestFQDN': '', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'swap_usage': '0', u'mem_cached': '549844', u'mem_free': '1054040', u'mem_buffers': '2080', u'swap_in': '0', u'swap_total': '4064252', u'pageflt': '148', u'mem_total': '1815524', u'mem_unused': '502116'}, 'session': 'Unknown', 'netIfaces': [], 'guestCPUCount': -1, 'appsList': (), 'guestIPs': '', 'disksUsage': []}} Nov 14 21:01:34 ovirt2.telia.ru vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|ca2815c5-f815-469d-869d-a8fe1cb8c2e7' args={u'ca2815c5-f815-469d-869d-a8fe1cb8c2e7': {'status': 'Up', 'username': 'Unknown', 'memUsage': '14', 'guestFQDN': '', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'swap_usage': '0', u'mem_cached': '497136', u'mem_free': '1801440', u'mem_buffers': '102108', u'swap_in': '0', u'swap_total': '1046524', u'pageflt': '64', u'mem_total': '2046116', u'mem_unused': '1202196'}, 'session': 'Unknown', 'netIfaces': [], 'guestCPUCount': -1, 'appsList': (), 'guestIPs': '', 'disksUsage': []}}
Above logs say that there is no connection from the engine so events won't be sent. Can you share engine logs?
On Tue, Nov 14, 2017 at 8:49 PM, Darrell Budic <budic@onholyground.com> wrote:
Try restarting vdsmd from the shell, “systemctl restart vdsmd”.
________________________________ From: Artem Tambovskiy <artem.tambovskiy@gmail.com> Subject: [ovirt-users] Non-responsive host, VM's are still running - how to resolve? Date: November 14, 2017 at 11:23:32 AM CST To: users
Apparently, i lost the host which was running hosted-engine and another
4
VM's exactly during migration of second host from bare-metal to second host in the cluster. For some reason first host entered the "Non reponsive" state. The interesting thing is that hosted-engine and all other VM's up and running, so its like a communication problem between hosted-engine and host.
The engine.log at hosted-engine is full of following messages:
2017-11-14 17:06:43,158Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:43,159Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler9) [50938c3] Command 'GetAllVmStatsVDSCommand(HostName = ovirt2.telia.ru, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:43,159Z INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler9) [50938c3] Failed to fetch vms info for host 'ovirt2.telia.ru' - skipping VMs monitoring. 2017-11-14 17:06:45,929Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:45,930Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [6080f1cc] Command 'GetCapabilitiesVDSCommand(HostName = ovirt2.telia.ru, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', vds='Host[ovirt2.telia.ru,3970247c-69eb-4bd8-b263-9100703a8243]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:45,930Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [6080f1cc] Failure to refresh host 'ovirt2.telia.ru' runtime info: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:48,933Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:48,934Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [1a64dfea] Command 'GetCapabilitiesVDSCommand(HostName = ovirt2.telia.ru, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', vds='Host[ovirt2.telia.ru,3970247c-69eb-4bd8-b263-9100703a8243]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:48,934Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [1a64dfea] Failure to refresh host 'ovirt2.telia.ru' runtime info: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:50,931Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:50,932Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler4) [6b19d168] Command 'SpmStatusVDSCommand(HostName = ovirt2.telia.ru, SpmStatusVDSCommandParameters:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', storagePoolId='5a044257-02ec-0382-0243-0000000001f2'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:50,939Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:50,940Z ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [6b19d168] IrsBroker::Failed::GetStoragePoolInfoVDS 2017-11-14 17:06:50,940Z ERROR [org.ovirt.engine.core.vdsbroker.irsbroker. GetStoragePoolInfoVDSCommand] (DefaultQuartzScheduler4) [6b19d168] Command 'GetStoragePoolInfoVDSCommand( GetStoragePoolInfoVDSCommandParameters:{runAsync='true', storagePoolId='5a044257-02ec-0382-0243-0000000001f2', ignoreFailoverLimit='true'})' execution failed: IRSProtocolException: 2017-11-14 17:06:51,937Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:51,938Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [7f23a3bd] Command 'GetCapabilitiesVDSCommand(HostName = ovirt2.telia.ru, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', vds='Host[ovirt2.telia.ru,3970247c-69eb-4bd8-b263-9100703a8243]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:51,938Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [7f23a3bd] Failure to refresh host 'ovirt2.telia.ru' runtime info: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:54,941Z INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/80.239.162.106 2017-11-14 17:06:54,942Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [7a769f6c] Command 'GetCapabilitiesVDSCommand(HostName = ovirt2.telia.ru, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='3970247c-69eb-4bd8-b263-9100703a8243', vds='Host[ovirt2.telia.ru,3970247c-69eb-4bd8-b263-9100703a8243]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-11-14 17:06:54,942Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [7a769f6c] Failure to refresh host 'ovirt2.telia.ru' runtime info: java.net.NoRouteToHostException: No route to host
Its a bit weird, since I can ping and login via ssh to the host from hosted-engine with no problem. I have added second host to the cluster, but it not running hosted-engine. Any suggestion for the further steps? Just reboot the host and hope for the best?
Regards, Artem _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (3)
-
Artem Tambovskiy -
Darrell Budic -
Piotr Kliczewski