<div dir="ltr">Thanks, Darrell!<br><br>Restarted vdsmd but it didn't helped. <br>systemctl status vdsmd -l showing following:<div><br><div>● vdsmd.service - Virtual Desktop Server Manager</div><div> Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled)</div><div> Active: active (running) since Tue 2017-11-14 21:01:31 MSK; 4min 53s ago</div><div> Process: 54674 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh --post-stop (code=exited, status=0/SUCCESS)</div><div> Process: 54677 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS)</div><div> Main PID: 54971 (vdsm)</div><div> CGroup: /system.slice/vdsmd.service</div><div> ├─54971 /usr/bin/python2 /usr/share/vdsm/vdsm</div><div> └─55099 /usr/libexec/ioprocess --read-pipe-fd 84 --write-pipe-fd 83 --max-threads 10 --max-queued-requests 10</div><div><br></div><div>Nov 14 21:01:33 <a href="http://ovirt2.telia.ru">ovirt2.telia.ru</a> vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|e0970bbf-11d8-4517-acff-0f8dccbb10a9' args={u'e0970bbf-11d8-4517-acff-0f8dccbb10a9': {'status': 'Up', 'displayInfo': [{'tlsPort': '5901', 'ipAddress': '80.239.162.106', 'type': u'spice', 'port': '-1'}], 'hash': '-6982259661244130819', 'displayIp': '80.239.162.106', 'displayPort': '-1', 'displaySecurePort': '5901', 'timeOffset': u'0', 'pauseCode': 'NOERR', 'vcpuQuota': '-1', 'cpuUser': '0.00', 'monitorResponse': '0', 'elapsedTime': '370019', 'displayType': 'qxl', 'cpuSys': '0.00', 'clientIp': '172.16.11.6', 'vcpuPeriod': 100000L}}</div><div>Nov 14 21:01:33 <a href="http://ovirt2.telia.ru">ovirt2.telia.ru</a> vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|b366e466-b0ea-4a09-866b-d0248d7523a6' args={u'b366e466-b0ea-4a09-866b-d0248d7523a6': {'status': 'Up', 'displayInfo': [{'tlsPort': '5900', 'ipAddress': '0', 'type': u'spice', 'port': '-1'}], 'hash': '1858968312777883492', 'displayIp': '0', 'displayPort': '-1', 'displaySecurePort': '5900', 'timeOffset': '0', 'pauseCode': 'NOERR', 'vcpuQuota': '-1', 'cpuUser': '0.00', 'monitorResponse': '0', 'elapsedTime': '453444', 'displayType': 'qxl', 'cpuSys': '0.00', 'clientIp': '', 'vcpuPeriod': 100000L}}</div><div>Nov 14 21:01:33 <a href="http://ovirt2.telia.ru">ovirt2.telia.ru</a> vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|ca2815c5-f815-469d-869d-a8fe1cb8c2e7' args={u'ca2815c5-f815-469d-869d-a8fe1cb8c2e7': {'status': 'Up', 'displayInfo': [{'tlsPort': '5904', 'ipAddress': '80.239.162.106', 'type': u'spice', 'port': '-1'}], 'hash': '1149212890076264321', 'displayIp': '80.239.162.106', 'displayPort': '-1', 'displaySecurePort': '5904', 'timeOffset': u'0', 'pauseCode': 'NOERR', 'vcpuQuota': '-1', 'cpuUser': '0.00', 'monitorResponse': '0', 'elapsedTime': '105160', 'displayType': 'qxl', 'cpuSys': '0.00', 'clientIp': '172.16.11.6', 'vcpuPeriod': 100000L}}</div><div>Nov 14 21:01:33 <a href="http://ovirt2.telia.ru">ovirt2.telia.ru</a> vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|a083da47-3e39-458c-8822-459af3d2d93a' args={u'a083da47-3e39-458c-8822-459af3d2d93a': {'status': 'Up', 'displayInfo': [{'tlsPort': '5902', 'ipAddress': '80.239.162.106', 'type': u'spice', 'port': '-1'}], 'hash': '5529949835126538749', 'displayIp': '80.239.162.106', 'displayPort': '-1', 'displaySecurePort': '5902', 'timeOffset': u'0', 'pauseCode': 'NOERR', 'vcpuQuota': '-1', 'cpuUser': '0.00', 'monitorResponse': '0', 'elapsedTime': '365326', 'displayType': 'qxl', 'cpuSys': '0.00', 'clientIp': '', 'vcpuPeriod': 100000L}}</div><div>Nov 14 21:01:33 <a href="http://ovirt2.telia.ru">ovirt2.telia.ru</a> vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|0b7d02df-0286-4e0e-a50b-1d02915ba81c' args={u'0b7d02df-0286-4e0e-a50b-1d02915ba81c': {'status': 'Up', 'displayInfo': [{'tlsPort': '5903', 'ipAddress': '80.239.162.106', 'type': u'spice', 'port': '-1'}], 'hash': '3267121054607612619', 'displayIp': '80.239.162.106', 'displayPort': '-1', 'displaySecurePort': '5903', 'timeOffset': '-1', 'pauseCode': 'NOERR', 'vcpuQuota': '-1', 'cpuUser': '0.00', 'monitorResponse': '0', 'elapsedTime': '275708', 'displayType': 'qxl', 'cpuSys': '0.00', 'clientIp': '', 'vcpuPeriod': 100000L}}</div><div>Nov 14 21:01:33 <a href="http://ovirt2.telia.ru">ovirt2.telia.ru</a> vdsm[54971]: vdsm throttled WARN MOM not available.</div><div>Nov 14 21:01:33 <a href="http://ovirt2.telia.ru">ovirt2.telia.ru</a> vdsm[54971]: vdsm throttled WARN MOM not available, KSM stats will be missing.</div><div>Nov 14 21:01:34 <a href="http://ovirt2.telia.ru">ovirt2.telia.ru</a> vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|0b7d02df-0286-4e0e-a50b-1d02915ba81c' args={u'0b7d02df-0286-4e0e-a50b-1d02915ba81c': {'status': 'Up', 'username': 'Unknown', 'memUsage': '36', 'guestFQDN': '', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'swap_usage': '0', u'mem_cached': '548192', u'mem_free': '2679664', u'mem_buffers': '231016', u'swap_in': '0', u'swap_total': '786428', u'pageflt': '4346', u'mem_total': '3922564', u'mem_unused': '1900456'}, 'session': 'Unknown', 'netIfaces': [], 'guestCPUCount': -1, 'appsList': (), 'guestIPs': '', 'disksUsage': []}}</div><div>Nov 14 21:01:34 <a href="http://ovirt2.telia.ru">ovirt2.telia.ru</a> vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|a083da47-3e39-458c-8822-459af3d2d93a' args={u'a083da47-3e39-458c-8822-459af3d2d93a': {'status': 'Up', 'username': 'Unknown', 'memUsage': '49', 'guestFQDN': '', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'swap_usage': '0', u'mem_cached': '549844', u'mem_free': '1054040', u'mem_buffers': '2080', u'swap_in': '0', u'swap_total': '4064252', u'pageflt': '148', u'mem_total': '1815524', u'mem_unused': '502116'}, 'session': 'Unknown', 'netIfaces': [], 'guestCPUCount': -1, 'appsList': (), 'guestIPs': '', 'disksUsage': []}}</div><div>Nov 14 21:01:34 <a href="http://ovirt2.telia.ru">ovirt2.telia.ru</a> vdsm[54971]: vdsm vds WARN Not ready yet, ignoring event u'|virt|VM_status|ca2815c5-f815-469d-869d-a8fe1cb8c2e7' args={u'ca2815c5-f815-469d-869d-a8fe1cb8c2e7': {'status': 'Up', 'username': 'Unknown', 'memUsage': '14', 'guestFQDN': '', 'memoryStats': {u'swap_out': '0', u'majflt': '0', u'swap_usage': '0', u'mem_cached': '497136', u'mem_free': '1801440', u'mem_buffers': '102108', u'swap_in': '0', u'swap_total': '1046524', u'pageflt': '64', u'mem_total': '2046116', u'mem_unused': '1202196'}, 'session': 'Unknown', 'netIfaces': [], 'guestCPUCount': -1, 'appsList': (), 'guestIPs': '', 'disksUsage': []}}</div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Nov 14, 2017 at 8:49 PM, Darrell Budic <span dir="ltr"><<a href="mailto:budic@onholyground.com" target="_blank">budic@onholyground.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word;line-break:after-white-space">Try restarting vdsmd from the shell, “systemctl restart vdsmd”.<div><br></div><div><br><div><blockquote type="cite"><hr style="border:none;border-top:solid #b5c4df 1.0pt;padding:0 0 0 0;margin:10px 0 5px 0"><span id="m_-5705628519254371738RwhHeaderAttributes"><font face="Helvetica" size="4" color="#000000" style="font:13.0px Helvetica;color:#000000"><b>From:</b> Artem Tambovskiy <<a href="mailto:artem.tambovskiy@gmail.com" target="_blank">artem.tambovskiy@gmail.com</a>></font></span><br>
<span><font face="Helvetica" size="4" color="#000000" style="font:13.0px Helvetica;color:#000000"><b>Subject:</b> [ovirt-users] Non-responsive host, VM's are still running - how to resolve?</font></span><br>
<span><font face="Helvetica" size="4" color="#000000" style="font:13.0px Helvetica;color:#000000"><b>Date:</b> November 14, 2017 at 11:23:32 AM CST</font></span><br>
<span><font face="Helvetica" size="4" color="#000000" style="font:13.0px Helvetica;color:#000000"><b>To:</b> users</font></span><br>
<br class="m_-5705628519254371738Apple-interchange-newline"><div><div><div class="h5"><div dir="ltr">Apparently, i lost the host which was running hosted-engine and another 4 VM's exactly during migration of second host from bare-metal to second host in the cluster. For some reason first host entered the "Non reponsive" state. The interesting thing is that hosted-engine and all other VM's up and running, so its like a communication problem between hosted-engine and host. <div><br></div><div>The engine.log at hosted-engine is full of following messages:</div><div><br></div><div><div>2017-11-14 17:06:43,158Z INFO [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/<a href="http://80.239.162.106/" target="_blank">80.239.162.106</a></div><div>2017-11-14 17:06:43,159Z ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetAllVmStatsVDSCommand] (DefaultQuartzScheduler9) [50938c3] Command 'GetAllVmStatsVDSCommand(<wbr>HostName = <a href="http://ovirt2.telia.ru/" target="_blank">ovirt2.telia.ru</a>, VdsIdVDSCommandParametersBase:<wbr>{runAsync='true', hostId='3970247c-69eb-4bd8-<wbr>b263-9100703a8243'})' execution failed: <a href="http://java.net">java.net</a>.<wbr>NoRouteToHostException: No route to host</div><div>2017-11-14 17:06:43,159Z INFO [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>PollVmStatsRefresher] (DefaultQuartzScheduler9) [50938c3] Failed to fetch vms info for host '<a href="http://ovirt2.telia.ru/" target="_blank">ovirt2.telia.ru</a>' - skipping VMs monitoring.</div><div>2017-11-14 17:06:45,929Z INFO [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/<a href="http://80.239.162.106/" target="_blank">80.239.162.106</a></div><div>2017-11-14 17:06:45,930Z ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [6080f1cc] Command 'GetCapabilitiesVDSCommand(<wbr>HostName = <a href="http://ovirt2.telia.ru/" target="_blank">ovirt2.telia.ru</a>, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='3970247c-69eb-4bd8-<wbr>b263-9100703a8243', vds='Host[<a href="http://ovirt2.telia.ru/" target="_blank">ovirt2.telia.ru</a>,<wbr>3970247c-69eb-4bd8-b263-<wbr>9100703a8243]'})' execution failed: <a href="http://java.net">java.net</a>.<wbr>NoRouteToHostException: No route to host</div><div>2017-11-14 17:06:45,930Z ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler2) [6080f1cc] Failure to refresh host '<a href="http://ovirt2.telia.ru/" target="_blank">ovirt2.telia.ru</a>' runtime info: <a href="http://java.net">java.net</a>.<wbr>NoRouteToHostException: No route to host</div><div>2017-11-14 17:06:48,933Z INFO [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/<a href="http://80.239.162.106/" target="_blank">80.239.162.106</a></div><div>2017-11-14 17:06:48,934Z ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [1a64dfea] Command 'GetCapabilitiesVDSCommand(<wbr>HostName = <a href="http://ovirt2.telia.ru/" target="_blank">ovirt2.telia.ru</a>, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='3970247c-69eb-4bd8-<wbr>b263-9100703a8243', vds='Host[<a href="http://ovirt2.telia.ru/" target="_blank">ovirt2.telia.ru</a>,<wbr>3970247c-69eb-4bd8-b263-<wbr>9100703a8243]'})' execution failed: <a href="http://java.net">java.net</a>.<wbr>NoRouteToHostException: No route to host</div><div>2017-11-14 17:06:48,934Z ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler6) [1a64dfea] Failure to refresh host '<a href="http://ovirt2.telia.ru/" target="_blank">ovirt2.telia.ru</a>' runtime info: <a href="http://java.net">java.net</a>.<wbr>NoRouteToHostException: No route to host</div><div>2017-11-14 17:06:50,931Z INFO [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/<a href="http://80.239.162.106/" target="_blank">80.239.162.106</a></div><div>2017-11-14 17:06:50,932Z ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>SpmStatusVDSCommand] (DefaultQuartzScheduler4) [6b19d168] Command 'SpmStatusVDSCommand(HostName = <a href="http://ovirt2.telia.ru/" target="_blank">ovirt2.telia.ru</a>, SpmStatusVDSCommandParameters:<wbr>{runAsync='true', hostId='3970247c-69eb-4bd8-<wbr>b263-9100703a8243', storagePoolId='5a044257-02ec-<wbr>0382-0243-0000000001f2'})' execution failed: <a href="http://java.net">java.net</a>.<wbr>NoRouteToHostException: No route to host</div><div>2017-11-14 17:06:50,939Z INFO [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/<a href="http://80.239.162.106/" target="_blank">80.239.162.106</a></div><div>2017-11-14 17:06:50,940Z ERROR [org.ovirt.engine.core.<wbr>vdsbroker.irsbroker.<wbr>IrsBrokerCommand] (DefaultQuartzScheduler4) [6b19d168] IrsBroker::Failed::<wbr>GetStoragePoolInfoVDS</div><div>2017-11-14 17:06:50,940Z ERROR [org.ovirt.engine.core.<wbr>vdsbroker.irsbroker.<wbr>GetStoragePoolInfoVDSCommand] (DefaultQuartzScheduler4) [6b19d168] Command 'GetStoragePoolInfoVDSCommand( GetStoragePoolInfoVDSCommandPa<wbr>rameters:{runAsync='true', storagePoolId='5a044257-02ec-<wbr>0382-0243-0000000001f2', ignoreFailoverLimit='true'})' execution failed: IRSProtocolException: </div><div>2017-11-14 17:06:51,937Z INFO [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/<a href="http://80.239.162.106/" target="_blank">80.239.162.106</a></div><div>2017-11-14 17:06:51,938Z ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [7f23a3bd] Command 'GetCapabilitiesVDSCommand(<wbr>HostName = <a href="http://ovirt2.telia.ru/" target="_blank">ovirt2.telia.ru</a>, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='3970247c-69eb-4bd8-<wbr>b263-9100703a8243', vds='Host[<a href="http://ovirt2.telia.ru/" target="_blank">ovirt2.telia.ru</a>,<wbr>3970247c-69eb-4bd8-b263-<wbr>9100703a8243]'})' execution failed: <a href="http://java.net">java.net</a>.<wbr>NoRouteToHostException: No route to host</div><div>2017-11-14 17:06:51,938Z ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler7) [7f23a3bd] Failure to refresh host '<a href="http://ovirt2.telia.ru/" target="_blank">ovirt2.telia.ru</a>' runtime info: <a href="http://java.net">java.net</a>.<wbr>NoRouteToHostException: No route to host</div><div>2017-11-14 17:06:54,941Z INFO [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt2/<a href="http://80.239.162.106/" target="_blank">80.239.162.106</a></div><div>2017-11-14 17:06:54,942Z ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [7a769f6c] Command 'GetCapabilitiesVDSCommand(<wbr>HostName = <a href="http://ovirt2.telia.ru/" target="_blank">ovirt2.telia.ru</a>, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='3970247c-69eb-4bd8-<wbr>b263-9100703a8243', vds='Host[<a href="http://ovirt2.telia.ru/" target="_blank">ovirt2.telia.ru</a>,<wbr>3970247c-69eb-4bd8-b263-<wbr>9100703a8243]'})' execution failed: <a href="http://java.net">java.net</a>.<wbr>NoRouteToHostException: No route to host</div><div>2017-11-14 17:06:54,942Z ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler2) [7a769f6c] Failure to refresh host '<a href="http://ovirt2.telia.ru/" target="_blank">ovirt2.telia.ru</a>' runtime info: <a href="http://java.net">java.net</a>.<wbr>NoRouteToHostException: No route to host</div></div><div><br></div><div>Its a bit weird, since I can ping and login via ssh to the host from hosted-engine with no problem. I have added second host to the cluster, but it not running hosted-engine. Any suggestion for the further steps? Just reboot the host and hope for the best?</div><div><br></div><div>Regards,</div><div>Artem</div></div></div></div>
______________________________<wbr>_________________<br>Users mailing list<br><a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br><a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br></div></blockquote></div><br></div></div></blockquote></div><br></div>