<div dir="auto">What 4.0 version do you use? <div dir="auto">Can you attach complete engine and vdsm logs? </div><div dir="auto">CC-ing Piotr as well. </div><div dir="auto"><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On May 11, 2017 6:44 PM, &quot;Alexis HAUSER&quot; &lt;<a href="mailto:alexis.hauser@imt-atlantique.fr">alexis.hauser@imt-atlantique.fr</a>&gt; wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:times new roman,new york,times,serif;font-size:12pt;color:#000000"><div>After rebooting the manager VM,  hosts are connecting/non responsive and data domains inactive. Here are the engine and vdsmd logs. Any ideas ?<br></div><div><br></div><div><br></div><div><br></div><div><br></div><div>Engine logs : <br></div><div><br></div><div><br></div><div><br></div><div>2017-05-11 17:28:09,302 WARN  [org.ovirt.engine.core.dal.<wbr>dbbroker.auditloghandling.<wbr>AuditLogDirector] (DefaultQuartzScheduler5) [55f1aab5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rhvserv-05.<br>2017-05-11 17:28:09,346 INFO  [org.ovirt.engine.core.bll.<wbr>HandleVdsVersionCommand] (DefaultQuartzScheduler5) [48bc69cd] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 04565f10-9abf-4709-9445-<wbr>9dc6ed97e136 Type: VDS<br>2017-05-11 17:28:09,349 WARN  [org.ovirt.engine.core.<wbr>vdsbroker.VdsManager] (org.ovirt.thread.pool-6-<wbr>thread-27) [639977e4] Host &#39;rhvserv-05&#39; is not responding.<br>2017-05-11 17:28:09,364 WARN  [org.ovirt.engine.core.dal.<wbr>dbbroker.auditloghandling.<wbr>AuditLogDirector] (org.ovirt.thread.pool-6-<wbr>thread-27) [639977e4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rhvserv-05 is not responding. Host cannot be fenced automatically because power management for the host is disabled.<br>2017-05-11 17:28:11,299 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [c0e6a2e] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;4036f027-8e90-49c0-<wbr>8ca5-3ddb8d586916&#39;, vds=&#39;Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:11,299 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler3) [c0e6a2e] Failure to refresh host &#39;rhvserv-03&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:11,327 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.<wbr>168.93.214</a><br>2017-05-11 17:28:12,484 INFO  [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [c0e6a2e] START, GetHardwareInfoVDSCommand(<wbr>HostName = rhvserv-05, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;04565f10-9abf-4709-<wbr>9445-9dc6ed97e136&#39;, vds=&#39;Host[rhvserv-05,04565f10-<wbr>9abf-4709-9445-9dc6ed97e136]&#39;}<wbr>), log id: f807ece<br>2017-05-11 17:28:12,487 INFO  [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [c0e6a2e] FINISH, GetHardwareInfoVDSCommand, log id: f807ece<br>2017-05-11 17:28:12,532 INFO  [org.ovirt.engine.core.bll.<wbr>HandleVdsCpuFlagsOrClusterChan<wbr>gedCommand] (DefaultQuartzScheduler3) [4e882ea0] Running command: HandleVdsCpuFlagsOrClusterChan<wbr>gedCommand internal: true. Entities affected :  ID: 04565f10-9abf-4709-9445-<wbr>9dc6ed97e136 Type: VDS<br>2017-05-11 17:28:12,539 INFO  [org.ovirt.engine.core.bll.<wbr>InitVdsOnUpCommand] (DefaultQuartzScheduler3) [75f25b35] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 58f8df36-019f-02bc-00e7-<wbr>000000000023 Type: StoragePool<br>2017-05-11 17:28:12,545 INFO  [org.ovirt.engine.core.bll.<wbr>storage.pool.<wbr>ConnectHostToStoragePoolServer<wbr>sCommand] (DefaultQuartzScheduler3) [46cc3f58] Running command: ConnectHostToStoragePoolServer<wbr>sCommand internal: true. Entities affected :  ID: 58f8df36-019f-02bc-00e7-<wbr>000000000023 Type: StoragePool<br>2017-05-11 17:28:12,556 INFO  [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>ConnectStorageServerVDSCommand<wbr>] (DefaultQuartzScheduler3) [46cc3f58] START, ConnectStorageServerVDSCommand<wbr>(HostName = rhvserv-05, StorageServerConnectionManagem<wbr>entVDSParameters:{runAsync=&#39;<wbr>true&#39;, hostId=&#39;04565f10-9abf-4709-<wbr>9445-9dc6ed97e136&#39;, storagePoolId=&#39;58f8df36-019f-<wbr>02bc-00e7-000000000023&#39;, storageType=&#39;ISCSI&#39;, connectionList=&#39;[<wbr>StorageServerConnections:{id=&#39;<wbr>10c0528b-f08d-4d1d-8c63-<wbr>8a05fd9d58b9&#39;, connection=&#39;10.35.21.1&#39;, iqn=&#39;iqn.1984-05.com.dell:<wbr>powervault.md3200i.<wbr>6782bcb00073e332000000004edde1<wbr>64&#39;, vfsType=&#39;null&#39;, mountOptions=&#39;null&#39;, nfsVersion=&#39;null&#39;, nfsRetrans=&#39;null&#39;, nfsTimeo=&#39;null&#39;, iface=&#39;null&#39;, netIfaceName=&#39;null&#39;}]&#39;}), log id: 1beb27b6<br>2017-05-11 17:28:13,031 INFO  [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>ConnectStorageServerVDSCommand<wbr>] (DefaultQuartzScheduler3) [46cc3f58] FINISH, ConnectStorageServerVDSCommand<wbr>, return: {10c0528b-f08d-4d1d-8c63-<wbr>8a05fd9d58b9=0}, log id: 1beb27b6<br>2017-05-11 17:28:13,032 INFO  [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>ConnectStorageServerVDSCommand<wbr>] (DefaultQuartzScheduler3) [46cc3f58] START, ConnectStorageServerVDSCommand<wbr>(HostName = rhvserv-05, StorageServerConnectionManagem<wbr>entVDSParameters:{runAsync=&#39;<wbr>true&#39;, hostId=&#39;04565f10-9abf-4709-<wbr>9445-9dc6ed97e136&#39;, storagePoolId=&#39;58f8df36-019f-<wbr>02bc-00e7-000000000023&#39;, storageType=&#39;NFS&#39;, connectionList=&#39;[<wbr>StorageServerConnections:{id=&#39;<wbr>e604d0d2-0810-4c25-b9ed-<wbr>610f9923cb1a&#39;, connection=&#39;nfsserv-01:/nfs/<wbr>export&#39;, iqn=&#39;null&#39;, vfsType=&#39;null&#39;, mountOptions=&#39;null&#39;, nfsVersion=&#39;V3&#39;, nfsRetrans=&#39;null&#39;, nfsTimeo=&#39;null&#39;, iface=&#39;null&#39;, netIfaceName=&#39;null&#39;}, StorageServerConnections:{id=&#39;<wbr>2c1aba4b-786b-4c76-a02f-<wbr>d9e09e8afff4&#39;, connection=&#39;nfsserv-01:/nfs/<wbr>iso&#39;, iqn=&#39;null&#39;, vfsType=&#39;null&#39;, mountOptions=&#39;null&#39;, nfsVersion=&#39;V3&#39;, nfsRetrans=&#39;null&#39;, nfsTimeo=&#39;null&#39;, iface=&#39;null&#39;, netIfaceName=&#39;null&#39;}]&#39;}), log id: 15c54cd7<br>2017-05-11 17:28:14,301 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.<wbr>168.93.213</a><br>2017-05-11 17:28:14,304 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [62006eaa] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;4036f027-8e90-49c0-<wbr>8ca5-3ddb8d586916&#39;, vds=&#39;Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:14,304 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler1) [62006eaa] Failure to refresh host &#39;rhvserv-03&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:14,334 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [1efe65a7] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;0d0cd690-b64a-42bb-<wbr>a167-43fcedd634e4&#39;, vds=&#39;Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:14,334 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler7) [1efe65a7] Failure to refresh host &#39;rhvserv-04&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:17,308 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.<wbr>168.93.213</a><br>2017-05-11 17:28:17,336 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.<wbr>168.93.214</a><br>2017-05-11 17:28:17,340 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;0d0cd690-b64a-42bb-<wbr>a167-43fcedd634e4&#39;, vds=&#39;Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:17,340 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh host &#39;rhvserv-04&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:20,315 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [62006eaa] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;4036f027-8e90-49c0-<wbr>8ca5-3ddb8d586916&#39;, vds=&#39;Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:20,315 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler1) [62006eaa] Failure to refresh host &#39;rhvserv-03&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:20,343 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.<wbr>168.93.214</a><br>2017-05-11 17:28:23,316 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.<wbr>168.93.213</a><br>2017-05-11 17:28:23,325 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;4036f027-8e90-49c0-<wbr>8ca5-3ddb8d586916&#39;, vds=&#39;Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:23,325 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh host &#39;rhvserv-03&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:23,348 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [1efe65a7] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;0d0cd690-b64a-42bb-<wbr>a167-43fcedd634e4&#39;, vds=&#39;Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:23,348 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler7) [1efe65a7] Failure to refresh host &#39;rhvserv-04&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:26,329 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.<wbr>168.93.213</a><br>2017-05-11 17:28:26,350 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.<wbr>168.93.214</a><br>2017-05-11 17:28:26,354 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [1efe65a7] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;0d0cd690-b64a-42bb-<wbr>a167-43fcedd634e4&#39;, vds=&#39;Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:26,354 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler7) [1efe65a7] Failure to refresh host &#39;rhvserv-04&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:29,335 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;4036f027-8e90-49c0-<wbr>8ca5-3ddb8d586916&#39;, vds=&#39;Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:29,335 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh host &#39;rhvserv-03&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:29,356 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.<wbr>168.93.214</a><br>2017-05-11 17:28:32,337 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.<wbr>168.93.213</a><br>2017-05-11 17:28:32,340 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;4036f027-8e90-49c0-<wbr>8ca5-3ddb8d586916&#39;, vds=&#39;Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:32,340 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler10) [] Failure to refresh host &#39;rhvserv-03&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:32,362 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [4cf3eed0] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;0d0cd690-b64a-42bb-<wbr>a167-43fcedd634e4&#39;, vds=&#39;Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:32,362 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler9) [4cf3eed0] Failure to refresh host &#39;rhvserv-04&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:35,342 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.<wbr>168.93.213</a><br>2017-05-11 17:28:35,370 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.<wbr>168.93.214</a><br>2017-05-11 17:28:38,349 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [1efe65a7] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;4036f027-8e90-49c0-<wbr>8ca5-3ddb8d586916&#39;, vds=&#39;Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:38,349 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler7) [1efe65a7] Failure to refresh host &#39;rhvserv-03&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:38,374 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;0d0cd690-b64a-42bb-<wbr>a167-43fcedd634e4&#39;, vds=&#39;Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:38,375 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh host &#39;rhvserv-04&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:41,352 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.<wbr>168.93.213</a><br>2017-05-11 17:28:41,354 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [4cf3eed0] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;4036f027-8e90-49c0-<wbr>8ca5-3ddb8d586916&#39;, vds=&#39;Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:41,354 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler9) [4cf3eed0] Failure to refresh host &#39;rhvserv-03&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:41,377 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.<wbr>168.93.214</a><br>2017-05-11 17:28:41,380 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [1efe65a7] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;0d0cd690-b64a-42bb-<wbr>a167-43fcedd634e4&#39;, vds=&#39;Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:41,380 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler7) [1efe65a7] Failure to refresh host &#39;rhvserv-04&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:44,358 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.<wbr>168.93.213</a><br>2017-05-11 17:28:44,382 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.<wbr>168.93.214</a><br>2017-05-11 17:28:47,362 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [62006eaa] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;4036f027-8e90-49c0-<wbr>8ca5-3ddb8d586916&#39;, vds=&#39;Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:47,363 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler1) [62006eaa] Failure to refresh host &#39;rhvserv-03&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:47,388 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [1efe65a7] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;0d0cd690-b64a-42bb-<wbr>a167-43fcedd634e4&#39;, vds=&#39;Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:47,388 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler7) [1efe65a7] Failure to refresh host &#39;rhvserv-04&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:50,365 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.<wbr>168.93.213</a><br>2017-05-11 17:28:50,368 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler5) [48bc69cd] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;4036f027-8e90-49c0-<wbr>8ca5-3ddb8d586916&#39;, vds=&#39;Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:50,368 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler5) [48bc69cd] Failure to refresh host &#39;rhvserv-03&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:50,390 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.<wbr>168.93.214</a><br>2017-05-11 17:28:50,394 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [590af365] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;0d0cd690-b64a-42bb-<wbr>a167-43fcedd634e4&#39;, vds=&#39;Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:50,394 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler2) [590af365] Failure to refresh host &#39;rhvserv-04&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:53,371 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.<wbr>168.93.213</a><br>2017-05-11 17:28:53,396 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.<wbr>168.93.214</a><br>2017-05-11 17:28:56,379 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;4036f027-8e90-49c0-<wbr>8ca5-3ddb8d586916&#39;, vds=&#39;Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:56,379 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler10) [] Failure to refresh host &#39;rhvserv-03&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:56,402 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler5) [48bc69cd] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;0d0cd690-b64a-42bb-<wbr>a167-43fcedd634e4&#39;, vds=&#39;Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:56,403 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler5) [48bc69cd] Failure to refresh host &#39;rhvserv-04&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:56,488 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetAllVmStatsVDSCommand] (DefaultQuartzScheduler6) [] Command &#39;GetAllVmStatsVDSCommand(<wbr>HostName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;0d0cd690-b64a-42bb-<wbr>a167-43fcedd634e4&#39;, vds=&#39;Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]&#39;}<wbr>)&#39; execution failed: VDSGenericException: VDSNetworkException: Vds timeout occured<br>2017-05-11 17:28:56,488 INFO  [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>PollVmStatsRefresher] (DefaultQuartzScheduler6) [] Failed to fetch vms info for host &#39;rhvserv-04&#39; - skipping VMs monitoring.<br>2017-05-11 17:28:59,382 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.<wbr>168.93.213</a><br>2017-05-11 17:28:59,384 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;4036f027-8e90-49c0-<wbr>8ca5-3ddb8d586916&#39;, vds=&#39;Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:59,384 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler10) [] Failure to refresh host &#39;rhvserv-03&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:59,404 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.<wbr>168.93.214</a><br>2017-05-11 17:28:59,408 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;0d0cd690-b64a-42bb-<wbr>a167-43fcedd634e4&#39;, vds=&#39;Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:59,408 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler6) [] Failure to refresh host &#39;rhvserv-04&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:02,386 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.<wbr>168.93.213</a><br>2017-05-11 17:29:02,409 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.<wbr>168.93.214</a><br>2017-05-11 17:29:02,490 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetAllVmStatsVDSCommand] (DefaultQuartzScheduler8) [56fb4c82] Command &#39;GetAllVmStatsVDSCommand(<wbr>HostName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;4036f027-8e90-49c0-<wbr>8ca5-3ddb8d586916&#39;, vds=&#39;Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]&#39;}<wbr>)&#39; execution failed: VDSGenericException: VDSNetworkException: Vds timeout occured<br>2017-05-11 17:29:02,490 INFO  [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>PollVmStatsRefresher] (DefaultQuartzScheduler8) [56fb4c82] Failed to fetch vms info for host &#39;rhvserv-03&#39; - skipping VMs monitoring.<br>2017-05-11 17:29:05,393 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;4036f027-8e90-49c0-<wbr>8ca5-3ddb8d586916&#39;, vds=&#39;Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:05,393 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh host &#39;rhvserv-03&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:05,415 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;0d0cd690-b64a-42bb-<wbr>a167-43fcedd634e4&#39;, vds=&#39;Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:05,415 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler10) [] Failure to refresh host &#39;rhvserv-04&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:08,395 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.<wbr>168.93.213</a><br>2017-05-11 17:29:08,398 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [56fb4c82] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;4036f027-8e90-49c0-<wbr>8ca5-3ddb8d586916&#39;, vds=&#39;Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:08,398 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler8) [56fb4c82] Failure to refresh host &#39;rhvserv-03&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:08,417 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.<wbr>168.93.214</a><br>2017-05-11 17:29:08,420 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [590af365] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;0d0cd690-b64a-42bb-<wbr>a167-43fcedd634e4&#39;, vds=&#39;Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:08,420 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler2) [590af365] Failure to refresh host &#39;rhvserv-04&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:11,402 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.<wbr>168.93.213</a><br>2017-05-11 17:29:11,423 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.<wbr>168.93.214</a><br>2017-05-11 17:29:14,409 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;4036f027-8e90-49c0-<wbr>8ca5-3ddb8d586916&#39;, vds=&#39;Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:14,409 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh host &#39;rhvserv-03&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:14,430 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler5) [48bc69cd] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;0d0cd690-b64a-42bb-<wbr>a167-43fcedd634e4&#39;, vds=&#39;Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:14,430 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler5) [48bc69cd] Failure to refresh host &#39;rhvserv-04&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:17,411 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.<wbr>168.93.213</a><br>2017-05-11 17:29:17,414 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;4036f027-8e90-49c0-<wbr>8ca5-3ddb8d586916&#39;, vds=&#39;Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:17,414 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler6) [] Failure to refresh host &#39;rhvserv-03&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:17,432 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.<wbr>168.93.214</a><br>2017-05-11 17:29:17,436 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [] Command &#39;GetCapabilitiesVDSCommand(<wbr>HostName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync=&#39;true&#39;, hostId=&#39;0d0cd690-b64a-42bb-<wbr>a167-43fcedd634e4&#39;, vds=&#39;Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]&#39;}<wbr>)&#39; execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:17,436 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.monitoring.<wbr>HostMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh host &#39;rhvserv-04&#39; runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:17,491 INFO  [org.ovirt.vdsm.jsonrpc.<wbr>client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.<wbr>168.93.213</a></div><div><br></div><div><br></div><div>vdsmd logs<br></div><div><br></div><div><br></div><div>Thread-113::ERROR::2017-05-11 17:41:33,734::sdc::146::<wbr>Storage.StorageDomainCache::(_<wbr>findDomain) domain 5b978dda-d1ef-46fe-9996-<wbr>20aee42cf303 not found<br>Traceback (most recent call last):<br>  File &quot;/usr/share/vdsm/storage/sdc.<wbr>py&quot;, line 144, in _findDomain<br>    dom = findMethod(sdUUID)<br>  File &quot;/usr/share/vdsm/storage/sdc.<wbr>py&quot;, line 174, in _findUnfetchedDomain<br>    raise se.StorageDomainDoesNotExist(<wbr>sdUUID)<br>StorageDomainDoesNotExist: Storage domain does not exist: (u&#39;5b978dda-d1ef-46fe-9996-<wbr>20aee42cf303&#39;,)<br>Thread-113::ERROR::2017-05-11 17:41:33,735::monitor::328::<wbr>Storage.Monitor::(_setupLoop) Setting up monitor for 5b978dda-d1ef-46fe-9996-<wbr>20aee42cf303 failed<br>Traceback (most recent call last):<br>  File &quot;/usr/share/vdsm/storage/<wbr>monitor.py&quot;, line 325, in _setupLoop<br>    self._setupMonitor()<br>  File &quot;/usr/share/vdsm/storage/<wbr>monitor.py&quot;, line 348, in _setupMonitor<br>    self._produceDomain()<br>  File &quot;/usr/lib/python2.7/site-<wbr>packages/vdsm/utils.py&quot;, line 405, in wrapper<br>    value = meth(self, *a, **kw)<br>  File &quot;/usr/share/vdsm/storage/<wbr>monitor.py&quot;, line 366, in _produceDomain<br>    self.domain = sdCache.produce(self.sdUUID)<br>  File &quot;/usr/share/vdsm/storage/sdc.<wbr>py&quot;, line 101, in produce<br>    domain.getRealDomain()<br>  File &quot;/usr/share/vdsm/storage/sdc.<wbr>py&quot;, line 53, in getRealDomain<br>    return self._cache._realProduce(self.<wbr>_sdUUID)<br>  File &quot;/usr/share/vdsm/storage/sdc.<wbr>py&quot;, line 125, in _realProduce<br>    domain = self._findDomain(sdUUID)<br>  File &quot;/usr/share/vdsm/storage/sdc.<wbr>py&quot;, line 144, in _findDomain<br>    dom = findMethod(sdUUID)<br>  File &quot;/usr/share/vdsm/storage/sdc.<wbr>py&quot;, line 174, in _findUnfetchedDomain<br>    raise se.StorageDomainDoesNotExist(<wbr>sdUUID)<br>StorageDomainDoesNotExist: Storage domain does not exist: (u&#39;5b978dda-d1ef-46fe-9996-<wbr>20aee42cf303&#39;,)<br>Thread-112::DEBUG::2017-05-11 17:41:33,783::lvm::288::<wbr>Storage.Misc.excCmd::(cmd) SUCCESS: &lt;err&gt; = &#39;&#39;; &lt;rc&gt; = 0<br>Thread-112::INFO::2017-05-11 17:41:33,783::sd::604::<wbr>Storage.StorageDomain::(_<wbr>registerResourceNamespaces) Resource namespace edd229cb-b72f-4988-8c10-<wbr>d83c84ef4a8a_imageNS already registered<br>Thread-112::INFO::2017-05-11 17:41:33,783::sd::612::<wbr>Storage.StorageDomain::(_<wbr>registerResourceNamespaces) Resource namespace edd229cb-b72f-4988-8c10-<wbr>d83c84ef4a8a_volumeNS already registered<br>Thread-112::INFO::2017-05-11 17:41:33,783::blockSD::846::<wbr>Storage.StorageDomain::(_<wbr>registerResourceNamespaces) Resource namespace edd229cb-b72f-4988-8c10-<wbr>d83c84ef4a8a_lvmActivationNS already registered<br>Thread-112::DEBUG::2017-05-11 17:41:33,784::lvm::288::<wbr>Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-15 /usr/bin/sudo -n /usr/sbin/lvm vgck --config &#39; devices { preferred_names = [&quot;^/dev/mapper/&quot;] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ &#39;\&#39;&#39;a|/dev/mapper/<wbr>36782bcb00073e33200002d3858fa1<wbr>a81|&#39;\&#39;&#39;, &#39;\&#39;&#39;r|.*|&#39;\&#39;&#39; ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } &#39; edd229cb-b72f-4988-8c10-<wbr>d83c84ef4a8a (cwd None)<br>jsonrpc.Executor/6::DEBUG::<wbr>2017-05-11 17:41:33,792::lvm::288::<wbr>Storage.Misc.excCmd::(cmd) SUCCESS: &lt;err&gt; = &#39;&#39;; &lt;rc&gt; = 0<br>jsonrpc.Executor/6::INFO::<wbr>2017-05-11 17:41:33,792::logUtils::52::<wbr>dispatcher::(wrapper) Run and protect: getStorageDomainStats, Return response: {&#39;stats&#39;: {&#39;mdasize&#39;: &#39;134217728&#39;, &#39;mdathreshold&#39;: True, &#39;mdavalid&#39;: True, &#39;diskfree&#39;: &#39;498484641792&#39;, &#39;disktotal&#39;: &#39;557943095296&#39;, &#39;mdafree&#39;: &#39;67102208&#39;}}<br>jsonrpc.Executor/6::DEBUG::<wbr>2017-05-11 17:41:33,793::task::1193::<wbr>Storage.TaskManager.Task::(<wbr>prepare) Task=`234cbec4-a422-47c1-a50a-<wbr>243526aeddc7`::finished: {&#39;stats&#39;: {&#39;mdasize&#39;: &#39;134217728&#39;, &#39;mdathreshold&#39;: True, &#39;mdavalid&#39;: True, &#39;diskfree&#39;: &#39;498484641792&#39;, &#39;disktotal&#39;: &#39;557943095296&#39;, &#39;mdafree&#39;: &#39;67102208&#39;}}<br>jsonrpc.Executor/6::DEBUG::<wbr>2017-05-11 17:41:33,793::task::597::<wbr>Storage.TaskManager.Task::(_<wbr>updateState) Task=`234cbec4-a422-47c1-a50a-<wbr>243526aeddc7`::moving from state preparing -&gt; state finished<br>jsonrpc.Executor/6::DEBUG::<wbr>2017-05-11 17:41:33,793::resourceManager:<wbr>:952::Storage.ResourceManager.<wbr>Owner::(releaseAll) Owner.releaseAll requests {} resources {u&#39;Storage.595de2cf-89ba-407b-<wbr>aae5-d0a7a0656ba1&#39;: &lt; ResourceRef &#39;Storage.595de2cf-89ba-407b-<wbr>aae5-d0a7a0656ba1&#39;, isValid: &#39;True&#39; obj: &#39;None&#39;&gt;}<br>jsonrpc.Executor/6::DEBUG::<wbr>2017-05-11 17:41:33,793::resourceManager:<wbr>:989::Storage.ResourceManager.<wbr>Owner::(cancelAll) Owner.cancelAll requests {}<br>jsonrpc.Executor/6::DEBUG::<wbr>2017-05-11 17:41:33,793::resourceManager:<wbr>:628::Storage.ResourceManager:<wbr>:(releaseResource) Trying to release resource &#39;Storage.595de2cf-89ba-407b-<wbr>aae5-d0a7a0656ba1&#39;<br>jsonrpc.Executor/6::DEBUG::<wbr>2017-05-11 17:41:33,793::resourceManager:<wbr>:647::Storage.ResourceManager:<wbr>:(releaseResource) Released resource &#39;Storage.595de2cf-89ba-407b-<wbr>aae5-d0a7a0656ba1&#39; (0 active users)<br>jsonrpc.Executor/6::DEBUG::<wbr>2017-05-11 17:41:33,793::resourceManager:<wbr>:653::Storage.ResourceManager:<wbr>:(releaseResource) Resource &#39;Storage.595de2cf-89ba-407b-<wbr>aae5-d0a7a0656ba1&#39; is free, finding out if anyone is waiting for it.<br>jsonrpc.Executor/6::DEBUG::<wbr>2017-05-11 17:41:33,793::resourceManager:<wbr>:661::Storage.ResourceManager:<wbr>:(releaseResource) No one is waiting for resource &#39;Storage.595de2cf-89ba-407b-<wbr>aae5-d0a7a0656ba1&#39;, Clearing records.<br>jsonrpc.Executor/6::DEBUG::<wbr>2017-05-11 17:41:33,793::task::995::<wbr>Storage.TaskManager.Task::(_<wbr>decref) Task=`234cbec4-a422-47c1-a50a-<wbr>243526aeddc7`::ref 0 aborting False<br>jsonrpc.Executor/6::DEBUG::<wbr>2017-05-11 17:41:33,794::__init__::555::<wbr>jsonrpc.JsonRpcServer::(_<wbr>handle_request) Return &#39;StorageDomain.getStats&#39; in bridge with {&#39;mdasize&#39;: &#39;134217728&#39;, &#39;mdathreshold&#39;: True, &#39;mdavalid&#39;: True, &#39;diskfree&#39;: &#39;498484641792&#39;, &#39;disktotal&#39;: &#39;557943095296&#39;, &#39;mdafree&#39;: &#39;67102208&#39;}<br>jsonrpc.Executor/6::INFO::<wbr>2017-05-11 17:41:33,794::__init__::513::<wbr>jsonrpc.JsonRpcServer::(_<wbr>serveRequest) RPC call StorageDomain.getStats succeeded in 1.05 seconds<br>JsonRpc (StompReactor)::ERROR::2017-<wbr>05-11 17:41:33,816::betterAsyncore::<wbr>113::vds.dispatcher::(recv) SSL error during reading data: unexpected eof<br>Thread-112::DEBUG::2017-05-11 17:41:33,846::lvm::288::<wbr>Storage.Misc.excCmd::(cmd) SUCCESS: &lt;err&gt; = &#39;&#39;; &lt;rc&gt; = 0<br>Thread-12::DEBUG::2017-05-11 17:41:34,913::check::296::<wbr>storage.check::(_start_<wbr>process) START check &#39;/dev/595de2cf-89ba-407b-aae5-<wbr>d0a7a0656ba1/metadata&#39; cmd=[&#39;/usr/bin/taskset&#39;, &#39;--cpu-list&#39;, &#39;0-15&#39;, &#39;/usr/bin/dd&#39;, &#39;if=/dev/595de2cf-89ba-407b-<wbr>aae5-d0a7a0656ba1/metadata&#39;, &#39;of=/dev/null&#39;, &#39;bs=4096&#39;, &#39;count=1&#39;, &#39;iflag=direct&#39;] delay=0.00<br>Thread-12::DEBUG::2017-05-11 17:41:34,954::asyncevent::564:<wbr>:storage.asyncevent::(reap) Process &lt;cpopen.CPopen object at 0x1b15b90&gt; terminated (count=1)<br>Thread-12::DEBUG::2017-05-11 17:41:34,954::check::327::<wbr>storage.check::(_check_<wbr>completed) FINISH check &#39;/dev/595de2cf-89ba-407b-aae5-<wbr>d0a7a0656ba1/metadata&#39; rc=0 err=bytearray(b&#39;1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000537649 s, 7.6 MB/s\n&#39;) elapsed=0.05<br>Reactor thread::INFO::2017-05-11 17:41:35,823::<wbr>protocoldetector::76::<wbr>ProtocolDetector.AcceptorImpl:<wbr>:(handle_accept) Accepted connection from ::1:49348<br>Reactor thread::DEBUG::2017-05-11 17:41:35,827::<wbr>protocoldetector::92::<wbr>ProtocolDetector.Detector::(__<wbr>init__) Using required_size=11<br>Reactor thread::INFO::2017-05-11 17:41:35,828::<wbr>protocoldetector::128::<wbr>ProtocolDetector.Detector::(<wbr>handle_read) Detected protocol stomp from ::1:49348<br>Reactor thread::INFO::2017-05-11 17:41:35,828::stompreactor::<wbr>101::Broker.StompAdapter::(_<wbr>cmd_connect) Processing CONNECT request<br>Reactor thread::DEBUG::2017-05-11 17:41:35,829::stompreactor::<wbr>492::protocoldetector.<wbr>StompDetector::(handle_socket) Stomp detected from (&#39;::1&#39;, 49348)<br>JsonRpc (StompReactor)::INFO::2017-05-<wbr>11 17:41:35,829::stompreactor::<wbr>128::Broker.StompAdapter::(_<wbr>cmd_subscribe) Subscribe command received</div></div></div><br>______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br>
<br></blockquote></div></div>