<div dir="ltr"><div><div>From provided log snippets I see that engine was unable to connect to both of your hosts. Is the network OK?<br><br></div>In vdsm log it seems that there is an issue with you storage domain.<br><br></div>I am not able to say anything about the issue you are facing without having more complete logs.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, May 11, 2017 at 6:28 PM, Oved Ourfali <span dir="ltr"><<a href="mailto:oourfali@redhat.com" target="_blank">oourfali@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto">What 4.0 version do you use? <div dir="auto">Can you attach complete engine and vdsm logs? </div><div dir="auto">CC-ing Piotr as well. </div><div dir="auto"><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On May 11, 2017 6:44 PM, "Alexis HAUSER" <<a href="mailto:alexis.hauser@imt-atlantique.fr" target="_blank">alexis.hauser@imt-atlantique.<wbr>fr</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:times new roman,new york,times,serif;font-size:12pt;color:#000000"><div>After rebooting the manager VM, hosts are connecting/non responsive and data domains inactive. Here are the engine and vdsmd logs. Any ideas ?<br></div><div><br></div><div><br></div><div><br></div><div><br></div><div>Engine logs : <br></div><div><br></div><div><br></div><div><br></div><div>2017-05-11 17:28:09,302 WARN [org.ovirt.engine.core.dal.dbb<wbr>roker.auditloghandling.AuditLo<wbr>gDirector] (DefaultQuartzScheduler5) [55f1aab5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host rhvserv-05.<br>2017-05-11 17:28:09,346 INFO [org.ovirt.engine.core.bll.Han<wbr>dleVdsVersionCommand] (DefaultQuartzScheduler5) [48bc69cd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 04565f10-9abf-4709-9445-9dc6ed<wbr>97e136 Type: VDS<br>2017-05-11 17:28:09,349 WARN [org.ovirt.engine.core.vdsbrok<wbr>er.VdsManager] (org.ovirt.thread.pool-6-threa<wbr>d-27) [639977e4] Host 'rhvserv-05' is not responding.<br>2017-05-11 17:28:09,364 WARN [org.ovirt.engine.core.dal.dbb<wbr>roker.auditloghandling.AuditLo<wbr>gDirector] (org.ovirt.thread.pool-6-threa<wbr>d-27) [639977e4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rhvserv-05 is not responding. Host cannot be fenced automatically because power management for the host is disabled.<br>2017-05-11 17:28:11,299 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler3) [c0e6a2e] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='4036f027-8e90-49c0-8ca<wbr>5-3ddb8d586916', vds='Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:11,299 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [c0e6a2e] Failure to refresh host 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:11,327 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.16<wbr>8.93.214</a><br>2017-05-11 17:28:12,484 INFO [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetHardwareInfoVD<wbr>SCommand] (DefaultQuartzScheduler3) [c0e6a2e] START, GetHardwareInfoVDSCommand(Host<wbr>Name = rhvserv-05, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='04565f10-9abf-4709-944<wbr>5-9dc6ed97e136', vds='Host[rhvserv-05,04565f10-<wbr>9abf-4709-9445-9dc6ed97e136]'}<wbr>), log id: f807ece<br>2017-05-11 17:28:12,487 INFO [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetHardwareInfoVD<wbr>SCommand] (DefaultQuartzScheduler3) [c0e6a2e] FINISH, GetHardwareInfoVDSCommand, log id: f807ece<br>2017-05-11 17:28:12,532 INFO [org.ovirt.engine.core.bll.Han<wbr>dleVdsCpuFlagsOrClusterChanged<wbr>Command] (DefaultQuartzScheduler3) [4e882ea0] Running command: HandleVdsCpuFlagsOrClusterChan<wbr>gedCommand internal: true. Entities affected : ID: 04565f10-9abf-4709-9445-9dc6ed<wbr>97e136 Type: VDS<br>2017-05-11 17:28:12,539 INFO [org.ovirt.engine.core.bll.Ini<wbr>tVdsOnUpCommand] (DefaultQuartzScheduler3) [75f25b35] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 58f8df36-019f-02bc-00e7-000000<wbr>000023 Type: StoragePool<br>2017-05-11 17:28:12,545 INFO [org.ovirt.engine.core.bll.sto<wbr>rage.pool.ConnectHostToStorage<wbr>PoolServersCommand] (DefaultQuartzScheduler3) [46cc3f58] Running command: ConnectHostToStoragePoolServer<wbr>sCommand internal: true. Entities affected : ID: 58f8df36-019f-02bc-00e7-000000<wbr>000023 Type: StoragePool<br>2017-05-11 17:28:12,556 INFO [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.ConnectStorageSer<wbr>verVDSCommand] (DefaultQuartzScheduler3) [46cc3f58] START, ConnectStorageServerVDSCommand<wbr>(HostName = rhvserv-05, StorageServerConnectionManagem<wbr>entVDSParameters:{runAsync='tr<wbr>ue', hostId='04565f10-9abf-4709-944<wbr>5-9dc6ed97e136', storagePoolId='58f8df36-019f-0<wbr>2bc-00e7-000000000023', storageType='ISCSI', connectionList='[StorageServer<wbr>Connections:{id='10c0528b-<wbr>f08d-4d1d-8c63-8a05fd9d58b9', connection='10.35.21.1', iqn='iqn.1984-05.com.dell:powe<wbr>rvault.md3200i.6782bcb00073e33<wbr>2000000004edde164', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1beb27b6<br>2017-05-11 17:28:13,031 INFO [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.ConnectStorageSer<wbr>verVDSCommand] (DefaultQuartzScheduler3) [46cc3f58] FINISH, ConnectStorageServerVDSCommand<wbr>, return: {10c0528b-f08d-4d1d-8c63-8a05f<wbr>d9d58b9=0}, log id: 1beb27b6<br>2017-05-11 17:28:13,032 INFO [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.ConnectStorageSer<wbr>verVDSCommand] (DefaultQuartzScheduler3) [46cc3f58] START, ConnectStorageServerVDSCommand<wbr>(HostName = rhvserv-05, StorageServerConnectionManagem<wbr>entVDSParameters:{runAsync='tr<wbr>ue', hostId='04565f10-9abf-4709-944<wbr>5-9dc6ed97e136', storagePoolId='58f8df36-019f-0<wbr>2bc-00e7-000000000023', storageType='NFS', connectionList='[StorageServer<wbr>Connections:{id='e604d0d2-<wbr>0810-4c25-b9ed-610f9923cb1a', connection='nfsserv-01:/nfs/ex<wbr>port', iqn='null', vfsType='null', mountOptions='null', nfsVersion='V3', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='<wbr>2c1aba4b-786b-4c76-a02f-d9e09e<wbr>8afff4', connection='nfsserv-01:/nfs/is<wbr>o', iqn='null', vfsType='null', mountOptions='null', nfsVersion='V3', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 15c54cd7<br>2017-05-11 17:28:14,301 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.16<wbr>8.93.213</a><br>2017-05-11 17:28:14,304 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler1) [62006eaa] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='4036f027-8e90-49c0-8ca<wbr>5-3ddb8d586916', vds='Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:14,304 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [62006eaa] Failure to refresh host 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:14,334 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler7) [1efe65a7] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='0d0cd690-b64a-42bb-a16<wbr>7-43fcedd634e4', vds='Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:14,334 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [1efe65a7] Failure to refresh host 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:17,308 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.16<wbr>8.93.213</a><br>2017-05-11 17:28:17,336 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.16<wbr>8.93.214</a><br>2017-05-11 17:28:17,340 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler4) [] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='0d0cd690-b64a-42bb-a16<wbr>7-43fcedd634e4', vds='Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:17,340 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh host 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:20,315 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler1) [62006eaa] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='4036f027-8e90-49c0-8ca<wbr>5-3ddb8d586916', vds='Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:20,315 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [62006eaa] Failure to refresh host 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:20,343 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.16<wbr>8.93.214</a><br>2017-05-11 17:28:23,316 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.16<wbr>8.93.213</a><br>2017-05-11 17:28:23,325 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler4) [] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='4036f027-8e90-49c0-8ca<wbr>5-3ddb8d586916', vds='Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:23,325 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh host 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:23,348 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler7) [1efe65a7] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='0d0cd690-b64a-42bb-a16<wbr>7-43fcedd634e4', vds='Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:23,348 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [1efe65a7] Failure to refresh host 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:26,329 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.16<wbr>8.93.213</a><br>2017-05-11 17:28:26,350 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.16<wbr>8.93.214</a><br>2017-05-11 17:28:26,354 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler7) [1efe65a7] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='0d0cd690-b64a-42bb-a16<wbr>7-43fcedd634e4', vds='Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:26,354 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [1efe65a7] Failure to refresh host 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:29,335 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler4) [] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='4036f027-8e90-49c0-8ca<wbr>5-3ddb8d586916', vds='Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:29,335 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh host 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:29,356 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.16<wbr>8.93.214</a><br>2017-05-11 17:28:32,337 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.16<wbr>8.93.213</a><br>2017-05-11 17:28:32,340 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler10) [] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='4036f027-8e90-49c0-8ca<wbr>5-3ddb8d586916', vds='Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:32,340 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [] Failure to refresh host 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:32,362 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler9) [4cf3eed0] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='0d0cd690-b64a-42bb-a16<wbr>7-43fcedd634e4', vds='Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:32,362 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [4cf3eed0] Failure to refresh host 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:35,342 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.16<wbr>8.93.213</a><br>2017-05-11 17:28:35,370 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.16<wbr>8.93.214</a><br>2017-05-11 17:28:38,349 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler7) [1efe65a7] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='4036f027-8e90-49c0-8ca<wbr>5-3ddb8d586916', vds='Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:38,349 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [1efe65a7] Failure to refresh host 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:38,374 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler4) [] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='0d0cd690-b64a-42bb-a16<wbr>7-43fcedd634e4', vds='Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:38,375 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh host 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:41,352 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.16<wbr>8.93.213</a><br>2017-05-11 17:28:41,354 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler9) [4cf3eed0] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='4036f027-8e90-49c0-8ca<wbr>5-3ddb8d586916', vds='Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:41,354 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [4cf3eed0] Failure to refresh host 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:41,377 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.16<wbr>8.93.214</a><br>2017-05-11 17:28:41,380 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler7) [1efe65a7] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='0d0cd690-b64a-42bb-a16<wbr>7-43fcedd634e4', vds='Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:41,380 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [1efe65a7] Failure to refresh host 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:44,358 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.16<wbr>8.93.213</a><br>2017-05-11 17:28:44,382 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.16<wbr>8.93.214</a><br>2017-05-11 17:28:47,362 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler1) [62006eaa] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='4036f027-8e90-49c0-8ca<wbr>5-3ddb8d586916', vds='Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:47,363 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [62006eaa] Failure to refresh host 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:47,388 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler7) [1efe65a7] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='0d0cd690-b64a-42bb-a16<wbr>7-43fcedd634e4', vds='Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:47,388 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [1efe65a7] Failure to refresh host 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:50,365 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.16<wbr>8.93.213</a><br>2017-05-11 17:28:50,368 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler5) [48bc69cd] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='4036f027-8e90-49c0-8ca<wbr>5-3ddb8d586916', vds='Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:50,368 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [48bc69cd] Failure to refresh host 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:50,390 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.16<wbr>8.93.214</a><br>2017-05-11 17:28:50,394 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler2) [590af365] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='0d0cd690-b64a-42bb-a16<wbr>7-43fcedd634e4', vds='Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:50,394 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [590af365] Failure to refresh host 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:53,371 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.16<wbr>8.93.213</a><br>2017-05-11 17:28:53,396 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.16<wbr>8.93.214</a><br>2017-05-11 17:28:56,379 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler10) [] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='4036f027-8e90-49c0-8ca<wbr>5-3ddb8d586916', vds='Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:56,379 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [] Failure to refresh host 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:56,402 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler5) [48bc69cd] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='0d0cd690-b64a-42bb-a16<wbr>7-43fcedd634e4', vds='Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:56,403 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [48bc69cd] Failure to refresh host 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:56,488 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetAllVmStatsVDSC<wbr>ommand] (DefaultQuartzScheduler6) [] Command 'GetAllVmStatsVDSCommand(HostN<wbr>ame = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='0d0cd690-b64a-42bb-a16<wbr>7-43fcedd634e4', vds='Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]'}<wbr>)' execution failed: VDSGenericException: VDSNetworkException: Vds timeout occured<br>2017-05-11 17:28:56,488 INFO [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.PollVmStatsRefre<wbr>sher] (DefaultQuartzScheduler6) [] Failed to fetch vms info for host 'rhvserv-04' - skipping VMs monitoring.<br>2017-05-11 17:28:59,382 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.16<wbr>8.93.213</a><br>2017-05-11 17:28:59,384 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler10) [] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='4036f027-8e90-49c0-8ca<wbr>5-3ddb8d586916', vds='Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:59,384 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [] Failure to refresh host 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:59,404 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.16<wbr>8.93.214</a><br>2017-05-11 17:28:59,408 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler6) [] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='0d0cd690-b64a-42bb-a16<wbr>7-43fcedd634e4', vds='Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:28:59,408 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [] Failure to refresh host 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:02,386 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.16<wbr>8.93.213</a><br>2017-05-11 17:29:02,409 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.16<wbr>8.93.214</a><br>2017-05-11 17:29:02,490 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetAllVmStatsVDSC<wbr>ommand] (DefaultQuartzScheduler8) [56fb4c82] Command 'GetAllVmStatsVDSCommand(HostN<wbr>ame = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='4036f027-8e90-49c0-8ca<wbr>5-3ddb8d586916', vds='Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]'}<wbr>)' execution failed: VDSGenericException: VDSNetworkException: Vds timeout occured<br>2017-05-11 17:29:02,490 INFO [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.PollVmStatsRefre<wbr>sher] (DefaultQuartzScheduler8) [56fb4c82] Failed to fetch vms info for host 'rhvserv-03' - skipping VMs monitoring.<br>2017-05-11 17:29:05,393 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler4) [] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='4036f027-8e90-49c0-8ca<wbr>5-3ddb8d586916', vds='Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:05,393 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh host 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:05,415 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler10) [] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='0d0cd690-b64a-42bb-a16<wbr>7-43fcedd634e4', vds='Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:05,415 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [] Failure to refresh host 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:08,395 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.16<wbr>8.93.213</a><br>2017-05-11 17:29:08,398 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler8) [56fb4c82] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='4036f027-8e90-49c0-8ca<wbr>5-3ddb8d586916', vds='Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:08,398 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [56fb4c82] Failure to refresh host 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:08,417 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.16<wbr>8.93.214</a><br>2017-05-11 17:29:08,420 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler2) [590af365] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='0d0cd690-b64a-42bb-a16<wbr>7-43fcedd634e4', vds='Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:08,420 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [590af365] Failure to refresh host 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:11,402 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.16<wbr>8.93.213</a><br>2017-05-11 17:29:11,423 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.16<wbr>8.93.214</a><br>2017-05-11 17:29:14,409 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler4) [] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='4036f027-8e90-49c0-8ca<wbr>5-3ddb8d586916', vds='Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:14,409 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh host 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:14,430 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler5) [48bc69cd] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='0d0cd690-b64a-42bb-a16<wbr>7-43fcedd634e4', vds='Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:14,430 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [48bc69cd] Failure to refresh host 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:17,411 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.16<wbr>8.93.213</a><br>2017-05-11 17:29:17,414 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler6) [] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-03, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='4036f027-8e90-49c0-8ca<wbr>5-3ddb8d586916', vds='Host[rhvserv-03,4036f027-<wbr>8e90-49c0-8ca5-3ddb8d586916]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:17,414 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [] Failure to refresh host 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:17,432 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-04.mydomain.com/192.168.93.214" target="_blank">rhvserv-04.mydomain.com/192.16<wbr>8.93.214</a><br>2017-05-11 17:29:17,436 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand] (DefaultQuartzScheduler4) [] Command 'GetCapabilitiesVDSCommand(Hos<wbr>tName = rhvserv-04, VdsIdAndVdsVDSCommandParameter<wbr>sBase:{runAsync='true', hostId='0d0cd690-b64a-42bb-a16<wbr>7-43fcedd634e4', vds='Host[rhvserv-04,0d0cd690-<wbr>b64a-42bb-a167-43fcedd634e4]'}<wbr>)' execution failed: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:17,436 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh host 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.<wbr>ClientConnectionException: Connection failed<br>2017-05-11 17:29:17,491 INFO [org.ovirt.vdsm.jsonrpc.client<wbr>.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to <a href="http://rhvserv-03.mydomain.com/192.168.93.213" target="_blank">rhvserv-03.mydomain.com/192.16<wbr>8.93.213</a></div><div><br></div><div><br></div><div>vdsmd logs<br></div><div><br></div><div><br></div><div>Thread-113::ERROR::2017-05-11 17:41:33,734::sdc::146::Storag<wbr>e.StorageDomainCache::(_findDo<wbr>main) domain 5b978dda-d1ef-46fe-9996-20aee4<wbr>2cf303 not found<br>Traceback (most recent call last):<br> File "/usr/share/vdsm/storage/sdc.p<wbr>y", line 144, in _findDomain<br> dom = findMethod(sdUUID)<br> File "/usr/share/vdsm/storage/sdc.p<wbr>y", line 174, in _findUnfetchedDomain<br> raise se.StorageDomainDoesNotExist(s<wbr>dUUID)<br>StorageDomainDoesNotExist: Storage domain does not exist: (u'5b978dda-d1ef-46fe-9996-20a<wbr>ee42cf303',)<br>Thread-113::ERROR::2017-05-11 17:41:33,735::monitor::328::St<wbr>orage.Monitor::(_setupLoop) Setting up monitor for 5b978dda-d1ef-46fe-9996-20aee4<wbr>2cf303 failed<br>Traceback (most recent call last):<br> File "/usr/share/vdsm/storage/monit<wbr>or.py", line 325, in _setupLoop<br> self._setupMonitor()<br> File "/usr/share/vdsm/storage/monit<wbr>or.py", line 348, in _setupMonitor<br> self._produceDomain()<br> File "/usr/lib/python2.7/site-packa<wbr>ges/vdsm/utils.py", line 405, in wrapper<br> value = meth(self, *a, **kw)<br> File "/usr/share/vdsm/storage/monit<wbr>or.py", line 366, in _produceDomain<br> self.domain = sdCache.produce(self.sdUUID)<br> File "/usr/share/vdsm/storage/sdc.p<wbr>y", line 101, in produce<br> domain.getRealDomain()<br> File "/usr/share/vdsm/storage/sdc.p<wbr>y", line 53, in getRealDomain<br> return self._cache._realProduce(self.<wbr>_sdUUID)<br> File "/usr/share/vdsm/storage/sdc.p<wbr>y", line 125, in _realProduce<br> domain = self._findDomain(sdUUID)<br> File "/usr/share/vdsm/storage/sdc.p<wbr>y", line 144, in _findDomain<br> dom = findMethod(sdUUID)<br> File "/usr/share/vdsm/storage/sdc.p<wbr>y", line 174, in _findUnfetchedDomain<br> raise se.StorageDomainDoesNotExist(s<wbr>dUUID)<br>StorageDomainDoesNotExist: Storage domain does not exist: (u'5b978dda-d1ef-46fe-9996-20a<wbr>ee42cf303',)<br>Thread-112::DEBUG::2017-05-11 17:41:33,783::lvm::288::Storag<wbr>e.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0<br>Thread-112::INFO::2017-05-11 17:41:33,783::sd::604::Storage<wbr>.StorageDomain::(_registerReso<wbr>urceNamespaces) Resource namespace edd229cb-b72f-4988-8c10-d83c84<wbr>ef4a8a_imageNS already registered<br>Thread-112::INFO::2017-05-11 17:41:33,783::sd::612::Storage<wbr>.StorageDomain::(_registerReso<wbr>urceNamespaces) Resource namespace edd229cb-b72f-4988-8c10-d83c84<wbr>ef4a8a_volumeNS already registered<br>Thread-112::INFO::2017-05-11 17:41:33,783::blockSD::846::St<wbr>orage.StorageDomain::(_registe<wbr>rResourceNamespaces) Resource namespace edd229cb-b72f-4988-8c10-d83c84<wbr>ef4a8a_lvmActivationNS already registered<br>Thread-112::DEBUG::2017-05-11 17:41:33,784::lvm::288::Storag<wbr>e.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-15 /usr/bin/sudo -n /usr/sbin/lvm vgck --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/36782bcb0007<wbr>3e33200002d3858fa1a81|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' edd229cb-b72f-4988-8c10-d83c84<wbr>ef4a8a (cwd None)<br>jsonrpc.Executor/6::DEBUG::201<wbr>7-05-11 17:41:33,792::lvm::288::Storag<wbr>e.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0<br>jsonrpc.Executor/6::INFO::2017<wbr>-05-11 17:41:33,792::logUtils::52::di<wbr>spatcher::(wrapper) Run and protect: getStorageDomainStats, Return response: {'stats': {'mdasize': '134217728', 'mdathreshold': True, 'mdavalid': True, 'diskfree': '498484641792', 'disktotal': '557943095296', 'mdafree': '67102208'}}<br>jsonrpc.Executor/6::DEBUG::201<wbr>7-05-11 17:41:33,793::task::1193::Stor<wbr>age.TaskManager.Task::(prepare<wbr>) Task=`234cbec4-a422-47c1-a50a-<wbr>243526aeddc7`::finished: {'stats': {'mdasize': '134217728', 'mdathreshold': True, 'mdavalid': True, 'diskfree': '498484641792', 'disktotal': '557943095296', 'mdafree': '67102208'}}<br>jsonrpc.Executor/6::DEBUG::201<wbr>7-05-11 17:41:33,793::task::597::Stora<wbr>ge.TaskManager.Task::(_updateS<wbr>tate) Task=`234cbec4-a422-47c1-a50a-<wbr>243526aeddc7`::moving from state preparing -> state finished<br>jsonrpc.Executor/6::DEBUG::201<wbr>7-05-11 17:41:33,793::resourceManager:<wbr>:952::Storage.ResourceManager.<wbr>Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.595de2cf-89ba-407b-<wbr>aae5-d0a7a0656ba1': < ResourceRef 'Storage.595de2cf-89ba-407b-aa<wbr>e5-d0a7a0656ba1', isValid: 'True' obj: 'None'>}<br>jsonrpc.Executor/6::DEBUG::201<wbr>7-05-11 17:41:33,793::resourceManager:<wbr>:989::Storage.ResourceManager.<wbr>Owner::(cancelAll) Owner.cancelAll requests {}<br>jsonrpc.Executor/6::DEBUG::201<wbr>7-05-11 17:41:33,793::resourceManager:<wbr>:628::Storage.ResourceManager:<wbr>:(releaseResource) Trying to release resource 'Storage.595de2cf-89ba-407b-aa<wbr>e5-d0a7a0656ba1'<br>jsonrpc.Executor/6::DEBUG::201<wbr>7-05-11 17:41:33,793::resourceManager:<wbr>:647::Storage.ResourceManager:<wbr>:(releaseResource) Released resource 'Storage.595de2cf-89ba-407b-aa<wbr>e5-d0a7a0656ba1' (0 active users)<br>jsonrpc.Executor/6::DEBUG::201<wbr>7-05-11 17:41:33,793::resourceManager:<wbr>:653::Storage.ResourceManager:<wbr>:(releaseResource) Resource 'Storage.595de2cf-89ba-407b-aa<wbr>e5-d0a7a0656ba1' is free, finding out if anyone is waiting for it.<br>jsonrpc.Executor/6::DEBUG::201<wbr>7-05-11 17:41:33,793::resourceManager:<wbr>:661::Storage.ResourceManager:<wbr>:(releaseResource) No one is waiting for resource 'Storage.595de2cf-89ba-407b-aa<wbr>e5-d0a7a0656ba1', Clearing records.<br>jsonrpc.Executor/6::DEBUG::201<wbr>7-05-11 17:41:33,793::task::995::Stora<wbr>ge.TaskManager.Task::(_decref) Task=`234cbec4-a422-47c1-a50a-<wbr>243526aeddc7`::ref 0 aborting False<br>jsonrpc.Executor/6::DEBUG::201<wbr>7-05-11 17:41:33,794::__init__::555::j<wbr>sonrpc.JsonRpcServer::(_handle<wbr>_request) Return 'StorageDomain.getStats' in bridge with {'mdasize': '134217728', 'mdathreshold': True, 'mdavalid': True, 'diskfree': '498484641792', 'disktotal': '557943095296', 'mdafree': '67102208'}<br>jsonrpc.Executor/6::INFO::2017<wbr>-05-11 17:41:33,794::__init__::513::j<wbr>sonrpc.JsonRpcServer::(_serveR<wbr>equest) RPC call StorageDomain.getStats succeeded in 1.05 seconds<br>JsonRpc (StompReactor)::ERROR::2017-05<wbr>-11 17:41:33,816::betterAsyncore::<wbr>113::vds.dispatcher::(recv) SSL error during reading data: unexpected eof<br>Thread-112::DEBUG::2017-05-11 17:41:33,846::lvm::288::Storag<wbr>e.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0<br>Thread-12::DEBUG::2017-05-11 17:41:34,913::check::296::stor<wbr>age.check::(_start_process) START check '/dev/595de2cf-89ba-407b-aae5-<wbr>d0a7a0656ba1/metadata' cmd=['/usr/bin/taskset', '--cpu-list', '0-15', '/usr/bin/dd', 'if=/dev/595de2cf-89ba-407b-aa<wbr>e5-d0a7a0656ba1/metadata', 'of=/dev/null', 'bs=4096', 'count=1', 'iflag=direct'] delay=0.00<br>Thread-12::DEBUG::2017-05-11 17:41:34,954::asyncevent::564:<wbr>:storage.asyncevent::(reap) Process <cpopen.CPopen object at 0x1b15b90> terminated (count=1)<br>Thread-12::DEBUG::2017-05-11 17:41:34,954::check::327::stor<wbr>age.check::(_check_completed) FINISH check '/dev/595de2cf-89ba-407b-aae5-<wbr>d0a7a0656ba1/metadata' rc=0 err=bytearray(b'1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000537649 s, 7.6 MB/s\n') elapsed=0.05<br>Reactor thread::INFO::2017-05-11 17:41:35,823::protocoldetector<wbr>::76::ProtocolDetector.<wbr>AcceptorImpl::(handle_accept) Accepted connection from ::1:49348<br>Reactor thread::DEBUG::2017-05-11 17:41:35,827::protocoldetector<wbr>::92::ProtocolDetector.<wbr>Detector::(__init__) Using required_size=11<br>Reactor thread::INFO::2017-05-11 17:41:35,828::protocoldetector<wbr>::128::ProtocolDetector.<wbr>Detector::(handle_read) Detected protocol stomp from ::1:49348<br>Reactor thread::INFO::2017-05-11 17:41:35,828::stompreactor::10<wbr>1::Broker.StompAdapter::(_cmd_<wbr>connect) Processing CONNECT request<br>Reactor thread::DEBUG::2017-05-11 17:41:35,829::stompreactor::49<wbr>2::protocoldetector.StompDetec<wbr>tor::(handle_socket) Stomp detected from ('::1', 49348)<br>JsonRpc (StompReactor)::INFO::2017-05-<wbr>11 17:41:35,829::stompreactor::12<wbr>8::Broker.StompAdapter::(_cmd_<wbr>subscribe) Subscribe command received</div></div></div><br>______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br>
<br></blockquote></div></div>
</blockquote></div><br></div>