2017-05-11 17:28:38,375 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.= HostMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh host 'rhvse= rv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionExceptio= n: Connection failed<br>2017-05-11 17:28:41,352 INFO  [org.ovirt.vdsm.= jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to= rhvserv-03.mydomain.com/192.168.93.213<br>2017-05-11 17:28:41,354 ERROR [o= rg.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (Defaul= tQuartzScheduler9) [4cf3eed0] Command 'GetCapabilitiesVDSCommand(HostName = =3D rhvserv-03, VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', hos= tId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916', vds=3D'Host[rhvserv-03,4036f0= 27-8e90-49c0-8ca5-3ddb8d586916]'})' execution failed: org.ovirt.vdsm.jsonrp= c.client.ClientConnectionException: Connection failed<br>2017-05-11 17:28:4= 1,354 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (De= faultQuartzScheduler9) [4cf3eed0] Failure to refresh host 'rhvserv-03' runt= ime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connecti= on failed<br>2017-05-11 17:28:41,377 INFO  [org.ovirt.vdsm.jsonrpc.cli= ent.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to rhvserv-04= .mydomain.com/192.168.93.214<br>2017-05-11 17:28:41,380 ERROR [org.ovirt.en= gine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzSche= duler7) [1efe65a7] Command 'GetCapabilitiesVDSCommand(HostName =3D rhvserv-= 04, VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', hostId=3D'0d0cd= 690-b64a-42bb-a167-43fcedd634e4', vds=3D'Host[rhvserv-04,0d0cd690-b64a-42bb= -a167-43fcedd634e4]'})' execution failed: org.ovirt.vdsm.jsonrpc.client.Cli= entConnectionException: Connection failed<br>2017-05-11 17:28:41,380 ERROR = [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzS= cheduler7) [1efe65a7] Failure to refresh host 'rhvserv-04' runtime info: or= g.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed<br= 2017-05-11 17:28:44,358 INFO  [org.ovirt.vdsm.jsonrpc.client.reactors= .ReactorClient] (SSL Stomp Reactor) [] Connecting to rhvserv-03.mydomain.co= m/192.168.93.213<br>2017-05-11 17:28:44,382 INFO  [org.ovirt.vdsm.json= rpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to rhv= serv-04.mydomain.com/192.168.93.214<br>2017-05-11 17:28:47,362 ERROR [org.o= virt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQua= rtzScheduler1) [62006eaa] Command 'GetCapabilitiesVDSCommand(HostName =3D r= hvserv-03, VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', hostId= =3D'4036f027-8e90-49c0-8ca5-3ddb8d586916', vds=3D'Host[rhvserv-03,4036f027-= 8e90-49c0-8ca5-3ddb8d586916]'})' execution failed: org.ovirt.vdsm.jsonrpc.c=

2017-05-11 17:29:17,411 INFO  [org.ovirt.vdsm.jsonrpc.client.reactors= .ReactorClient] (SSL Stomp Reactor) [] Connecting to rhvserv-03.mydomain.co= m/192.168.93.213<br>2017-05-11 17:29:17,414 ERROR [org.ovirt.engine.core.vd= sbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [] C= ommand 'GetCapabilitiesVDSCommand(HostName =3D rhvserv-03, VdsIdAndVdsVDSCo= mmandParametersBase:{runAsync=3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3=

vdsmd logs<br></div><div><br></div><div><br></div><div>Thread-113::ERROR::= 2017-05-11 17:41:33,734::sdc::146::Storage.StorageDomainCache::(_findDomain= ) domain 5b978dda-d1ef-46fe-9996-20aee42cf303 not found<br>Traceback (most = recent call last):<br>  File "/usr/share/vdsm/storage/sdc.py", line 14= 4, in _findDomain<br>    dom =3D findMethod(sdUUID)<br> = ; File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain<= br>    raise se.StorageDomainDoesNotExist(sdUUID)<br>Storage= DomainDoesNotExist: Storage domain does not exist: (u'5b978dda-d1ef-46fe-99= 96-20aee42cf303',)<br>Thread-113::ERROR::2017-05-11 17:41:33,735::monitor::= 328::Storage.Monitor::(_setupLoop) Setting up monitor for 5b978dda-d1ef-46f= e-9996-20aee42cf303 failed<br>Traceback (most recent call last):<br>  = File "/usr/share/vdsm/storage/monitor.py", line 325, in _setupLoop<br> = ;   self._setupMonitor()<br>  File "/usr/share/vdsm/storage/= monitor.py", line 348, in _setupMonitor<br>    self._produce= Domain()<br>  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", l= ine 405, in wrapper<br>    value =3D meth(self, *a, **kw)<br=   File "/usr/share/vdsm/storage/monitor.py", line 366, in _produceDom= ain<br>    self.domain =3D sdCache.produce(self.sdUUID)<br>&= nbsp; File "/usr/share/vdsm/storage/sdc.py", line 101, in produce<br> =    domain.getRealDomain()<br>  File "/usr/share/vdsm/storage= /sdc.py", line 53, in getRealDomain<br>    return self._cach= e._realProduce(self._sdUUID)<br>  File "/usr/share/vdsm/storage/sdc.py= ", line 125, in _realProduce<br>    domain =3D self._findDom= ain(sdUUID)<br>  File "/usr/share/vdsm/storage/sdc.py", line 144, in _= findDomain<br>    dom =3D findMethod(sdUUID)<br>  File = "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain<br>&nbs=

------=_Part_11446852_617310615.1494517429918 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable After rebooting the manager VM, hosts are connecting/non responsive and dat= a domains inactive. Here are the engine and vdsmd logs. Any ideas ?=20 Engine logs :=20 2017-05-11 17:28:09,302 WARN [org.ovirt.engine.core.dal.dbbroker.auditlogha= ndling.AuditLogDirector] (DefaultQuartzScheduler5) [55f1aab5] Correlation I= D: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify P= ower Management configuration for Host rhvserv-05.=20 2017-05-11 17:28:09,346 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCom= mand] (DefaultQuartzScheduler5) [48bc69cd] Running command: HandleVdsVersio= nCommand internal: true. Entities affected : ID: 04565f10-9abf-4709-9445-9d= c6ed97e136 Type: VDS=20 2017-05-11 17:28:09,349 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (= org.ovirt.thread.pool-6-thread-27) [639977e4] Host 'rhvserv-05' is not resp= onding.=20 2017-05-11 17:28:09,364 WARN [org.ovirt.engine.core.dal.dbbroker.auditlogha= ndling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [639977e4] Cor= relation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host rhv= serv-05 is not responding. Host cannot be fenced automatically because powe= r management for the host is disabled.=20 2017-05-11 17:28:11,299 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [c0e6a2e] Command 'GetCa= pabilitiesVDSCommand(HostName =3D rhvserv-03, VdsIdAndVdsVDSCommandParamete= rsBase:{runAsync=3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916',= vds=3D'Host[rhvserv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'})' execution= failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connectio= n failed=20 2017-05-11 17:28:11,299 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler3) [c0e6a2e] Failure to refresh host = 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionEx= ception: Connection failed=20 2017-05-11 17:28:11,327 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-04.mydomain.com/192.1= 68.93.214=20 2017-05-11 17:28:12,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.Get= HardwareInfoVDSCommand] (DefaultQuartzScheduler3) [c0e6a2e] START, GetHardw= areInfoVDSCommand(HostName =3D rhvserv-05, VdsIdAndVdsVDSCommandParametersB= ase:{runAsync=3D'true', hostId=3D'04565f10-9abf-4709-9445-9dc6ed97e136', vd= s=3D'Host[rhvserv-05,04565f10-9abf-4709-9445-9dc6ed97e136]'}), log id: f807= ece=20 2017-05-11 17:28:12,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.Get= HardwareInfoVDSCommand] (DefaultQuartzScheduler3) [c0e6a2e] FINISH, GetHard= wareInfoVDSCommand, log id: f807ece=20 2017-05-11 17:28:12,532 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOr= ClusterChangedCommand] (DefaultQuartzScheduler3) [4e882ea0] Running command= : HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affecte= d : ID: 04565f10-9abf-4709-9445-9dc6ed97e136 Type: VDS=20 2017-05-11 17:28:12,539 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand]= (DefaultQuartzScheduler3) [75f25b35] Running command: InitVdsOnUpCommand i= nternal: true. Entities affected : ID: 58f8df36-019f-02bc-00e7-000000000023= Type: StoragePool=20 2017-05-11 17:28:12,545 INFO [org.ovirt.engine.core.bll.storage.pool.Connec= tHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [46cc3f58] Runn= ing command: ConnectHostToStoragePoolServersCommand internal: true. Entitie= s affected : ID: 58f8df36-019f-02bc-00e7-000000000023 Type: StoragePool=20 2017-05-11 17:28:12,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.Con= nectStorageServerVDSCommand] (DefaultQuartzScheduler3) [46cc3f58] START, Co= nnectStorageServerVDSCommand(HostName =3D rhvserv-05, StorageServerConnecti= onManagementVDSParameters:{runAsync=3D'true', hostId=3D'04565f10-9abf-4709-= 9445-9dc6ed97e136', storagePoolId=3D'58f8df36-019f-02bc-00e7-000000000023',= storageType=3D'ISCSI', connectionList=3D'[StorageServerConnections:{id=3D'= 10c0528b-f08d-4d1d-8c63-8a05fd9d58b9', connection=3D'10.35.21.1', iqn=3D'iq= n.1984-05.com.dell:powervault.md3200i.6782bcb00073e332000000004edde164', vf= sType=3D'null', mountOptions=3D'null', nfsVersion=3D'null', nfsRetrans=3D'n= ull', nfsTimeo=3D'null', iface=3D'null', netIfaceName=3D'null'}]'}), log id= : 1beb27b6=20 2017-05-11 17:28:13,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.Con= nectStorageServerVDSCommand] (DefaultQuartzScheduler3) [46cc3f58] FINISH, C= onnectStorageServerVDSCommand, return: {10c0528b-f08d-4d1d-8c63-8a05fd9d58b= 9=3D0}, log id: 1beb27b6=20 2017-05-11 17:28:13,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.Con= nectStorageServerVDSCommand] (DefaultQuartzScheduler3) [46cc3f58] START, Co= nnectStorageServerVDSCommand(HostName =3D rhvserv-05, StorageServerConnecti= onManagementVDSParameters:{runAsync=3D'true', hostId=3D'04565f10-9abf-4709-= 9445-9dc6ed97e136', storagePoolId=3D'58f8df36-019f-02bc-00e7-000000000023',= storageType=3D'NFS', connectionList=3D'[StorageServerConnections:{id=3D'e6= 04d0d2-0810-4c25-b9ed-610f9923cb1a', connection=3D'nfsserv-01:/nfs/export',= iqn=3D'null', vfsType=3D'null', mountOptions=3D'null', nfsVersion=3D'V3', = nfsRetrans=3D'null', nfsTimeo=3D'null', iface=3D'null', netIfaceName=3D'nul= l'}, StorageServerConnections:{id=3D'2c1aba4b-786b-4c76-a02f-d9e09e8afff4',= connection=3D'nfsserv-01:/nfs/iso', iqn=3D'null', vfsType=3D'null', mountO= ptions=3D'null', nfsVersion=3D'V3', nfsRetrans=3D'null', nfsTimeo=3D'null',= iface=3D'null', netIfaceName=3D'null'}]'}), log id: 15c54cd7=20 2017-05-11 17:28:14,301 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-03.mydomain.com/192.1= 68.93.213=20 2017-05-11 17:28:14,304 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [62006eaa] Command 'GetC= apabilitiesVDSCommand(HostName =3D rhvserv-03, VdsIdAndVdsVDSCommandParamet= ersBase:{runAsync=3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916'= , vds=3D'Host[rhvserv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'})' executio= n failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connecti= on failed=20 2017-05-11 17:28:14,304 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler1) [62006eaa] Failure to refresh host= 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionE= xception: Connection failed=20 2017-05-11 17:28:14,334 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [1efe65a7] Command 'GetC= apabilitiesVDSCommand(HostName =3D rhvserv-04, VdsIdAndVdsVDSCommandParamet= ersBase:{runAsync=3D'true', hostId=3D'0d0cd690-b64a-42bb-a167-43fcedd634e4'= , vds=3D'Host[rhvserv-04,0d0cd690-b64a-42bb-a167-43fcedd634e4]'})' executio= n failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connecti= on failed=20 2017-05-11 17:28:14,334 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler7) [1efe65a7] Failure to refresh host= 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionE= xception: Connection failed=20 2017-05-11 17:28:17,308 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-03.mydomain.com/192.1= 68.93.213=20 2017-05-11 17:28:17,336 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-04.mydomain.com/192.1= 68.93.214=20 2017-05-11 17:28:17,340 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [] Command 'GetCapabilit= iesVDSCommand(HostName =3D rhvserv-04, VdsIdAndVdsVDSCommandParametersBase:= {runAsync=3D'true', hostId=3D'0d0cd690-b64a-42bb-a167-43fcedd634e4', vds=3D= 'Host[rhvserv-04,0d0cd690-b64a-42bb-a167-43fcedd634e4]'})' execution failed= : org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection faile= d=20 2017-05-11 17:28:17,340 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh host 'rhvser= v-04' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException= : Connection failed=20 2017-05-11 17:28:20,315 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [62006eaa] Command 'GetC= apabilitiesVDSCommand(HostName =3D rhvserv-03, VdsIdAndVdsVDSCommandParamet= ersBase:{runAsync=3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916'= , vds=3D'Host[rhvserv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'})' executio= n failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connecti= on failed=20 2017-05-11 17:28:20,315 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler1) [62006eaa] Failure to refresh host= 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionE= xception: Connection failed=20 2017-05-11 17:28:20,343 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-04.mydomain.com/192.1= 68.93.214=20 2017-05-11 17:28:23,316 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-03.mydomain.com/192.1= 68.93.213=20 2017-05-11 17:28:23,325 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [] Command 'GetCapabilit= iesVDSCommand(HostName =3D rhvserv-03, VdsIdAndVdsVDSCommandParametersBase:= {runAsync=3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916', vds=3D= 'Host[rhvserv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'})' execution failed= : org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection faile= d=20 2017-05-11 17:28:23,325 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh host 'rhvser= v-03' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException= : Connection failed=20 2017-05-11 17:28:23,348 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [1efe65a7] Command 'GetC= apabilitiesVDSCommand(HostName =3D rhvserv-04, VdsIdAndVdsVDSCommandParamet= ersBase:{runAsync=3D'true', hostId=3D'0d0cd690-b64a-42bb-a167-43fcedd634e4'= , vds=3D'Host[rhvserv-04,0d0cd690-b64a-42bb-a167-43fcedd634e4]'})' executio= n failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connecti= on failed=20 2017-05-11 17:28:23,348 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler7) [1efe65a7] Failure to refresh host= 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionE= xception: Connection failed=20 2017-05-11 17:28:26,329 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-03.mydomain.com/192.1= 68.93.213=20 2017-05-11 17:28:26,350 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-04.mydomain.com/192.1= 68.93.214=20 2017-05-11 17:28:26,354 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [1efe65a7] Command 'GetC= apabilitiesVDSCommand(HostName =3D rhvserv-04, VdsIdAndVdsVDSCommandParamet= ersBase:{runAsync=3D'true', hostId=3D'0d0cd690-b64a-42bb-a167-43fcedd634e4'= , vds=3D'Host[rhvserv-04,0d0cd690-b64a-42bb-a167-43fcedd634e4]'})' executio= n failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connecti= on failed=20 2017-05-11 17:28:26,354 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler7) [1efe65a7] Failure to refresh host= 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionE= xception: Connection failed=20 2017-05-11 17:28:29,335 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [] Command 'GetCapabilit= iesVDSCommand(HostName =3D rhvserv-03, VdsIdAndVdsVDSCommandParametersBase:= {runAsync=3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916', vds=3D= 'Host[rhvserv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'})' execution failed= : org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection faile= d=20 2017-05-11 17:28:29,335 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh host 'rhvser= v-03' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException= : Connection failed=20 2017-05-11 17:28:29,356 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-04.mydomain.com/192.1= 68.93.214=20 2017-05-11 17:28:32,337 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-03.mydomain.com/192.1= 68.93.213=20 2017-05-11 17:28:32,340 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [] Command 'GetCapabili= tiesVDSCommand(HostName =3D rhvserv-03, VdsIdAndVdsVDSCommandParametersBase= :{runAsync=3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916', vds= =3D'Host[rhvserv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'})' execution fai= led: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection fa= iled=20 2017-05-11 17:28:32,340 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler10) [] Failure to refresh host 'rhvse= rv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionExceptio= n: Connection failed=20 2017-05-11 17:28:32,362 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [4cf3eed0] Command 'GetC= apabilitiesVDSCommand(HostName =3D rhvserv-04, VdsIdAndVdsVDSCommandParamet= ersBase:{runAsync=3D'true', hostId=3D'0d0cd690-b64a-42bb-a167-43fcedd634e4'= , vds=3D'Host[rhvserv-04,0d0cd690-b64a-42bb-a167-43fcedd634e4]'})' executio= n failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connecti= on failed=20 2017-05-11 17:28:32,362 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler9) [4cf3eed0] Failure to refresh host= 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionE= xception: Connection failed=20 2017-05-11 17:28:35,342 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-03.mydomain.com/192.1= 68.93.213=20 2017-05-11 17:28:35,370 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-04.mydomain.com/192.1= 68.93.214=20 2017-05-11 17:28:38,349 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [1efe65a7] Command 'GetC= apabilitiesVDSCommand(HostName =3D rhvserv-03, VdsIdAndVdsVDSCommandParamet= ersBase:{runAsync=3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916'= , vds=3D'Host[rhvserv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'})' executio= n failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connecti= on failed=20 2017-05-11 17:28:38,349 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler7) [1efe65a7] Failure to refresh host= 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionE= xception: Connection failed=20 2017-05-11 17:28:38,374 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [] Command 'GetCapabilit= iesVDSCommand(HostName =3D rhvserv-04, VdsIdAndVdsVDSCommandParametersBase:= {runAsync=3D'true', hostId=3D'0d0cd690-b64a-42bb-a167-43fcedd634e4', vds=3D= 'Host[rhvserv-04,0d0cd690-b64a-42bb-a167-43fcedd634e4]'})' execution failed= : org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection faile= d=20 2017-05-11 17:28:38,375 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh host 'rhvser= v-04' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException= : Connection failed=20 2017-05-11 17:28:41,352 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-03.mydomain.com/192.1= 68.93.213=20 2017-05-11 17:28:41,354 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [4cf3eed0] Command 'GetC= apabilitiesVDSCommand(HostName =3D rhvserv-03, VdsIdAndVdsVDSCommandParamet= ersBase:{runAsync=3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916'= , vds=3D'Host[rhvserv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'})' executio= n failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connecti= on failed=20 2017-05-11 17:28:41,354 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler9) [4cf3eed0] Failure to refresh host= 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionE= xception: Connection failed=20 2017-05-11 17:28:41,377 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-04.mydomain.com/192.1= 68.93.214=20 2017-05-11 17:28:41,380 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [1efe65a7] Command 'GetC= apabilitiesVDSCommand(HostName =3D rhvserv-04, VdsIdAndVdsVDSCommandParamet= ersBase:{runAsync=3D'true', hostId=3D'0d0cd690-b64a-42bb-a167-43fcedd634e4'= , vds=3D'Host[rhvserv-04,0d0cd690-b64a-42bb-a167-43fcedd634e4]'})' executio= n failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connecti= on failed=20 2017-05-11 17:28:41,380 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler7) [1efe65a7] Failure to refresh host= 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionE= xception: Connection failed=20 2017-05-11 17:28:44,358 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-03.mydomain.com/192.1= 68.93.213=20 2017-05-11 17:28:44,382 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-04.mydomain.com/192.1= 68.93.214=20 2017-05-11 17:28:47,362 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [62006eaa] Command 'GetC= apabilitiesVDSCommand(HostName =3D rhvserv-03, VdsIdAndVdsVDSCommandParamet= ersBase:{runAsync=3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916'= , vds=3D'Host[rhvserv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'})' executio= n failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connecti= on failed=20 2017-05-11 17:28:47,363 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler1) [62006eaa] Failure to refresh host= 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionE= xception: Connection failed=20 2017-05-11 17:28:47,388 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [1efe65a7] Command 'GetC= apabilitiesVDSCommand(HostName =3D rhvserv-04, VdsIdAndVdsVDSCommandParamet= ersBase:{runAsync=3D'true', hostId=3D'0d0cd690-b64a-42bb-a167-43fcedd634e4'= , vds=3D'Host[rhvserv-04,0d0cd690-b64a-42bb-a167-43fcedd634e4]'})' executio= n failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connecti= on failed=20 2017-05-11 17:28:47,388 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler7) [1efe65a7] Failure to refresh host= 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionE= xception: Connection failed=20 2017-05-11 17:28:50,365 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-03.mydomain.com/192.1= 68.93.213=20 2017-05-11 17:28:50,368 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler5) [48bc69cd] Command 'GetC= apabilitiesVDSCommand(HostName =3D rhvserv-03, VdsIdAndVdsVDSCommandParamet= ersBase:{runAsync=3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916'= , vds=3D'Host[rhvserv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'})' executio= n failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connecti= on failed=20 2017-05-11 17:28:50,368 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler5) [48bc69cd] Failure to refresh host= 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionE= xception: Connection failed=20 2017-05-11 17:28:50,390 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-04.mydomain.com/192.1= 68.93.214=20 2017-05-11 17:28:50,394 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [590af365] Command 'GetC= apabilitiesVDSCommand(HostName =3D rhvserv-04, VdsIdAndVdsVDSCommandParamet= ersBase:{runAsync=3D'true', hostId=3D'0d0cd690-b64a-42bb-a167-43fcedd634e4'= , vds=3D'Host[rhvserv-04,0d0cd690-b64a-42bb-a167-43fcedd634e4]'})' executio= n failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connecti= on failed=20 2017-05-11 17:28:50,394 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler2) [590af365] Failure to refresh host= 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionE= xception: Connection failed=20 2017-05-11 17:28:53,371 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-03.mydomain.com/192.1= 68.93.213=20 2017-05-11 17:28:53,396 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-04.mydomain.com/192.1= 68.93.214=20 2017-05-11 17:28:56,379 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [] Command 'GetCapabili= tiesVDSCommand(HostName =3D rhvserv-03, VdsIdAndVdsVDSCommandParametersBase= :{runAsync=3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916', vds= =3D'Host[rhvserv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'})' execution fai= led: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection fa= iled=20 2017-05-11 17:28:56,379 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler10) [] Failure to refresh host 'rhvse= rv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionExceptio= n: Connection failed=20 2017-05-11 17:28:56,402 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler5) [48bc69cd] Command 'GetC= apabilitiesVDSCommand(HostName =3D rhvserv-04, VdsIdAndVdsVDSCommandParamet= ersBase:{runAsync=3D'true', hostId=3D'0d0cd690-b64a-42bb-a167-43fcedd634e4'= , vds=3D'Host[rhvserv-04,0d0cd690-b64a-42bb-a167-43fcedd634e4]'})' executio= n failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connecti= on failed=20 2017-05-11 17:28:56,403 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler5) [48bc69cd] Failure to refresh host= 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionE= xception: Connection failed=20 2017-05-11 17:28:56,488 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tAllVmStatsVDSCommand] (DefaultQuartzScheduler6) [] Command 'GetAllVmStatsV= DSCommand(HostName =3D rhvserv-04, VdsIdAndVdsVDSCommandParametersBase:{run= Async=3D'true', hostId=3D'0d0cd690-b64a-42bb-a167-43fcedd634e4', vds=3D'Hos= t[rhvserv-04,0d0cd690-b64a-42bb-a167-43fcedd634e4]'})' execution failed: VD= SGenericException: VDSNetworkException: Vds timeout occured=20 2017-05-11 17:28:56,488 INFO [org.ovirt.engine.core.vdsbroker.monitoring.Po= llVmStatsRefresher] (DefaultQuartzScheduler6) [] Failed to fetch vms info f= or host 'rhvserv-04' - skipping VMs monitoring.=20 2017-05-11 17:28:59,382 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-03.mydomain.com/192.1= 68.93.213=20 2017-05-11 17:28:59,384 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [] Command 'GetCapabili= tiesVDSCommand(HostName =3D rhvserv-03, VdsIdAndVdsVDSCommandParametersBase= :{runAsync=3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916', vds= =3D'Host[rhvserv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'})' execution fai= led: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection fa= iled=20 2017-05-11 17:28:59,384 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler10) [] Failure to refresh host 'rhvse= rv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionExceptio= n: Connection failed=20 2017-05-11 17:28:59,404 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-04.mydomain.com/192.1= 68.93.214=20 2017-05-11 17:28:59,408 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [] Command 'GetCapabilit= iesVDSCommand(HostName =3D rhvserv-04, VdsIdAndVdsVDSCommandParametersBase:= {runAsync=3D'true', hostId=3D'0d0cd690-b64a-42bb-a167-43fcedd634e4', vds=3D= 'Host[rhvserv-04,0d0cd690-b64a-42bb-a167-43fcedd634e4]'})' execution failed= : org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection faile= d=20 2017-05-11 17:28:59,408 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler6) [] Failure to refresh host 'rhvser= v-04' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException= : Connection failed=20 2017-05-11 17:29:02,386 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-03.mydomain.com/192.1= 68.93.213=20 2017-05-11 17:29:02,409 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-04.mydomain.com/192.1= 68.93.214=20 2017-05-11 17:29:02,490 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tAllVmStatsVDSCommand] (DefaultQuartzScheduler8) [56fb4c82] Command 'GetAll= VmStatsVDSCommand(HostName =3D rhvserv-03, VdsIdAndVdsVDSCommandParametersB= ase:{runAsync=3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916', vd= s=3D'Host[rhvserv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'})' execution fa= iled: VDSGenericException: VDSNetworkException: Vds timeout occured=20 2017-05-11 17:29:02,490 INFO [org.ovirt.engine.core.vdsbroker.monitoring.Po= llVmStatsRefresher] (DefaultQuartzScheduler8) [56fb4c82] Failed to fetch vm= s info for host 'rhvserv-03' - skipping VMs monitoring.=20 2017-05-11 17:29:05,393 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [] Command 'GetCapabilit= iesVDSCommand(HostName =3D rhvserv-03, VdsIdAndVdsVDSCommandParametersBase:= {runAsync=3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916', vds=3D= 'Host[rhvserv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'})' execution failed= : org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection faile= d=20 2017-05-11 17:29:05,393 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh host 'rhvser= v-03' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException= : Connection failed=20 2017-05-11 17:29:05,415 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [] Command 'GetCapabili= tiesVDSCommand(HostName =3D rhvserv-04, VdsIdAndVdsVDSCommandParametersBase= :{runAsync=3D'true', hostId=3D'0d0cd690-b64a-42bb-a167-43fcedd634e4', vds= =3D'Host[rhvserv-04,0d0cd690-b64a-42bb-a167-43fcedd634e4]'})' execution fai= led: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection fa= iled=20 2017-05-11 17:29:05,415 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler10) [] Failure to refresh host 'rhvse= rv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionExceptio= n: Connection failed=20 2017-05-11 17:29:08,395 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-03.mydomain.com/192.1= 68.93.213=20 2017-05-11 17:29:08,398 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [56fb4c82] Command 'GetC= apabilitiesVDSCommand(HostName =3D rhvserv-03, VdsIdAndVdsVDSCommandParamet= ersBase:{runAsync=3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916'= , vds=3D'Host[rhvserv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'})' executio= n failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connecti= on failed=20 2017-05-11 17:29:08,398 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler8) [56fb4c82] Failure to refresh host= 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionE= xception: Connection failed=20 2017-05-11 17:29:08,417 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-04.mydomain.com/192.1= 68.93.214=20 2017-05-11 17:29:08,420 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [590af365] Command 'GetC= apabilitiesVDSCommand(HostName =3D rhvserv-04, VdsIdAndVdsVDSCommandParamet= ersBase:{runAsync=3D'true', hostId=3D'0d0cd690-b64a-42bb-a167-43fcedd634e4'= , vds=3D'Host[rhvserv-04,0d0cd690-b64a-42bb-a167-43fcedd634e4]'})' executio= n failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connecti= on failed=20 2017-05-11 17:29:08,420 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler2) [590af365] Failure to refresh host= 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionE= xception: Connection failed=20 2017-05-11 17:29:11,402 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-03.mydomain.com/192.1= 68.93.213=20 2017-05-11 17:29:11,423 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-04.mydomain.com/192.1= 68.93.214=20 2017-05-11 17:29:14,409 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [] Command 'GetCapabilit= iesVDSCommand(HostName =3D rhvserv-03, VdsIdAndVdsVDSCommandParametersBase:= {runAsync=3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916', vds=3D= 'Host[rhvserv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'})' execution failed= : org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection faile= d=20 2017-05-11 17:29:14,409 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh host 'rhvser= v-03' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException= : Connection failed=20 2017-05-11 17:29:14,430 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler5) [48bc69cd] Command 'GetC= apabilitiesVDSCommand(HostName =3D rhvserv-04, VdsIdAndVdsVDSCommandParamet= ersBase:{runAsync=3D'true', hostId=3D'0d0cd690-b64a-42bb-a167-43fcedd634e4'= , vds=3D'Host[rhvserv-04,0d0cd690-b64a-42bb-a167-43fcedd634e4]'})' executio= n failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connecti= on failed=20 2017-05-11 17:29:14,430 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler5) [48bc69cd] Failure to refresh host= 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionE= xception: Connection failed=20 2017-05-11 17:29:17,411 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-03.mydomain.com/192.1= 68.93.213=20 2017-05-11 17:29:17,414 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [] Command 'GetCapabilit= iesVDSCommand(HostName =3D rhvserv-03, VdsIdAndVdsVDSCommandParametersBase:= {runAsync=3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916', vds=3D= 'Host[rhvserv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'})' execution failed= : org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection faile= d=20 2017-05-11 17:29:17,414 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler6) [] Failure to refresh host 'rhvser= v-03' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException= : Connection failed=20 2017-05-11 17:29:17,432 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-04.mydomain.com/192.1= 68.93.214=20 2017-05-11 17:29:17,436 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.Ge= tCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [] Command 'GetCapabilit= iesVDSCommand(HostName =3D rhvserv-04, VdsIdAndVdsVDSCommandParametersBase:= {runAsync=3D'true', hostId=3D'0d0cd690-b64a-42bb-a167-43fcedd634e4', vds=3D= 'Host[rhvserv-04,0d0cd690-b64a-42bb-a167-43fcedd634e4]'})' execution failed= : org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection faile= d=20 2017-05-11 17:29:17,436 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh host 'rhvser= v-04' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException= : Connection failed=20 2017-05-11 17:29:17,491 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.Reacto= rClient] (SSL Stomp Reactor) [] Connecting to rhvserv-03.mydomain.com/192.1= 68.93.213=20 vdsmd logs=20 Thread-113::ERROR::2017-05-11 17:41:33,734::sdc::146::Storage.StorageDomain= Cache::(_findDomain) domain 5b978dda-d1ef-46fe-9996-20aee42cf303 not found= =20 Traceback (most recent call last):=20 File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain=20 dom =3D findMethod(sdUUID)=20 File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain=20 raise se.StorageDomainDoesNotExist(sdUUID)=20 StorageDomainDoesNotExist: Storage domain does not exist: (u'5b978dda-d1ef-= 46fe-9996-20aee42cf303',)=20 Thread-113::ERROR::2017-05-11 17:41:33,735::monitor::328::Storage.Monitor::= (_setupLoop) Setting up monitor for 5b978dda-d1ef-46fe-9996-20aee42cf303 fa= iled=20 Traceback (most recent call last):=20 File "/usr/share/vdsm/storage/monitor.py", line 325, in _setupLoop=20 self._setupMonitor()=20 File "/usr/share/vdsm/storage/monitor.py", line 348, in _setupMonitor=20 self._produceDomain()=20 File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 405, in wrapper= =20 value =3D meth(self, *a, **kw)=20 File "/usr/share/vdsm/storage/monitor.py", line 366, in _produceDomain=20 self.domain =3D sdCache.produce(self.sdUUID)=20 File "/usr/share/vdsm/storage/sdc.py", line 101, in produce=20 domain.getRealDomain()=20 File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain=20 return self._cache._realProduce(self._sdUUID)=20 File "/usr/share/vdsm/storage/sdc.py", line 125, in _realProduce=20 domain =3D self._findDomain(sdUUID)=20 File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain=20 dom =3D findMethod(sdUUID)=20 File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain=20 raise se.StorageDomainDoesNotExist(sdUUID)=20 StorageDomainDoesNotExist: Storage domain does not exist: (u'5b978dda-d1ef-= 46fe-9996-20aee42cf303',)=20 Thread-112::DEBUG::2017-05-11 17:41:33,783::lvm::288::Storage.Misc.excCmd::= (cmd) SUCCESS: <err> =3D ''; <rc> =3D 0=20 Thread-112::INFO::2017-05-11 17:41:33,783::sd::604::Storage.StorageDomain::= (_registerResourceNamespaces) Resource namespace edd229cb-b72f-4988-8c10-d8= 3c84ef4a8a_imageNS already registered=20 Thread-112::INFO::2017-05-11 17:41:33,783::sd::612::Storage.StorageDomain::= (_registerResourceNamespaces) Resource namespace edd229cb-b72f-4988-8c10-d8= 3c84ef4a8a_volumeNS already registered=20 Thread-112::INFO::2017-05-11 17:41:33,783::blockSD::846::Storage.StorageDom= ain::(_registerResourceNamespaces) Resource namespace edd229cb-b72f-4988-8c= 10-d83c84ef4a8a_lvmActivationNS already registered=20 Thread-112::DEBUG::2017-05-11 17:41:33,784::lvm::288::Storage.Misc.excCmd::= (cmd) /usr/bin/taskset --cpu-list 0-15 /usr/bin/sudo -n /usr/sbin/lvm vgck = --config ' devices { preferred_names =3D ["^/dev/mapper/"] ignore_suspended= _devices=3D1 write_cache_state=3D0 disable_after_error_count=3D3 filter =3D= [ '\''a|/dev/mapper/36782bcb00073e33200002d3858fa1a81|'\'', '\''r|.*|'\'' = ] } global { locking_type=3D1 prioritise_write_locks=3D1 wait_for_locks=3D1= use_lvmetad=3D0 } backup { retain_min =3D 50 retain_days =3D 0 } ' edd229c= b-b72f-4988-8c10-d83c84ef4a8a (cwd None)=20 jsonrpc.Executor/6::DEBUG::2017-05-11 17:41:33,792::lvm::288::Storage.Misc.= excCmd::(cmd) SUCCESS: <err> =3D ''; <rc> =3D 0=20 jsonrpc.Executor/6::INFO::2017-05-11 17:41:33,792::logUtils::52::dispatcher= ::(wrapper) Run and protect: getStorageDomainStats, Return response: {'stat= s': {'mdasize': '134217728', 'mdathreshold': True, 'mdavalid': True, 'diskf= ree': '498484641792', 'disktotal': '557943095296', 'mdafree': '67102208'}}= =20 jsonrpc.Executor/6::DEBUG::2017-05-11 17:41:33,793::task::1193::Storage.Tas= kManager.Task::(prepare) Task=3D`234cbec4-a422-47c1-a50a-243526aeddc7`::fin= ished: {'stats': {'mdasize': '134217728', 'mdathreshold': True, 'mdavalid':= True, 'diskfree': '498484641792', 'disktotal': '557943095296', 'mdafree': = '67102208'}}=20 jsonrpc.Executor/6::DEBUG::2017-05-11 17:41:33,793::task::597::Storage.Task= Manager.Task::(_updateState) Task=3D`234cbec4-a422-47c1-a50a-243526aeddc7`:= :moving from state preparing -> state finished=20 jsonrpc.Executor/6::DEBUG::2017-05-11 17:41:33,793::resourceManager::952::S= torage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} res= ources {u'Storage.595de2cf-89ba-407b-aae5-d0a7a0656ba1': < ResourceRef 'Sto= rage.595de2cf-89ba-407b-aae5-d0a7a0656ba1', isValid: 'True' obj: 'None'>}= =20 jsonrpc.Executor/6::DEBUG::2017-05-11 17:41:33,793::resourceManager::989::S= torage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}=20 jsonrpc.Executor/6::DEBUG::2017-05-11 17:41:33,793::resourceManager::628::S= torage.ResourceManager::(releaseResource) Trying to release resource 'Stora= ge.595de2cf-89ba-407b-aae5-d0a7a0656ba1'=20 jsonrpc.Executor/6::DEBUG::2017-05-11 17:41:33,793::resourceManager::647::S= torage.ResourceManager::(releaseResource) Released resource 'Storage.595de2= cf-89ba-407b-aae5-d0a7a0656ba1' (0 active users)=20 jsonrpc.Executor/6::DEBUG::2017-05-11 17:41:33,793::resourceManager::653::S= torage.ResourceManager::(releaseResource) Resource 'Storage.595de2cf-89ba-4= 07b-aae5-d0a7a0656ba1' is free, finding out if anyone is waiting for it.=20 jsonrpc.Executor/6::DEBUG::2017-05-11 17:41:33,793::resourceManager::661::S= torage.ResourceManager::(releaseResource) No one is waiting for resource 'S= torage.595de2cf-89ba-407b-aae5-d0a7a0656ba1', Clearing records.=20 jsonrpc.Executor/6::DEBUG::2017-05-11 17:41:33,793::task::995::Storage.Task= Manager.Task::(_decref) Task=3D`234cbec4-a422-47c1-a50a-243526aeddc7`::ref = 0 aborting False=20 jsonrpc.Executor/6::DEBUG::2017-05-11 17:41:33,794::__init__::555::jsonrpc.= JsonRpcServer::(_handle_request) Return 'StorageDomain.getStats' in bridge = with {'mdasize': '134217728', 'mdathreshold': True, 'mdavalid': True, 'disk= free': '498484641792', 'disktotal': '557943095296', 'mdafree': '67102208'}= =20 jsonrpc.Executor/6::INFO::2017-05-11 17:41:33,794::__init__::513::jsonrpc.J= sonRpcServer::(_serveRequest) RPC call StorageDomain.getStats succeeded in = 1.05 seconds=20 JsonRpc (StompReactor)::ERROR::2017-05-11 17:41:33,816::betterAsyncore::113= ::vds.dispatcher::(recv) SSL error during reading data: unexpected eof=20 Thread-112::DEBUG::2017-05-11 17:41:33,846::lvm::288::Storage.Misc.excCmd::= (cmd) SUCCESS: <err> =3D ''; <rc> =3D 0=20 Thread-12::DEBUG::2017-05-11 17:41:34,913::check::296::storage.check::(_sta= rt_process) START check '/dev/595de2cf-89ba-407b-aae5-d0a7a0656ba1/metadata= ' cmd=3D['/usr/bin/taskset', '--cpu-list', '0-15', '/usr/bin/dd', 'if=3D/de= v/595de2cf-89ba-407b-aae5-d0a7a0656ba1/metadata', 'of=3D/dev/null', 'bs=3D4= 096', 'count=3D1', 'iflag=3Ddirect'] delay=3D0.00=20 Thread-12::DEBUG::2017-05-11 17:41:34,954::asyncevent::564::storage.asyncev= ent::(reap) Process <cpopen.CPopen object at 0x1b15b90> terminated (count= =3D1)=20 Thread-12::DEBUG::2017-05-11 17:41:34,954::check::327::storage.check::(_che= ck_completed) FINISH check '/dev/595de2cf-89ba-407b-aae5-d0a7a0656ba1/metad= ata' rc=3D0 err=3Dbytearray(b'1+0 records in\n1+0 records out\n4096 bytes (= 4.1 kB) copied, 0.000537649 s, 7.6 MB/s\n') elapsed=3D0.05=20 Reactor thread::INFO::2017-05-11 17:41:35,823::protocoldetector::76::Protoc= olDetector.AcceptorImpl::(handle_accept) Accepted connection from ::1:49348= =20 Reactor thread::DEBUG::2017-05-11 17:41:35,827::protocoldetector::92::Proto= colDetector.Detector::(__init__) Using required_size=3D11=20 Reactor thread::INFO::2017-05-11 17:41:35,828::protocoldetector::128::Proto= colDetector.Detector::(handle_read) Detected protocol stomp from ::1:49348= =20 Reactor thread::INFO::2017-05-11 17:41:35,828::stompreactor::101::Broker.St= ompAdapter::(_cmd_connect) Processing CONNECT request=20 Reactor thread::DEBUG::2017-05-11 17:41:35,829::stompreactor::492::protocol= detector.StompDetector::(handle_socket) Stomp detected from ('::1', 49348)= =20 JsonRpc (StompReactor)::INFO::2017-05-11 17:41:35,829::stompreactor::128::B= roker.StompAdapter::(_cmd_subscribe) Subscribe command received=20 ------=_Part_11446852_617310615.1494517429918 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><body><div style=3D"font-family: times new roman, new york, times, se= rif; font-size: 12pt; color: #000000"><div>After rebooting the manager VM, =  hosts are connecting/non responsive and data domains inactive. Here a= re the engine and vdsmd logs. Any ideas ?<br></div><div><br></div><div><br>= </div><div><br></div><div><br></div><div>Engine logs : <br></div><div><br><= /div><div><br></div><div><br></div><div>2017-05-11 17:28:09,302 WARN  = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (Def= aultQuartzScheduler5) [55f1aab5] Correlation ID: null, Call Stack: null, Cu= stom Event ID: -1, Message: Failed to verify Power Management configuration= for Host rhvserv-05.<br>2017-05-11 17:28:09,346 INFO  [org.ovirt.engi= ne.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler5) [48bc69cd] R= unning command: HandleVdsVersionCommand internal: true. Entities affected :=   ID: 04565f10-9abf-4709-9445-9dc6ed97e136 Type: VDS<br>2017-05-11 17:= 28:09,349 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovir= t.thread.pool-6-thread-27) [639977e4] Host 'rhvserv-05' is not responding.<= br>2017-05-11 17:28:09,364 WARN  [org.ovirt.engine.core.dal.dbbroker.a= uditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-27) [6399= 77e4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message:= Host rhvserv-05 is not responding. Host cannot be fenced automatically bec= ause power management for the host is disabled.<br>2017-05-11 17:28:11,299 = ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]= (DefaultQuartzScheduler3) [c0e6a2e] Command 'GetCapabilitiesVDSCommand(Hos= tName =3D rhvserv-03, VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true= ', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916', vds=3D'Host[rhvserv-03,= 4036f027-8e90-49c0-8ca5-3ddb8d586916]'})' execution failed: org.ovirt.vdsm.= jsonrpc.client.ClientConnectionException: Connection failed<br>2017-05-11 1= 7:28:11,299 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitorin= g] (DefaultQuartzScheduler3) [c0e6a2e] Failure to refresh host 'rhvserv-03'= runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Con= nection failed<br>2017-05-11 17:28:11,327 INFO  [org.ovirt.vdsm.jsonrp= c.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to rhvse= rv-04.mydomain.com/192.168.93.214<br>2017-05-11 17:28:12,484 INFO  [or= g.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (Default= QuartzScheduler3) [c0e6a2e] START, GetHardwareInfoVDSCommand(HostName =3D r= hvserv-05, VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', hostId= =3D'04565f10-9abf-4709-9445-9dc6ed97e136', vds=3D'Host[rhvserv-05,04565f10-= 9abf-4709-9445-9dc6ed97e136]'}), log id: f807ece<br>2017-05-11 17:28:12,487= INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCo= mmand] (DefaultQuartzScheduler3) [c0e6a2e] FINISH, GetHardwareInfoVDSComman= d, log id: f807ece<br>2017-05-11 17:28:12,532 INFO  [org.ovirt.engine.= core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3= ) [4e882ea0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand inte= rnal: true. Entities affected :  ID: 04565f10-9abf-4709-9445-9dc6ed97e= 136 Type: VDS<br>2017-05-11 17:28:12,539 INFO  [org.ovirt.engine.core.= bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [75f25b35] Running comman= d: InitVdsOnUpCommand internal: true. Entities affected :  ID: 58f8df3= 6-019f-02bc-00e7-000000000023 Type: StoragePool<br>2017-05-11 17:28:12,545 = INFO  [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePool= ServersCommand] (DefaultQuartzScheduler3) [46cc3f58] Running command: Conne= ctHostToStoragePoolServersCommand internal: true. Entities affected : = ID: 58f8df36-019f-02bc-00e7-000000000023 Type: StoragePool<br>2017-05-11 1= 7:28:12,556 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectSt= orageServerVDSCommand] (DefaultQuartzScheduler3) [46cc3f58] START, ConnectS= torageServerVDSCommand(HostName =3D rhvserv-05, StorageServerConnectionMana= gementVDSParameters:{runAsync=3D'true', hostId=3D'04565f10-9abf-4709-9445-9= dc6ed97e136', storagePoolId=3D'58f8df36-019f-02bc-00e7-000000000023', stora= geType=3D'ISCSI', connectionList=3D'[StorageServerConnections:{id=3D'10c052= 8b-f08d-4d1d-8c63-8a05fd9d58b9', connection=3D'10.35.21.1', iqn=3D'iqn.1984= -05.com.dell:powervault.md3200i.6782bcb00073e332000000004edde164', vfsType= =3D'null', mountOptions=3D'null', nfsVersion=3D'null', nfsRetrans=3D'null',= nfsTimeo=3D'null', iface=3D'null', netIfaceName=3D'null'}]'}), log id: 1be= b27b6<br>2017-05-11 17:28:13,031 INFO  [org.ovirt.engine.core.vdsbroke= r.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [46cc= 3f58] FINISH, ConnectStorageServerVDSCommand, return: {10c0528b-f08d-4d1d-8= c63-8a05fd9d58b9=3D0}, log id: 1beb27b6<br>2017-05-11 17:28:13,032 INFO&nbs= p; [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSComman= d] (DefaultQuartzScheduler3) [46cc3f58] START, ConnectStorageServerVDSComma= nd(HostName =3D rhvserv-05, StorageServerConnectionManagementVDSParameters:= {runAsync=3D'true', hostId=3D'04565f10-9abf-4709-9445-9dc6ed97e136', storag= ePoolId=3D'58f8df36-019f-02bc-00e7-000000000023', storageType=3D'NFS', conn= ectionList=3D'[StorageServerConnections:{id=3D'e604d0d2-0810-4c25-b9ed-610f= 9923cb1a', connection=3D'nfsserv-01:/nfs/export', iqn=3D'null', vfsType=3D'= null', mountOptions=3D'null', nfsVersion=3D'V3', nfsRetrans=3D'null', nfsTi= meo=3D'null', iface=3D'null', netIfaceName=3D'null'}, StorageServerConnecti= ons:{id=3D'2c1aba4b-786b-4c76-a02f-d9e09e8afff4', connection=3D'nfsserv-01:= /nfs/iso', iqn=3D'null', vfsType=3D'null', mountOptions=3D'null', nfsVersio= n=3D'V3', nfsRetrans=3D'null', nfsTimeo=3D'null', iface=3D'null', netIfaceN= ame=3D'null'}]'}), log id: 15c54cd7<br>2017-05-11 17:28:14,301 INFO  [= org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [= ] Connecting to rhvserv-03.mydomain.com/192.168.93.213<br>2017-05-11 17:28:= 14,304 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSC= ommand] (DefaultQuartzScheduler1) [62006eaa] Command 'GetCapabilitiesVDSCom= mand(HostName =3D rhvserv-03, VdsIdAndVdsVDSCommandParametersBase:{runAsync= =3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916', vds=3D'Host[rhv= serv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'})' execution failed: org.ovi= rt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed<br>2017= -05-11 17:28:14,304 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostM= onitoring] (DefaultQuartzScheduler1) [62006eaa] Failure to refresh host 'rh= vserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionExcep= tion: Connection failed<br>2017-05-11 17:28:14,334 ERROR [org.ovirt.engine.= core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler= 7) [1efe65a7] Command 'GetCapabilitiesVDSCommand(HostName =3D rhvserv-04, V= dsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', hostId=3D'0d0cd690-b= 64a-42bb-a167-43fcedd634e4', vds=3D'Host[rhvserv-04,0d0cd690-b64a-42bb-a167= -43fcedd634e4]'})' execution failed: org.ovirt.vdsm.jsonrpc.client.ClientCo= nnectionException: Connection failed<br>2017-05-11 17:28:14,334 ERROR [org.= ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzSchedu= ler7) [1efe65a7] Failure to refresh host 'rhvserv-04' runtime info: org.ovi= rt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed<br>2017= -05-11 17:28:17,308 INFO  [org.ovirt.vdsm.jsonrpc.client.reactors.Reac= torClient] (SSL Stomp Reactor) [] Connecting to rhvserv-03.mydomain.com/192= .168.93.213<br>2017-05-11 17:28:17,336 INFO  [org.ovirt.vdsm.jsonrpc.c= lient.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to rhvserv-= 04.mydomain.com/192.168.93.214<br>2017-05-11 17:28:17,340 ERROR [org.ovirt.= engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzSc= heduler4) [] Command 'GetCapabilitiesVDSCommand(HostName =3D rhvserv-04, Vd= sIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', hostId=3D'0d0cd690-b6= 4a-42bb-a167-43fcedd634e4', vds=3D'Host[rhvserv-04,0d0cd690-b64a-42bb-a167-= 43fcedd634e4]'})' execution failed: org.ovirt.vdsm.jsonrpc.client.ClientCon= nectionException: Connection failed<br>2017-05-11 17:28:17,340 ERROR [org.o= virt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzSchedul= er4) [] Failure to refresh host 'rhvserv-04' runtime info: org.ovirt.vdsm.j= sonrpc.client.ClientConnectionException: Connection failed<br>2017-05-11 17= :28:20,315 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilities= VDSCommand] (DefaultQuartzScheduler1) [62006eaa] Command 'GetCapabilitiesVD= SCommand(HostName =3D rhvserv-03, VdsIdAndVdsVDSCommandParametersBase:{runA= sync=3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916', vds=3D'Host= [rhvserv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'})' execution failed: org= .ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed<br>= 2017-05-11 17:28:20,315 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.H= ostMonitoring] (DefaultQuartzScheduler1) [62006eaa] Failure to refresh host= 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionE= xception: Connection failed<br>2017-05-11 17:28:20,343 INFO  [org.ovir= t.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connec= ting to rhvserv-04.mydomain.com/192.168.93.214<br>2017-05-11 17:28:23,316 I= NFO  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp= Reactor) [] Connecting to rhvserv-03.mydomain.com/192.168.93.213<br>2017-0= 5-11 17:28:23,325 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapab= ilitiesVDSCommand] (DefaultQuartzScheduler4) [] Command 'GetCapabilitiesVDS= Command(HostName =3D rhvserv-03, VdsIdAndVdsVDSCommandParametersBase:{runAs= ync=3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916', vds=3D'Host[= rhvserv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'})' execution failed: org.= ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed<br>2= 017-05-11 17:28:23,325 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.Ho= stMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh host 'rhvserv= -03' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException:= Connection failed<br>2017-05-11 17:28:23,348 ERROR [org.ovirt.engine.core.= vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [1= efe65a7] Command 'GetCapabilitiesVDSCommand(HostName =3D rhvserv-04, VdsIdA= ndVdsVDSCommandParametersBase:{runAsync=3D'true', hostId=3D'0d0cd690-b64a-4= 2bb-a167-43fcedd634e4', vds=3D'Host[rhvserv-04,0d0cd690-b64a-42bb-a167-43fc= edd634e4]'})' execution failed: org.ovirt.vdsm.jsonrpc.client.ClientConnect= ionException: Connection failed<br>2017-05-11 17:28:23,348 ERROR [org.ovirt= .engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7)= [1efe65a7] Failure to refresh host 'rhvserv-04' runtime info: org.ovirt.vd= sm.jsonrpc.client.ClientConnectionException: Connection failed<br>2017-05-1= 1 17:28:26,329 INFO  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorCl= ient] (SSL Stomp Reactor) [] Connecting to rhvserv-03.mydomain.com/192.168.= 93.213<br>2017-05-11 17:28:26,350 INFO  [org.ovirt.vdsm.jsonrpc.client= .reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to rhvserv-04.my= domain.com/192.168.93.214<br>2017-05-11 17:28:26,354 ERROR [org.ovirt.engin= e.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzSchedul= er7) [1efe65a7] Command 'GetCapabilitiesVDSCommand(HostName =3D rhvserv-04,= VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', hostId=3D'0d0cd690= -b64a-42bb-a167-43fcedd634e4', vds=3D'Host[rhvserv-04,0d0cd690-b64a-42bb-a1= 67-43fcedd634e4]'})' execution failed: org.ovirt.vdsm.jsonrpc.client.Client= ConnectionException: Connection failed<br>2017-05-11 17:28:26,354 ERROR [or= g.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzSche= duler7) [1efe65a7] Failure to refresh host 'rhvserv-04' runtime info: org.o= virt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed<br>20= 17-05-11 17:28:29,335 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetC= apabilitiesVDSCommand] (DefaultQuartzScheduler4) [] Command 'GetCapabilitie= sVDSCommand(HostName =3D rhvserv-03, VdsIdAndVdsVDSCommandParametersBase:{r= unAsync=3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916', vds=3D'H= ost[rhvserv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'})' execution failed: = org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed<= br>2017-05-11 17:28:29,335 ERROR [org.ovirt.engine.core.vdsbroker.monitorin= g.HostMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh host 'rhv= serv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionExcept= ion: Connection failed<br>2017-05-11 17:28:29,356 INFO  [org.ovirt.vds= m.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting = to rhvserv-04.mydomain.com/192.168.93.214<br>2017-05-11 17:28:32,337 INFO&n= bsp; [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reac= tor) [] Connecting to rhvserv-03.mydomain.com/192.168.93.213<br>2017-05-11 = 17:28:32,340 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabiliti= esVDSCommand] (DefaultQuartzScheduler10) [] Command 'GetCapabilitiesVDSComm= and(HostName =3D rhvserv-03, VdsIdAndVdsVDSCommandParametersBase:{runAsync= =3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916', vds=3D'Host[rhv= serv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'})' execution failed: org.ovi= rt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed<br>2017= -05-11 17:28:32,340 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostM= onitoring] (DefaultQuartzScheduler10) [] Failure to refresh host 'rhvserv-0= 3' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: C= onnection failed<br>2017-05-11 17:28:32,362 ERROR [org.ovirt.engine.core.vd= sbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [4cf= 3eed0] Command 'GetCapabilitiesVDSCommand(HostName =3D rhvserv-04, VdsIdAnd= VdsVDSCommandParametersBase:{runAsync=3D'true', hostId=3D'0d0cd690-b64a-42b= b-a167-43fcedd634e4', vds=3D'Host[rhvserv-04,0d0cd690-b64a-42bb-a167-43fced= d634e4]'})' execution failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectio= nException: Connection failed<br>2017-05-11 17:28:32,362 ERROR [org.ovirt.e= ngine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [= 4cf3eed0] Failure to refresh host 'rhvserv-04' runtime info: org.ovirt.vdsm= .jsonrpc.client.ClientConnectionException: Connection failed<br>2017-05-11 = 17:28:35,342 INFO  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClie= nt] (SSL Stomp Reactor) [] Connecting to rhvserv-03.mydomain.com/192.168.93= .213<br>2017-05-11 17:28:35,370 INFO  [org.ovirt.vdsm.jsonrpc.client.r= eactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to rhvserv-04.mydo= main.com/192.168.93.214<br>2017-05-11 17:28:38,349 ERROR [org.ovirt.engine.= core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler= 7) [1efe65a7] Command 'GetCapabilitiesVDSCommand(HostName =3D rhvserv-03, V= dsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', hostId=3D'4036f027-8= e90-49c0-8ca5-3ddb8d586916', vds=3D'Host[rhvserv-03,4036f027-8e90-49c0-8ca5= -3ddb8d586916]'})' execution failed: org.ovirt.vdsm.jsonrpc.client.ClientCo= nnectionException: Connection failed<br>2017-05-11 17:28:38,349 ERROR [org.= ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzSchedu= ler7) [1efe65a7] Failure to refresh host 'rhvserv-03' runtime info: org.ovi= rt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed<br>2017= -05-11 17:28:38,374 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCap= abilitiesVDSCommand] (DefaultQuartzScheduler4) [] Command 'GetCapabilitiesV= DSCommand(HostName =3D rhvserv-04, VdsIdAndVdsVDSCommandParametersBase:{run= Async=3D'true', hostId=3D'0d0cd690-b64a-42bb-a167-43fcedd634e4', vds=3D'Hos= t[rhvserv-04,0d0cd690-b64a-42bb-a167-43fcedd634e4]'})' execution failed: or= g.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed<br= lient.ClientConnectionException: Connection failed<br>2017-05-11 17:28:47,3= 63 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (Defau= ltQuartzScheduler1) [62006eaa] Failure to refresh host 'rhvserv-03' runtime= info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection = failed<br>2017-05-11 17:28:47,388 ERROR [org.ovirt.engine.core.vdsbroker.vd= sbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [1efe65a7] Com= mand 'GetCapabilitiesVDSCommand(HostName =3D rhvserv-04, VdsIdAndVdsVDSComm= andParametersBase:{runAsync=3D'true', hostId=3D'0d0cd690-b64a-42bb-a167-43f= cedd634e4', vds=3D'Host[rhvserv-04,0d0cd690-b64a-42bb-a167-43fcedd634e4]'})= ' execution failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException= : Connection failed<br>2017-05-11 17:28:47,388 ERROR [org.ovirt.engine.core= .vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [1efe65a7] = Failure to refresh host 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.c= lient.ClientConnectionException: Connection failed<br>2017-05-11 17:28:50,3= 65 INFO  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL S= tomp Reactor) [] Connecting to rhvserv-03.mydomain.com/192.168.93.213<br>20= 17-05-11 17:28:50,368 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetC= apabilitiesVDSCommand] (DefaultQuartzScheduler5) [48bc69cd] Command 'GetCap= abilitiesVDSCommand(HostName =3D rhvserv-03, VdsIdAndVdsVDSCommandParameter= sBase:{runAsync=3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916', = vds=3D'Host[rhvserv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'})' execution = failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection= failed<br>2017-05-11 17:28:50,368 ERROR [org.ovirt.engine.core.vdsbroker.m= onitoring.HostMonitoring] (DefaultQuartzScheduler5) [48bc69cd] Failure to r= efresh host 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.Client= ConnectionException: Connection failed<br>2017-05-11 17:28:50,390 INFO = ; [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor= ) [] Connecting to rhvserv-04.mydomain.com/192.168.93.214<br>2017-05-11 17:= 28:50,394 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesV= DSCommand] (DefaultQuartzScheduler2) [590af365] Command 'GetCapabilitiesVDS= Command(HostName =3D rhvserv-04, VdsIdAndVdsVDSCommandParametersBase:{runAs= ync=3D'true', hostId=3D'0d0cd690-b64a-42bb-a167-43fcedd634e4', vds=3D'Host[= rhvserv-04,0d0cd690-b64a-42bb-a167-43fcedd634e4]'})' execution failed: org.= ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed<br>2= 017-05-11 17:28:50,394 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.Ho= stMonitoring] (DefaultQuartzScheduler2) [590af365] Failure to refresh host = 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionEx= ception: Connection failed<br>2017-05-11 17:28:53,371 INFO  [org.ovirt= .vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connect= ing to rhvserv-03.mydomain.com/192.168.93.213<br>2017-05-11 17:28:53,396 IN= FO  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp = Reactor) [] Connecting to rhvserv-04.mydomain.com/192.168.93.214<br>2017-05= -11 17:28:56,379 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabi= litiesVDSCommand] (DefaultQuartzScheduler10) [] Command 'GetCapabilitiesVDS= Command(HostName =3D rhvserv-03, VdsIdAndVdsVDSCommandParametersBase:{runAs= ync=3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916', vds=3D'Host[= rhvserv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'})' execution failed: org.= ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed<br>2= 017-05-11 17:28:56,379 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.Ho= stMonitoring] (DefaultQuartzScheduler10) [] Failure to refresh host 'rhvser= v-03' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException= : Connection failed<br>2017-05-11 17:28:56,402 ERROR [org.ovirt.engine.core= .vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler5) [= 48bc69cd] Command 'GetCapabilitiesVDSCommand(HostName =3D rhvserv-04, VdsId= AndVdsVDSCommandParametersBase:{runAsync=3D'true', hostId=3D'0d0cd690-b64a-= 42bb-a167-43fcedd634e4', vds=3D'Host[rhvserv-04,0d0cd690-b64a-42bb-a167-43f= cedd634e4]'})' execution failed: org.ovirt.vdsm.jsonrpc.client.ClientConnec= tionException: Connection failed<br>2017-05-11 17:28:56,403 ERROR [org.ovir= t.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5= ) [48bc69cd] Failure to refresh host 'rhvserv-04' runtime info: org.ovirt.v= dsm.jsonrpc.client.ClientConnectionException: Connection failed<br>2017-05-= 11 17:28:56,488 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmSt= atsVDSCommand] (DefaultQuartzScheduler6) [] Command 'GetAllVmStatsVDSComman= d(HostName =3D rhvserv-04, VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D= 'true', hostId=3D'0d0cd690-b64a-42bb-a167-43fcedd634e4', vds=3D'Host[rhvser= v-04,0d0cd690-b64a-42bb-a167-43fcedd634e4]'})' execution failed: VDSGeneric= Exception: VDSNetworkException: Vds timeout occured<br>2017-05-11 17:28:56,= 488 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefre= sher] (DefaultQuartzScheduler6) [] Failed to fetch vms info for host 'rhvse= rv-04' - skipping VMs monitoring.<br>2017-05-11 17:28:59,382 INFO  [or= g.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] = Connecting to rhvserv-03.mydomain.com/192.168.93.213<br>2017-05-11 17:28:59= ,384 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCom= mand] (DefaultQuartzScheduler10) [] Command 'GetCapabilitiesVDSCommand(Host= Name =3D rhvserv-03, VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true'= , hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916', vds=3D'Host[rhvserv-03,4= 036f027-8e90-49c0-8ca5-3ddb8d586916]'})' execution failed: org.ovirt.vdsm.j= sonrpc.client.ClientConnectionException: Connection failed<br>2017-05-11 17= :28:59,384 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring= ] (DefaultQuartzScheduler10) [] Failure to refresh host 'rhvserv-03' runtim= e info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection= failed<br>2017-05-11 17:28:59,404 INFO  [org.ovirt.vdsm.jsonrpc.clien= t.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to rhvserv-04.m= ydomain.com/192.168.93.214<br>2017-05-11 17:28:59,408 ERROR [org.ovirt.engi= ne.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzSchedu= ler6) [] Command 'GetCapabilitiesVDSCommand(HostName =3D rhvserv-04, VdsIdA= ndVdsVDSCommandParametersBase:{runAsync=3D'true', hostId=3D'0d0cd690-b64a-4= 2bb-a167-43fcedd634e4', vds=3D'Host[rhvserv-04,0d0cd690-b64a-42bb-a167-43fc= edd634e4]'})' execution failed: org.ovirt.vdsm.jsonrpc.client.ClientConnect= ionException: Connection failed<br>2017-05-11 17:28:59,408 ERROR [org.ovirt= .engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6)= [] Failure to refresh host 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonr= pc.client.ClientConnectionException: Connection failed<br>2017-05-11 17:29:= 02,386 INFO  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (S= SL Stomp Reactor) [] Connecting to rhvserv-03.mydomain.com/192.168.93.213<b= r>2017-05-11 17:29:02,409 INFO  [org.ovirt.vdsm.jsonrpc.client.reactor= s.ReactorClient] (SSL Stomp Reactor) [] Connecting to rhvserv-04.mydomain.c= om/192.168.93.214<br>2017-05-11 17:29:02,490 ERROR [org.ovirt.engine.core.v= dsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler8) [56fb= 4c82] Command 'GetAllVmStatsVDSCommand(HostName =3D rhvserv-03, VdsIdAndVds= VDSCommandParametersBase:{runAsync=3D'true', hostId=3D'4036f027-8e90-49c0-8= ca5-3ddb8d586916', vds=3D'Host[rhvserv-03,4036f027-8e90-49c0-8ca5-3ddb8d586= 916]'})' execution failed: VDSGenericException: VDSNetworkException: Vds ti= meout occured<br>2017-05-11 17:29:02,490 INFO  [org.ovirt.engine.core.= vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler8) [56fb4= c82] Failed to fetch vms info for host 'rhvserv-03' - skipping VMs monitori= ng.<br>2017-05-11 17:29:05,393 ERROR [org.ovirt.engine.core.vdsbroker.vdsbr= oker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [] Command 'GetCa= pabilitiesVDSCommand(HostName =3D rhvserv-03, VdsIdAndVdsVDSCommandParamete= rsBase:{runAsync=3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916',= vds=3D'Host[rhvserv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'})' execution= failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connectio= n failed<br>2017-05-11 17:29:05,393 ERROR [org.ovirt.engine.core.vdsbroker.= monitoring.HostMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh = host 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnect= ionException: Connection failed<br>2017-05-11 17:29:05,415 ERROR [org.ovirt= .engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzS= cheduler10) [] Command 'GetCapabilitiesVDSCommand(HostName =3D rhvserv-04, = VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', hostId=3D'0d0cd690-= b64a-42bb-a167-43fcedd634e4', vds=3D'Host[rhvserv-04,0d0cd690-b64a-42bb-a16= 7-43fcedd634e4]'})' execution failed: org.ovirt.vdsm.jsonrpc.client.ClientC= onnectionException: Connection failed<br>2017-05-11 17:29:05,415 ERROR [org= .ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzSched= uler10) [] Failure to refresh host 'rhvserv-04' runtime info: org.ovirt.vds= m.jsonrpc.client.ClientConnectionException: Connection failed<br>2017-05-11= 17:29:08,395 INFO  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorCli= ent] (SSL Stomp Reactor) [] Connecting to rhvserv-03.mydomain.com/192.168.9= 3.213<br>2017-05-11 17:29:08,398 ERROR [org.ovirt.engine.core.vdsbroker.vds= broker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [56fb4c82] Comm= and 'GetCapabilitiesVDSCommand(HostName =3D rhvserv-03, VdsIdAndVdsVDSComma= ndParametersBase:{runAsync=3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb= 8d586916', vds=3D'Host[rhvserv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'})'= execution failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException:= Connection failed<br>2017-05-11 17:29:08,398 ERROR [org.ovirt.engine.core.= vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [56fb4c82] F= ailure to refresh host 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.cl= ient.ClientConnectionException: Connection failed<br>2017-05-11 17:29:08,41= 7 INFO  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL St= omp Reactor) [] Connecting to rhvserv-04.mydomain.com/192.168.93.214<br>201= 7-05-11 17:29:08,420 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCa= pabilitiesVDSCommand] (DefaultQuartzScheduler2) [590af365] Command 'GetCapa= bilitiesVDSCommand(HostName =3D rhvserv-04, VdsIdAndVdsVDSCommandParameters= Base:{runAsync=3D'true', hostId=3D'0d0cd690-b64a-42bb-a167-43fcedd634e4', v= ds=3D'Host[rhvserv-04,0d0cd690-b64a-42bb-a167-43fcedd634e4]'})' execution f= ailed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection = failed<br>2017-05-11 17:29:08,420 ERROR [org.ovirt.engine.core.vdsbroker.mo= nitoring.HostMonitoring] (DefaultQuartzScheduler2) [590af365] Failure to re= fresh host 'rhvserv-04' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientC= onnectionException: Connection failed<br>2017-05-11 17:29:11,402 INFO = [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor)= [] Connecting to rhvserv-03.mydomain.com/192.168.93.213<br>2017-05-11 17:2= 9:11,423 INFO  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] = (SSL Stomp Reactor) [] Connecting to rhvserv-04.mydomain.com/192.168.93.214= <br>2017-05-11 17:29:14,409 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroke= r.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [] Command 'GetCapab= ilitiesVDSCommand(HostName =3D rhvserv-03, VdsIdAndVdsVDSCommandParametersB= ase:{runAsync=3D'true', hostId=3D'4036f027-8e90-49c0-8ca5-3ddb8d586916', vd= s=3D'Host[rhvserv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'})' execution fa= iled: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection f= ailed<br>2017-05-11 17:29:14,409 ERROR [org.ovirt.engine.core.vdsbroker.mon= itoring.HostMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh hos= t 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnection= Exception: Connection failed<br>2017-05-11 17:29:14,430 ERROR [org.ovirt.en= gine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzSche= duler5) [48bc69cd] Command 'GetCapabilitiesVDSCommand(HostName =3D rhvserv-= 04, VdsIdAndVdsVDSCommandParametersBase:{runAsync=3D'true', hostId=3D'0d0cd= 690-b64a-42bb-a167-43fcedd634e4', vds=3D'Host[rhvserv-04,0d0cd690-b64a-42bb= -a167-43fcedd634e4]'})' execution failed: org.ovirt.vdsm.jsonrpc.client.Cli= entConnectionException: Connection failed<br>2017-05-11 17:29:14,430 ERROR = [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzS= cheduler5) [48bc69cd] Failure to refresh host 'rhvserv-04' runtime info: or= g.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed<br= ddb8d586916', vds=3D'Host[rhvserv-03,4036f027-8e90-49c0-8ca5-3ddb8d586916]'= })' execution failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionExcepti= on: Connection failed<br>2017-05-11 17:29:17,414 ERROR [org.ovirt.engine.co= re.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [] Failur= e to refresh host 'rhvserv-03' runtime info: org.ovirt.vdsm.jsonrpc.client.= ClientConnectionException: Connection failed<br>2017-05-11 17:29:17,432 INF= O  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp R= eactor) [] Connecting to rhvserv-04.mydomain.com/192.168.93.214<br>2017-05-= 11 17:29:17,436 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabil= itiesVDSCommand] (DefaultQuartzScheduler4) [] Command 'GetCapabilitiesVDSCo= mmand(HostName =3D rhvserv-04, VdsIdAndVdsVDSCommandParametersBase:{runAsyn= c=3D'true', hostId=3D'0d0cd690-b64a-42bb-a167-43fcedd634e4', vds=3D'Host[rh= vserv-04,0d0cd690-b64a-42bb-a167-43fcedd634e4]'})' execution failed: org.ov= irt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed<br>201= 7-05-11 17:29:17,436 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.Host= Monitoring] (DefaultQuartzScheduler4) [] Failure to refresh host 'rhvserv-0= 4' runtime info: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: C= onnection failed<br>2017-05-11 17:29:17,491 INFO  [org.ovirt.vdsm.json= rpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to rhv= serv-03.mydomain.com/192.168.93.213</div><div><br></div><div><br></div><div= p;   raise se.StorageDomainDoesNotExist(sdUUID)<br>StorageDomainD= oesNotExist: Storage domain does not exist: (u'5b978dda-d1ef-46fe-9996-20ae= e42cf303',)<br>Thread-112::DEBUG::2017-05-11 17:41:33,783::lvm::288::Storag= e.Misc.excCmd::(cmd) SUCCESS: <err> =3D ''; <rc> =3D 0<br>Threa= d-112::INFO::2017-05-11 17:41:33,783::sd::604::Storage.StorageDomain::(_reg= isterResourceNamespaces) Resource namespace edd229cb-b72f-4988-8c10-d83c84e= f4a8a_imageNS already registered<br>Thread-112::INFO::2017-05-11 17:41:33,7= 83::sd::612::Storage.StorageDomain::(_registerResourceNamespaces) Resource = namespace edd229cb-b72f-4988-8c10-d83c84ef4a8a_volumeNS already registered<= br>Thread-112::INFO::2017-05-11 17:41:33,783::blockSD::846::Storage.Storage= Domain::(_registerResourceNamespaces) Resource namespace edd229cb-b72f-4988= -8c10-d83c84ef4a8a_lvmActivationNS already registered<br>Thread-112::DEBUG:= :2017-05-11 17:41:33,784::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/tas= kset --cpu-list 0-15 /usr/bin/sudo -n /usr/sbin/lvm vgck --config ' devices= { preferred_names =3D ["^/dev/mapper/"] ignore_suspended_devices=3D1 write= _cache_state=3D0 disable_after_error_count=3D3 filter =3D [ '\''a|/dev/mapp= er/36782bcb00073e33200002d3858fa1a81|'\'', '\''r|.*|'\'' ] }  global {=   locking_type=3D1  prioritise_write_locks=3D1  wait_for_loc= ks=3D1  use_lvmetad=3D0 }  backup {  retain_min =3D 50 = retain_days =3D 0 } ' edd229cb-b72f-4988-8c10-d83c84ef4a8a (cwd None)<br>j= sonrpc.Executor/6::DEBUG::2017-05-11 17:41:33,792::lvm::288::Storage.Misc.e= xcCmd::(cmd) SUCCESS: <err> =3D ''; <rc> =3D 0<br>jsonrpc.Execu= tor/6::INFO::2017-05-11 17:41:33,792::logUtils::52::dispatcher::(wrapper) R= un and protect: getStorageDomainStats, Return response: {'stats': {'mdasize= ': '134217728', 'mdathreshold': True, 'mdavalid': True, 'diskfree': '498484= 641792', 'disktotal': '557943095296', 'mdafree': '67102208'}}<br>jsonrpc.Ex= ecutor/6::DEBUG::2017-05-11 17:41:33,793::task::1193::Storage.TaskManager.T= ask::(prepare) Task=3D`234cbec4-a422-47c1-a50a-243526aeddc7`::finished: {'s= tats': {'mdasize': '134217728', 'mdathreshold': True, 'mdavalid': True, 'di= skfree': '498484641792', 'disktotal': '557943095296', 'mdafree': '67102208'= }}<br>jsonrpc.Executor/6::DEBUG::2017-05-11 17:41:33,793::task::597::Storag= e.TaskManager.Task::(_updateState) Task=3D`234cbec4-a422-47c1-a50a-243526ae= ddc7`::moving from state preparing -> state finished<br>jsonrpc.Executor= /6::DEBUG::2017-05-11 17:41:33,793::resourceManager::952::Storage.ResourceM= anager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storag= e.595de2cf-89ba-407b-aae5-d0a7a0656ba1': < ResourceRef 'Storage.595de2cf= -89ba-407b-aae5-d0a7a0656ba1', isValid: 'True' obj: 'None'>}<br>jsonrpc.= Executor/6::DEBUG::2017-05-11 17:41:33,793::resourceManager::989::Storage.R= esourceManager.Owner::(cancelAll) Owner.cancelAll requests {}<br>jsonrpc.Ex= ecutor/6::DEBUG::2017-05-11 17:41:33,793::resourceManager::628::Storage.Res= ourceManager::(releaseResource) Trying to release resource 'Storage.595de2c= f-89ba-407b-aae5-d0a7a0656ba1'<br>jsonrpc.Executor/6::DEBUG::2017-05-11 17:= 41:33,793::resourceManager::647::Storage.ResourceManager::(releaseResource)= Released resource 'Storage.595de2cf-89ba-407b-aae5-d0a7a0656ba1' (0 active= users)<br>jsonrpc.Executor/6::DEBUG::2017-05-11 17:41:33,793::resourceMana= ger::653::Storage.ResourceManager::(releaseResource) Resource 'Storage.595d= e2cf-89ba-407b-aae5-d0a7a0656ba1' is free, finding out if anyone is waiting= for it.<br>jsonrpc.Executor/6::DEBUG::2017-05-11 17:41:33,793::resourceMan= ager::661::Storage.ResourceManager::(releaseResource) No one is waiting for= resource 'Storage.595de2cf-89ba-407b-aae5-d0a7a0656ba1', Clearing records.= <br>jsonrpc.Executor/6::DEBUG::2017-05-11 17:41:33,793::task::995::Storage.= TaskManager.Task::(_decref) Task=3D`234cbec4-a422-47c1-a50a-243526aeddc7`::= ref 0 aborting False<br>jsonrpc.Executor/6::DEBUG::2017-05-11 17:41:33,794:= :__init__::555::jsonrpc.JsonRpcServer::(_handle_request) Return 'StorageDom= ain.getStats' in bridge with {'mdasize': '134217728', 'mdathreshold': True,= 'mdavalid': True, 'diskfree': '498484641792', 'disktotal': '557943095296',= 'mdafree': '67102208'}<br>jsonrpc.Executor/6::INFO::2017-05-11 17:41:33,79= 4::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call StorageDo= main.getStats succeeded in 1.05 seconds<br>JsonRpc (StompReactor)::ERROR::2= 017-05-11 17:41:33,816::betterAsyncore::113::vds.dispatcher::(recv) SSL err= or during reading data: unexpected eof<br>Thread-112::DEBUG::2017-05-11 17:= 41:33,846::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err> =3D ''= ; <rc> =3D 0<br>Thread-12::DEBUG::2017-05-11 17:41:34,913::check::296= ::storage.check::(_start_process) START check '/dev/595de2cf-89ba-407b-aae5= -d0a7a0656ba1/metadata' cmd=3D['/usr/bin/taskset', '--cpu-list', '0-15', '/= usr/bin/dd', 'if=3D/dev/595de2cf-89ba-407b-aae5-d0a7a0656ba1/metadata', 'of= =3D/dev/null', 'bs=3D4096', 'count=3D1', 'iflag=3Ddirect'] delay=3D0.00<br>= Thread-12::DEBUG::2017-05-11 17:41:34,954::asyncevent::564::storage.asyncev= ent::(reap) Process <cpopen.CPopen object at 0x1b15b90> terminated (c= ount=3D1)<br>Thread-12::DEBUG::2017-05-11 17:41:34,954::check::327::storage= .check::(_check_completed) FINISH check '/dev/595de2cf-89ba-407b-aae5-d0a7a= 0656ba1/metadata' rc=3D0 err=3Dbytearray(b'1+0 records in\n1+0 records out\= n4096 bytes (4.1 kB) copied, 0.000537649 s, 7.6 MB/s\n') elapsed=3D0.05<br>= Reactor thread::INFO::2017-05-11 17:41:35,823::protocoldetector::76::Protoc= olDetector.AcceptorImpl::(handle_accept) Accepted connection from ::1:49348= <br>Reactor thread::DEBUG::2017-05-11 17:41:35,827::protocoldetector::92::P= rotocolDetector.Detector::(__init__) Using required_size=3D11<br>Reactor th= read::INFO::2017-05-11 17:41:35,828::protocoldetector::128::ProtocolDetecto= r.Detector::(handle_read) Detected protocol stomp from ::1:49348<br>Reactor= thread::INFO::2017-05-11 17:41:35,828::stompreactor::101::Broker.StompAdap= ter::(_cmd_connect) Processing CONNECT request<br>Reactor thread::DEBUG::20= 17-05-11 17:41:35,829::stompreactor::492::protocoldetector.StompDetector::(= handle_socket) Stomp detected from ('::1', 49348)<br>JsonRpc (StompReactor)= ::INFO::2017-05-11 17:41:35,829::stompreactor::128::Broker.StompAdapter::(_= cmd_subscribe) Subscribe command received</div></div></body></html> ------=_Part_11446852_617310615.1494517429918--