[ovirt-users] Re: VM HostedEngine is down with error

Thank you very much for your reply. I checked the NTP and realized the service wasn't working properly on two of the three nodes, but despite that the clocks seemed to have the correct time(date and hwdate). I switched to chronyd and stopped the ntpd service and now it seems the servers' clocks are synchronized. The time in BIOS has different time than the systems. Does this affect the behaviour of the overall performance? This is what I could gather from the logs: Node1: [root@ov-no1 ~]# more /var/log/messages |egrep "(WARN|error)"|more d Aug 27 17:53:08 ov-no1 libvirtd: 2020-08-27 14:53:08.947+0000: 5613: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Aug 27 17:58:08 ov-no1 libvirtd: 2020-08-27 14:58:08.943+0000: 5613: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Aug 27 18:03:08 ov-no1 libvirtd: 2020-08-27 15:03:08.937+0000: 5614: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Aug 27 18:08:08 ov-no1 libvirtd: 2020-08-27 15:08:08.951+0000: 5617: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Aug 27 18:13:08 ov-no1 libvirtd: 2020-08-27 15:13:08.951+0000: 5616: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Aug 27 18:18:08 ov-no1 libvirtd: 2020-08-27 15:18:08.942+0000: 5618: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected . . [Around that time Node 3, who is the arbiter node, was placed in local Maintenance mode. It was shut down for maintenance and when we boot it up again all seemed right. We removed it from the maintenance mode and when the healing processes finished, Node 1 became NonResponsive. Long story short, the VDSM agent sent Node1 a restart command. Node1 rebooted, HostedEngine was up on Node1 and the rest of the VMs that were hosted by Node1 had to be manually brought up. Since then everything seemed to be working as it should. The HostedEngine VM shutdown with no apparent reason 5 days later, making us believe there was no connection between the two incidents.] . . . . . Sep 1 05:53:30 ov-no1 vdsm[6706]: WARN Worker blocked: <Worker name=jsonrpc/3 running <Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0', 'method': u'GlusterVolume.geoRepSessionList', 'id': u'e1f4db88-5d36-4ba1-8848-3855c1dbc6ee'} at 0x7f1ed7381650> timeout=60, duration=60.00 at 0x7f1ed7381dd0> t ask#=76268 at 0x7f1ebc0797d0>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012Fil e: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run #012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/p ython2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 3 91, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler(sel f._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_request (req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: "/usr /lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-package s/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012 remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", l ine 93, in wrapper#012 rv = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in volumeGeoRepSessionL ist#012 remoteUserName,#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in glusterVolumeGeoRep Status#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() Sep 1 05:54:30 ov-no1 vdsm[6706]: WARN Worker blocked: <Worker name=jsonrpc/3 running <Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0', 'method': u'GlusterVolume.geoRepSessionList', 'id': u'e1f4db88-5d36-4ba1-8848-3855c1dbc6ee'} at 0x7f1ed7381650> timeout=60, duration=120.00 at 0x7f1ed7381dd0> task#=76268 at 0x7f1ebc0797d0>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012Fi le: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in ru n#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/ python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler(se lf._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_reques t(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: "/us r/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-packag es/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012 remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 93, in wrapper#012 rv = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in volumeGeoRepSession List#012 remoteUserName,#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012File : "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in glusterVolumeGeoRe pStatus#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() [root@ov-no1 ~]# xzcat /var/log/vdsm/vdsm.log.57.xz |egrep "(WARN|error)" |more 2020-09-01 09:40:50,023+0300 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:40:50,023+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:41:00,061+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:41:00,061+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:41:10,078+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:41:10,078+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:41:20,086+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:41:20,086+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:41:30,107+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:41:30,107+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:41:38,710+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:41:38,711+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:41:40,115+0300 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:41:40,115+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.01 seconds (__init__:312) 2020-09-01 09:41:44,541+0300 WARN (monitor/e2e7282) [storage.PersistentDict] Could not parse line ` 2020-09-01 09:41:44,544+0300 WARN (monitor/0a5e552) [storage.PersistentDict] Could not parse line ` [root@ov-no1 ~]# xzcat /var/log/vdsm/supervdsm.log.29.xz |egrep "(WARN|error|ERR)" |more MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,182::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,183::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,185::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,304::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,306::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,308::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,440::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,441::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,442::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,442::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,443::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,443::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,444::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,445::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,445::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,358::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,361::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,363::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,365::udev::53::blivet::(get_device) No device at '' [root@ov-no1 ~]# xzcat /var/log/vdsm/supervdsm.log.28.xz |egrep "(WARN|error|ERR)" |more MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,730::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,731::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,732::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,847::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,849::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,850::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,974::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,975::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,975::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,976::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,977::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,977::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,978::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,979::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,979::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/7::ERROR::2020-09-01 11:32:40,951::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/7::ERROR::2020-09-01 11:32:40,954::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/7::ERROR::2020-09-01 11:32:40,956::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/7::ERROR::2020-09-01 11:32:40,958::udev::53::blivet::(get_device) No device at '' [root@ov-no1 ~]# cat /var/log/ovirt-hosted-engine-ha/broker.log.2020-09-01 |egrep -i "(WARN|ERR)" |more [nothing in the output] [root@ov-no1 ~]# gluster volume status Status of volume: data Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ov-no1.ariadne-t.local:/gluster_brick s/data/data 49152 0 Y 6070 Brick ov-no2.ariadne-t.local:/gluster_brick s/data/data 49152 0 Y 28860 Brick ov-no3.ariadne-t.local:/gluster_brick s/data/data 49152 0 Y 5752 Self-heal Daemon on localhost N/A N/A Y 6539 Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813 Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353 Task Status of Volume data ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: engine Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ov-no1.ariadne-t.local:/gluster_brick s/engine/engine 49153 0 Y 6098 Brick ov-no2.ariadne-t.local:/gluster_brick s/engine/engine 49156 0 Y 28867 Brick ov-no3.ariadne-t.local:/gluster_brick s/engine/engine 49153 0 Y 5789 Self-heal Daemon on localhost N/A N/A Y 6539 Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813 Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353 Task Status of Volume engine ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: vmstore Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ov-no1.ariadne-t.local:/gluster_brick s/vmstore/vmstore 49154 0 Y 6102 Brick ov-no2.ariadne-t.local:/gluster_brick s/vmstore/vmstore 49157 0 Y 28878 Brick ov-no3.ariadne-t.local:/gluster_brick s/vmstore/vmstore 49154 0 Y 5772 Self-heal Daemon on localhost N/A N/A Y 6539 Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813 Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353 Task Status of Volume vmstore ------------------------------------------------------------------------------ There are no active volume tasks [root@ov-no1 ~]# gluster volume heal engine info Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 [root@ov-no1 ~]# gluster volume heal engine info summary Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [root@ov-no1 ~]# gluster volume heal data info Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data Status: Connected Number of entries: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data Status: Connected Number of entries: 0 [root@ov-no1 ~]# gluster volume heal data info summary Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [root@ov-no1 ~]# gluster volume heal vmstore info Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 [root@ov-no1 ~]# gluster volume heal vmstore info summary Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [root@ov-no1 ~]# ps -A|grep gluster 5522 ? 00:01:40 glustereventsd 5591 ? 00:05:04 glusterd 6070 ? 02:16:28 glusterfsd 6098 ? 05:44:14 glusterfsd 6102 ? 00:23:48 glusterfsd 6285 ? 00:00:38 glustereventsd 6539 ? 01:32:19 glusterfs 10470 ? 00:34:10 glusterfs 17763 ? 00:50:18 glusterfs 18101 ? 00:10:23 glusterfs Node2: [root@ov-no2 ~]# more /var/log/messages |egrep "(WARN|error)"|more Sep 1 07:58:30 ov-no2 vdsm[13586]: WARN Worker blocked: <Worker name=jsonrpc/7 running <Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0', 'method': u'GlusterVolume.geoRepSessionList', 'id': u'ec8af6cc-6109-4ddb-a6fb-ad9f4fbc1e7b'} at 0x7f44602a7c90> timeout=60, duration=120.00 at 0x7f44602a7510> task#=5773350 at 0x7f44c00c71d0>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#01 2File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 r et = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/l ib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", li ne 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler (self._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_req uest(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: " /usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-pac kages/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012 remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py ", line 93, in wrapper#012 rv = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in volumeGeoRepSess ionList#012 remoteUserName,#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012F ile: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in glusterVolumeGe oRepStatus#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() Sep 1 09:48:02 ov-no2 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Unhandled monitoring loop exception#012T raceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 449, in start_moni toring#012 self._monitoring_loop()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 468, in _monit oring_loop#012 for old_state, state, delay in self.fsm:#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py", li ne 127, in next#012 new_data = self.refresh(self._state.data)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/state_machi ne.py", line 81, in refresh#012 stats.update(self.hosted_engine.collect_stats())#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_h a/agent/hosted_engine.py", line 756, in collect_stats#012 all_stats = self._broker.get_stats_from_storage()#012 File "/usr/lib/python2.7/site-pac kages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 143, in get_stats_from_storage#012 result = self._proxy.get_stats()#012 File "/usr/lib64/py thon2.7/xmlrpclib.py", line 1233, in __call__#012 return self.__send(self.__name, args)#012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in __request#012 verbose=self.__verbose#012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request#012 return self.single_request(hos t, handler, request_body, verbose)#012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1301, in single_request#012 self.send_content(h, request_bo dy)#012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content#012 connection.endheaders(request_body)#012 File "/usr/lib64/python 2.7/httplib.py", line 1037, in endheaders#012 self._send_output(message_body)#012 File "/usr/lib64/python2.7/httplib.py", line 881, in _send_outp ut#012 self.send(msg)#012 File "/usr/lib64/python2.7/httplib.py", line 843, in send#012 self.connect()#012 File "/usr/lib/python2.7/site-pack ages/ovirt_hosted_engine_ha/lib/unixrpc.py", line 52, in connect#012 self.sock.connect(base64.b16decode(self.host))#012 File "/usr/lib64/python2. 7/socket.py", line 224, in meth#012 return getattr(self._sock,name)(*args)#012error: [Errno 2] No such file or directory Sep 1 09:48:02 ov-no2 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/ python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-p ackages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-pack ages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 456, in start_monitoring#012 self.publish(stopped)#012 File "/usr/lib/python2.7/site-pa ckages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 356, in publish#012 self._push_to_storage(blocks)#012 File "/usr/lib/python2.7/site-p ackages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 727, in _push_to_storage#012 self._broker.put_stats_on_storage(self.host_id, blocks)# 012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 113, in put_stats_on_storage#012 self._proxy.put_stats (host_id, xmlrpclib.Binary(data))#012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__#012 return self.__send(self.__name, args)# 012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in __request#012 verbose=self.__verbose#012 File "/usr/lib64/python2.7/xmlrpclib.py", l ine 1273, in request#012 return self.single_request(host, handler, request_body, verbose)#012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1301 , in single_request#012 self.send_content(h, request_body)#012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content#012 connec tion.endheaders(request_body)#012 File "/usr/lib64/python2.7/httplib.py", line 1037, in endheaders#012 self._send_output(message_body)#012 File "/usr/lib64/python2.7/httplib.py", line 881, in _send_output#012 self.send(msg)#012 File "/usr/lib64/python2.7/httplib.py", line 843, in send#012 self.connect()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py", line 52, in connect#012 self.sock.connect(b ase64.b16decode(self.host))#012 File "/usr/lib64/python2.7/socket.py", line 224, in meth#012 return getattr(self._sock,name)(*args)#012error: [Er rno 2] No such file or directory Sep 1 09:48:15 ov-no2 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to host#012Trace back (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py", line 26, in send_email#012 timeout=float(cfg["smtp-timeout"]))#012 File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) = self.connect(host, po rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012 self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No route to host Sep 1 09:48:25 ov-no2 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to host#012Trace back (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py", line 26, in send_email#012 timeout=float(cfg["smtp-timeout"]))#012 File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) = self.connect(host, po rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012 self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No route to host Sep 1 09:49:15 ov-no2 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to host#012Trace back (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py", line 26, in send_email#012 timeout=float(cfg["smtp-timeout"]))#012 File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) = self.connect(host, po rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012 self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No route to host Sep 1 09:49:16 ov-no2 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to host#012Trace back (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py", line 26, in send_email#012 timeout=float(cfg["smtp-timeout"]))#012 File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) = self.connect(host, po rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012 self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No route to host Sep 1 09:49:17 ov-no2 vdsm[13586]: WARN Attempting to add an existing net user: ovirtmgmt/baa3d4d1-6918-4447-80fa-64b86dbcb1f9 Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet0 -j libvirt-J-vnet0' f ailed: Illegal target name 'libvirt-J-vnet0'. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-P-vnet0' failed: Illegal target name 'libvirt-P-vnet0'. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet0' failed: Chain 'libvirt-J -vnet0' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet0' failed: Chain 'libvirt-P -vnet0' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet0' failed: Chain 'libvirt-J -vnet0' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet0' failed: Chain 'libvirt-J -vnet0' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet0' failed: Chain 'libvirt-P -vnet0' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet0' failed: Chain 'libvirt-P -vnet0' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet0-mac' failed: Chain 'J-vnet0-mac' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet0-mac' failed: Chain 'J-vnet0-mac' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet0-arp-mac' failed: Chain 'J-vnet0-a rp-mac' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet0-arp-mac' failed: Chain 'J-vnet0-a rp-mac' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev- out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0 ' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: iptables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vn et0' failed: iptables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FO-vnet0' failed: iptables: No chain/target/match by t hat name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FO-vnet0' failed: iptables: No chain/target/match by t hat name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FI-vnet0' failed: iptables: No chain/target/match by t hat name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FI-vnet0' failed: iptables: No chain/target/match by t hat name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HI-vnet0' failed: iptables: No chain/target/match by t hat name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HI-vnet0' failed: iptables: No chain/target/match by t hat name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FP-vnet0 FO-vnet0' failed: iptables: No chain/target/m atch by that name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FJ-vnet0 FI-vnet0' failed: iptables: No chain/target/m atch by that name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E HJ-vnet0 HI-vnet0' failed: iptables: No chain/target/m atch by that name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev -out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more informatio n. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet 0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: ip6tables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-v net0' failed: ip6tables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FO-vnet0' failed: ip6tables: No chain/target/match by that name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FO-vnet0' failed: ip6tables: No chain/target/match by that name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FI-vnet0' failed: ip6tables: No chain/target/match by that name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FI-vnet0' failed: ip6tables: No chain/target/match by that name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HI-vnet0' failed: ip6tables: No chain/target/match by that name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HI-vnet0' failed: ip6tables: No chain/target/match by that name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E FP-vnet0 FO-vnet0' failed: ip6tables: No chain/target /match by that name. Sep 1 09:49:20 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E FJ-vnet0 FI-vnet0' failed: ip6tables: No chain/target /match by that name. Sep 1 09:49:20 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E HJ-vnet0 HI-vnet0' failed: ip6tables: No chain/target /match by that name. . . . . . Sep 1 09:49:22 ov-no2 vdsm[13586]: WARN File: /var/lib/libvirt/qemu/channels/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.ovirt-guest-agent.0 already remove d Sep 1 09:49:22 ov-no2 vdsm[13586]: WARN Attempting to remove a non existing net user: ovirtmgmt/baa3d4d1-6918-4447-80fa-64b86dbcb1f9 Sep 1 09:49:22 ov-no2 vdsm[13586]: WARN File: /var/lib/libvirt/qemu/channels/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.org.qemu.guest_agent.0 already rem oved Sep 1 09:49:22 ov-no2 vdsm[13586]: WARN File: /var/run/ovirt-vmconsole-console/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.sock already removed Sep 1 09:49:27 ov-no2 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to host#012Trace back (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py", line 26, in send_email#012 timeout=float(cfg["smtp-timeout"]))#012 File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) = self.connect(host, po rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012 self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No route to host Sep 1 09:49:27 ov-no2 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to host#012Trace back (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py", line 26, in send_email#012 timeout=float(cfg["smtp-timeout"]))#012 File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) = self.connect(host, po rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012 self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No route to host Sep 1 09:49:37 ov-no2 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to host#012Trace back (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py", line 26, in send_email#012 timeout=float(cfg["smtp-timeout"]))#012 File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) = self.connect(host, po rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012 self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No route to host Sep 1 10:01:31 ov-no2 vdsm[13586]: WARN Worker blocked: <Worker name=jsonrpc/4 running <Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0', 'method': u'GlusterVolume.geoRepSessionList', 'id': u'7f800636-f107-4b98-a35f-2175ec72b7b8'} at 0x7f44805aa110> timeout=60, duration=60.00 at 0x7f44805aad10> task#=5775347 at 0x7f44d81c4110>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012 File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 re t = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/li b/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", lin e 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler( self._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_requ est(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: "/ usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-pack ages/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012 remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py" , line 93, in wrapper#012 rv = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in volumeGeoRepSessi onList#012 remoteUserName,#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012Fi le: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in glusterVolumeGeo RepStatus#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() [root@ov-no2 ~]# xzcat /var/log/vdsm/vdsm.log.57.xz |egrep "(WARN|error)" |more 2020-09-01 09:24:06,889+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:24:16,910+0300 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:24:16,911+0300 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:24:26,932+0300 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:24:26,933+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.01 seconds (__init__:312) 2020-09-01 09:24:30,375+0300 WARN (monitor/0a5e552) [storage.PersistentDict] Could not parse line ` 2020-09-01 09:24:30,379+0300 WARN (monitor/e2e7282) [storage.PersistentDict] Could not parse line ` . . . . 2020-09-01 09:47:37,008+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:47:47,030+0300 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:47:47,030+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:48:01,712+0300 WARN (jsonrpc/6) [storage.PersistentDict] Could not parse line ` 2020-09-01 09:48:01,743+0300 WARN (jsonrpc/1) [storage.PersistentDict] Could not parse line ` 2020-09-01 09:48:02,220+0300 WARN (jsonrpc/0) [storage.PersistentDict] Could not parse line ` 2020-09-01 09:48:05,192+0300 WARN (periodic/22) [storage.PersistentDict] Could not parse line ` 2020-09-01 09:48:11,745+0300 WARN (monitor/e2e7282) [storage.PersistentDict] Could not parse line ` 2020-09-01 09:48:13,711+0300 INFO (jsonrpc/7) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:48:13,711+0300 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:48:23,591+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:48:23,591+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:48:23,731+0300 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:48:23,731+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:48:33,754+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:48:33,754+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:48:43,787+0300 INFO (jsonrpc/2) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:48:43,787+0300 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:48:53,808+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:48:53,809+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:49:03,830+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:49:03,831+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:49:13,852+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:49:13,853+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.01 seconds (__init__:312) 2020-09-01 09:49:16,042+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:49:16,042+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.01 seconds (__init__:312) 2020-09-01 09:49:16,405+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:49:16,405+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:49:16,797+0300 INFO (jsonrpc/7) [api.virt] START create(vmParams={u'xml': u'<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n<domain xmlns: ovirt-tune="http://ovirt.org/vm/tune/1.0" xmlns:ovirt-vm="http://ovirt.org/vm/1.0" type="kvm"><name>HostedEngine</name><uuid>baa3d4d1-6918-4447-80fa- 64b86dbcb1f9</uuid><memory>16777216</memory><currentMemory>16777216</currentMemory><iothreads>1</iothreads><maxMemory slots="16">67108864</maxMemory> <vcpu current="4">64</vcpu><sysinfo type="smbios"><system><entry name="manufacturer">oVirt</entry><entry name="product">OS-NAME:</entry><entry name=" version">OS-VERSION:</entry><entry name="serial">HOST-SERIAL:</entry><entry name="uuid">baa3d4d1-6918-4447-80fa-64b86dbcb1f9</entry></system></sysinf o><clock offset="variable" adjustment="0"><timer name="rtc" tickpolicy="catchup"/><timer name="pit" tickpolicy="delay"/><timer name="hpet" present="n o"/></clock><features><acpi/></features><cpu match="exact"><model>Westmere</model><feature name="pcid" policy="require"/><feature name="spec-ctrl" po licy="require"/><feature name="ssbd" policy="require"/><topology cores="4" threads="1" sockets="16"/><numa><cell id="0" cpus="0,1,2,3" memory="167772 16"/></numa></cpu><cputune/><devices><input type="tablet" bus="usb"/><channel type="unix"><target type="virtio" name="ovirt-guest-agent.0"/><source m ode="bind" path="/var/lib/libvirt/qemu/channels/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.ovirt-guest-agent.0"/></channel><channel type="unix"><target typ e="virtio" name="org.qemu.guest_agent.0"/><source mode="bind" path="/var/lib/libvirt/qemu/channels/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.org.qemu.gues t_agent.0"/></channel><controller type="usb" model="piix3-uhci" index="0"><address bus="0x00" domain="0x0000" function="0x2" slot="0x01" type="pci"/> </controller><console type="unix"><source path="/var/run/ovirt-vmconsole-console/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.sock" mode="bind"/><target type ="serial" port="0"/><alias name="ua-228f9c56-c195-4f1e-8cd2-5f7ef60a78f1"/></console><memballoon model="virtio"><stats period="5"/><alias name="ua-42 10968c-acf1-46cc-8a35-6e29e3c58b38"/><address bus="0x00" domain="0x0000" function="0x0" slot="0x08" type="pci"/></memballoon><controller type="scsi" model="virtio-scsi" index="0"><driver iothread="1"/><alias name="ua-473cb0f7-09da-4077-a0e2-8eb5a5efdbe4"/><address bus="0x00" domain="0x0000" functi on="0x0" slot="0x05" type="pci"/></controller><sound model="ich6"><alias name="ua-496f9a72-5484-4dc0-83f4-caa8225100ae"/><address bus="0x00" domain=" 0x0000" function="0x0" slot="0x04" type="pci"/></sound><graphics type="vnc" port="-1" autoport="yes" passwd="*****" passwdValidTo="1970-01-01T00:00:0 1" keymap="en-us"><listen type="network" network="vdsm-ovirtmgmt"/></graphics><controller type="ide" index="0"><address bus="0x00" domain="0x0000" fu nction="0x1" slot="0x01" type="pci"/></controller><graphics type="spice" port="-1" autoport="yes" passwd="*****" passwdValidTo="1970-01-01T00:00:01" tlsPort="-1"><channel name="main" mode="secure"/><channel name="inputs" mode="secure"/><channel name="cursor" mode="secure"/><channel name="playback" mode="secure"/><channel name="record" mode="secure"/><channel name="display" mode="secure"/><channel name="smartcard" mode="secure"/><channel name=" usbredir" mode="secure"/><listen type="network" network="vdsm-ovirtmgmt"/></graphics><rng model="virtio"><backend model="random">/dev/urandom</backen d><alias name="ua-9edfa822-3b11-424e-83f8-4846c6221a6e"/></rng><video><model type="qxl" vram="32768" heads="1" ram="65536" vgamem="16384"/><alias nam e="ua-a4ad75db-c91d-45df-a767-33a71009587f"/><address bus="0x00" domain="0x0000" function="0x0" slot="0x02" type="pci"/></video><controller type="vir tio-serial" index="0" ports="16"><alias name="ua-f60f1742-07fe-464e-a160-1a16c94db61a"/><address bus="0x00" domain="0x0000" function="0x0" slot="0x06 " type="pci"/></controller><serial type="unix"><source path="/var/run/ovirt-vmconsole-console/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.sock" mode="bind"/ "virtio"/><link state="up"/><source bridge="ovirtmgmt"/><driver queues="4" name="vhost"/><alias name="ua-e97c94c0-ffc2-46dd-b5cd-9d9c123f4db0"/><addr ess bus="0x00" domain="0x0000" function="0x0" slot="0x03" type="pci"/><mac address="00:16:3e:27:6e:ef"/><mtu size="1500"/><filterref filter="vdsm-no- mac-spoofing"/><bandwidth/></interface><disk type="file" device="cdrom" snapshot="no"><driver name="qemu" type="raw" error_policy="report"/><source f ile="" startupPolicy="optional"><seclabel model="dac" type="none" relabel="no"/></source><target dev="hdc" bus="ide"/><readonly/><alias name="ua-9034 9138-d20e-4306-b0e2-271e6baaad4a"/><address bus="1" controller="0" unit="0" type="drive" target="0"/></disk><disk snapshot="no" type="file" device="d isk"><target dev="vda" bus="virtio"/><source file="/rhev/data-center/00000000-0000-0000-0000-000000000000/80f6e393-9718-4738-a14a-64cf43c3d8c2/images /d5de54b6-9f8e-4fba-819b-ebf6780757d2/a48555f4-be23-4467-8a54-400ae7baf9d7"><seclabel model="dac" type="none" relabel="no"/></source><driver name="qe mu" iothread="1" io="threads" type="raw" error_policy="stop" cache="none"/><alias name="ua-d5de54b6-9f8e-4fba-819b-ebf6780757d2"/><address bus="0x00" domain="0x0000" function="0x0" slot="0x07" type="pci"/><serial>d5de54b6-9f8e-4fba-819b-ebf6780757d2</serial></disk><lease><key>a48555f4-be23-4467-8a 54-400ae7baf9d7</key><lockspace>80f6e393-9718-4738-a14a-64cf43c3d8c2</lockspace><target offset="LEASE-OFFSET:a48555f4-be23-4467-8a54-400ae7baf9d7:80f 6e393-9718-4738-a14a-64cf43c3d8c2" path="LEASE-PATH:a48555f4-be23-4467-8a54-400ae7baf9d7:80f6e393-9718-4738-a14a-64cf43c3d8c2"/></lease></devices><pm nfo"/><bios useserial="yes"/></os><metadata><ovirt-tune:qos/><ovirt-vm:vm><ovirt-vm:minGuaranteedMemoryMb type="int">1024</ovirt-vm:minGuaranteedMemo ryMb><ovirt-vm:clusterVersion>4.3</ovirt-vm:clusterVersion><ovirt-vm:custom/><ovirt-vm:device mac_address="00:16:3e:27:6e:ef"><ovirt-vm:custom/></ovi rt-vm:device><ovirt-vm:device devtype="disk" name="vda"><ovirt-vm:poolID>00000000-0000-0000-0000-000000000000</ovirt-vm:poolID><ovirt-vm:volumeID>a48 555f4-be23-4467-8a54-400ae7baf9d7</ovirt-vm:volumeID><ovirt-vm:shared>exclusive</ovirt-vm:shared><ovirt-vm:imageID>d5de54b6-9f8e-4fba-819b-ebf6780757 d2</ovirt-vm:imageID><ovirt-vm:domainID>80f6e393-9718-4738-a14a-64cf43c3d8c2</ovirt-vm:domainID></ovirt-vm:device><ovirt-vm:launchPaused>false</ovirt -vm:launchPaused><ovirt-vm:resumeBehavior>auto_resume</ovirt-vm:resumeBehavior></ovirt-vm:vm></metadata></domain>'}) from=::1,36826, vmId=baa3d4d1-69 18-4447-80fa-64b86dbcb1f9 (api:48) [root@ov-no2 ~]# xzcat /var/log/vdsm/supervdsm.log.30.xz |egrep "(WARN|error|ERR)" |more MainProcess|jsonrpc/4::ERROR::2020-09-01 06:56:30,652::supervdsm_server::103::SuperVdsm.ServerCallback::(wrapper) Error in volumeGeoRepStatus MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,206::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,213::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,217::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,565::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,571::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,573::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,735::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,735::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,736::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,736::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,737::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,738::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,740::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,740::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,741::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/3::ERROR::2020-09-01 07:32:31,889::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/3::ERROR::2020-09-01 07:32:31,893::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/3::ERROR::2020-09-01 07:32:31,894::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/3::ERROR::2020-09-01 07:32:31,896::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/7::ERROR::2020-09-01 07:58:30,879::supervdsm_server::103::SuperVdsm.ServerCallback::(wrapper) Error in volumeGeoRepStatus [root@ov-no2 ~]# xzcat /var/log/vdsm/supervdsm.log.29.xz |egrep "(WARN|error|ERR)" |more MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,179::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,180::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,181::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,390::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,393::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,395::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,510::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,510::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,511::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,511::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,512::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,512::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,513::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,513::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,514::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,440::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,444::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,446::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,448::udev::53::blivet::(get_device) No device at '' [root@ov-no2 ~]# cat /var/log/vdsm/supervdsm.log |egrep "(WARN|error|ERR)" |more MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,056::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,057::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,059::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,177::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,179::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,180::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,292::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,292::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,293::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,294::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,294::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,295::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,295::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,296::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,296::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,217::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,220::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,222::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,223::udev::53::blivet::(get_device) No device at '' [root@ov-no2 ~]# cat /var/log/ovirt-hosted-engine-ha/agent.log.2020-09-01 |egrep -i "(WARN|ERR)" |more MainThread::ERROR::2020-09-01 09:48:02,766::hosted_engine::452::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Unhandled monitoring loop exception error: [Errno 2] No such file or directory MainThread::ERROR::2020-09-01 09:48:02,786::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): error: [Errno 2] No such file or directory MainThread::ERROR::2020-09-01 09:48:02,786::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2020-09-01 09:49:16,925::hosted_engine::854::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) stderr: Co mmand VM.getStats with args {'vmID': 'baa3d4d1-6918-4447-80fa-64b86dbcb1f9'} failed: MainThread::INFO::2020-09-01 09:49:27,616::hosted_engine::931::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) stderr: Com mand VM.destroy with args {'vmID': 'baa3d4d1-6918-4447-80fa-64b86dbcb1f9'} failed: MainThread::ERROR::2020-09-01 09:49:27,616::hosted_engine::936::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) Failed to stop engine vm with /usr/sbin/hosted-engine --vm-poweroff: Command VM.destroy with args {'vmID': 'baa3d4d1-6918-4447-80fa-64b86dbcb1f9'} failed: MainThread::ERROR::2020-09-01 09:49:27,616::hosted_engine::942::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) Failed to stop engine VM: Command VM.destroy with args {'vmID': 'baa3d4d1-6918-4447-80fa-64b86dbcb1f9'} failed: [root@ov-no2 ~]# cat /var/log/ovirt-hosted-engine-ha/broker.log.2020-09-01 |egrep -i "(WARN|ERR)" |more StatusStorageThread::ERROR::2020-09-01 09:47:52,786::status_broker::90::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(run) Failed to update state. raise ex.HostIdNotLockedError("Host id is not set") HostIdNotLockedError: Host id is not set StatusStorageThread::ERROR::2020-09-01 09:47:52,787::status_broker::70::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(trigger_res tart) Trying to restart the broker Listener::ERROR::2020-09-01 09:48:15,543::notifications::39::ovirt_hosted_engine_ha.broker.notifications.Notifications::(send_email) [Errno 113] No r oute to host raise err error: [Errno 113] No route to host Listener::ERROR::2020-09-01 09:48:25,768::notifications::39::ovirt_hosted_engine_ha.broker.notifications.Notifications::(send_email) [Errno 113] No r oute to host raise err error: [Errno 113] No route to host Listener::ERROR::2020-09-01 09:49:15,838::notifications::39::ovirt_hosted_engine_ha.broker.notifications.Notifications::(send_email) [Errno 113] No r oute to host raise err The Gluster status and info I could gather for the present are the following: [root@ov-no2 ~]# gluster volume status Status of volume: data Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ov-no1.ariadne-t.local:/gluster_brick s/data/data 49152 0 Y 6070 Brick ov-no2.ariadne-t.local:/gluster_brick s/data/data 49152 0 Y 28860 Brick ov-no3.ariadne-t.local:/gluster_brick s/data/data 49152 0 Y 5752 Self-heal Daemon on localhost N/A N/A Y 813 Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539 Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353 Task Status of Volume data ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: engine Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ov-no1.ariadne-t.local:/gluster_brick s/engine/engine 49153 0 Y 6098 Brick ov-no2.ariadne-t.local:/gluster_brick s/engine/engine 49156 0 Y 28867 Brick ov-no3.ariadne-t.local:/gluster_brick s/engine/engine 49153 0 Y 5789 Self-heal Daemon on localhost N/A N/A Y 813 Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539 Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353 Task Status of Volume engine ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: vmstore Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ov-no1.ariadne-t.local:/gluster_brick s/vmstore/vmstore 49154 0 Y 6102 Brick ov-no2.ariadne-t.local:/gluster_brick s/vmstore/vmstore 49157 0 Y 28878 Brick ov-no3.ariadne-t.local:/gluster_brick s/vmstore/vmstore 49154 0 Y 5772 Self-heal Daemon on localhost N/A N/A Y 813 Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539 Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353 Task Status of Volume vmstore ------------------------------------------------------------------------------ There are no active volume tasks [root@ov-no2 ~]# gluster volume heal engine info Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 [root@ov-no2 ~]# gluster volume heal engine info summary Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [root@ov-no2 ~]# gluster volume heal data info Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data Status: Connected Number of entries: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data Status: Connected Number of entries: 0 [root@ov-no2 ~]# gluster volume heal data info summary Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [root@ov-no2 ~]# gluster volume heal vmstore info Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 [root@ov-no2 ~]# gluster volume heal vmstore info summary Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [root@ov-no2 ~]# ps -A |grep gluster 813 ? 02:18:00 glusterfs 14183 ? 1-12:02:26 glusterfs 17944 ? 01:19:03 glustereventsd 17995 ? 00:31:50 glustereventsd 28722 ? 00:02:09 glusterd 28860 ? 02:19:48 glusterfsd 28867 ? 05:23:00 glusterfsd 28878 ? 00:20:15 glusterfsd 32187 ? 1-14:42:54 glusterfs 32309 ? 09:59:54 glusterfs Node3: [root@ov-no3 ~]# virsh qemu-agent-command HostedEngine '{"execute":"guest-info"}' Please enter your authentication name: root Please enter your password: {"return":{"version":"2.12.0","supported_commands":[{"enabled":true,"name":"guest-get-osinfo","success-response":true},{"enabled":true,"name":"guest-get-timezone","success-response":true},{"enabled":true,"name":"guest-get-users","success-response":true},{"enabled":true,"name":"guest-get-host-name","success-response":true},{"enabled":false,"name":"guest-exec","success-response":true},{"enabled":false,"name":"guest-exec-status","success-response":true},{"enabled":true,"name":"guest-get-memory-block-info","success-response":true},{"enabled":true,"name":"guest-set-memory-blocks","success-response":true},{"enabled":true,"name":"guest-get-memory-blocks","success-response":true},{"enabled":true,"name":"guest-set-user-password","success-response":true},{"enabled":true,"name":"guest-get-fsinfo","success-response":true},{"enabled":true,"name":"guest-set-vcpus","success-response":true},{"enabled":true,"name":"guest-get-vcpus","success-response":true},{"enabled":true,"name":"guest-network-get-in terfaces","success-response":true},{"enabled":true,"name":"guest-suspend-hybrid","success-response":false},{"enabled":true,"name":"guest-suspend-ram","success-response":false},{"enabled":true,"name":"guest-suspend-disk","success-response":false},{"enabled":true,"name":"guest-fstrim","success-response":true},{"enabled":true,"name":"guest-fsfreeze-thaw","success-response":true},{"enabled":true,"name":"guest-fsfreeze-freeze-list","success-response":true},{"enabled":true,"name":"guest-fsfreeze-freeze","success-response":true},{"enabled":true,"name":"guest-fsfreeze-status","success-response":true},{"enabled":false,"name":"guest-file-flush","success-response":true},{"enabled":false,"name":"guest-file-seek","success-response":true},{"enabled":false,"name":"guest-file-write","success-response":true},{"enabled":false,"name":"guest-file-read","success-response":true},{"enabled":false,"name":"guest-file-close","success-response":true},{"enabled":false,"name":"guest-file-open","success-response ":true},{"enabled":true,"name":"guest-shutdown","success-response":false},{"enabled":true,"name":"guest-info","success-response":true},{"enabled":true,"name":"guest-set-time","success-response":true},{"enabled":true,"name":"guest-get-time","success-response":true},{"enabled":true,"name":"guest-ping","success-response":true},{"enabled":true,"name":"guest-sync","success-response":true},{"enabled":true,"name":"guest-sync-delimited","success-response":true}]}} [root@ov-no3 ~]# cat /var/log/messages|egrep -i "(WARN|ERR)" |more Sep 1 08:55:17 ov-no3 libvirtd: 2020-09-01 05:55:17.816+0000: 5284: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 08:59:30 ov-no3 vdsm[6112]: WARN Worker blocked: <Worker name=jsonrpc/3 running <Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0', 'method': u'GlusterVolume.geoRepSessionList', 'id': u'8298ea63-eadc-47b8-b8b4-d54982200010'} at 0x7f3c14739d90> timeout=60, duration=60.00 at 0x7f3c14739dd0> t ask#=72655 at 0x7f3c34306550>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012Fil e: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run #012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/p ython2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 3 91, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler(sel f._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_request (req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: "/usr /lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-package s/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012 remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", l ine 93, in wrapper#012 rv = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in volumeGeoRepSessionL ist#012 remoteUserName,#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in glusterVolumeGeoRep Status#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() Sep 1 09:00:17 ov-no3 libvirtd: 2020-09-01 06:00:17.817+0000: 5281: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:00:30 ov-no3 vdsm[6112]: WARN Worker blocked: <Worker name=jsonrpc/3 running <Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0', 'method': u'GlusterVolume.geoRepSessionList', 'id': u'8298ea63-eadc-47b8-b8b4-d54982200010'} at 0x7f3c14739d90> timeout=60, duration=120.00 at 0x7f3c14739dd0> task#=72655 at 0x7f3c34306550>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012Fi le: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in ru n#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/ python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler(se lf._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_reques t(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: "/us r/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-packag es/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012 remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 93, in wrapper#012 rv = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in volumeGeoRepSession List#012 remoteUserName,#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012File : "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in glusterVolumeGeoRe pStatus#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() Sep 1 09:05:17 ov-no3 libvirtd: 2020-09-01 06:05:17.816+0000: 5283: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:10:17 ov-no3 libvirtd: 2020-09-01 06:10:17.823+0000: 5282: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:15:17 ov-no3 libvirtd: 2020-09-01 06:15:17.817+0000: 5281: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:20:17 ov-no3 libvirtd: 2020-09-01 06:20:17.820+0000: 5282: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:25:17 ov-no3 libvirtd: 2020-09-01 06:25:17.818+0000: 5280: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:30:17 ov-no3 libvirtd: 2020-09-01 06:30:17.820+0000: 5282: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:35:17 ov-no3 libvirtd: 2020-09-01 06:35:17.815+0000: 5281: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:40:17 ov-no3 libvirtd: 2020-09-01 06:40:17.823+0000: 5284: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:45:17 ov-no3 libvirtd: 2020-09-01 06:45:17.819+0000: 5280: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:50:17 ov-no3 libvirtd: 2020-09-01 06:50:17.824+0000: 5280: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:55:17 ov-no3 libvirtd: 2020-09-01 06:55:17.818+0000: 5284: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 10:00:17 ov-no3 libvirtd: 2020-09-01 07:00:17.822+0000: 5284: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 10:05:17 ov-no3 libvirtd: 2020-09-01 07:05:17.824+0000: 5282: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected . . . . [which continues as so 'till now] [root@ov-no3 ~]# xzcat /var/log/vdsm/vdsm.log.57.xz |egrep "(WARN|error)" |more 2020-09-01 09:31:35,023+0300 WARN (monitor/0a5e552) [storage.PersistentDict] Could not parse line ``. (persistent:241) 2020-09-01 09:31:35,026+0300 WARN (monitor/e2e7282) [storage.PersistentDict] Could not parse line ``. (persistent:241) 2020-09-01 09:35:19,868+0300 WARN (monitor/80f6e39) [storage.PersistentDict] Could not parse line ``. (persistent:241) 2020-09-01 09:35:22,210+0300 WARN (qgapoller/1) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:35:32,207+0300 WARN (qgapoller/3) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:35:42,213+0300 WARN (qgapoller/1) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:35:52,212+0300 WARN (qgapoller/2) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:36:02,207+0300 WARN (qgapoller/1) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:36:12,215+0300 WARN (qgapoller/2) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:36:35,276+0300 WARN (monitor/0a5e552) [storage.PersistentDict] Could not parse line ``. (persistent:241) 2020-09-01 09:36:35,281+0300 WARN (monitor/e2e7282) [storage.PersistentDict] Could not parse line ``. (persistent:241) 2020-09-01 09:40:18,494+0300 WARN (qgapoller/3) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c6803ff50> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:40:18,534+0300 WARN (qgapoller/1) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c6803fe60> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:40:20,244+0300 WARN (monitor/80f6e39) [storage.PersistentDict] Could not parse line ``. (persistent:241) 2020-09-01 09:40:22,209+0300 WARN (qgapoller/3) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:40:32,207+0300 WARN (qgapoller/1) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:40:42,214+0300 WARN (qgapoller/3) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:40:52,215+0300 WARN (qgapoller/2) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) [root@ov-no3 ~]# xzcat /var/log/vdsm/supervdsm.log.29.xz|egrep "(WARN|ERR)" |more MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,165::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,166::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,168::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,285::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,287::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,288::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,396::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,397::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,397::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,398::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,398::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,399::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,399::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,400::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,400::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/4::ERROR::2020-09-01 09:32:36,337::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/4::ERROR::2020-09-01 09:32:36,341::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/4::ERROR::2020-09-01 09:32:36,342::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/4::ERROR::2020-09-01 09:32:36,344::udev::53::blivet::(get_device) No device at '' [root@ov-no3 ~]# cat /var/log/vdsm/supervdsm.log|egrep "(WARN|ERR)" |more MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,041::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,043::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,044::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,158::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,160::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,161::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,265::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,266::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,266::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,267::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,267::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,268::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,269::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,269::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,270::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,399::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,402::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,404::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,406::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/7::ERROR::2020-09-03 19:54:43,983::supervdsm_server::103::SuperVdsm.ServerCallback::(wrapper) Error in volumeGeoRepStatus [root@ov-no3 ~]# cat /var/log/ovirt-hosted-engine-ha/agent.log.2020-09-01 |egrep -i "(WARN|ERR)" |more [nothing in the output] [root@ov-no3 ~]# cat /var/log/ovirt-hosted-engine-ha/broker.log.2020-09-01 |egrep -i "(WARN|ERR)" |more Thread-1::WARNING::2020-09-02 12:46:59,657::network::92::network.Network::(action) Failed to verify network status, (4 out of 5) [root@ov-no3 ~]# gluster volume status Status of volume: data Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ov-no1.ariadne-t.local:/gluster_brick s/data/data 49152 0 Y 6070 Brick ov-no2.ariadne-t.local:/gluster_brick s/data/data 49152 0 Y 28860 Brick ov-no3.ariadne-t.local:/gluster_brick s/data/data 49152 0 Y 5752 Self-heal Daemon on localhost N/A N/A Y 19353 Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539 Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813 Task Status of Volume data ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: engine Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ov-no1.ariadne-t.local:/gluster_brick s/engine/engine 49153 0 Y 6098 Brick ov-no2.ariadne-t.local:/gluster_brick s/engine/engine 49156 0 Y 28867 Brick ov-no3.ariadne-t.local:/gluster_brick s/engine/engine 49153 0 Y 5789 Self-heal Daemon on localhost N/A N/A Y 19353 Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539 Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813 Task Status of Volume engine ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: vmstore Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ov-no1.ariadne-t.local:/gluster_brick s/vmstore/vmstore 49154 0 Y 6102 Brick ov-no2.ariadne-t.local:/gluster_brick s/vmstore/vmstore 49157 0 Y 28878 Brick ov-no3.ariadne-t.local:/gluster_brick s/vmstore/vmstore 49154 0 Y 5772 Self-heal Daemon on localhost N/A N/A Y 19353 Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539 Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813 Task Status of Volume vmstore ------------------------------------------------------------------------------ There are no active volume tasks [root@ov-no3 ~]# gluster volume heal engine info Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 [root@ov-no3 ~]# gluster volume heal engine info summary Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [root@ov-no3 ~]# gluster volume heal data info Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data Status: Connected Number of entries: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data Status: Connected Number of entries: 0 [root@ov-no3 ~]# gluster volume heal data info summary Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [root@ov-no3 ~]# gluster volume heal vmstore info Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 [root@ov-no3 ~]# gluster volume heal vmstore info summary Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [root@ov-no3 ~]# ps -A|grep gluster 5143 ? 00:01:31 glustereventsd 5752 ? 02:01:08 glusterfsd 5772 ? 00:17:26 glusterfsd 5789 ? 05:12:24 glusterfsd 5906 ? 00:00:38 glustereventsd 6382 ? 02:37:00 glusterfs 9843 ? 00:02:18 glusterd 10613 ? 00:22:08 glusterfs 10750 ? 00:13:24 glusterfs 19353 ? 01:20:36 glusterfs Hosted Engine: [root@ov-eng ~]# grep ERROR /var/log/ovirt-engine/engine.log |more 2020-09-02 03:36:35,199+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler8) [1a87b58 d] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 04:38:35,429+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler2) [15fadfa 2] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 05:40:35,655+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler9) [50fa85f 0] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 06:42:35,885+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler7) [3092fdc c] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 07:44:36,119+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler6) [696ffca 7] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 08:46:36,341+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler9) [180bb50 ] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 09:48:36,562+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler6) [3a45e1b 3] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 10:50:36,788+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler4) [7da1197 3] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 11:52:37,011+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler6) [76b3cb6 5] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 12:54:37,234+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler5) [6a0d904 9] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 13:56:37,459+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler4) [1e25ecf f] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 14:58:37,674+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler1) [64144e3 4] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' How can I find why the HE failover happened? I believe it's the gluster geo replication, but I haven't configured geo replication in the enviroment. Can you help me how to proceed? Thank you all in advance, Maria Souvalioti