Hello everyone, I have a replica 2 + arbiter installation and this morning the Hosted Engine gave the following error on the UI and resumed on a different node (node3) than the one it was originally running(node1). (The original node has more memory than the one it ended up, but it had a better memory usage percentage at the time). Also, the only way I discovered the migration had happened and there was an Error in Events, was because I logged in the web interface of ovirt for a routine inspection. Βesides that, everything was working properly and still is. The error that popped is the following: VM HostedEngine is down with error. Exit message: internal error: qemu unexpectedly closed the monitor: 2020-09-01T06:49:20.749126Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future 2020-09-01T06:49:20.927274Z qemu-kvm: -device virtio-blk-pci,iothread=iothread1,scsi=off,bus=pci.0,addr=0x7,drive=drive-ua-d5de54b6-9f8e-4fba-819b-ebf6780757d2,id=ua-d5de54b6-9f8e-4fba-819b-ebf6780757d2,bootindex=1,write-cache=on: Failed to get "write" lock Is another process using the image?. Which from what I could gather concerns the following snippet from the HostedEngine.xml and it's the virtio disk of the Hosted Engine: <disk type='file' device='disk' snapshot='no'> <driver name='qemu' type='raw' cache='none' error_policy='stop' io='threads' iothread='1'/> <source file='/var/run/vdsm/storage/80f6e393-9718-4738-a14a-64cf43c3d8c2/d5de54b6-9f8e-4fba-819b-ebf6780757d2/a48555f4-be23-4467-8a54-400ae7baf9d7'> <seclabel model='dac' relabel='no'/> </source> <target dev='vda' bus='virtio'/> <serial>d5de54b6-9f8e-4fba-819b-ebf6780757d2</serial> <alias name='ua-d5de54b6-9f8e-4fba-819b-ebf6780757d2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </disk> I've tried looking into the logs and the sar command but I couldn't find anything to relate with the above errors and determining the reason for it to happen. Is this a Gluster or a QEMU problem? The Hosted Engine was manually migrated five days before on node1. Is there a standard practice I could follow to determine what happened and secure my system? Thank you very much for your time, Maria Souvalioti
On Tue, Sep 1, 2020 at 7:17 PM <souvaliotimaria@mail.com> wrote:
Hello everyone,
I have a replica 2 + arbiter installation and this morning the Hosted Engine gave the following error on the UI and resumed on a different node (node3) than the one it was originally running(node1). (The original node has more memory than the one it ended up, but it had a better memory usage percentage at the time). Also, the only way I discovered the migration had happened and there was an Error in Events, was because I logged in the web interface of ovirt for a routine inspection. Βesides that, everything was working properly and still is.
The error that popped is the following:
VM HostedEngine is down with error. Exit message: internal error: qemu unexpectedly closed the monitor: 2020-09-01T06:49:20.749126Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future 2020-09-01T06:49:20.927274Z qemu-kvm: -device virtio-blk-pci,iothread=iothread1,scsi=off,bus=pci.0,addr=0x7,drive=drive-ua-d5de54b6-9f8e-4fba-819b-ebf6780757d2,id=ua-d5de54b6-9f8e-4fba-819b-ebf6780757d2,bootindex=1,write-cache=on: Failed to get "write" lock Is another process using the image?.
It's quite likely that this isn't the root cause. Please check your logs from before that. Above looks like something (ovirt-ha-agent?) tried to start the hosted engine VM, but failed due to locking - most likely, because it was already up elsewhere (on some other host?). So you want to check when/where the VM was started before this error, and then carefully any errors before it was started. Also, check that the clocks on all your machines are in sync.
Which from what I could gather concerns the following snippet from the HostedEngine.xml and it's the virtio disk of the Hosted Engine:
<disk type='file' device='disk' snapshot='no'> <driver name='qemu' type='raw' cache='none' error_policy='stop' io='threads' iothread='1'/> <source file='/var/run/vdsm/storage/80f6e393-9718-4738-a14a-64cf43c3d8c2/d5de54b6-9f8e-4fba-819b-ebf6780757d2/a48555f4-be23-4467-8a54-400ae7baf9d7'> <seclabel model='dac' relabel='no'/> </source> <target dev='vda' bus='virtio'/> <serial>d5de54b6-9f8e-4fba-819b-ebf6780757d2</serial> <alias name='ua-d5de54b6-9f8e-4fba-819b-ebf6780757d2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </disk>
I've tried looking into the logs and the sar command but I couldn't find anything to relate with the above errors and determining the reason for it to happen. Is this a Gluster or a QEMU problem?
Likely, but hard to tell without more information.
The Hosted Engine was manually migrated five days before on node1.
Is there a standard practice I could follow to determine what happened and secure my system?
Nothing, other than checking the logs. Check, on all of your hosts: /var/log/messages /var/log/vdsm/* /var/log/ovirt-hosted-engine-ha/* And on the engine (likely won't help in this case, but just in case): /var/log/ovirt-engine/*
Thank you very much for your time,
Good luck and best regards, -- Didi
Thank you very much for your reply. I checked the NTP and realized the service wasn't working properly on two of the three nodes, but despite that the clocks seemed to have the correct time(date and hwdate). I switched to chronyd and stopped the ntpd service and now it seems the servers' clocks are synchronized. The time in BIOS has different time than the systems. Does this affect the behaviour of the overall performance? This is what I could gather from the logs: Node1: [root@ov-no1 ~]# more /var/log/messages |egrep "(WARN|error)"|more d Aug 27 17:53:08 ov-no1 libvirtd: 2020-08-27 14:53:08.947+0000: 5613: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Aug 27 17:58:08 ov-no1 libvirtd: 2020-08-27 14:58:08.943+0000: 5613: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Aug 27 18:03:08 ov-no1 libvirtd: 2020-08-27 15:03:08.937+0000: 5614: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Aug 27 18:08:08 ov-no1 libvirtd: 2020-08-27 15:08:08.951+0000: 5617: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Aug 27 18:13:08 ov-no1 libvirtd: 2020-08-27 15:13:08.951+0000: 5616: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Aug 27 18:18:08 ov-no1 libvirtd: 2020-08-27 15:18:08.942+0000: 5618: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected . . [Around that time Node 3, who is the arbiter node, was placed in local Maintenance mode. It was shut down for maintenance and when we boot it up again all seemed right. We removed it from the maintenance mode and when the healing processes finished, Node 1 became NonResponsive. Long story short, the VDSM agent sent Node1 a restart command. Node1 rebooted, HostedEngine was up on Node1 and the rest of the VMs that were hosted by Node1 had to be manually brought up. Since then everything seemed to be working as it should. The HostedEngine VM shutdown with no apparent reason 5 days later, making us believe there was no connection between the two incidents.] . . . . . Sep 1 05:53:30 ov-no1 vdsm[6706]: WARN Worker blocked: <Worker name=jsonrpc/3 running <Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0', 'method': u'GlusterVolume.geoRepSessionList', 'id': u'e1f4db88-5d36-4ba1-8848-3855c1dbc6ee'} at 0x7f1ed7381650> timeout=60, duration=60.00 at 0x7f1ed7381dd0> t ask#=76268 at 0x7f1ebc0797d0>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012Fil e: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run #012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/p ython2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 3 91, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler(sel f._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_request (req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: "/usr /lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-package s/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012 remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", l ine 93, in wrapper#012 rv = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in volumeGeoRepSessionL ist#012 remoteUserName,#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in glusterVolumeGeoRep Status#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() Sep 1 05:54:30 ov-no1 vdsm[6706]: WARN Worker blocked: <Worker name=jsonrpc/3 running <Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0', 'method': u'GlusterVolume.geoRepSessionList', 'id': u'e1f4db88-5d36-4ba1-8848-3855c1dbc6ee'} at 0x7f1ed7381650> timeout=60, duration=120.00 at 0x7f1ed7381dd0> task#=76268 at 0x7f1ebc0797d0>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012Fi le: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in ru n#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/ python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler(se lf._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_reques t(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: "/us r/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-packag es/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012 remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 93, in wrapper#012 rv = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in volumeGeoRepSession List#012 remoteUserName,#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012File : "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in glusterVolumeGeoRe pStatus#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() [root@ov-no1 ~]# xzcat /var/log/vdsm/vdsm.log.57.xz |egrep "(WARN|error)" |more 2020-09-01 09:40:50,023+0300 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:40:50,023+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:41:00,061+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:41:00,061+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:41:10,078+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:41:10,078+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:41:20,086+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:41:20,086+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:41:30,107+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:41:30,107+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:41:38,710+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:41:38,711+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:41:40,115+0300 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:41:40,115+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.01 seconds (__init__:312) 2020-09-01 09:41:44,541+0300 WARN (monitor/e2e7282) [storage.PersistentDict] Could not parse line ` 2020-09-01 09:41:44,544+0300 WARN (monitor/0a5e552) [storage.PersistentDict] Could not parse line ` [root@ov-no1 ~]# xzcat /var/log/vdsm/supervdsm.log.29.xz |egrep "(WARN|error|ERR)" |more MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,182::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,183::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,185::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,304::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,306::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,308::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,440::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,441::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,442::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,442::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,443::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,443::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,444::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,445::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,445::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,358::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,361::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,363::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,365::udev::53::blivet::(get_device) No device at '' [root@ov-no1 ~]# xzcat /var/log/vdsm/supervdsm.log.28.xz |egrep "(WARN|error|ERR)" |more MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,730::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,731::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,732::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,847::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,849::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,850::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,974::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,975::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,975::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,976::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,977::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,977::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,978::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,979::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,979::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/7::ERROR::2020-09-01 11:32:40,951::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/7::ERROR::2020-09-01 11:32:40,954::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/7::ERROR::2020-09-01 11:32:40,956::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/7::ERROR::2020-09-01 11:32:40,958::udev::53::blivet::(get_device) No device at '' [root@ov-no1 ~]# cat /var/log/ovirt-hosted-engine-ha/broker.log.2020-09-01 |egrep -i "(WARN|ERR)" |more [nothing in the output] [root@ov-no1 ~]# gluster volume status Status of volume: data Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ov-no1.ariadne-t.local:/gluster_brick s/data/data 49152 0 Y 6070 Brick ov-no2.ariadne-t.local:/gluster_brick s/data/data 49152 0 Y 28860 Brick ov-no3.ariadne-t.local:/gluster_brick s/data/data 49152 0 Y 5752 Self-heal Daemon on localhost N/A N/A Y 6539 Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813 Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353 Task Status of Volume data ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: engine Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ov-no1.ariadne-t.local:/gluster_brick s/engine/engine 49153 0 Y 6098 Brick ov-no2.ariadne-t.local:/gluster_brick s/engine/engine 49156 0 Y 28867 Brick ov-no3.ariadne-t.local:/gluster_brick s/engine/engine 49153 0 Y 5789 Self-heal Daemon on localhost N/A N/A Y 6539 Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813 Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353 Task Status of Volume engine ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: vmstore Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ov-no1.ariadne-t.local:/gluster_brick s/vmstore/vmstore 49154 0 Y 6102 Brick ov-no2.ariadne-t.local:/gluster_brick s/vmstore/vmstore 49157 0 Y 28878 Brick ov-no3.ariadne-t.local:/gluster_brick s/vmstore/vmstore 49154 0 Y 5772 Self-heal Daemon on localhost N/A N/A Y 6539 Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813 Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353 Task Status of Volume vmstore ------------------------------------------------------------------------------ There are no active volume tasks [root@ov-no1 ~]# gluster volume heal engine info Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 [root@ov-no1 ~]# gluster volume heal engine info summary Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [root@ov-no1 ~]# gluster volume heal data info Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data Status: Connected Number of entries: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data Status: Connected Number of entries: 0 [root@ov-no1 ~]# gluster volume heal data info summary Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [root@ov-no1 ~]# gluster volume heal vmstore info Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 [root@ov-no1 ~]# gluster volume heal vmstore info summary Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [root@ov-no1 ~]# ps -A|grep gluster 5522 ? 00:01:40 glustereventsd 5591 ? 00:05:04 glusterd 6070 ? 02:16:28 glusterfsd 6098 ? 05:44:14 glusterfsd 6102 ? 00:23:48 glusterfsd 6285 ? 00:00:38 glustereventsd 6539 ? 01:32:19 glusterfs 10470 ? 00:34:10 glusterfs 17763 ? 00:50:18 glusterfs 18101 ? 00:10:23 glusterfs Node2: [root@ov-no2 ~]# more /var/log/messages |egrep "(WARN|error)"|more Sep 1 07:58:30 ov-no2 vdsm[13586]: WARN Worker blocked: <Worker name=jsonrpc/7 running <Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0', 'method': u'GlusterVolume.geoRepSessionList', 'id': u'ec8af6cc-6109-4ddb-a6fb-ad9f4fbc1e7b'} at 0x7f44602a7c90> timeout=60, duration=120.00 at 0x7f44602a7510> task#=5773350 at 0x7f44c00c71d0>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#01 2File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 r et = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/l ib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", li ne 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler (self._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_req uest(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: " /usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-pac kages/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012 remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py ", line 93, in wrapper#012 rv = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in volumeGeoRepSess ionList#012 remoteUserName,#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012F ile: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in glusterVolumeGe oRepStatus#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() Sep 1 09:48:02 ov-no2 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Unhandled monitoring loop exception#012T raceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 449, in start_moni toring#012 self._monitoring_loop()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 468, in _monit oring_loop#012 for old_state, state, delay in self.fsm:#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py", li ne 127, in next#012 new_data = self.refresh(self._state.data)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/state_machi ne.py", line 81, in refresh#012 stats.update(self.hosted_engine.collect_stats())#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_h a/agent/hosted_engine.py", line 756, in collect_stats#012 all_stats = self._broker.get_stats_from_storage()#012 File "/usr/lib/python2.7/site-pac kages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 143, in get_stats_from_storage#012 result = self._proxy.get_stats()#012 File "/usr/lib64/py thon2.7/xmlrpclib.py", line 1233, in __call__#012 return self.__send(self.__name, args)#012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in __request#012 verbose=self.__verbose#012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request#012 return self.single_request(hos t, handler, request_body, verbose)#012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1301, in single_request#012 self.send_content(h, request_bo dy)#012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content#012 connection.endheaders(request_body)#012 File "/usr/lib64/python 2.7/httplib.py", line 1037, in endheaders#012 self._send_output(message_body)#012 File "/usr/lib64/python2.7/httplib.py", line 881, in _send_outp ut#012 self.send(msg)#012 File "/usr/lib64/python2.7/httplib.py", line 843, in send#012 self.connect()#012 File "/usr/lib/python2.7/site-pack ages/ovirt_hosted_engine_ha/lib/unixrpc.py", line 52, in connect#012 self.sock.connect(base64.b16decode(self.host))#012 File "/usr/lib64/python2. 7/socket.py", line 224, in meth#012 return getattr(self._sock,name)(*args)#012error: [Errno 2] No such file or directory Sep 1 09:48:02 ov-no2 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/ python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-p ackages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-pack ages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 456, in start_monitoring#012 self.publish(stopped)#012 File "/usr/lib/python2.7/site-pa ckages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 356, in publish#012 self._push_to_storage(blocks)#012 File "/usr/lib/python2.7/site-p ackages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 727, in _push_to_storage#012 self._broker.put_stats_on_storage(self.host_id, blocks)# 012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 113, in put_stats_on_storage#012 self._proxy.put_stats (host_id, xmlrpclib.Binary(data))#012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__#012 return self.__send(self.__name, args)# 012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in __request#012 verbose=self.__verbose#012 File "/usr/lib64/python2.7/xmlrpclib.py", l ine 1273, in request#012 return self.single_request(host, handler, request_body, verbose)#012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1301 , in single_request#012 self.send_content(h, request_body)#012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content#012 connec tion.endheaders(request_body)#012 File "/usr/lib64/python2.7/httplib.py", line 1037, in endheaders#012 self._send_output(message_body)#012 File "/usr/lib64/python2.7/httplib.py", line 881, in _send_output#012 self.send(msg)#012 File "/usr/lib64/python2.7/httplib.py", line 843, in send#012 self.connect()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py", line 52, in connect#012 self.sock.connect(b ase64.b16decode(self.host))#012 File "/usr/lib64/python2.7/socket.py", line 224, in meth#012 return getattr(self._sock,name)(*args)#012error: [Er rno 2] No such file or directory Sep 1 09:48:15 ov-no2 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to host#012Trace back (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py", line 26, in send_email#012 timeout=float(cfg["smtp-timeout"]))#012 File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) = self.connect(host, po rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012 self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No route to host Sep 1 09:48:25 ov-no2 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to host#012Trace back (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py", line 26, in send_email#012 timeout=float(cfg["smtp-timeout"]))#012 File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) = self.connect(host, po rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012 self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No route to host Sep 1 09:49:15 ov-no2 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to host#012Trace back (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py", line 26, in send_email#012 timeout=float(cfg["smtp-timeout"]))#012 File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) = self.connect(host, po rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012 self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No route to host Sep 1 09:49:16 ov-no2 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to host#012Trace back (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py", line 26, in send_email#012 timeout=float(cfg["smtp-timeout"]))#012 File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) = self.connect(host, po rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012 self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No route to host Sep 1 09:49:17 ov-no2 vdsm[13586]: WARN Attempting to add an existing net user: ovirtmgmt/baa3d4d1-6918-4447-80fa-64b86dbcb1f9 Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet0 -j libvirt-J-vnet0' f ailed: Illegal target name 'libvirt-J-vnet0'. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-P-vnet0' failed: Illegal target name 'libvirt-P-vnet0'. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet0' failed: Chain 'libvirt-J -vnet0' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet0' failed: Chain 'libvirt-P -vnet0' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet0' failed: Chain 'libvirt-J -vnet0' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet0' failed: Chain 'libvirt-J -vnet0' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet0' failed: Chain 'libvirt-P -vnet0' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet0' failed: Chain 'libvirt-P -vnet0' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet0-mac' failed: Chain 'J-vnet0-mac' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet0-mac' failed: Chain 'J-vnet0-mac' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet0-arp-mac' failed: Chain 'J-vnet0-a rp-mac' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet0-arp-mac' failed: Chain 'J-vnet0-a rp-mac' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev- out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0 ' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: iptables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vn et0' failed: iptables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FO-vnet0' failed: iptables: No chain/target/match by t hat name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FO-vnet0' failed: iptables: No chain/target/match by t hat name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FI-vnet0' failed: iptables: No chain/target/match by t hat name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FI-vnet0' failed: iptables: No chain/target/match by t hat name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HI-vnet0' failed: iptables: No chain/target/match by t hat name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HI-vnet0' failed: iptables: No chain/target/match by t hat name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FP-vnet0 FO-vnet0' failed: iptables: No chain/target/m atch by that name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FJ-vnet0 FI-vnet0' failed: iptables: No chain/target/m atch by that name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E HJ-vnet0 HI-vnet0' failed: iptables: No chain/target/m atch by that name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev -out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more informatio n. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet 0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: ip6tables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-v net0' failed: ip6tables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FO-vnet0' failed: ip6tables: No chain/target/match by that name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FO-vnet0' failed: ip6tables: No chain/target/match by that name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FI-vnet0' failed: ip6tables: No chain/target/match by that name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FI-vnet0' failed: ip6tables: No chain/target/match by that name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HI-vnet0' failed: ip6tables: No chain/target/match by that name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HI-vnet0' failed: ip6tables: No chain/target/match by that name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E FP-vnet0 FO-vnet0' failed: ip6tables: No chain/target /match by that name. Sep 1 09:49:20 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E FJ-vnet0 FI-vnet0' failed: ip6tables: No chain/target /match by that name. Sep 1 09:49:20 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E HJ-vnet0 HI-vnet0' failed: ip6tables: No chain/target /match by that name. . . . . . Sep 1 09:49:22 ov-no2 vdsm[13586]: WARN File: /var/lib/libvirt/qemu/channels/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.ovirt-guest-agent.0 already remove d Sep 1 09:49:22 ov-no2 vdsm[13586]: WARN Attempting to remove a non existing net user: ovirtmgmt/baa3d4d1-6918-4447-80fa-64b86dbcb1f9 Sep 1 09:49:22 ov-no2 vdsm[13586]: WARN File: /var/lib/libvirt/qemu/channels/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.org.qemu.guest_agent.0 already rem oved Sep 1 09:49:22 ov-no2 vdsm[13586]: WARN File: /var/run/ovirt-vmconsole-console/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.sock already removed Sep 1 09:49:27 ov-no2 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to host#012Trace back (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py", line 26, in send_email#012 timeout=float(cfg["smtp-timeout"]))#012 File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) = self.connect(host, po rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012 self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No route to host Sep 1 09:49:27 ov-no2 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to host#012Trace back (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py", line 26, in send_email#012 timeout=float(cfg["smtp-timeout"]))#012 File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) = self.connect(host, po rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012 self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No route to host Sep 1 09:49:37 ov-no2 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to host#012Trace back (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py", line 26, in send_email#012 timeout=float(cfg["smtp-timeout"]))#012 File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) = self.connect(host, po rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012 self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No route to host Sep 1 10:01:31 ov-no2 vdsm[13586]: WARN Worker blocked: <Worker name=jsonrpc/4 running <Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0', 'method': u'GlusterVolume.geoRepSessionList', 'id': u'7f800636-f107-4b98-a35f-2175ec72b7b8'} at 0x7f44805aa110> timeout=60, duration=60.00 at 0x7f44805aad10> task#=5775347 at 0x7f44d81c4110>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012 File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 re t = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/li b/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", lin e 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler( self._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_requ est(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: "/ usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-pack ages/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012 remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py" , line 93, in wrapper#012 rv = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in volumeGeoRepSessi onList#012 remoteUserName,#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012Fi le: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in glusterVolumeGeo RepStatus#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() [root@ov-no2 ~]# xzcat /var/log/vdsm/vdsm.log.57.xz |egrep "(WARN|error)" |more 2020-09-01 09:24:06,889+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:24:16,910+0300 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:24:16,911+0300 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:24:26,932+0300 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:24:26,933+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.01 seconds (__init__:312) 2020-09-01 09:24:30,375+0300 WARN (monitor/0a5e552) [storage.PersistentDict] Could not parse line ` 2020-09-01 09:24:30,379+0300 WARN (monitor/e2e7282) [storage.PersistentDict] Could not parse line ` . . . . 2020-09-01 09:47:37,008+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:47:47,030+0300 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:47:47,030+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:48:01,712+0300 WARN (jsonrpc/6) [storage.PersistentDict] Could not parse line ` 2020-09-01 09:48:01,743+0300 WARN (jsonrpc/1) [storage.PersistentDict] Could not parse line ` 2020-09-01 09:48:02,220+0300 WARN (jsonrpc/0) [storage.PersistentDict] Could not parse line ` 2020-09-01 09:48:05,192+0300 WARN (periodic/22) [storage.PersistentDict] Could not parse line ` 2020-09-01 09:48:11,745+0300 WARN (monitor/e2e7282) [storage.PersistentDict] Could not parse line ` 2020-09-01 09:48:13,711+0300 INFO (jsonrpc/7) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:48:13,711+0300 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:48:23,591+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:48:23,591+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:48:23,731+0300 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:48:23,731+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:48:33,754+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:48:33,754+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:48:43,787+0300 INFO (jsonrpc/2) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:48:43,787+0300 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:48:53,808+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:48:53,809+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:49:03,830+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:49:03,831+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:49:13,852+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:49:13,853+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.01 seconds (__init__:312) 2020-09-01 09:49:16,042+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:49:16,042+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.01 seconds (__init__:312) 2020-09-01 09:49:16,405+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:49:16,405+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:49:16,797+0300 INFO (jsonrpc/7) [api.virt] START create(vmParams={u'xml': u'<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n<domain xmlns: ovirt-tune="http://ovirt.org/vm/tune/1.0" xmlns:ovirt-vm="http://ovirt.org/vm/1.0" type="kvm"><name>HostedEngine</name><uuid>baa3d4d1-6918-4447-80fa- 64b86dbcb1f9</uuid><memory>16777216</memory><currentMemory>16777216</currentMemory><iothreads>1</iothreads><maxMemory slots="16">67108864</maxMemory> <vcpu current="4">64</vcpu><sysinfo type="smbios"><system><entry name="manufacturer">oVirt</entry><entry name="product">OS-NAME:</entry><entry name=" version">OS-VERSION:</entry><entry name="serial">HOST-SERIAL:</entry><entry name="uuid">baa3d4d1-6918-4447-80fa-64b86dbcb1f9</entry></system></sysinf o><clock offset="variable" adjustment="0"><timer name="rtc" tickpolicy="catchup"/><timer name="pit" tickpolicy="delay"/><timer name="hpet" present="n o"/></clock><features><acpi/></features><cpu match="exact"><model>Westmere</model><feature name="pcid" policy="require"/><feature name="spec-ctrl" po licy="require"/><feature name="ssbd" policy="require"/><topology cores="4" threads="1" sockets="16"/><numa><cell id="0" cpus="0,1,2,3" memory="167772 16"/></numa></cpu><cputune/><devices><input type="tablet" bus="usb"/><channel type="unix"><target type="virtio" name="ovirt-guest-agent.0"/><source m ode="bind" path="/var/lib/libvirt/qemu/channels/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.ovirt-guest-agent.0"/></channel><channel type="unix"><target typ e="virtio" name="org.qemu.guest_agent.0"/><source mode="bind" path="/var/lib/libvirt/qemu/channels/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.org.qemu.gues t_agent.0"/></channel><controller type="usb" model="piix3-uhci" index="0"><address bus="0x00" domain="0x0000" function="0x2" slot="0x01" type="pci"/> </controller><console type="unix"><source path="/var/run/ovirt-vmconsole-console/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.sock" mode="bind"/><target type ="serial" port="0"/><alias name="ua-228f9c56-c195-4f1e-8cd2-5f7ef60a78f1"/></console><memballoon model="virtio"><stats period="5"/><alias name="ua-42 10968c-acf1-46cc-8a35-6e29e3c58b38"/><address bus="0x00" domain="0x0000" function="0x0" slot="0x08" type="pci"/></memballoon><controller type="scsi" model="virtio-scsi" index="0"><driver iothread="1"/><alias name="ua-473cb0f7-09da-4077-a0e2-8eb5a5efdbe4"/><address bus="0x00" domain="0x0000" functi on="0x0" slot="0x05" type="pci"/></controller><sound model="ich6"><alias name="ua-496f9a72-5484-4dc0-83f4-caa8225100ae"/><address bus="0x00" domain=" 0x0000" function="0x0" slot="0x04" type="pci"/></sound><graphics type="vnc" port="-1" autoport="yes" passwd="*****" passwdValidTo="1970-01-01T00:00:0 1" keymap="en-us"><listen type="network" network="vdsm-ovirtmgmt"/></graphics><controller type="ide" index="0"><address bus="0x00" domain="0x0000" fu nction="0x1" slot="0x01" type="pci"/></controller><graphics type="spice" port="-1" autoport="yes" passwd="*****" passwdValidTo="1970-01-01T00:00:01" tlsPort="-1"><channel name="main" mode="secure"/><channel name="inputs" mode="secure"/><channel name="cursor" mode="secure"/><channel name="playback" mode="secure"/><channel name="record" mode="secure"/><channel name="display" mode="secure"/><channel name="smartcard" mode="secure"/><channel name=" usbredir" mode="secure"/><listen type="network" network="vdsm-ovirtmgmt"/></graphics><rng model="virtio"><backend model="random">/dev/urandom</backen d><alias name="ua-9edfa822-3b11-424e-83f8-4846c6221a6e"/></rng><video><model type="qxl" vram="32768" heads="1" ram="65536" vgamem="16384"/><alias nam e="ua-a4ad75db-c91d-45df-a767-33a71009587f"/><address bus="0x00" domain="0x0000" function="0x0" slot="0x02" type="pci"/></video><controller type="vir tio-serial" index="0" ports="16"><alias name="ua-f60f1742-07fe-464e-a160-1a16c94db61a"/><address bus="0x00" domain="0x0000" function="0x0" slot="0x06 " type="pci"/></controller><serial type="unix"><source path="/var/run/ovirt-vmconsole-console/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.sock" mode="bind"/
<target port="0"/></serial><channel type="spicevmc"><target type="virtio" name="com.redhat.spice.0"/></channel><interface type="bridge"><model type= "virtio"/><link state="up"/><source bridge="ovirtmgmt"/><driver queues="4" name="vhost"/><alias name="ua-e97c94c0-ffc2-46dd-b5cd-9d9c123f4db0"/><addr ess bus="0x00" domain="0x0000" function="0x0" slot="0x03" type="pci"/><mac address="00:16:3e:27:6e:ef"/><mtu size="1500"/><filterref filter="vdsm-no- mac-spoofing"/><bandwidth/></interface><disk type="file" device="cdrom" snapshot="no"><driver name="qemu" type="raw" error_policy="report"/><source f ile="" startupPolicy="optional"><seclabel model="dac" type="none" relabel="no"/></source><target dev="hdc" bus="ide"/><readonly/><alias name="ua-9034 9138-d20e-4306-b0e2-271e6baaad4a"/><address bus="1" controller="0" unit="0" type="drive" target="0"/></disk><disk snapshot="no" type="file" device="d isk"><target dev="vda" bus="virtio"/><source file="/rhev/data-center/00000000-0000-0000-0000-000000000000/80f6e393-9718-4738-a14a-64cf43c3d8c2/images /d5de54b6-9f8e-4fba-819b-ebf6780757d2/a48555f4-be23-4467-8a54-400ae7baf9d7"><seclabel model="dac" type="none" relabel="no"/></source><driver name="qe mu" iothread="1" io="threads" type="raw" error_policy="stop" cache="none"/><alias name="ua-d5de54b6-9f8e-4fba-819b-ebf6780757d2"/><address bus="0x00" domain="0x0000" function="0x0" slot="0x07" type="pci"/><serial>d5de54b6-9f8e-4fba-819b-ebf6780757d2</serial></disk><lease><key>a48555f4-be23-4467-8a 54-400ae7baf9d7</key><lockspace>80f6e393-9718-4738-a14a-64cf43c3d8c2</lockspace><target offset="LEASE-OFFSET:a48555f4-be23-4467-8a54-400ae7baf9d7:80f 6e393-9718-4738-a14a-64cf43c3d8c2" path="LEASE-PATH:a48555f4-be23-4467-8a54-400ae7baf9d7:80f6e393-9718-4738-a14a-64cf43c3d8c2"/></lease></devices><pm <suspend-to-disk enabled="no"/><suspend-to-mem enabled="no"/></pm><os><type arch="x86_64" machine="pc-i440fx-rhel7.6.0">hvm</type><smbios mode="sysi nfo"/><bios useserial="yes"/></os><metadata><ovirt-tune:qos/><ovirt-vm:vm><ovirt-vm:minGuaranteedMemoryMb type="int">1024</ovirt-vm:minGuaranteedMemo ryMb><ovirt-vm:clusterVersion>4.3</ovirt-vm:clusterVersion><ovirt-vm:custom/><ovirt-vm:device mac_address="00:16:3e:27:6e:ef"><ovirt-vm:custom/></ovi rt-vm:device><ovirt-vm:device devtype="disk" name="vda"><ovirt-vm:poolID>00000000-0000-0000-0000-000000000000</ovirt-vm:poolID><ovirt-vm:volumeID>a48 555f4-be23-4467-8a54-400ae7baf9d7</ovirt-vm:volumeID><ovirt-vm:shared>exclusive</ovirt-vm:shared><ovirt-vm:imageID>d5de54b6-9f8e-4fba-819b-ebf6780757 d2</ovirt-vm:imageID><ovirt-vm:domainID>80f6e393-9718-4738-a14a-64cf43c3d8c2</ovirt-vm:domainID></ovirt-vm:device><ovirt-vm:launchPaused>false</ovirt -vm:launchPaused><ovirt-vm:resumeBehavior>auto_resume</ovirt-vm:resumeBehavior></ovirt-vm:vm></metadata></domain>'}) from=::1,36826, vmId=baa3d4d1-69 18-4447-80fa-64b86dbcb1f9 (api:48)
[root@ov-no2 ~]# xzcat /var/log/vdsm/supervdsm.log.30.xz |egrep "(WARN|error|ERR)" |more MainProcess|jsonrpc/4::ERROR::2020-09-01 06:56:30,652::supervdsm_server::103::SuperVdsm.ServerCallback::(wrapper) Error in volumeGeoRepStatus MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,206::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,213::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,217::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,565::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,571::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,573::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,735::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,735::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,736::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,736::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,737::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,738::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,740::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,740::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,741::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/3::ERROR::2020-09-01 07:32:31,889::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/3::ERROR::2020-09-01 07:32:31,893::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/3::ERROR::2020-09-01 07:32:31,894::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/3::ERROR::2020-09-01 07:32:31,896::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/7::ERROR::2020-09-01 07:58:30,879::supervdsm_server::103::SuperVdsm.ServerCallback::(wrapper) Error in volumeGeoRepStatus [root@ov-no2 ~]# xzcat /var/log/vdsm/supervdsm.log.29.xz |egrep "(WARN|error|ERR)" |more MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,179::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,180::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,181::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,390::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,393::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,395::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,510::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,510::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,511::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,511::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,512::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,512::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,513::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,513::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,514::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,440::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,444::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,446::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,448::udev::53::blivet::(get_device) No device at '' [root@ov-no2 ~]# cat /var/log/vdsm/supervdsm.log |egrep "(WARN|error|ERR)" |more MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,056::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,057::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,059::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,177::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,179::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,180::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,292::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,292::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,293::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,294::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,294::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,295::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,295::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,296::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,296::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,217::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,220::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,222::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,223::udev::53::blivet::(get_device) No device at '' [root@ov-no2 ~]# cat /var/log/ovirt-hosted-engine-ha/agent.log.2020-09-01 |egrep -i "(WARN|ERR)" |more MainThread::ERROR::2020-09-01 09:48:02,766::hosted_engine::452::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Unhandled monitoring loop exception error: [Errno 2] No such file or directory MainThread::ERROR::2020-09-01 09:48:02,786::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): error: [Errno 2] No such file or directory MainThread::ERROR::2020-09-01 09:48:02,786::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2020-09-01 09:49:16,925::hosted_engine::854::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) stderr: Co mmand VM.getStats with args {'vmID': 'baa3d4d1-6918-4447-80fa-64b86dbcb1f9'} failed: MainThread::INFO::2020-09-01 09:49:27,616::hosted_engine::931::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) stderr: Com mand VM.destroy with args {'vmID': 'baa3d4d1-6918-4447-80fa-64b86dbcb1f9'} failed: MainThread::ERROR::2020-09-01 09:49:27,616::hosted_engine::936::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) Failed to stop engine vm with /usr/sbin/hosted-engine --vm-poweroff: Command VM.destroy with args {'vmID': 'baa3d4d1-6918-4447-80fa-64b86dbcb1f9'} failed: MainThread::ERROR::2020-09-01 09:49:27,616::hosted_engine::942::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) Failed to stop engine VM: Command VM.destroy with args {'vmID': 'baa3d4d1-6918-4447-80fa-64b86dbcb1f9'} failed: [root@ov-no2 ~]# cat /var/log/ovirt-hosted-engine-ha/broker.log.2020-09-01 |egrep -i "(WARN|ERR)" |more StatusStorageThread::ERROR::2020-09-01 09:47:52,786::status_broker::90::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(run) Failed to update state. raise ex.HostIdNotLockedError("Host id is not set") HostIdNotLockedError: Host id is not set StatusStorageThread::ERROR::2020-09-01 09:47:52,787::status_broker::70::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(trigger_res tart) Trying to restart the broker Listener::ERROR::2020-09-01 09:48:15,543::notifications::39::ovirt_hosted_engine_ha.broker.notifications.Notifications::(send_email) [Errno 113] No r oute to host raise err error: [Errno 113] No route to host Listener::ERROR::2020-09-01 09:48:25,768::notifications::39::ovirt_hosted_engine_ha.broker.notifications.Notifications::(send_email) [Errno 113] No r oute to host raise err error: [Errno 113] No route to host Listener::ERROR::2020-09-01 09:49:15,838::notifications::39::ovirt_hosted_engine_ha.broker.notifications.Notifications::(send_email) [Errno 113] No r oute to host raise err The Gluster status and info I could gather for the present are the following: [root@ov-no2 ~]# gluster volume status Status of volume: data Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ov-no1.ariadne-t.local:/gluster_brick s/data/data 49152 0 Y 6070 Brick ov-no2.ariadne-t.local:/gluster_brick s/data/data 49152 0 Y 28860 Brick ov-no3.ariadne-t.local:/gluster_brick s/data/data 49152 0 Y 5752 Self-heal Daemon on localhost N/A N/A Y 813 Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539 Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353 Task Status of Volume data ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: engine Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ov-no1.ariadne-t.local:/gluster_brick s/engine/engine 49153 0 Y 6098 Brick ov-no2.ariadne-t.local:/gluster_brick s/engine/engine 49156 0 Y 28867 Brick ov-no3.ariadne-t.local:/gluster_brick s/engine/engine 49153 0 Y 5789 Self-heal Daemon on localhost N/A N/A Y 813 Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539 Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353 Task Status of Volume engine ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: vmstore Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ov-no1.ariadne-t.local:/gluster_brick s/vmstore/vmstore 49154 0 Y 6102 Brick ov-no2.ariadne-t.local:/gluster_brick s/vmstore/vmstore 49157 0 Y 28878 Brick ov-no3.ariadne-t.local:/gluster_brick s/vmstore/vmstore 49154 0 Y 5772 Self-heal Daemon on localhost N/A N/A Y 813 Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539 Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353 Task Status of Volume vmstore ------------------------------------------------------------------------------ There are no active volume tasks [root@ov-no2 ~]# gluster volume heal engine info Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 [root@ov-no2 ~]# gluster volume heal engine info summary Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [root@ov-no2 ~]# gluster volume heal data info Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data Status: Connected Number of entries: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data Status: Connected Number of entries: 0 [root@ov-no2 ~]# gluster volume heal data info summary Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [root@ov-no2 ~]# gluster volume heal vmstore info Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 [root@ov-no2 ~]# gluster volume heal vmstore info summary Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [root@ov-no2 ~]# ps -A |grep gluster 813 ? 02:18:00 glusterfs 14183 ? 1-12:02:26 glusterfs 17944 ? 01:19:03 glustereventsd 17995 ? 00:31:50 glustereventsd 28722 ? 00:02:09 glusterd 28860 ? 02:19:48 glusterfsd 28867 ? 05:23:00 glusterfsd 28878 ? 00:20:15 glusterfsd 32187 ? 1-14:42:54 glusterfs 32309 ? 09:59:54 glusterfs Node3: [root@ov-no3 ~]# virsh qemu-agent-command HostedEngine '{"execute":"guest-info"}' Please enter your authentication name: root Please enter your password: {"return":{"version":"2.12.0","supported_commands":[{"enabled":true,"name":"guest-get-osinfo","success-response":true},{"enabled":true,"name":"guest-get-timezone","success-response":true},{"enabled":true,"name":"guest-get-users","success-response":true},{"enabled":true,"name":"guest-get-host-name","success-response":true},{"enabled":false,"name":"guest-exec","success-response":true},{"enabled":false,"name":"guest-exec-status","success-response":true},{"enabled":true,"name":"guest-get-memory-block-info","success-response":true},{"enabled":true,"name":"guest-set-memory-blocks","success-response":true},{"enabled":true,"name":"guest-get-memory-blocks","success-response":true},{"enabled":true,"name":"guest-set-user-password","success-response":true},{"enabled":true,"name":"guest-get-fsinfo","success-response":true},{"enabled":true,"name":"guest-set-vcpus","success-response":true},{"enabled":true,"name":"guest-get-vcpus","success-response":true},{"enabled":true,"name":"guest-network-get-in terfaces","success-response":true},{"enabled":true,"name":"guest-suspend-hybrid","success-response":false},{"enabled":true,"name":"guest-suspend-ram","success-response":false},{"enabled":true,"name":"guest-suspend-disk","success-response":false},{"enabled":true,"name":"guest-fstrim","success-response":true},{"enabled":true,"name":"guest-fsfreeze-thaw","success-response":true},{"enabled":true,"name":"guest-fsfreeze-freeze-list","success-response":true},{"enabled":true,"name":"guest-fsfreeze-freeze","success-response":true},{"enabled":true,"name":"guest-fsfreeze-status","success-response":true},{"enabled":false,"name":"guest-file-flush","success-response":true},{"enabled":false,"name":"guest-file-seek","success-response":true},{"enabled":false,"name":"guest-file-write","success-response":true},{"enabled":false,"name":"guest-file-read","success-response":true},{"enabled":false,"name":"guest-file-close","success-response":true},{"enabled":false,"name":"guest-file-open","success-response ":true},{"enabled":true,"name":"guest-shutdown","success-response":false},{"enabled":true,"name":"guest-info","success-response":true},{"enabled":true,"name":"guest-set-time","success-response":true},{"enabled":true,"name":"guest-get-time","success-response":true},{"enabled":true,"name":"guest-ping","success-response":true},{"enabled":true,"name":"guest-sync","success-response":true},{"enabled":true,"name":"guest-sync-delimited","success-response":true}]}} [root@ov-no3 ~]# cat /var/log/messages|egrep -i "(WARN|ERR)" |more Sep 1 08:55:17 ov-no3 libvirtd: 2020-09-01 05:55:17.816+0000: 5284: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 08:59:30 ov-no3 vdsm[6112]: WARN Worker blocked: <Worker name=jsonrpc/3 running <Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0', 'method': u'GlusterVolume.geoRepSessionList', 'id': u'8298ea63-eadc-47b8-b8b4-d54982200010'} at 0x7f3c14739d90> timeout=60, duration=60.00 at 0x7f3c14739dd0> t ask#=72655 at 0x7f3c34306550>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012Fil e: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run #012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/p ython2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 3 91, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler(sel f._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_request (req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: "/usr /lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-package s/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012 remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", l ine 93, in wrapper#012 rv = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in volumeGeoRepSessionL ist#012 remoteUserName,#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in glusterVolumeGeoRep Status#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() Sep 1 09:00:17 ov-no3 libvirtd: 2020-09-01 06:00:17.817+0000: 5281: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:00:30 ov-no3 vdsm[6112]: WARN Worker blocked: <Worker name=jsonrpc/3 running <Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0', 'method': u'GlusterVolume.geoRepSessionList', 'id': u'8298ea63-eadc-47b8-b8b4-d54982200010'} at 0x7f3c14739d90> timeout=60, duration=120.00 at 0x7f3c14739dd0> task#=72655 at 0x7f3c34306550>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012Fi le: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in ru n#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/ python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler(se lf._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_reques t(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: "/us r/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-packag es/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012 remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 93, in wrapper#012 rv = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in volumeGeoRepSession List#012 remoteUserName,#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012File : "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in glusterVolumeGeoRe pStatus#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() Sep 1 09:05:17 ov-no3 libvirtd: 2020-09-01 06:05:17.816+0000: 5283: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:10:17 ov-no3 libvirtd: 2020-09-01 06:10:17.823+0000: 5282: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:15:17 ov-no3 libvirtd: 2020-09-01 06:15:17.817+0000: 5281: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:20:17 ov-no3 libvirtd: 2020-09-01 06:20:17.820+0000: 5282: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:25:17 ov-no3 libvirtd: 2020-09-01 06:25:17.818+0000: 5280: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:30:17 ov-no3 libvirtd: 2020-09-01 06:30:17.820+0000: 5282: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:35:17 ov-no3 libvirtd: 2020-09-01 06:35:17.815+0000: 5281: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:40:17 ov-no3 libvirtd: 2020-09-01 06:40:17.823+0000: 5284: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:45:17 ov-no3 libvirtd: 2020-09-01 06:45:17.819+0000: 5280: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:50:17 ov-no3 libvirtd: 2020-09-01 06:50:17.824+0000: 5280: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:55:17 ov-no3 libvirtd: 2020-09-01 06:55:17.818+0000: 5284: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 10:00:17 ov-no3 libvirtd: 2020-09-01 07:00:17.822+0000: 5284: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 10:05:17 ov-no3 libvirtd: 2020-09-01 07:05:17.824+0000: 5282: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected . . . . [which continues as so 'till now] [root@ov-no3 ~]# xzcat /var/log/vdsm/vdsm.log.57.xz |egrep "(WARN|error)" |more 2020-09-01 09:31:35,023+0300 WARN (monitor/0a5e552) [storage.PersistentDict] Could not parse line ``. (persistent:241) 2020-09-01 09:31:35,026+0300 WARN (monitor/e2e7282) [storage.PersistentDict] Could not parse line ``. (persistent:241) 2020-09-01 09:35:19,868+0300 WARN (monitor/80f6e39) [storage.PersistentDict] Could not parse line ``. (persistent:241) 2020-09-01 09:35:22,210+0300 WARN (qgapoller/1) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:35:32,207+0300 WARN (qgapoller/3) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:35:42,213+0300 WARN (qgapoller/1) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:35:52,212+0300 WARN (qgapoller/2) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:36:02,207+0300 WARN (qgapoller/1) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:36:12,215+0300 WARN (qgapoller/2) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:36:35,276+0300 WARN (monitor/0a5e552) [storage.PersistentDict] Could not parse line ``. (persistent:241) 2020-09-01 09:36:35,281+0300 WARN (monitor/e2e7282) [storage.PersistentDict] Could not parse line ``. (persistent:241) 2020-09-01 09:40:18,494+0300 WARN (qgapoller/3) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c6803ff50> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:40:18,534+0300 WARN (qgapoller/1) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c6803fe60> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:40:20,244+0300 WARN (monitor/80f6e39) [storage.PersistentDict] Could not parse line ``. (persistent:241) 2020-09-01 09:40:22,209+0300 WARN (qgapoller/3) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:40:32,207+0300 WARN (qgapoller/1) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:40:42,214+0300 WARN (qgapoller/3) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:40:52,215+0300 WARN (qgapoller/2) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) [root@ov-no3 ~]# xzcat /var/log/vdsm/supervdsm.log.29.xz|egrep "(WARN|ERR)" |more MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,165::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,166::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,168::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,285::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,287::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,288::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,396::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,397::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,397::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,398::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,398::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,399::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,399::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,400::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,400::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/4::ERROR::2020-09-01 09:32:36,337::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/4::ERROR::2020-09-01 09:32:36,341::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/4::ERROR::2020-09-01 09:32:36,342::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/4::ERROR::2020-09-01 09:32:36,344::udev::53::blivet::(get_device) No device at '' [root@ov-no3 ~]# cat /var/log/vdsm/supervdsm.log|egrep "(WARN|ERR)" |more MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,041::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,043::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,044::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,158::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,160::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,161::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,265::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,266::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,266::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,267::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,267::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,268::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,269::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,269::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,270::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,399::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,402::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,404::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,406::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/7::ERROR::2020-09-03 19:54:43,983::supervdsm_server::103::SuperVdsm.ServerCallback::(wrapper) Error in volumeGeoRepStatus [root@ov-no3 ~]# cat /var/log/ovirt-hosted-engine-ha/agent.log.2020-09-01 |egrep -i "(WARN|ERR)" |more [nothing in the output] [root@ov-no3 ~]# cat /var/log/ovirt-hosted-engine-ha/broker.log.2020-09-01 |egrep -i "(WARN|ERR)" |more Thread-1::WARNING::2020-09-02 12:46:59,657::network::92::network.Network::(action) Failed to verify network status, (4 out of 5) [root@ov-no3 ~]# gluster volume status Status of volume: data Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ov-no1.ariadne-t.local:/gluster_brick s/data/data 49152 0 Y 6070 Brick ov-no2.ariadne-t.local:/gluster_brick s/data/data 49152 0 Y 28860 Brick ov-no3.ariadne-t.local:/gluster_brick s/data/data 49152 0 Y 5752 Self-heal Daemon on localhost N/A N/A Y 19353 Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539 Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813 Task Status of Volume data ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: engine Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ov-no1.ariadne-t.local:/gluster_brick s/engine/engine 49153 0 Y 6098 Brick ov-no2.ariadne-t.local:/gluster_brick s/engine/engine 49156 0 Y 28867 Brick ov-no3.ariadne-t.local:/gluster_brick s/engine/engine 49153 0 Y 5789 Self-heal Daemon on localhost N/A N/A Y 19353 Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539 Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813 Task Status of Volume engine ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: vmstore Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ov-no1.ariadne-t.local:/gluster_brick s/vmstore/vmstore 49154 0 Y 6102 Brick ov-no2.ariadne-t.local:/gluster_brick s/vmstore/vmstore 49157 0 Y 28878 Brick ov-no3.ariadne-t.local:/gluster_brick s/vmstore/vmstore 49154 0 Y 5772 Self-heal Daemon on localhost N/A N/A Y 19353 Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539 Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813 Task Status of Volume vmstore ------------------------------------------------------------------------------ There are no active volume tasks [root@ov-no3 ~]# gluster volume heal engine info Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 [root@ov-no3 ~]# gluster volume heal engine info summary Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [root@ov-no3 ~]# gluster volume heal data info Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data Status: Connected Number of entries: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data Status: Connected Number of entries: 0 [root@ov-no3 ~]# gluster volume heal data info summary Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [root@ov-no3 ~]# gluster volume heal vmstore info Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 [root@ov-no3 ~]# gluster volume heal vmstore info summary Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [root@ov-no3 ~]# ps -A|grep gluster 5143 ? 00:01:31 glustereventsd 5752 ? 02:01:08 glusterfsd 5772 ? 00:17:26 glusterfsd 5789 ? 05:12:24 glusterfsd 5906 ? 00:00:38 glustereventsd 6382 ? 02:37:00 glusterfs 9843 ? 00:02:18 glusterd 10613 ? 00:22:08 glusterfs 10750 ? 00:13:24 glusterfs 19353 ? 01:20:36 glusterfs Hosted Engine: [root@ov-eng ~]# grep ERROR /var/log/ovirt-engine/engine.log |more 2020-09-02 03:36:35,199+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler8) [1a87b58 d] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 04:38:35,429+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler2) [15fadfa 2] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 05:40:35,655+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler9) [50fa85f 0] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 06:42:35,885+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler7) [3092fdc c] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 07:44:36,119+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler6) [696ffca 7] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 08:46:36,341+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler9) [180bb50 ] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 09:48:36,562+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler6) [3a45e1b 3] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 10:50:36,788+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler4) [7da1197 3] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 11:52:37,011+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler6) [76b3cb6 5] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 12:54:37,234+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler5) [6a0d904 9] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 13:56:37,459+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler4) [1e25ecf f] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 14:58:37,674+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler1) [64144e3 4] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' How can I find why the HE failover happened? I believe it's the gluster geo replication, but I haven't configured geo replication in the enviroment. Can you help me how to proceed? Thank you all in advance, Maria Souvalioti
What about the gluster logs ? Best Regards, Strahil Nikolov В четвъртък, 3 септември 2020 г., 20:57:05 Гринуич+3, souvaliotimaria@mail.com <souvaliotimaria@mail.com> написа: Thank you very much for your reply. I checked the NTP and realized the service wasn't working properly on two of the three nodes, but despite that the clocks seemed to have the correct time(date and hwdate). I switched to chronyd and stopped the ntpd service and now it seems the servers' clocks are synchronized. The time in BIOS has different time than the systems. Does this affect the behaviour of the overall performance? This is what I could gather from the logs: Node1: [root@ov-no1 ~]# more /var/log/messages |egrep "(WARN|error)"|more d Aug 27 17:53:08 ov-no1 libvirtd: 2020-08-27 14:53:08.947+0000: 5613: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Aug 27 17:58:08 ov-no1 libvirtd: 2020-08-27 14:58:08.943+0000: 5613: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Aug 27 18:03:08 ov-no1 libvirtd: 2020-08-27 15:03:08.937+0000: 5614: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Aug 27 18:08:08 ov-no1 libvirtd: 2020-08-27 15:08:08.951+0000: 5617: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Aug 27 18:13:08 ov-no1 libvirtd: 2020-08-27 15:13:08.951+0000: 5616: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Aug 27 18:18:08 ov-no1 libvirtd: 2020-08-27 15:18:08.942+0000: 5618: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected . . [Around that time Node 3, who is the arbiter node, was placed in local Maintenance mode. It was shut down for maintenance and when we boot it up again all seemed right. We removed it from the maintenance mode and when the healing processes finished, Node 1 became NonResponsive. Long story short, the VDSM agent sent Node1 a restart command. Node1 rebooted, HostedEngine was up on Node1 and the rest of the VMs that were hosted by Node1 had to be manually brought up. Since then everything seemed to be working as it should. The HostedEngine VM shutdown with no apparent reason 5 days later, making us believe there was no connection between the two incidents.] . . . . . Sep 1 05:53:30 ov-no1 vdsm[6706]: WARN Worker blocked: <Worker name=jsonrpc/3 running <Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0', 'method': u'GlusterVolume.geoRepSessionList', 'id': u'e1f4db88-5d36-4ba1-8848-3855c1dbc6ee'} at 0x7f1ed7381650> timeout=60, duration=60.00 at 0x7f1ed7381dd0> t ask#=76268 at 0x7f1ebc0797d0>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012Fil e: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run #012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/p ython2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 3 91, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler(sel f._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_request (req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: "/usr /lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-package s/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012 remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", l ine 93, in wrapper#012 rv = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in volumeGeoRepSessionL ist#012 remoteUserName,#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in glusterVolumeGeoRep Status#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() Sep 1 05:54:30 ov-no1 vdsm[6706]: WARN Worker blocked: <Worker name=jsonrpc/3 running <Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0', 'method': u'GlusterVolume.geoRepSessionList', 'id': u'e1f4db88-5d36-4ba1-8848-3855c1dbc6ee'} at 0x7f1ed7381650> timeout=60, duration=120.00 at 0x7f1ed7381dd0> task#=76268 at 0x7f1ebc0797d0>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012Fi le: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in ru n#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/ python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler(se lf._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_reques t(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: "/us r/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-packag es/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012 remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 93, in wrapper#012 rv = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in volumeGeoRepSession List#012 remoteUserName,#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012File : "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in glusterVolumeGeoRe pStatus#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() [root@ov-no1 ~]# xzcat /var/log/vdsm/vdsm.log.57.xz |egrep "(WARN|error)" |more 2020-09-01 09:40:50,023+0300 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:40:50,023+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:41:00,061+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:41:00,061+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:41:10,078+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:41:10,078+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:41:20,086+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:41:20,086+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:41:30,107+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:41:30,107+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:41:38,710+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:41:38,711+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:41:40,115+0300 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:41:40,115+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.01 seconds (__init__:312) 2020-09-01 09:41:44,541+0300 WARN (monitor/e2e7282) [storage.PersistentDict] Could not parse line ` 2020-09-01 09:41:44,544+0300 WARN (monitor/0a5e552) [storage.PersistentDict] Could not parse line ` [root@ov-no1 ~]# xzcat /var/log/vdsm/supervdsm.log.29.xz |egrep "(WARN|error|ERR)" |more MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,182::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,183::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,185::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,304::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,306::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,308::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,440::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,441::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,442::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,442::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,443::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,443::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,444::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,445::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,445::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,358::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,361::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,363::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,365::udev::53::blivet::(get_device) No device at '' [root@ov-no1 ~]# xzcat /var/log/vdsm/supervdsm.log.28.xz |egrep "(WARN|error|ERR)" |more MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,730::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,731::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,732::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,847::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,849::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,850::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,974::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,975::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,975::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,976::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,977::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,977::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,978::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,979::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,979::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/7::ERROR::2020-09-01 11:32:40,951::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/7::ERROR::2020-09-01 11:32:40,954::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/7::ERROR::2020-09-01 11:32:40,956::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/7::ERROR::2020-09-01 11:32:40,958::udev::53::blivet::(get_device) No device at '' [root@ov-no1 ~]# cat /var/log/ovirt-hosted-engine-ha/broker.log.2020-09-01 |egrep -i "(WARN|ERR)" |more [nothing in the output] [root@ov-no1 ~]# gluster volume status Status of volume: data Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ov-no1.ariadne-t.local:/gluster_brick s/data/data 49152 0 Y 6070 Brick ov-no2.ariadne-t.local:/gluster_brick s/data/data 49152 0 Y 28860 Brick ov-no3.ariadne-t.local:/gluster_brick s/data/data 49152 0 Y 5752 Self-heal Daemon on localhost N/A N/A Y 6539 Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813 Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353 Task Status of Volume data ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: engine Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ov-no1.ariadne-t.local:/gluster_brick s/engine/engine 49153 0 Y 6098 Brick ov-no2.ariadne-t.local:/gluster_brick s/engine/engine 49156 0 Y 28867 Brick ov-no3.ariadne-t.local:/gluster_brick s/engine/engine 49153 0 Y 5789 Self-heal Daemon on localhost N/A N/A Y 6539 Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813 Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353 Task Status of Volume engine ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: vmstore Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ov-no1.ariadne-t.local:/gluster_brick s/vmstore/vmstore 49154 0 Y 6102 Brick ov-no2.ariadne-t.local:/gluster_brick s/vmstore/vmstore 49157 0 Y 28878 Brick ov-no3.ariadne-t.local:/gluster_brick s/vmstore/vmstore 49154 0 Y 5772 Self-heal Daemon on localhost N/A N/A Y 6539 Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813 Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353 Task Status of Volume vmstore ------------------------------------------------------------------------------ There are no active volume tasks [root@ov-no1 ~]# gluster volume heal engine info Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 [root@ov-no1 ~]# gluster volume heal engine info summary Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [root@ov-no1 ~]# gluster volume heal data info Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data Status: Connected Number of entries: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data Status: Connected Number of entries: 0 [root@ov-no1 ~]# gluster volume heal data info summary Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [root@ov-no1 ~]# gluster volume heal vmstore info Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 [root@ov-no1 ~]# gluster volume heal vmstore info summary Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [root@ov-no1 ~]# ps -A|grep gluster 5522 ? 00:01:40 glustereventsd 5591 ? 00:05:04 glusterd 6070 ? 02:16:28 glusterfsd 6098 ? 05:44:14 glusterfsd 6102 ? 00:23:48 glusterfsd 6285 ? 00:00:38 glustereventsd 6539 ? 01:32:19 glusterfs 10470 ? 00:34:10 glusterfs 17763 ? 00:50:18 glusterfs 18101 ? 00:10:23 glusterfs Node2: [root@ov-no2 ~]# more /var/log/messages |egrep "(WARN|error)"|more Sep 1 07:58:30 ov-no2 vdsm[13586]: WARN Worker blocked: <Worker name=jsonrpc/7 running <Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0', 'method': u'GlusterVolume.geoRepSessionList', 'id': u'ec8af6cc-6109-4ddb-a6fb-ad9f4fbc1e7b'} at 0x7f44602a7c90> timeout=60, duration=120.00 at 0x7f44602a7510> task#=5773350 at 0x7f44c00c71d0>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#01 2File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 r et = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/l ib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", li ne 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler (self._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_req uest(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: " /usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-pac kages/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012 remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py ", line 93, in wrapper#012 rv = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in volumeGeoRepSess ionList#012 remoteUserName,#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012F ile: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in glusterVolumeGe oRepStatus#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() Sep 1 09:48:02 ov-no2 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Unhandled monitoring loop exception#012T raceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 449, in start_moni toring#012 self._monitoring_loop()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 468, in _monit oring_loop#012 for old_state, state, delay in self.fsm:#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py", li ne 127, in next#012 new_data = self.refresh(self._state.data)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/state_machi ne.py", line 81, in refresh#012 stats.update(self.hosted_engine.collect_stats())#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_h a/agent/hosted_engine.py", line 756, in collect_stats#012 all_stats = self._broker.get_stats_from_storage()#012 File "/usr/lib/python2.7/site-pac kages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 143, in get_stats_from_storage#012 result = self._proxy.get_stats()#012 File "/usr/lib64/py thon2.7/xmlrpclib.py", line 1233, in __call__#012 return self.__send(self.__name, args)#012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in __request#012 verbose=self.__verbose#012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request#012 return self.single_request(hos t, handler, request_body, verbose)#012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1301, in single_request#012 self.send_content(h, request_bo dy)#012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content#012 connection.endheaders(request_body)#012 File "/usr/lib64/python 2.7/httplib.py", line 1037, in endheaders#012 self._send_output(message_body)#012 File "/usr/lib64/python2.7/httplib.py", line 881, in _send_outp ut#012 self.send(msg)#012 File "/usr/lib64/python2.7/httplib.py", line 843, in send#012 self.connect()#012 File "/usr/lib/python2.7/site-pack ages/ovirt_hosted_engine_ha/lib/unixrpc.py", line 52, in connect#012 self.sock.connect(base64.b16decode(self.host))#012 File "/usr/lib64/python2. 7/socket.py", line 224, in meth#012 return getattr(self._sock,name)(*args)#012error: [Errno 2] No such file or directory Sep 1 09:48:02 ov-no2 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/ python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-p ackages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-pack ages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 456, in start_monitoring#012 self.publish(stopped)#012 File "/usr/lib/python2.7/site-pa ckages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 356, in publish#012 self._push_to_storage(blocks)#012 File "/usr/lib/python2.7/site-p ackages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 727, in _push_to_storage#012 self._broker.put_stats_on_storage(self.host_id, blocks)# 012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 113, in put_stats_on_storage#012 self._proxy.put_stats (host_id, xmlrpclib.Binary(data))#012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__#012 return self.__send(self.__name, args)# 012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in __request#012 verbose=self.__verbose#012 File "/usr/lib64/python2.7/xmlrpclib.py", l ine 1273, in request#012 return self.single_request(host, handler, request_body, verbose)#012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1301 , in single_request#012 self.send_content(h, request_body)#012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content#012 connec tion.endheaders(request_body)#012 File "/usr/lib64/python2.7/httplib.py", line 1037, in endheaders#012 self._send_output(message_body)#012 File "/usr/lib64/python2.7/httplib.py", line 881, in _send_output#012 self.send(msg)#012 File "/usr/lib64/python2.7/httplib.py", line 843, in send#012 self.connect()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py", line 52, in connect#012 self.sock.connect(b ase64.b16decode(self.host))#012 File "/usr/lib64/python2.7/socket.py", line 224, in meth#012 return getattr(self._sock,name)(*args)#012error: [Er rno 2] No such file or directory Sep 1 09:48:15 ov-no2 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to host#012Trace back (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py", line 26, in send_email#012 timeout=float(cfg["smtp-timeout"]))#012 File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) = self.connect(host, po rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012 self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No route to host Sep 1 09:48:25 ov-no2 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to host#012Trace back (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py", line 26, in send_email#012 timeout=float(cfg["smtp-timeout"]))#012 File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) = self.connect(host, po rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012 self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No route to host Sep 1 09:49:15 ov-no2 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to host#012Trace back (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py", line 26, in send_email#012 timeout=float(cfg["smtp-timeout"]))#012 File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) = self.connect(host, po rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012 self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No route to host Sep 1 09:49:16 ov-no2 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to host#012Trace back (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py", line 26, in send_email#012 timeout=float(cfg["smtp-timeout"]))#012 File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) = self.connect(host, po rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012 self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No route to host Sep 1 09:49:17 ov-no2 vdsm[13586]: WARN Attempting to add an existing net user: ovirtmgmt/baa3d4d1-6918-4447-80fa-64b86dbcb1f9 Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet0 -j libvirt-J-vnet0' f ailed: Illegal target name 'libvirt-J-vnet0'. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-P-vnet0' failed: Illegal target name 'libvirt-P-vnet0'. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet0' failed: Chain 'libvirt-J -vnet0' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet0' failed: Chain 'libvirt-P -vnet0' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet0' failed: Chain 'libvirt-J -vnet0' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet0' failed: Chain 'libvirt-J -vnet0' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet0' failed: Chain 'libvirt-P -vnet0' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet0' failed: Chain 'libvirt-P -vnet0' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet0-mac' failed: Chain 'J-vnet0-mac' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet0-mac' failed: Chain 'J-vnet0-mac' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet0-arp-mac' failed: Chain 'J-vnet0-a rp-mac' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet0-arp-mac' failed: Chain 'J-vnet0-a rp-mac' doesn't exist. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev- out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0 ' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: iptables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vn et0' failed: iptables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FO-vnet0' failed: iptables: No chain/target/match by t hat name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FO-vnet0' failed: iptables: No chain/target/match by t hat name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FI-vnet0' failed: iptables: No chain/target/match by t hat name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FI-vnet0' failed: iptables: No chain/target/match by t hat name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HI-vnet0' failed: iptables: No chain/target/match by t hat name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HI-vnet0' failed: iptables: No chain/target/match by t hat name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FP-vnet0 FO-vnet0' failed: iptables: No chain/target/m atch by that name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FJ-vnet0 FI-vnet0' failed: iptables: No chain/target/m atch by that name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E HJ-vnet0 HI-vnet0' failed: iptables: No chain/target/m atch by that name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev -out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more informatio n. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet 0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: ip6tables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-v net0' failed: ip6tables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FO-vnet0' failed: ip6tables: No chain/target/match by that name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FO-vnet0' failed: ip6tables: No chain/target/match by that name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FI-vnet0' failed: ip6tables: No chain/target/match by that name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FI-vnet0' failed: ip6tables: No chain/target/match by that name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HI-vnet0' failed: ip6tables: No chain/target/match by that name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HI-vnet0' failed: ip6tables: No chain/target/match by that name. Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E FP-vnet0 FO-vnet0' failed: ip6tables: No chain/target /match by that name. Sep 1 09:49:20 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E FJ-vnet0 FI-vnet0' failed: ip6tables: No chain/target /match by that name. Sep 1 09:49:20 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E HJ-vnet0 HI-vnet0' failed: ip6tables: No chain/target /match by that name. . . . . . Sep 1 09:49:22 ov-no2 vdsm[13586]: WARN File: /var/lib/libvirt/qemu/channels/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.ovirt-guest-agent.0 already remove d Sep 1 09:49:22 ov-no2 vdsm[13586]: WARN Attempting to remove a non existing net user: ovirtmgmt/baa3d4d1-6918-4447-80fa-64b86dbcb1f9 Sep 1 09:49:22 ov-no2 vdsm[13586]: WARN File: /var/lib/libvirt/qemu/channels/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.org.qemu.guest_agent.0 already rem oved Sep 1 09:49:22 ov-no2 vdsm[13586]: WARN File: /var/run/ovirt-vmconsole-console/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.sock already removed Sep 1 09:49:27 ov-no2 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to host#012Trace back (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py", line 26, in send_email#012 timeout=float(cfg["smtp-timeout"]))#012 File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) = self.connect(host, po rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012 self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No route to host Sep 1 09:49:27 ov-no2 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to host#012Trace back (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py", line 26, in send_email#012 timeout=float(cfg["smtp-timeout"]))#012 File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) = self.connect(host, po rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012 self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No route to host Sep 1 09:49:37 ov-no2 journal: ovirt-ha-broker ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to host#012Trace back (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py", line 26, in send_email#012 timeout=float(cfg["smtp-timeout"]))#012 File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) = self.connect(host, po rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012 self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No route to host Sep 1 10:01:31 ov-no2 vdsm[13586]: WARN Worker blocked: <Worker name=jsonrpc/4 running <Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0', 'method': u'GlusterVolume.geoRepSessionList', 'id': u'7f800636-f107-4b98-a35f-2175ec72b7b8'} at 0x7f44805aa110> timeout=60, duration=60.00 at 0x7f44805aad10> task#=5775347 at 0x7f44d81c4110>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012 File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 re t = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/li b/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", lin e 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler( self._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_requ est(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: "/ usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-pack ages/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012 remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py" , line 93, in wrapper#012 rv = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in volumeGeoRepSessi onList#012 remoteUserName,#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012Fi le: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in glusterVolumeGeo RepStatus#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() [root@ov-no2 ~]# xzcat /var/log/vdsm/vdsm.log.57.xz |egrep "(WARN|error)" |more 2020-09-01 09:24:06,889+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:24:16,910+0300 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:24:16,911+0300 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:24:26,932+0300 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:24:26,933+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.01 seconds (__init__:312) 2020-09-01 09:24:30,375+0300 WARN (monitor/0a5e552) [storage.PersistentDict] Could not parse line ` 2020-09-01 09:24:30,379+0300 WARN (monitor/e2e7282) [storage.PersistentDict] Could not parse line ` . . . . 2020-09-01 09:47:37,008+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:47:47,030+0300 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:47:47,030+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:48:01,712+0300 WARN (jsonrpc/6) [storage.PersistentDict] Could not parse line ` 2020-09-01 09:48:01,743+0300 WARN (jsonrpc/1) [storage.PersistentDict] Could not parse line ` 2020-09-01 09:48:02,220+0300 WARN (jsonrpc/0) [storage.PersistentDict] Could not parse line ` 2020-09-01 09:48:05,192+0300 WARN (periodic/22) [storage.PersistentDict] Could not parse line ` 2020-09-01 09:48:11,745+0300 WARN (monitor/e2e7282) [storage.PersistentDict] Could not parse line ` 2020-09-01 09:48:13,711+0300 INFO (jsonrpc/7) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:48:13,711+0300 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:48:23,591+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:48:23,591+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:48:23,731+0300 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:48:23,731+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:48:33,754+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:48:33,754+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:48:43,787+0300 INFO (jsonrpc/2) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:48:43,787+0300 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:48:53,808+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:48:53,809+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:49:03,830+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:49:03,831+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:49:13,852+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:49:13,853+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.01 seconds (__init__:312) 2020-09-01 09:49:16,042+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:49:16,042+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.01 seconds (__init__:312) 2020-09-01 09:49:16,405+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db cb1f9'} (api:129) 2020-09-01 09:49:16,405+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2020-09-01 09:49:16,797+0300 INFO (jsonrpc/7) [api.virt] START create(vmParams={u'xml': u'<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n<domain xmlns: ovirt-tune="http://ovirt.org/vm/tune/1.0" xmlns:ovirt-vm="http://ovirt.org/vm/1.0" type="kvm"><name>HostedEngine</name><uuid>baa3d4d1-6918-4447-80fa- 64b86dbcb1f9</uuid><memory>16777216</memory><currentMemory>16777216</currentMemory><iothreads>1</iothreads><maxMemory slots="16">67108864</maxMemory> <vcpu current="4">64</vcpu><sysinfo type="smbios"><system><entry name="manufacturer">oVirt</entry><entry name="product">OS-NAME:</entry><entry name=" version">OS-VERSION:</entry><entry name="serial">HOST-SERIAL:</entry><entry name="uuid">baa3d4d1-6918-4447-80fa-64b86dbcb1f9</entry></system></sysinf o><clock offset="variable" adjustment="0"><timer name="rtc" tickpolicy="catchup"/><timer name="pit" tickpolicy="delay"/><timer name="hpet" present="n o"/></clock><features><acpi/></features><cpu match="exact"><model>Westmere</model><feature name="pcid" policy="require"/><feature name="spec-ctrl" po licy="require"/><feature name="ssbd" policy="require"/><topology cores="4" threads="1" sockets="16"/><numa><cell id="0" cpus="0,1,2,3" memory="167772 16"/></numa></cpu><cputune/><devices><input type="tablet" bus="usb"/><channel type="unix"><target type="virtio" name="ovirt-guest-agent.0"/><source m ode="bind" path="/var/lib/libvirt/qemu/channels/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.ovirt-guest-agent.0"/></channel><channel type="unix"><target typ e="virtio" name="org.qemu.guest_agent.0"/><source mode="bind" path="/var/lib/libvirt/qemu/channels/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.org.qemu.gues t_agent.0"/></channel><controller type="usb" model="piix3-uhci" index="0"><address bus="0x00" domain="0x0000" function="0x2" slot="0x01" type="pci"/> </controller><console type="unix"><source path="/var/run/ovirt-vmconsole-console/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.sock" mode="bind"/><target type ="serial" port="0"/><alias name="ua-228f9c56-c195-4f1e-8cd2-5f7ef60a78f1"/></console><memballoon model="virtio"><stats period="5"/><alias name="ua-42 10968c-acf1-46cc-8a35-6e29e3c58b38"/><address bus="0x00" domain="0x0000" function="0x0" slot="0x08" type="pci"/></memballoon><controller type="scsi" model="virtio-scsi" index="0"><driver iothread="1"/><alias name="ua-473cb0f7-09da-4077-a0e2-8eb5a5efdbe4"/><address bus="0x00" domain="0x0000" functi on="0x0" slot="0x05" type="pci"/></controller><sound model="ich6"><alias name="ua-496f9a72-5484-4dc0-83f4-caa8225100ae"/><address bus="0x00" domain=" 0x0000" function="0x0" slot="0x04" type="pci"/></sound><graphics type="vnc" port="-1" autoport="yes" passwd="*****" passwdValidTo="1970-01-01T00:00:0 1" keymap="en-us"><listen type="network" network="vdsm-ovirtmgmt"/></graphics><controller type="ide" index="0"><address bus="0x00" domain="0x0000" fu nction="0x1" slot="0x01" type="pci"/></controller><graphics type="spice" port="-1" autoport="yes" passwd="*****" passwdValidTo="1970-01-01T00:00:01" tlsPort="-1"><channel name="main" mode="secure"/><channel name="inputs" mode="secure"/><channel name="cursor" mode="secure"/><channel name="playback" mode="secure"/><channel name="record" mode="secure"/><channel name="display" mode="secure"/><channel name="smartcard" mode="secure"/><channel name=" usbredir" mode="secure"/><listen type="network" network="vdsm-ovirtmgmt"/></graphics><rng model="virtio"><backend model="random">/dev/urandom</backen d><alias name="ua-9edfa822-3b11-424e-83f8-4846c6221a6e"/></rng><video><model type="qxl" vram="32768" heads="1" ram="65536" vgamem="16384"/><alias nam e="ua-a4ad75db-c91d-45df-a767-33a71009587f"/><address bus="0x00" domain="0x0000" function="0x0" slot="0x02" type="pci"/></video><controller type="vir tio-serial" index="0" ports="16"><alias name="ua-f60f1742-07fe-464e-a160-1a16c94db61a"/><address bus="0x00" domain="0x0000" function="0x0" slot="0x06 " type="pci"/></controller><serial type="unix"><source path="/var/run/ovirt-vmconsole-console/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.sock" mode="bind"/
<target port="0"/></serial><channel type="spicevmc"><target type="virtio" name="com.redhat.spice.0"/></channel><interface type="bridge"><model type= "virtio"/><link state="up"/><source bridge="ovirtmgmt"/><driver queues="4" name="vhost"/><alias name="ua-e97c94c0-ffc2-46dd-b5cd-9d9c123f4db0"/><addr ess bus="0x00" domain="0x0000" function="0x0" slot="0x03" type="pci"/><mac address="00:16:3e:27:6e:ef"/><mtu size="1500"/><filterref filter="vdsm-no- mac-spoofing"/><bandwidth/></interface><disk type="file" device="cdrom" snapshot="no"><driver name="qemu" type="raw" error_policy="report"/><source f ile="" startupPolicy="optional"><seclabel model="dac" type="none" relabel="no"/></source><target dev="hdc" bus="ide"/><readonly/><alias name="ua-9034 9138-d20e-4306-b0e2-271e6baaad4a"/><address bus="1" controller="0" unit="0" type="drive" target="0"/></disk><disk snapshot="no" type="file" device="d isk"><target dev="vda" bus="virtio"/><source file="/rhev/data-center/00000000-0000-0000-0000-000000000000/80f6e393-9718-4738-a14a-64cf43c3d8c2/images /d5de54b6-9f8e-4fba-819b-ebf6780757d2/a48555f4-be23-4467-8a54-400ae7baf9d7"><seclabel model="dac" type="none" relabel="no"/></source><driver name="qe mu" iothread="1" io="threads" type="raw" error_policy="stop" cache="none"/><alias name="ua-d5de54b6-9f8e-4fba-819b-ebf6780757d2"/><address bus="0x00" domain="0x0000" function="0x0" slot="0x07" type="pci"/><serial>d5de54b6-9f8e-4fba-819b-ebf6780757d2</serial></disk><lease><key>a48555f4-be23-4467-8a 54-400ae7baf9d7</key><lockspace>80f6e393-9718-4738-a14a-64cf43c3d8c2</lockspace><target offset="LEASE-OFFSET:a48555f4-be23-4467-8a54-400ae7baf9d7:80f 6e393-9718-4738-a14a-64cf43c3d8c2" path="LEASE-PATH:a48555f4-be23-4467-8a54-400ae7baf9d7:80f6e393-9718-4738-a14a-64cf43c3d8c2"/></lease></devices><pm <suspend-to-disk enabled="no"/><suspend-to-mem enabled="no"/></pm><os><type arch="x86_64" machine="pc-i440fx-rhel7.6.0">hvm</type><smbios mode="sysi nfo"/><bios useserial="yes"/></os><metadata><ovirt-tune:qos/><ovirt-vm:vm><ovirt-vm:minGuaranteedMemoryMb type="int">1024</ovirt-vm:minGuaranteedMemo ryMb><ovirt-vm:clusterVersion>4.3</ovirt-vm:clusterVersion><ovirt-vm:custom/><ovirt-vm:device mac_address="00:16:3e:27:6e:ef"><ovirt-vm:custom/></ovi rt-vm:device><ovirt-vm:device devtype="disk" name="vda"><ovirt-vm:poolID>00000000-0000-0000-0000-000000000000</ovirt-vm:poolID><ovirt-vm:volumeID>a48 555f4-be23-4467-8a54-400ae7baf9d7</ovirt-vm:volumeID><ovirt-vm:shared>exclusive</ovirt-vm:shared><ovirt-vm:imageID>d5de54b6-9f8e-4fba-819b-ebf6780757 d2</ovirt-vm:imageID><ovirt-vm:domainID>80f6e393-9718-4738-a14a-64cf43c3d8c2</ovirt-vm:domainID></ovirt-vm:device><ovirt-vm:launchPaused>false</ovirt -vm:launchPaused><ovirt-vm:resumeBehavior>auto_resume</ovirt-vm:resumeBehavior></ovirt-vm:vm></metadata></domain>'}) from=::1,36826, vmId=baa3d4d1-69 18-4447-80fa-64b86dbcb1f9 (api:48)
[root@ov-no2 ~]# xzcat /var/log/vdsm/supervdsm.log.30.xz |egrep "(WARN|error|ERR)" |more MainProcess|jsonrpc/4::ERROR::2020-09-01 06:56:30,652::supervdsm_server::103::SuperVdsm.ServerCallback::(wrapper) Error in volumeGeoRepStatus MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,206::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,213::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,217::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,565::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,571::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,573::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,735::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,735::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,736::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,736::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,737::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,738::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,740::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,740::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,741::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/3::ERROR::2020-09-01 07:32:31,889::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/3::ERROR::2020-09-01 07:32:31,893::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/3::ERROR::2020-09-01 07:32:31,894::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/3::ERROR::2020-09-01 07:32:31,896::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/7::ERROR::2020-09-01 07:58:30,879::supervdsm_server::103::SuperVdsm.ServerCallback::(wrapper) Error in volumeGeoRepStatus [root@ov-no2 ~]# xzcat /var/log/vdsm/supervdsm.log.29.xz |egrep "(WARN|error|ERR)" |more MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,179::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,180::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,181::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,390::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,393::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,395::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,510::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,510::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,511::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,511::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,512::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,512::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,513::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,513::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,514::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,440::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,444::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,446::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,448::udev::53::blivet::(get_device) No device at '' [root@ov-no2 ~]# cat /var/log/vdsm/supervdsm.log |egrep "(WARN|error|ERR)" |more MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,056::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,057::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,059::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,177::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,179::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,180::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,292::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,292::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,293::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,294::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,294::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,295::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,295::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,296::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,296::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,217::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,220::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,222::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,223::udev::53::blivet::(get_device) No device at '' [root@ov-no2 ~]# cat /var/log/ovirt-hosted-engine-ha/agent.log.2020-09-01 |egrep -i "(WARN|ERR)" |more MainThread::ERROR::2020-09-01 09:48:02,766::hosted_engine::452::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Unhandled monitoring loop exception error: [Errno 2] No such file or directory MainThread::ERROR::2020-09-01 09:48:02,786::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): error: [Errno 2] No such file or directory MainThread::ERROR::2020-09-01 09:48:02,786::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2020-09-01 09:49:16,925::hosted_engine::854::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) stderr: Co mmand VM.getStats with args {'vmID': 'baa3d4d1-6918-4447-80fa-64b86dbcb1f9'} failed: MainThread::INFO::2020-09-01 09:49:27,616::hosted_engine::931::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) stderr: Com mand VM.destroy with args {'vmID': 'baa3d4d1-6918-4447-80fa-64b86dbcb1f9'} failed: MainThread::ERROR::2020-09-01 09:49:27,616::hosted_engine::936::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) Failed to stop engine vm with /usr/sbin/hosted-engine --vm-poweroff: Command VM.destroy with args {'vmID': 'baa3d4d1-6918-4447-80fa-64b86dbcb1f9'} failed: MainThread::ERROR::2020-09-01 09:49:27,616::hosted_engine::942::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) Failed to stop engine VM: Command VM.destroy with args {'vmID': 'baa3d4d1-6918-4447-80fa-64b86dbcb1f9'} failed: [root@ov-no2 ~]# cat /var/log/ovirt-hosted-engine-ha/broker.log.2020-09-01 |egrep -i "(WARN|ERR)" |more StatusStorageThread::ERROR::2020-09-01 09:47:52,786::status_broker::90::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(run) Failed to update state. raise ex.HostIdNotLockedError("Host id is not set") HostIdNotLockedError: Host id is not set StatusStorageThread::ERROR::2020-09-01 09:47:52,787::status_broker::70::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(trigger_res tart) Trying to restart the broker Listener::ERROR::2020-09-01 09:48:15,543::notifications::39::ovirt_hosted_engine_ha.broker.notifications.Notifications::(send_email) [Errno 113] No r oute to host raise err error: [Errno 113] No route to host Listener::ERROR::2020-09-01 09:48:25,768::notifications::39::ovirt_hosted_engine_ha.broker.notifications.Notifications::(send_email) [Errno 113] No r oute to host raise err error: [Errno 113] No route to host Listener::ERROR::2020-09-01 09:49:15,838::notifications::39::ovirt_hosted_engine_ha.broker.notifications.Notifications::(send_email) [Errno 113] No r oute to host raise err The Gluster status and info I could gather for the present are the following: [root@ov-no2 ~]# gluster volume status Status of volume: data Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ov-no1.ariadne-t.local:/gluster_brick s/data/data 49152 0 Y 6070 Brick ov-no2.ariadne-t.local:/gluster_brick s/data/data 49152 0 Y 28860 Brick ov-no3.ariadne-t.local:/gluster_brick s/data/data 49152 0 Y 5752 Self-heal Daemon on localhost N/A N/A Y 813 Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539 Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353 Task Status of Volume data ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: engine Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ov-no1.ariadne-t.local:/gluster_brick s/engine/engine 49153 0 Y 6098 Brick ov-no2.ariadne-t.local:/gluster_brick s/engine/engine 49156 0 Y 28867 Brick ov-no3.ariadne-t.local:/gluster_brick s/engine/engine 49153 0 Y 5789 Self-heal Daemon on localhost N/A N/A Y 813 Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539 Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353 Task Status of Volume engine ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: vmstore Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ov-no1.ariadne-t.local:/gluster_brick s/vmstore/vmstore 49154 0 Y 6102 Brick ov-no2.ariadne-t.local:/gluster_brick s/vmstore/vmstore 49157 0 Y 28878 Brick ov-no3.ariadne-t.local:/gluster_brick s/vmstore/vmstore 49154 0 Y 5772 Self-heal Daemon on localhost N/A N/A Y 813 Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539 Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353 Task Status of Volume vmstore ------------------------------------------------------------------------------ There are no active volume tasks [root@ov-no2 ~]# gluster volume heal engine info Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 [root@ov-no2 ~]# gluster volume heal engine info summary Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [root@ov-no2 ~]# gluster volume heal data info Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data Status: Connected Number of entries: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data Status: Connected Number of entries: 0 [root@ov-no2 ~]# gluster volume heal data info summary Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [root@ov-no2 ~]# gluster volume heal vmstore info Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 [root@ov-no2 ~]# gluster volume heal vmstore info summary Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [root@ov-no2 ~]# ps -A |grep gluster 813 ? 02:18:00 glusterfs 14183 ? 1-12:02:26 glusterfs 17944 ? 01:19:03 glustereventsd 17995 ? 00:31:50 glustereventsd 28722 ? 00:02:09 glusterd 28860 ? 02:19:48 glusterfsd 28867 ? 05:23:00 glusterfsd 28878 ? 00:20:15 glusterfsd 32187 ? 1-14:42:54 glusterfs 32309 ? 09:59:54 glusterfs Node3: [root@ov-no3 ~]# virsh qemu-agent-command HostedEngine '{"execute":"guest-info"}' Please enter your authentication name: root Please enter your password: {"return":{"version":"2.12.0","supported_commands":[{"enabled":true,"name":"guest-get-osinfo","success-response":true},{"enabled":true,"name":"guest-get-timezone","success-response":true},{"enabled":true,"name":"guest-get-users","success-response":true},{"enabled":true,"name":"guest-get-host-name","success-response":true},{"enabled":false,"name":"guest-exec","success-response":true},{"enabled":false,"name":"guest-exec-status","success-response":true},{"enabled":true,"name":"guest-get-memory-block-info","success-response":true},{"enabled":true,"name":"guest-set-memory-blocks","success-response":true},{"enabled":true,"name":"guest-get-memory-blocks","success-response":true},{"enabled":true,"name":"guest-set-user-password","success-response":true},{"enabled":true,"name":"guest-get-fsinfo","success-response":true},{"enabled":true,"name":"guest-set-vcpus","success-response":true},{"enabled":true,"name":"guest-get-vcpus","success-response":true},{"enabled":true,"name":"guest-network-get-in terfaces","success-response":true},{"enabled":true,"name":"guest-suspend-hybrid","success-response":false},{"enabled":true,"name":"guest-suspend-ram","success-response":false},{"enabled":true,"name":"guest-suspend-disk","success-response":false},{"enabled":true,"name":"guest-fstrim","success-response":true},{"enabled":true,"name":"guest-fsfreeze-thaw","success-response":true},{"enabled":true,"name":"guest-fsfreeze-freeze-list","success-response":true},{"enabled":true,"name":"guest-fsfreeze-freeze","success-response":true},{"enabled":true,"name":"guest-fsfreeze-status","success-response":true},{"enabled":false,"name":"guest-file-flush","success-response":true},{"enabled":false,"name":"guest-file-seek","success-response":true},{"enabled":false,"name":"guest-file-write","success-response":true},{"enabled":false,"name":"guest-file-read","success-response":true},{"enabled":false,"name":"guest-file-close","success-response":true},{"enabled":false,"name":"guest-file-open","success-response ":true},{"enabled":true,"name":"guest-shutdown","success-response":false},{"enabled":true,"name":"guest-info","success-response":true},{"enabled":true,"name":"guest-set-time","success-response":true},{"enabled":true,"name":"guest-get-time","success-response":true},{"enabled":true,"name":"guest-ping","success-response":true},{"enabled":true,"name":"guest-sync","success-response":true},{"enabled":true,"name":"guest-sync-delimited","success-response":true}]}} [root@ov-no3 ~]# cat /var/log/messages|egrep -i "(WARN|ERR)" |more Sep 1 08:55:17 ov-no3 libvirtd: 2020-09-01 05:55:17.816+0000: 5284: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 08:59:30 ov-no3 vdsm[6112]: WARN Worker blocked: <Worker name=jsonrpc/3 running <Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0', 'method': u'GlusterVolume.geoRepSessionList', 'id': u'8298ea63-eadc-47b8-b8b4-d54982200010'} at 0x7f3c14739d90> timeout=60, duration=60.00 at 0x7f3c14739dd0> t ask#=72655 at 0x7f3c34306550>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012Fil e: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run #012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/p ython2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 3 91, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler(sel f._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_request (req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: "/usr /lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-package s/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012 remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", l ine 93, in wrapper#012 rv = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in volumeGeoRepSessionL ist#012 remoteUserName,#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in glusterVolumeGeoRep Status#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() Sep 1 09:00:17 ov-no3 libvirtd: 2020-09-01 06:00:17.817+0000: 5281: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:00:30 ov-no3 vdsm[6112]: WARN Worker blocked: <Worker name=jsonrpc/3 running <Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0', 'method': u'GlusterVolume.geoRepSessionList', 'id': u'8298ea63-eadc-47b8-b8b4-d54982200010'} at 0x7f3c14739d90> timeout=60, duration=120.00 at 0x7f3c14739dd0> task#=72655 at 0x7f3c34306550>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012Fi le: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in ru n#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/ python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler(se lf._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_reques t(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: "/us r/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-packag es/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012 remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 93, in wrapper#012 rv = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in volumeGeoRepSession List#012 remoteUserName,#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012File : "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in glusterVolumeGeoRe pStatus#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() Sep 1 09:05:17 ov-no3 libvirtd: 2020-09-01 06:05:17.816+0000: 5283: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:10:17 ov-no3 libvirtd: 2020-09-01 06:10:17.823+0000: 5282: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:15:17 ov-no3 libvirtd: 2020-09-01 06:15:17.817+0000: 5281: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:20:17 ov-no3 libvirtd: 2020-09-01 06:20:17.820+0000: 5282: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:25:17 ov-no3 libvirtd: 2020-09-01 06:25:17.818+0000: 5280: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:30:17 ov-no3 libvirtd: 2020-09-01 06:30:17.820+0000: 5282: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:35:17 ov-no3 libvirtd: 2020-09-01 06:35:17.815+0000: 5281: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:40:17 ov-no3 libvirtd: 2020-09-01 06:40:17.823+0000: 5284: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:45:17 ov-no3 libvirtd: 2020-09-01 06:45:17.819+0000: 5280: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:50:17 ov-no3 libvirtd: 2020-09-01 06:50:17.824+0000: 5280: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 09:55:17 ov-no3 libvirtd: 2020-09-01 06:55:17.818+0000: 5284: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 10:00:17 ov-no3 libvirtd: 2020-09-01 07:00:17.822+0000: 5284: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected Sep 1 10:05:17 ov-no3 libvirtd: 2020-09-01 07:05:17.824+0000: 5282: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues t agent is not connected . . . . [which continues as so 'till now] [root@ov-no3 ~]# xzcat /var/log/vdsm/vdsm.log.57.xz |egrep "(WARN|error)" |more 2020-09-01 09:31:35,023+0300 WARN (monitor/0a5e552) [storage.PersistentDict] Could not parse line ``. (persistent:241) 2020-09-01 09:31:35,026+0300 WARN (monitor/e2e7282) [storage.PersistentDict] Could not parse line ``. (persistent:241) 2020-09-01 09:35:19,868+0300 WARN (monitor/80f6e39) [storage.PersistentDict] Could not parse line ``. (persistent:241) 2020-09-01 09:35:22,210+0300 WARN (qgapoller/1) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:35:32,207+0300 WARN (qgapoller/3) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:35:42,213+0300 WARN (qgapoller/1) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:35:52,212+0300 WARN (qgapoller/2) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:36:02,207+0300 WARN (qgapoller/1) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:36:12,215+0300 WARN (qgapoller/2) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:36:35,276+0300 WARN (monitor/0a5e552) [storage.PersistentDict] Could not parse line ``. (persistent:241) 2020-09-01 09:36:35,281+0300 WARN (monitor/e2e7282) [storage.PersistentDict] Could not parse line ``. (persistent:241) 2020-09-01 09:40:18,494+0300 WARN (qgapoller/3) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c6803ff50> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:40:18,534+0300 WARN (qgapoller/1) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c6803fe60> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:40:20,244+0300 WARN (monitor/80f6e39) [storage.PersistentDict] Could not parse line ``. (persistent:241) 2020-09-01 09:40:22,209+0300 WARN (qgapoller/3) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:40:32,207+0300 WARN (qgapoller/1) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:40:42,214+0300 WARN (qgapoller/3) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) 2020-09-01 09:40:52,215+0300 WARN (qgapoller/2) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f3c680470c8> on ['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289) [root@ov-no3 ~]# xzcat /var/log/vdsm/supervdsm.log.29.xz|egrep "(WARN|ERR)" |more MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,165::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,166::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,168::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,285::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,287::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,288::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,396::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,397::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,397::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,398::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,398::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,399::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,399::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,400::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,400::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/4::ERROR::2020-09-01 09:32:36,337::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/4::ERROR::2020-09-01 09:32:36,341::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/4::ERROR::2020-09-01 09:32:36,342::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/4::ERROR::2020-09-01 09:32:36,344::udev::53::blivet::(get_device) No device at '' [root@ov-no3 ~]# cat /var/log/vdsm/supervdsm.log|egrep "(WARN|ERR)" |more MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,041::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,043::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,044::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,158::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,160::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,161::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid attribute MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,265::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,266::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,266::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,267::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,267::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,268::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,269::lvm::360::blivet::(pvinfo) ignoring pvs output line: Read-only locking type set. Write loc ks are prohibited. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,269::lvm::360::blivet::(pvinfo) ignoring pvs output line: Recovery of standalone physical volum es failed. MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,270::lvm::360::blivet::(pvinfo) ignoring pvs output line: Cannot process standalone physical vo lumes MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,399::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,402::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,404::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,406::udev::53::blivet::(get_device) No device at '' MainProcess|jsonrpc/7::ERROR::2020-09-03 19:54:43,983::supervdsm_server::103::SuperVdsm.ServerCallback::(wrapper) Error in volumeGeoRepStatus [root@ov-no3 ~]# cat /var/log/ovirt-hosted-engine-ha/agent.log.2020-09-01 |egrep -i "(WARN|ERR)" |more [nothing in the output] [root@ov-no3 ~]# cat /var/log/ovirt-hosted-engine-ha/broker.log.2020-09-01 |egrep -i "(WARN|ERR)" |more Thread-1::WARNING::2020-09-02 12:46:59,657::network::92::network.Network::(action) Failed to verify network status, (4 out of 5) [root@ov-no3 ~]# gluster volume status Status of volume: data Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ov-no1.ariadne-t.local:/gluster_brick s/data/data 49152 0 Y 6070 Brick ov-no2.ariadne-t.local:/gluster_brick s/data/data 49152 0 Y 28860 Brick ov-no3.ariadne-t.local:/gluster_brick s/data/data 49152 0 Y 5752 Self-heal Daemon on localhost N/A N/A Y 19353 Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539 Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813 Task Status of Volume data ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: engine Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ov-no1.ariadne-t.local:/gluster_brick s/engine/engine 49153 0 Y 6098 Brick ov-no2.ariadne-t.local:/gluster_brick s/engine/engine 49156 0 Y 28867 Brick ov-no3.ariadne-t.local:/gluster_brick s/engine/engine 49153 0 Y 5789 Self-heal Daemon on localhost N/A N/A Y 19353 Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539 Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813 Task Status of Volume engine ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: vmstore Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ov-no1.ariadne-t.local:/gluster_brick s/vmstore/vmstore 49154 0 Y 6102 Brick ov-no2.ariadne-t.local:/gluster_brick s/vmstore/vmstore 49157 0 Y 28878 Brick ov-no3.ariadne-t.local:/gluster_brick s/vmstore/vmstore 49154 0 Y 5772 Self-heal Daemon on localhost N/A N/A Y 19353 Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539 Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813 Task Status of Volume vmstore ------------------------------------------------------------------------------ There are no active volume tasks [root@ov-no3 ~]# gluster volume heal engine info Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 [root@ov-no3 ~]# gluster volume heal engine info summary Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [root@ov-no3 ~]# gluster volume heal data info Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data Status: Connected Number of entries: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data Status: Connected Number of entries: 0 [root@ov-no3 ~]# gluster volume heal data info summary Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [root@ov-no3 ~]# gluster volume heal vmstore info Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Number of entries: 0 [root@ov-no3 ~]# gluster volume heal vmstore info summary Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 [root@ov-no3 ~]# ps -A|grep gluster 5143 ? 00:01:31 glustereventsd 5752 ? 02:01:08 glusterfsd 5772 ? 00:17:26 glusterfsd 5789 ? 05:12:24 glusterfsd 5906 ? 00:00:38 glustereventsd 6382 ? 02:37:00 glusterfs 9843 ? 00:02:18 glusterd 10613 ? 00:22:08 glusterfs 10750 ? 00:13:24 glusterfs 19353 ? 01:20:36 glusterfs Hosted Engine: [root@ov-eng ~]# grep ERROR /var/log/ovirt-engine/engine.log |more 2020-09-02 03:36:35,199+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler8) [1a87b58 d] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 04:38:35,429+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler2) [15fadfa 2] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 05:40:35,655+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler9) [50fa85f 0] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 06:42:35,885+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler7) [3092fdc c] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 07:44:36,119+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler6) [696ffca 7] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 08:46:36,341+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler9) [180bb50 ] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 09:48:36,562+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler6) [3a45e1b 3] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 10:50:36,788+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler4) [7da1197 3] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 11:52:37,011+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler6) [76b3cb6 5] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 12:54:37,234+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler5) [6a0d904 9] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 13:56:37,459+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler4) [1e25ecf f] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' 2020-09-02 14:58:37,674+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob] (DefaultQuartzScheduler1) [64144e3 4] VDS error Command execution failed: rc=1 out='Error : Request timed out\ngeo-replication command failed\n' err='' How can I find why the HE failover happened? I believe it's the gluster geo replication, but I haven't configured geo replication in the enviroment. Can you help me how to proceed? Thank you all in advance, Maria Souvalioti _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/IZOXEF6MGTZUMC...
Hello, This is what I could gather from the gluster logs around the time frame of the HE shutdown. NODE1: [root@ov-no1 glusterfs]# more bricks/gluster_bricks-vmstore-vmstore.log-20200830 |egrep "( W | E )"|more [2020-08-27 15:35:03.090477] W [glusterfsd.c:1570:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dd5) [0x7fa6e04a3dd5] -->/usr/sbin/glusterfsd(glus terfs_sigwaiter+0xe5) [0x55a40138d1b5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x55a40138d01b] ) 0-: received signum (15), shutting down [2020-08-27 15:35:14.926794] E [MSGID: 100018] [glusterfsd.c:2333:glusterfs_pidfile_update] 0-glusterfsd: pidfile /var/run/gluster/vols/vmstore/ov-no 1.ariadne-t.local-gluster_bricks-vmstore-vmstore.pid lock failed [Resource temporarily unavailable] [root@ov-no1 glusterfs]# more bricks/gluster_bricks-data-data.log-20200830 |egrep "( W | E )"|more [2020-08-27 15:35:01.087875] W [glusterfsd.c:1570:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dd5) [0x7fc3cbf69dd5] -->/usr/sbin/glusterfsd(glus terfs_sigwaiter+0xe5) [0x555e313711b5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x555e3137101b] ) 0-: received signum (15), shutting down [2020-08-27 15:35:14.890471] E [MSGID: 100018] [glusterfsd.c:2333:glusterfs_pidfile_update] 0-glusterfsd: pidfile /var/run/gluster/vols/data/ov-no1.a riadne-t.local-gluster_bricks-data-data.pid lock failed [Resource temporarily unavailable] [root@ov-no1 glusterfs]# more bricks/gluster_bricks-engine-engine.log-20200830 |egrep "( W | E )"|more [2020-08-27 15:35:02.088732] W [glusterfsd.c:1570:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dd5) [0x7f70b99cbdd5] -->/usr/sbin/glusterfsd(glus terfs_sigwaiter+0xe5) [0x55ebd132b1b5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x55ebd132b01b] ) 0-: received signum (15), shutting down [2020-08-27 15:35:14.907603] E [MSGID: 100018] [glusterfsd.c:2333:glusterfs_pidfile_update] 0-glusterfsd: pidfile /var/run/gluster/vols/engine/ov-no1 .ariadne-t.local-gluster_bricks-engine-engine.pid lock failed [Resource temporarily unavailable] [root@ov-no1 glusterfs]# more bricks/gluster_bricks-vmstore-vmstore.log |egrep "( W | E )"|more [nothing in the output] [root@ov-no1 glusterfs]# more bricks/gluster_bricks-data-data.log |egrep "( W | E )"|more [nothing in the output] [root@ov-no1 glusterfs]# more bricks/gluster_bricks-engine-engine.log |egrep "( W | E )"|more [nothing in the output] [root@ov-no1 glusterfs]# more cmd_history.log | egrep "(WARN|error|fail)" |more [2020-09-01 02:00:38.685251] : volume geo-replication status : FAILED : Commit failed on ov-no2.ariadne-t.local. Please check log file for details. Commit failed on ov-no3.ariadne-t.local. Please check log file for details. [2020-09-01 03:02:39.094984] : volume geo-replication status : FAILED : Commit failed on ov-no2.ariadne-t.local. Please check log file for details. Commit failed on ov-no3.ariadne-t.local. Please check log file for details. [2020-09-01 11:18:32.510224] : volume geo-replication status : FAILED : Commit failed on ov-no2.ariadne-t.local. Please check log file for details. Commit failed on ov-no3.ariadne-t.local. Please check log file for details. [2020-09-01 14:24:33.778942] : volume geo-replication status : FAILED : Commit failed on ov-no2.ariadne-t.local. Please check log file for details. Commit failed on ov-no3.ariadne-t.local. Please check log file for details. [root@ov-no1 glusterfs]# cat glusterd.log | egrep "( W | E )" |more [2020-09-01 07:00:31.326169] E [glusterd-op-sm.c:8132:glusterd_op_sm] (-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f23d8ac8a1e] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f23d8ac11be] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f23 d8ae806f] ) 0-management: Unable to get transaction opinfo for transaction ID :435d3780-aa0c-4a64-bc28-56ae394159d0 [2020-09-01 08:02:31.551563] E [glusterd-op-sm.c:8132:glusterd_op_sm] (-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f23d8ac8a1e] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f23d8ac11be] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f23 d8ae806f] ) 0-management: Unable to get transaction opinfo for transaction ID :930a8a08-1044-41cf-b921-913b982e0c72 [2020-09-01 09:04:31.786157] E [glusterd-op-sm.c:8132:glusterd_op_sm] (-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f23d8ac8a1e] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f23d8ac11be] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f23 d8ae806f] ) 0-management: Unable to get transaction opinfo for transaction ID :9942b579-5240-4fee-bb4c-78b9a1c98da8 [2020-09-01 10:06:32.014362] E [glusterd-op-sm.c:8132:glusterd_op_sm] (-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f23d8ac8a1e] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f23d8ac11be] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f23 d8ae806f] ) 0-management: Unable to get transaction opinfo for transaction ID :324fc904-b147-4801-89d5-485120e1fb84 [2020-09-01 11:08:32.240906] W [MSGID: 106061] [glusterd-geo-rep.c:2388:glusterd_op_gsync_args_get] 0-management: master not found [2020-09-01 11:18:32.510041] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame type(glusterd mgmt), op(--(4)), xid = 0x812c, unique = 0, s ent = 2020-09-01 11:08:32.242199, timeout = 600 for 192.168.48.202:24007 [root@ov-no1 glusterfs]# cat events.log-20200828 | egrep "( W | E )" |more [nothing in the output] [root@ov-no1 glusterfs]# cat events.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no1 glusterfs]# cat glfsheal-data.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no1 glusterfs]# cat glfsheal-vmstore.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no1 glusterfs]# cat glfsheal-engine.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no1 glusterfs]# cat glfsheal-data.log-20200830 | egrep "( W | E )" |more [nothing in the output] [root@ov-no1 glusterfs]# cat glfsheal-vmstore.log-20200830 | egrep "( W | E )" |more [nothing in the output] [root@ov-no1 glusterfs]# cat glfsheal-engine.log-20200830 | egrep "( W | E )" |more [nothing in the output] [root@ov-no1 glusterfs]# cat glustershd.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no1 glusterfs]# cat rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_engine.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no1 glusterfs]# cat rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_data.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no1 glusterfs]# cat rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_vmstore.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no1 glusterfs]# cat rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_data.log [2020-08-31 00:41:02.169424] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-08-31 00:41:02.170418] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-08-31 08:59:07.694673] I [MSGID: 133022] [shard.c:3673:shard_delete_shards] 0-data-shard: Deleted shards of gfid=d25c9bb6-4a25-4025-8652-686be0a4bdf1 from backend [2020-08-31 08:59:07.890175] I [MSGID: 133017] [shard.c:6696:shard_seek] 0-data-shard: seek called on 0e4c3fb8-7617-4e43-879a-cc784198216d. [Operation not supported] [2020-08-31 08:59:07.892803] I [MSGID: 133017] [shard.c:6696:shard_seek] 0-data-shard: seek called on 17c220e1-fa2c-42d7-a948-6ea2bf5a93d7. [Operation not supported] [2020-08-31 08:59:07.897929] I [MSGID: 133017] [shard.c:6696:shard_seek] 0-data-shard: seek called on f9315204-ec16-4c46-847e-d84f0169ea8f. [Operation not supported] [2020-08-31 08:59:07.928488] I [MSGID: 133022] [shard.c:3673:shard_delete_shards] 0-data-shard: Deleted shards of gfid=17c220e1-fa2c-42d7-a948-6ea2bf5a93d7 from backend [2020-09-01 00:36:01.667677] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-09-01 00:36:01.668986] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-09-02 00:48:02.278423] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-09-02 00:48:02.279637] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-09-04 00:36:01.546102] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-09-04 00:36:01.547101] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [root@ov-no1 glusterfs]# cat rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_vmstore.log [2020-08-28 00:12:02.424709] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-08-29 00:29:02.375537] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-08-29 00:29:02.377088] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-08-30 00:13:03.449758] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-08-30 00:13:03.451232] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-08-31 00:41:02.169556] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-08-31 00:41:02.171040] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-09-01 00:36:01.667768] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-09-01 00:36:01.669572] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-09-02 00:48:02.278490] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-09-02 00:48:02.280253] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-09-04 00:36:01.546134] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-09-04 00:36:01.547694] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [root@ov-no1 glusterfs]# [root@ov-no1 glusterfs]# cat rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_engine.log [2020-08-30 00:13:03.447882] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-08-30 00:13:03.449756] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-08-31 00:41:02.167497] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-08-31 00:41:02.169892] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-09-01 00:36:01.666356] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-09-01 00:36:01.668276] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-09-02 00:48:02.277024] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-09-02 00:48:02.279273] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-09-04 00:36:01.543799] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-09-04 00:36:01.545779] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [root@ov-no1 glusterfs]# cat glustershd.log | egrep "( W | E )" |more [root@ov-no1 glusterfs]# cat glustershd.log [2020-08-30 20:31:26.884546] I [MSGID: 108026] [afr-self-heal-common.c:1741:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on 3fafabf3-d0cd-4b9a-8dd7-43145451f7cf. sources=[1] 2 sinks=0 . . . [every ten minutes the same log appears] . . [2020-08-31 15:03:57.893224] I [MSGID: 108026] [afr-self-heal-common.c:1741:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on 3fafabf3-d0cd-4b9a-8dd7-43145451f7cf. sources=[1] 2 sinks=0 The message "I [MSGID: 108026] [afr-self-heal-common.c:1741:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on 3fafabf3-d0cd-4b9a-8dd7-43145451f7cf. sources=[1] 2 sinks=0 " repeated 15 times between [2020-08-31 15:03:57.893224] and [2020-08-31 15:04:42.605183] [2020-08-31 15:14:46.171044] I [MSGID: 108026] [afr-self-heal-common.c:1741:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on 3fafabf3-d0cd-4b9a-8dd7-43145451f7cf. sources=[1] 2 sinks=0 [2020-09-01 00:36:01.663267] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-09-01 00:36:01.665109] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-09-02 00:48:02.274383] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-09-02 00:48:02.275562] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [root@ov-no1 glusterfs]# zcat cli.log-20200902.gz |egrep "( W | E )" |more [2020-09-01 09:04:39.799900] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:04:39.801917] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:24:41.581140] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 09:39:43.395156] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:39:43.397147] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:49:44.128540] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 09:54:44.629071] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 10:04:45.642437] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 10:04:45.904006] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 10:04:45.905279] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 10:39:49.183655] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 10:44:49.968219] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 10:44:49.970487] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 10:49:50.206005] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 10:59:51.489242] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 10:59:51.491277] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [root@ov-no1 glusterfs]# cat cli.log |grep "( W | E )" |more [nothing in the output] [root@ov-no1 glusterfs]# cat cmd_history.log|egrep "( W| E|fail)" |more [2020-09-01 00:58:38.265743] : volume geo-replication status : FAILED : Commit failed on ov-no2.ariadne-t.local. Please check log file for details. Commit failed on ov-no3.ariadne-t.local. Please check log file for details. [2020-09-01 02:00:38.685251] : volume geo-replication status : FAILED : Commit failed on ov-no2.ariadne-t.local. Please check log file for details. Commit failed on ov-no3.ariadne-t.local. Please check log file for details. [2020-09-01 03:02:39.094984] : volume geo-replication status : FAILED : Commit failed on ov-no2.ariadne-t.local. Please check log file for details. Commit failed on ov-no3.ariadne-t.local. Please check log file for details. [2020-09-01 11:18:32.510224] : volume geo-replication status : FAILED : Commit failed on ov-no2.ariadne-t.local. Please check log file for details. Commit failed on ov-no3.ariadne-t.local. Please check log file for details. [2020-09-01 14:24:33.778942] : volume geo-replication status : FAILED : Commit failed on ov-no2.ariadne-t.local. Please check log file for details. Commit failed on ov-no3.ariadne-t.local. Please check log file for details. [2020-09-01 16:28:34.610703] : volume geo-replication status : FAILED : Commit failed on ov-no2.ariadne-t.local. Please check log file for details. Commit failed on ov-no3.ariadne-t.local. Please check log file for details. NODE2: [root@ov-no2 ~]# cd /var/log/glusterfs/ [root@ov-no2 glusterfs]# more bricks/gluster_bricks-data-data.log-20200830 |egrep "( W | E )"|more [2020-08-27 15:35:07.532064] W [glusterfsd.c:1570:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dd5) [0x7f01ef674dd5] -->/usr/sbin/glusterfsd(glus terfs_sigwaiter+0xe5) [0x55d96c9791b5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x55d96c97901b] ) 0-: received signum (15), shutting down [2020-08-27 15:35:07.545460] W [rpc-clnt.c:1704:rpc_clnt_submit] 0-glusterfs: failed to submit rpc-request (unique: 0, XID: 0x33 Program: Gluster Por tmap, ProgVers: 1, Proc: 5) to rpc-transport (glusterfs) [2020-08-27 15:35:07.545626] E [timer.c:34:gf_timer_call_after] (-->/usr/sbin/glusterfsd(mgmt_submit_request+0x13d) [0x55d96c97ecdd] -->/lib64/libgfr pc.so.0(rpc_clnt_submit+0x72d) [0x7f01f05dfc1d] -->/lib64/libglusterfs.so.0(gf_timer_call_after+0x229) [0x7f01f0844349] ) 0-timer: Either ctx is NULL or ctx cleanup started [Invalid argument] [2020-08-27 15:35:07.545724] E [timer.c:34:gf_timer_call_after] (-->/usr/sbin/glusterfsd(mgmt_submit_request+0x13d) [0x55d96c97ecdd] -->/lib64/libgfr pc.so.0(rpc_clnt_submit+0x72d) [0x7f01f05dfc1d] -->/lib64/libglusterfs.so.0(gf_timer_call_after+0x229) [0x7f01f0844349] ) 0-timer: Either ctx is NULL or ctx cleanup started [Invalid argument] [2020-08-27 16:25:42.954545] W [inodelk.c:609:pl_inodelk_log_cleanup] 0-data-server: releasing lock on 03848819-f37b-4fa7-81d9-3f5dc3eef2da held by { client=0x7fd84400f350, pid=7725 lk-owner=e8100a64447f0000} [root@ov-no2 glusterfs]# [root@ov-no2 glusterfs]# [root@ov-no2 glusterfs]# [root@ov-no2 glusterfs]# [root@ov-no2 glusterfs]# more bricks/gluster_bricks-engine-engine.log-20200830 |egrep "( W | E )"|more [2020-08-27 15:35:08.532812] W [glusterfsd.c:1570:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dd5) [0x7ff67fa1edd5] -->/usr/sbin/glusterfsd(glus terfs_sigwaiter+0xe5) [0x55d4f15951b5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x55d4f159501b] ) 0-: received signum (15), shutting down [2020-08-27 15:35:08.536715] W [rpc-clnt.c:1704:rpc_clnt_submit] 0-glusterfs: failed to submit rpc-request (unique: 0, XID: 0x33 Program: Gluster Por tmap, ProgVers: 1, Proc: 5) to rpc-transport (glusterfs) [2020-08-27 15:35:08.536903] E [timer.c:34:gf_timer_call_after] (-->/usr/sbin/glusterfsd(mgmt_submit_request+0x13d) [0x55d4f159acdd] -->/lib64/libgfr pc.so.0(rpc_clnt_submit+0x72d) [0x7ff680989c1d] -->/lib64/libglusterfs.so.0(gf_timer_call_after+0x229) [0x7ff680bee349] ) 0-timer: Either ctx is NULL or ctx cleanup started [Invalid argument] [2020-08-27 15:35:08.537018] E [timer.c:34:gf_timer_call_after] (-->/usr/sbin/glusterfsd(mgmt_submit_request+0x13d) [0x55d4f159acdd] -->/lib64/libgfr pc.so.0(rpc_clnt_submit+0x72d) [0x7ff680989c1d] -->/lib64/libglusterfs.so.0(gf_timer_call_after+0x229) [0x7ff680bee349] ) 0-timer: Either ctx is NULL or ctx cleanup started [Invalid argument] [root@ov-no2 glusterfs]# [root@ov-no2 glusterfs]# [root@ov-no2 glusterfs]# [root@ov-no2 glusterfs]# [root@ov-no2 glusterfs]# [root@ov-no2 glusterfs]# more bricks/gluster_bricks-vmstore-vmstore.log-20200830 |egrep "( W | E )"|more [2020-08-27 15:35:09.533389] W [glusterfsd.c:1570:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dd5) [0x7f2d8f5f8dd5] -->/usr/sbin/glusterfsd(glus terfs_sigwaiter+0xe5) [0x563cbb1081b5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x563cbb10801b] ) 0-: received signum (15), shutting down [2020-08-27 15:35:09.540492] W [rpc-clnt.c:1704:rpc_clnt_submit] 0-glusterfs: failed to submit rpc-request (unique: 0, XID: 0x33 Program: Gluster Por tmap, ProgVers: 1, Proc: 5) to rpc-transport (glusterfs) [2020-08-27 15:35:09.540657] E [timer.c:34:gf_timer_call_after] (-->/usr/sbin/glusterfsd(mgmt_submit_request+0x13d) [0x563cbb10dcdd] -->/lib64/libgfr pc.so.0(rpc_clnt_submit+0x72d) [0x7f2d90563c1d] -->/lib64/libglusterfs.so.0(gf_timer_call_after+0x229) [0x7f2d907c8349] ) 0-timer: Either ctx is NULL or ctx cleanup started [Invalid argument] [2020-08-27 15:35:09.540762] E [timer.c:34:gf_timer_call_after] (-->/usr/sbin/glusterfsd(mgmt_submit_request+0x13d) [0x563cbb10dcdd] -->/lib64/libgfr pc.so.0(rpc_clnt_submit+0x72d) [0x7f2d90563c1d] -->/lib64/libglusterfs.so.0(gf_timer_call_after+0x229) [0x7f2d907c8349] ) 0-timer: Either ctx is NULL or ctx cleanup started [Invalid argument] [root@ov-no2 glusterfs]# more bricks/gluster_bricks-data-data.log |egrep "( W | E )"|more [root@ov-no2 glusterfs]# more bricks/gluster_bricks-vmstore-vmstore.log |egrep "( W | E )"|more [root@ov-no2 glusterfs]# more bricks/gluster_bricks-engine-engine.log |egrep "( W | E )"|more [root@ov-no2 glusterfs]# more cmd_history.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no2 glusterfs]# cat glusterd.log | egrep "( W | E )" |more [2020-09-01 07:00:31.325533] W [MSGID: 106061] [glusterd-geo-rep.c:2388:glusterd_op_gsync_args_get] 0-management: master not found [2020-09-01 07:10:36.677633] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame type(glusterd mgmt), op(--(4)), xid = 0x7f74, unique = 0, s ent = 2020-09-01 07:00:31.326798, timeout = 600 for 192.168.48.203:24007 [2020-09-01 07:10:36.677697] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors] 0-glusterd: Commit failed on ov-no3.ariadne-t.local. Please check log file for details. [2020-09-01 07:10:40.678204] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame type(glusterd mgmt), op(--(4)), xid = 0x804d, unique = 0, s ent = 2020-09-01 07:00:31.326774, timeout = 600 for 192.168.48.201:24007 [2020-09-01 07:10:40.678259] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors] 0-glusterd: Commit failed on ov-no1.ariadne-t.local. Please check log file for details. [2020-09-01 07:10:40.678395] E [rpcsvc.c:1577:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x2, Program: GlusterD svc cli, Pr ogVers: 2, Proc: 22) to rpc-transport (socket.management) [2020-09-01 07:10:40.678414] E [MSGID: 106430] [glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission failed [2020-09-01 08:02:31.553170] E [glusterd-op-sm.c:8132:glusterd_op_sm] (-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7efc4e68ea1e] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7efc4e6871be] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7efc 4e6ae06f] ) 0-management: Unable to get transaction opinfo for transaction ID :930a8a08-1044-41cf-b921-913b982e0c72 [2020-09-01 09:04:31.784847] W [MSGID: 106061] [glusterd-geo-rep.c:2388:glusterd_op_gsync_args_get] 0-management: master not found [2020-09-01 09:14:31.577944] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame type(glusterd mgmt), op(--(4)), xid = 0x82ac, unique = 0, s ent = 2020-09-01 09:04:31.786089, timeout = 600 for 192.168.48.201:24007 [2020-09-01 09:14:31.578010] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors] 0-glusterd: Commit failed on ov-no1.ariadne-t.local. Please check log file for details. [2020-09-01 09:14:37.578887] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame type(glusterd mgmt), op(--(4)), xid = 0x81d3, unique = 0, s ent = 2020-09-01 09:04:31.786116, timeout = 600 for 192.168.48.203:24007 [2020-09-01 09:14:37.578945] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors] 0-glusterd: Commit failed on ov-no3.ariadne-t.local. Please check log file for details. [2020-09-01 09:14:37.579189] E [rpcsvc.c:1577:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x2, Program: GlusterD svc cli, Pr ogVers: 2, Proc: 22) to rpc-transport (socket.management) [2020-09-01 09:14:37.579210] E [MSGID: 106430] [glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission failed [2020-09-01 10:06:32.014210] W [MSGID: 106061] [glusterd-geo-rep.c:2388:glusterd_op_gsync_args_get] 0-management: master not found [2020-09-01 10:16:32.031372] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame type(glusterd mgmt), op(--(4)), xid = 0x83d7, unique = 0, s ent = 2020-09-01 10:06:32.015520, timeout = 600 for 192.168.48.201:24007 [2020-09-01 10:16:32.031415] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors] 0-glusterd: Commit failed on ov-no1.ariadne-t.local. Please check log file for details. [2020-09-01 10:16:38.032229] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame type(glusterd mgmt), op(--(4)), xid = 0x82fe, unique = 0, s ent = 2020-09-01 10:06:32.015546, timeout = 600 for 192.168.48.203:24007 [2020-09-01 10:16:38.032281] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors] 0-glusterd: Commit failed on ov-no3.ariadne-t.local. Please check log file for details. [2020-09-01 10:16:38.032491] E [rpcsvc.c:1577:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x2, Program: GlusterD svc cli, Pr ogVers: 2, Proc: 22) to rpc-transport (socket.management) [2020-09-01 10:16:38.032519] E [MSGID: 106430] [glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission failed [root@ov-no2 glusterfs]# cat rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_engine.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no2 glusterfs]# cat rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_data.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no2 glusterfs]# cat rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_vmstore.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no2 glusterfs]# cat cli.log | egrep "( W | E )" |more [2020-09-01 08:49:38.289905] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 08:49:38.291840] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:04:39.528219] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 09:09:40.049281] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 09:09:40.318414] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:09:40.319842] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:14:40.557140] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 09:19:41.064083] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 09:19:41.338070] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:19:41.340016] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:24:41.860173] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:24:41.862096] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:29:42.110479] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 09:29:42.380124] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:29:42.381580] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:34:42.628085] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 09:44:43.895269] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:44:43.897282] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:54:44.902854] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:54:44.904765] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [root@ov-no2 glusterfs]# cat cmd_history.log | egrep "( W | E |fail)" |more [2020-09-01 04:04:39.342142] : volume geo-replication status : FAILED : Commit failed on ov-no3.ariadne-t.local. Please check log file for details. Commit failed on ov-no1.ariadne-t.local. Please check log file for details. [2020-09-01 05:06:39.783310] : volume geo-replication status : FAILED : Commit failed on ov-no3.ariadne-t.local. Please check log file for details. Commit failed on ov-no1.ariadne-t.local. Please check log file for details. [2020-09-01 07:10:40.678333] : volume geo-replication status : FAILED : Commit failed on ov-no3.ariadne-t.local. Please check log file for details. Commit failed on ov-no1.ariadne-t.local. Please check log file for details. [2020-09-01 09:14:37.579120] : volume geo-replication status : FAILED : Commit failed on ov-no1.ariadne-t.local. Please check log file for details. Commit failed on ov-no3.ariadne-t.local. Please check log file for details. [2020-09-01 10:16:38.032427] : volume geo-replication status : FAILED : Commit failed on ov-no1.ariadne-t.local. Please check log file for details. Commit failed on ov-no3.ariadne-t.local. Please check log file for details. NODE3: [root@ov-no3 glusterfs]# more bricks/gluster_bricks-engine-engine.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no3 glusterfs]# more bricks/gluster_bricks-vmstore-vmstore.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no3 glusterfs]# more bricks/gluster_bricks-data-data.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no3 glusterfs]# zcat cli.log |egrep "( W | E )" |more [2020-09-01 07:29:30.165436] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 07:29:30.166677] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 07:34:30.666721] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 07:34:30.668801] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 07:44:31.431679] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 07:49:31.936985] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 07:54:32.447922] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 08:09:34.242293] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 08:09:34.244149] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 08:29:36.002322] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 08:34:36.773092] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 08:34:36.775476] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 08:44:37.512700] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 08:54:38.528131] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 08:54:38.797499] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 08:54:38.798780] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 08:59:39.030389] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 08:59:39.293311] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 08:59:39.294612] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:14:40.823633] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:14:40.825547] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:34:42.890897] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:34:42.893012] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:39:43.129137] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 09:44:43.629375] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 09:49:44.395649] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:49:44.397886] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:59:45.148111] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 09:59:45.409361] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:59:45.410701] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 10:14:46.647785] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 10:14:46.908152] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 10:14:46.909401] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [root@ov-no3 glusterfs]# more cmd_history.log | egrep "( W | E )" |more [root@ov-no3 glusterfs]# cat glusterd.log | egrep "( W | E )" |more [2020-09-01 08:02:31.550060] W [MSGID: 106061] [glusterd-geo-rep.c:2388:glusterd_op_gsync_args_get] 0-management: master not found [2020-09-01 08:12:31.582984] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame type(glusterd mgmt), op(--(4)), xid = 0x7fe0, unique = 0, s ent = 2020-09-01 08:02:31.551222, timeout = 600 for 192.168.48.202:24007 [2020-09-01 08:12:31.583040] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors] 0-glusterd: Commit failed on ov-no2.ariadne-t.local. Please check log file for details. [2020-09-01 08:12:31.583086] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame type(glusterd mgmt), op(--(4)), xid = 0x7fd9, unique = 0, s ent = 2020-09-01 08:02:31.551255, timeout = 600 for 192.168.48.201:24007 [2020-09-01 08:12:31.583100] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors] 0-glusterd: Commit failed on ov-no1.ariadne-t.local. Please check log file for details. [2020-09-01 08:12:31.583253] E [rpcsvc.c:1577:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x2, Program: GlusterD svc cli, Pr ogVers: 2, Proc: 22) to rpc-transport (socket.management) [2020-09-01 08:12:31.583269] E [MSGID: 106430] [glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission failed [2020-09-01 09:04:31.786592] E [glusterd-op-sm.c:8132:glusterd_op_sm] (-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f56fecfba1e] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f56fecf41be] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f56 fed1b06f] ) 0-management: Unable to get transaction opinfo for transaction ID :9942b579-5240-4fee-bb4c-78b9a1c98da8 [2020-09-01 10:06:32.014980] E [glusterd-op-sm.c:8132:glusterd_op_sm] (-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f56fecfba1e] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f56fecf41be] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f56 fed1b06f] ) 0-management: Unable to get transaction opinfo for transaction ID :324fc904-b147-4801-89d5-485120e1fb84 [2020-09-01 11:08:32.243189] E [glusterd-op-sm.c:8132:glusterd_op_sm] (-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f56fecfba1e] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f56fecf41be] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f56 fed1b06f] ) 0-management: Unable to get transaction opinfo for transaction ID :3b7b79b3-344c-4a67-b66e-c8fb8ddb2a2a [2020-09-01 12:10:32.473993] E [glusterd-op-sm.c:8132:glusterd_op_sm] (-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f56fecfba1e] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f56fecf41be] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f56 fed1b06f] ) 0-management: Unable to get transaction opinfo for transaction ID :2521f708-6bc4-4be4-b37f-41d67839e941 [2020-09-01 13:12:32.707217] W [MSGID: 106061] [glusterd-geo-rep.c:2388:glusterd_op_gsync_args_get] 0-management: master not found [2020-09-01 13:22:33.887995] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame type(glusterd mgmt), op(--(4)), xid = 0x866e, unique = 0, s ent = 2020-09-01 13:12:32.708475, timeout = 600 for 192.168.48.202:24007 [2020-09-01 13:22:33.888057] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors] 0-glusterd: Commit failed on ov-no2.ariadne-t.local. Please check log file for details. [2020-09-01 13:22:33.888095] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame type(glusterd mgmt), op(--(4)), xid = 0x8667, unique = 0, s ent = 2020-09-01 13:12:32.708500, timeout = 600 for 192.168.48.201:24007 [2020-09-01 13:22:33.888113] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors] 0-glusterd: Commit failed on ov-no1.ariadne-t.local. Please check log file for details. [2020-09-01 13:22:33.888234] E [rpcsvc.c:1577:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x2, Program: GlusterD svc cli, Pr ogVers: 2, Proc: 22) to rpc-transport (socket.management) [2020-09-01 13:22:33.888269] E [MSGID: 106430] [glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission failed [root@ov-no3 glusterfs]# cat events.log-20200828 | egrep "( W | E )" |more [nothing in the output] [root@ov-no3 glusterfs]# cat glfsheal-data.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no3 glusterfs]# cat glfsheal-vmstore.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no3 glusterfs]# cat glfsheal-engine.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no3 glusterfs]# cat glustershd.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no3 glusterfs]# cat rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_engine.log | egrep "( W | E )" |more [2020-08-31 14:49:13.805467] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2] 0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf) remote_fd is -1. EBADFD [File descriptor in bad state] [2020-08-31 14:51:13.882683] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2] 0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf) remote_fd is -1. EBADFD [File descriptor in bad state] [2020-08-31 14:53:12.691884] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2] 0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf) remote_fd is -1. EBADFD [File descriptor in bad state] [2020-08-31 14:55:12.713320] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2] 0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf) remote_fd is -1. EBADFD [File descriptor in bad state] [2020-08-31 14:57:13.146570] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2] 0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf) remote_fd is -1. EBADFD [File descriptor in bad state] [2020-08-31 14:59:13.502830] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2] 0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf) remote_fd is -1. EBADFD [File descriptor in bad state] [2020-08-31 15:01:13.039274] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2] 0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf) remote_fd is -1. EBADFD [File descriptor in bad state] [2020-08-31 15:03:13.401571] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2] 0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf) remote_fd is -1. EBADFD [File descriptor in bad state] [2020-08-31 15:05:13.764951] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2] 0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf) remote_fd is -1. EBADFD [File descriptor in bad state] [2020-08-31 15:07:14.140156] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2] 0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf) remote_fd is -1. EBADFD [File descriptor in bad state] [2020-08-31 15:09:13.061657] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2] 0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf) remote_fd is -1. EBADFD [File descriptor in bad state] [2020-08-31 15:11:14.616511] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2] 0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf) remote_fd is -1. EBADFD [File descriptor in bad state] [root@ov-no3 glusterfs]# cat rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_data.log | egrep "( W | E )" |more [nothing recent in the output] [root@ov-no3 glusterfs]# cat rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_vmstore.log | egrep "( W | E )" |more [nothing recent in the output] [root@ov-no3 glusterfs]# cat glustershd.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no3 glusterfs]# cat events.log | egrep "( W | E )" |more [nothing in the output] Journalctl results on the nodes about the gluster service: NODE1: [root@ov-no1 ~]# journalctl --unit glusterd -l -- Logs begin at Thu 2020-08-27 19:30:14 EEST, end at Fri 2020-09-04 16:34:42 EEST. -- Aug 27 19:30:52 ov-no1.ariadne-t.local systemd[1]: Starting GlusterFS, a clustered file-system server... Aug 27 19:30:56 ov-no1.ariadne-t.local systemd[1]: Started GlusterFS, a clustered file-system server. Aug 27 19:30:56 ov-no1.ariadne-t.local glusterd[5591]: [2020-08-27 16:30:56.544861] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum Aug 27 19:30:56 ov-no1.ariadne-t.local glusterd[5591]: [2020-08-27 16:30:56.558812] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum Aug 27 19:30:56 ov-no1.ariadne-t.local glusterd[5591]: [2020-08-27 16:30:56.571473] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum NODE2: [root@ov-no2 ~]# journalctl --unit glusterd -l -- Logs begin at Wed 2019-09-11 17:40:13 EEST, end at Fri 2020-09-04 16:34:27 EEST. -- Aug 27 18:35:01 ov-no2.ariadne-t.local systemd[1]: Stopping GlusterFS, a clustered file-system server... Aug 27 18:35:01 ov-no2.ariadne-t.local systemd[1]: Stopped GlusterFS, a clustered file-system server. Aug 27 18:35:01 ov-no2.ariadne-t.local systemd[1]: Starting GlusterFS, a clustered file-system server... Aug 27 18:35:05 ov-no2.ariadne-t.local systemd[1]: Started GlusterFS, a clustered file-system server. Aug 27 18:35:07 ov-no2.ariadne-t.local glusterd[28722]: [2020-08-27 15:35:07.519155] C [MSGID: 106002] [glusterd-server-quorum.c:355:glusterd_do_volu Aug 27 18:35:08 ov-no2.ariadne-t.local glusterd[28722]: [2020-08-27 15:35:08.532018] C [MSGID: 106002] [glusterd-server-quorum.c:355:glusterd_do_volu Aug 27 18:35:09 ov-no2.ariadne-t.local glusterd[28722]: [2020-08-27 15:35:09.532867] C [MSGID: 106002] [glusterd-server-quorum.c:355:glusterd_do_volu Aug 27 18:35:10 ov-no2.ariadne-t.local glusterd[28722]: [2020-08-27 15:35:10.535976] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volu Aug 27 18:35:10 ov-no2.ariadne-t.local glusterd[28722]: [2020-08-27 15:35:10.550171] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volu Aug 27 18:35:10 ov-no2.ariadne-t.local glusterd[28722]: [2020-08-27 15:35:10.563481] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volu Aug 27 19:25:23 ov-no2.ariadne-t.local glustershd[22187]: [2020-08-27 16:25:23.640530] C [rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 0-data-cli NODE3: [root@ov-no3 glusterfs]# journalctl --unit glusterd -l -- Logs begin at Thu 2020-08-27 19:09:30 EEST, end at Fri 2020-09-04 16:34:02 EEST. -- Aug 27 19:10:08 ov-no3.ariadne-t.local systemd[1]: Starting GlusterFS, a clustered file-system server... Aug 27 19:10:13 ov-no3.ariadne-t.local systemd[1]: Started GlusterFS, a clustered file-system server. Aug 27 19:10:13 ov-no3.ariadne-t.local glusterd[5233]: [2020-08-27 16:10:13.274001] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum Aug 27 19:10:13 ov-no3.ariadne-t.local glusterd[5233]: [2020-08-27 16:10:13.288391] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum Aug 27 19:10:13 ov-no3.ariadne-t.local glusterd[5233]: [2020-08-27 16:10:13.300866] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum Aug 27 19:14:38 ov-no3.ariadne-t.local systemd[1]: Stopping GlusterFS, a clustered file-system server... Aug 27 19:14:38 ov-no3.ariadne-t.local systemd[1]: Stopped GlusterFS, a clustered file-system server. Aug 27 19:14:38 ov-no3.ariadne-t.local systemd[1]: Starting GlusterFS, a clustered file-system server... Aug 27 19:14:41 ov-no3.ariadne-t.local systemd[1]: Started GlusterFS, a clustered file-system server. Aug 27 19:14:48 ov-no3.ariadne-t.local glusterd[9843]: [2020-08-27 16:14:48.117530] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum Aug 27 19:14:48 ov-no3.ariadne-t.local glusterd[9843]: [2020-08-27 16:14:48.131054] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum Aug 27 19:14:48 ov-no3.ariadne-t.local glusterd[9843]: [2020-08-27 16:14:48.141323] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum Aug 27 19:25:25 ov-no3.ariadne-t.local glustershd[10081]: [2020-08-27 16:25:25.286189] C [rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 0-data-cli Thanks for your help, Maria Souvalioti
Hi Maria, I am quite puzzled about:
/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x555e3137101b] ) 0-: received signum (15), shutting down [2020-08-27 15:35:14.890471] E [MSGID: 100018] [glusterfsd.c:2333:glusterfs_pidfile_update] 0-glusterfsd: pidfile /var/run/gluster/vols/data/ov-no1.a riadne-t.local-gluster_bricks-data-data.pid lock failed [Resource temporarily unavailable]
That doesn't make sense. Can you share the logs in a separate thread at gluster-users@gluster.org ? Best Regards, Strahil Nikolov В петък, 4 септември 2020 г., 19:55:24 Гринуич+3, souvaliotimaria@mail.com <souvaliotimaria@mail.com> написа: Hello, This is what I could gather from the gluster logs around the time frame of the HE shutdown. NODE1: [root@ov-no1 glusterfs]# more bricks/gluster_bricks-vmstore-vmstore.log-20200830 |egrep "( W | E )"|more [2020-08-27 15:35:03.090477] W [glusterfsd.c:1570:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dd5) [0x7fa6e04a3dd5] -->/usr/sbin/glusterfsd(glus terfs_sigwaiter+0xe5) [0x55a40138d1b5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x55a40138d01b] ) 0-: received signum (15), shutting down [2020-08-27 15:35:14.926794] E [MSGID: 100018] [glusterfsd.c:2333:glusterfs_pidfile_update] 0-glusterfsd: pidfile /var/run/gluster/vols/vmstore/ov-no 1.ariadne-t.local-gluster_bricks-vmstore-vmstore.pid lock failed [Resource temporarily unavailable] [root@ov-no1 glusterfs]# more bricks/gluster_bricks-data-data.log-20200830 |egrep "( W | E )"|more [2020-08-27 15:35:01.087875] W [glusterfsd.c:1570:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dd5) [0x7fc3cbf69dd5] -->/usr/sbin/glusterfsd(glus terfs_sigwaiter+0xe5) [0x555e313711b5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x555e3137101b] ) 0-: received signum (15), shutting down [2020-08-27 15:35:14.890471] E [MSGID: 100018] [glusterfsd.c:2333:glusterfs_pidfile_update] 0-glusterfsd: pidfile /var/run/gluster/vols/data/ov-no1.a riadne-t.local-gluster_bricks-data-data.pid lock failed [Resource temporarily unavailable] [root@ov-no1 glusterfs]# more bricks/gluster_bricks-engine-engine.log-20200830 |egrep "( W | E )"|more [2020-08-27 15:35:02.088732] W [glusterfsd.c:1570:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dd5) [0x7f70b99cbdd5] -->/usr/sbin/glusterfsd(glus terfs_sigwaiter+0xe5) [0x55ebd132b1b5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x55ebd132b01b] ) 0-: received signum (15), shutting down [2020-08-27 15:35:14.907603] E [MSGID: 100018] [glusterfsd.c:2333:glusterfs_pidfile_update] 0-glusterfsd: pidfile /var/run/gluster/vols/engine/ov-no1 .ariadne-t.local-gluster_bricks-engine-engine.pid lock failed [Resource temporarily unavailable] [root@ov-no1 glusterfs]# more bricks/gluster_bricks-vmstore-vmstore.log |egrep "( W | E )"|more [nothing in the output] [root@ov-no1 glusterfs]# more bricks/gluster_bricks-data-data.log |egrep "( W | E )"|more [nothing in the output] [root@ov-no1 glusterfs]# more bricks/gluster_bricks-engine-engine.log |egrep "( W | E )"|more [nothing in the output] [root@ov-no1 glusterfs]# more cmd_history.log | egrep "(WARN|error|fail)" |more [2020-09-01 02:00:38.685251] : volume geo-replication status : FAILED : Commit failed on ov-no2.ariadne-t.local. Please check log file for details. Commit failed on ov-no3.ariadne-t.local. Please check log file for details. [2020-09-01 03:02:39.094984] : volume geo-replication status : FAILED : Commit failed on ov-no2.ariadne-t.local. Please check log file for details. Commit failed on ov-no3.ariadne-t.local. Please check log file for details. [2020-09-01 11:18:32.510224] : volume geo-replication status : FAILED : Commit failed on ov-no2.ariadne-t.local. Please check log file for details. Commit failed on ov-no3.ariadne-t.local. Please check log file for details. [2020-09-01 14:24:33.778942] : volume geo-replication status : FAILED : Commit failed on ov-no2.ariadne-t.local. Please check log file for details. Commit failed on ov-no3.ariadne-t.local. Please check log file for details. [root@ov-no1 glusterfs]# cat glusterd.log | egrep "( W | E )" |more [2020-09-01 07:00:31.326169] E [glusterd-op-sm.c:8132:glusterd_op_sm] (-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f23d8ac8a1e] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f23d8ac11be] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f23 d8ae806f] ) 0-management: Unable to get transaction opinfo for transaction ID :435d3780-aa0c-4a64-bc28-56ae394159d0 [2020-09-01 08:02:31.551563] E [glusterd-op-sm.c:8132:glusterd_op_sm] (-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f23d8ac8a1e] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f23d8ac11be] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f23 d8ae806f] ) 0-management: Unable to get transaction opinfo for transaction ID :930a8a08-1044-41cf-b921-913b982e0c72 [2020-09-01 09:04:31.786157] E [glusterd-op-sm.c:8132:glusterd_op_sm] (-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f23d8ac8a1e] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f23d8ac11be] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f23 d8ae806f] ) 0-management: Unable to get transaction opinfo for transaction ID :9942b579-5240-4fee-bb4c-78b9a1c98da8 [2020-09-01 10:06:32.014362] E [glusterd-op-sm.c:8132:glusterd_op_sm] (-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f23d8ac8a1e] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f23d8ac11be] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f23 d8ae806f] ) 0-management: Unable to get transaction opinfo for transaction ID :324fc904-b147-4801-89d5-485120e1fb84 [2020-09-01 11:08:32.240906] W [MSGID: 106061] [glusterd-geo-rep.c:2388:glusterd_op_gsync_args_get] 0-management: master not found [2020-09-01 11:18:32.510041] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame type(glusterd mgmt), op(--(4)), xid = 0x812c, unique = 0, s ent = 2020-09-01 11:08:32.242199, timeout = 600 for 192.168.48.202:24007 [root@ov-no1 glusterfs]# cat events.log-20200828 | egrep "( W | E )" |more [nothing in the output] [root@ov-no1 glusterfs]# cat events.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no1 glusterfs]# cat glfsheal-data.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no1 glusterfs]# cat glfsheal-vmstore.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no1 glusterfs]# cat glfsheal-engine.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no1 glusterfs]# cat glfsheal-data.log-20200830 | egrep "( W | E )" |more [nothing in the output] [root@ov-no1 glusterfs]# cat glfsheal-vmstore.log-20200830 | egrep "( W | E )" |more [nothing in the output] [root@ov-no1 glusterfs]# cat glfsheal-engine.log-20200830 | egrep "( W | E )" |more [nothing in the output] [root@ov-no1 glusterfs]# cat glustershd.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no1 glusterfs]# cat rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_engine.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no1 glusterfs]# cat rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_data.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no1 glusterfs]# cat rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_vmstore.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no1 glusterfs]# cat rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_data.log [2020-08-31 00:41:02.169424] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-08-31 00:41:02.170418] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-08-31 08:59:07.694673] I [MSGID: 133022] [shard.c:3673:shard_delete_shards] 0-data-shard: Deleted shards of gfid=d25c9bb6-4a25-4025-8652-686be0a4bdf1 from backend [2020-08-31 08:59:07.890175] I [MSGID: 133017] [shard.c:6696:shard_seek] 0-data-shard: seek called on 0e4c3fb8-7617-4e43-879a-cc784198216d. [Operation not supported] [2020-08-31 08:59:07.892803] I [MSGID: 133017] [shard.c:6696:shard_seek] 0-data-shard: seek called on 17c220e1-fa2c-42d7-a948-6ea2bf5a93d7. [Operation not supported] [2020-08-31 08:59:07.897929] I [MSGID: 133017] [shard.c:6696:shard_seek] 0-data-shard: seek called on f9315204-ec16-4c46-847e-d84f0169ea8f. [Operation not supported] [2020-08-31 08:59:07.928488] I [MSGID: 133022] [shard.c:3673:shard_delete_shards] 0-data-shard: Deleted shards of gfid=17c220e1-fa2c-42d7-a948-6ea2bf5a93d7 from backend [2020-09-01 00:36:01.667677] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-09-01 00:36:01.668986] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-09-02 00:48:02.278423] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-09-02 00:48:02.279637] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-09-04 00:36:01.546102] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-09-04 00:36:01.547101] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [root@ov-no1 glusterfs]# cat rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_vmstore.log [2020-08-28 00:12:02.424709] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-08-29 00:29:02.375537] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-08-29 00:29:02.377088] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-08-30 00:13:03.449758] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-08-30 00:13:03.451232] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-08-31 00:41:02.169556] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-08-31 00:41:02.171040] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-09-01 00:36:01.667768] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-09-01 00:36:01.669572] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-09-02 00:48:02.278490] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-09-02 00:48:02.280253] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-09-04 00:36:01.546134] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-09-04 00:36:01.547694] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [root@ov-no1 glusterfs]# [root@ov-no1 glusterfs]# cat rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_engine.log [2020-08-30 00:13:03.447882] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-08-30 00:13:03.449756] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-08-31 00:41:02.167497] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-08-31 00:41:02.169892] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-09-01 00:36:01.666356] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-09-01 00:36:01.668276] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-09-02 00:48:02.277024] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-09-02 00:48:02.279273] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-09-04 00:36:01.543799] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-09-04 00:36:01.545779] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [root@ov-no1 glusterfs]# cat glustershd.log | egrep "( W | E )" |more [root@ov-no1 glusterfs]# cat glustershd.log [2020-08-30 20:31:26.884546] I [MSGID: 108026] [afr-self-heal-common.c:1741:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on 3fafabf3-d0cd-4b9a-8dd7-43145451f7cf. sources=[1] 2 sinks=0 . . . [every ten minutes the same log appears] . . [2020-08-31 15:03:57.893224] I [MSGID: 108026] [afr-self-heal-common.c:1741:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on 3fafabf3-d0cd-4b9a-8dd7-43145451f7cf. sources=[1] 2 sinks=0 The message "I [MSGID: 108026] [afr-self-heal-common.c:1741:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on 3fafabf3-d0cd-4b9a-8dd7-43145451f7cf. sources=[1] 2 sinks=0 " repeated 15 times between [2020-08-31 15:03:57.893224] and [2020-08-31 15:04:42.605183] [2020-08-31 15:14:46.171044] I [MSGID: 108026] [afr-self-heal-common.c:1741:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on 3fafabf3-d0cd-4b9a-8dd7-43145451f7cf. sources=[1] 2 sinks=0 [2020-09-01 00:36:01.663267] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-09-01 00:36:01.665109] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [2020-09-02 00:48:02.274383] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2020-09-02 00:48:02.275562] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing [root@ov-no1 glusterfs]# zcat cli.log-20200902.gz |egrep "( W | E )" |more [2020-09-01 09:04:39.799900] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:04:39.801917] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:24:41.581140] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 09:39:43.395156] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:39:43.397147] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:49:44.128540] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 09:54:44.629071] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 10:04:45.642437] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 10:04:45.904006] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 10:04:45.905279] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 10:39:49.183655] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 10:44:49.968219] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 10:44:49.970487] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 10:49:50.206005] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 10:59:51.489242] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 10:59:51.491277] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [root@ov-no1 glusterfs]# cat cli.log |grep "( W | E )" |more [nothing in the output] [root@ov-no1 glusterfs]# cat cmd_history.log|egrep "( W| E|fail)" |more [2020-09-01 00:58:38.265743] : volume geo-replication status : FAILED : Commit failed on ov-no2.ariadne-t.local. Please check log file for details. Commit failed on ov-no3.ariadne-t.local. Please check log file for details. [2020-09-01 02:00:38.685251] : volume geo-replication status : FAILED : Commit failed on ov-no2.ariadne-t.local. Please check log file for details. Commit failed on ov-no3.ariadne-t.local. Please check log file for details. [2020-09-01 03:02:39.094984] : volume geo-replication status : FAILED : Commit failed on ov-no2.ariadne-t.local. Please check log file for details. Commit failed on ov-no3.ariadne-t.local. Please check log file for details. [2020-09-01 11:18:32.510224] : volume geo-replication status : FAILED : Commit failed on ov-no2.ariadne-t.local. Please check log file for details. Commit failed on ov-no3.ariadne-t.local. Please check log file for details. [2020-09-01 14:24:33.778942] : volume geo-replication status : FAILED : Commit failed on ov-no2.ariadne-t.local. Please check log file for details. Commit failed on ov-no3.ariadne-t.local. Please check log file for details. [2020-09-01 16:28:34.610703] : volume geo-replication status : FAILED : Commit failed on ov-no2.ariadne-t.local. Please check log file for details. Commit failed on ov-no3.ariadne-t.local. Please check log file for details. NODE2: [root@ov-no2 ~]# cd /var/log/glusterfs/ [root@ov-no2 glusterfs]# more bricks/gluster_bricks-data-data.log-20200830 |egrep "( W | E )"|more [2020-08-27 15:35:07.532064] W [glusterfsd.c:1570:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dd5) [0x7f01ef674dd5] -->/usr/sbin/glusterfsd(glus terfs_sigwaiter+0xe5) [0x55d96c9791b5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x55d96c97901b] ) 0-: received signum (15), shutting down [2020-08-27 15:35:07.545460] W [rpc-clnt.c:1704:rpc_clnt_submit] 0-glusterfs: failed to submit rpc-request (unique: 0, XID: 0x33 Program: Gluster Por tmap, ProgVers: 1, Proc: 5) to rpc-transport (glusterfs) [2020-08-27 15:35:07.545626] E [timer.c:34:gf_timer_call_after] (-->/usr/sbin/glusterfsd(mgmt_submit_request+0x13d) [0x55d96c97ecdd] -->/lib64/libgfr pc.so.0(rpc_clnt_submit+0x72d) [0x7f01f05dfc1d] -->/lib64/libglusterfs.so.0(gf_timer_call_after+0x229) [0x7f01f0844349] ) 0-timer: Either ctx is NULL or ctx cleanup started [Invalid argument] [2020-08-27 15:35:07.545724] E [timer.c:34:gf_timer_call_after] (-->/usr/sbin/glusterfsd(mgmt_submit_request+0x13d) [0x55d96c97ecdd] -->/lib64/libgfr pc.so.0(rpc_clnt_submit+0x72d) [0x7f01f05dfc1d] -->/lib64/libglusterfs.so.0(gf_timer_call_after+0x229) [0x7f01f0844349] ) 0-timer: Either ctx is NULL or ctx cleanup started [Invalid argument] [2020-08-27 16:25:42.954545] W [inodelk.c:609:pl_inodelk_log_cleanup] 0-data-server: releasing lock on 03848819-f37b-4fa7-81d9-3f5dc3eef2da held by { client=0x7fd84400f350, pid=7725 lk-owner=e8100a64447f0000} [root@ov-no2 glusterfs]# [root@ov-no2 glusterfs]# [root@ov-no2 glusterfs]# [root@ov-no2 glusterfs]# [root@ov-no2 glusterfs]# more bricks/gluster_bricks-engine-engine.log-20200830 |egrep "( W | E )"|more [2020-08-27 15:35:08.532812] W [glusterfsd.c:1570:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dd5) [0x7ff67fa1edd5] -->/usr/sbin/glusterfsd(glus terfs_sigwaiter+0xe5) [0x55d4f15951b5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x55d4f159501b] ) 0-: received signum (15), shutting down [2020-08-27 15:35:08.536715] W [rpc-clnt.c:1704:rpc_clnt_submit] 0-glusterfs: failed to submit rpc-request (unique: 0, XID: 0x33 Program: Gluster Por tmap, ProgVers: 1, Proc: 5) to rpc-transport (glusterfs) [2020-08-27 15:35:08.536903] E [timer.c:34:gf_timer_call_after] (-->/usr/sbin/glusterfsd(mgmt_submit_request+0x13d) [0x55d4f159acdd] -->/lib64/libgfr pc.so.0(rpc_clnt_submit+0x72d) [0x7ff680989c1d] -->/lib64/libglusterfs.so.0(gf_timer_call_after+0x229) [0x7ff680bee349] ) 0-timer: Either ctx is NULL or ctx cleanup started [Invalid argument] [2020-08-27 15:35:08.537018] E [timer.c:34:gf_timer_call_after] (-->/usr/sbin/glusterfsd(mgmt_submit_request+0x13d) [0x55d4f159acdd] -->/lib64/libgfr pc.so.0(rpc_clnt_submit+0x72d) [0x7ff680989c1d] -->/lib64/libglusterfs.so.0(gf_timer_call_after+0x229) [0x7ff680bee349] ) 0-timer: Either ctx is NULL or ctx cleanup started [Invalid argument] [root@ov-no2 glusterfs]# [root@ov-no2 glusterfs]# [root@ov-no2 glusterfs]# [root@ov-no2 glusterfs]# [root@ov-no2 glusterfs]# [root@ov-no2 glusterfs]# more bricks/gluster_bricks-vmstore-vmstore.log-20200830 |egrep "( W | E )"|more [2020-08-27 15:35:09.533389] W [glusterfsd.c:1570:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dd5) [0x7f2d8f5f8dd5] -->/usr/sbin/glusterfsd(glus terfs_sigwaiter+0xe5) [0x563cbb1081b5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x563cbb10801b] ) 0-: received signum (15), shutting down [2020-08-27 15:35:09.540492] W [rpc-clnt.c:1704:rpc_clnt_submit] 0-glusterfs: failed to submit rpc-request (unique: 0, XID: 0x33 Program: Gluster Por tmap, ProgVers: 1, Proc: 5) to rpc-transport (glusterfs) [2020-08-27 15:35:09.540657] E [timer.c:34:gf_timer_call_after] (-->/usr/sbin/glusterfsd(mgmt_submit_request+0x13d) [0x563cbb10dcdd] -->/lib64/libgfr pc.so.0(rpc_clnt_submit+0x72d) [0x7f2d90563c1d] -->/lib64/libglusterfs.so.0(gf_timer_call_after+0x229) [0x7f2d907c8349] ) 0-timer: Either ctx is NULL or ctx cleanup started [Invalid argument] [2020-08-27 15:35:09.540762] E [timer.c:34:gf_timer_call_after] (-->/usr/sbin/glusterfsd(mgmt_submit_request+0x13d) [0x563cbb10dcdd] -->/lib64/libgfr pc.so.0(rpc_clnt_submit+0x72d) [0x7f2d90563c1d] -->/lib64/libglusterfs.so.0(gf_timer_call_after+0x229) [0x7f2d907c8349] ) 0-timer: Either ctx is NULL or ctx cleanup started [Invalid argument] [root@ov-no2 glusterfs]# more bricks/gluster_bricks-data-data.log |egrep "( W | E )"|more [root@ov-no2 glusterfs]# more bricks/gluster_bricks-vmstore-vmstore.log |egrep "( W | E )"|more [root@ov-no2 glusterfs]# more bricks/gluster_bricks-engine-engine.log |egrep "( W | E )"|more [root@ov-no2 glusterfs]# more cmd_history.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no2 glusterfs]# cat glusterd.log | egrep "( W | E )" |more [2020-09-01 07:00:31.325533] W [MSGID: 106061] [glusterd-geo-rep.c:2388:glusterd_op_gsync_args_get] 0-management: master not found [2020-09-01 07:10:36.677633] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame type(glusterd mgmt), op(--(4)), xid = 0x7f74, unique = 0, s ent = 2020-09-01 07:00:31.326798, timeout = 600 for 192.168.48.203:24007 [2020-09-01 07:10:36.677697] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors] 0-glusterd: Commit failed on ov-no3.ariadne-t.local. Please check log file for details. [2020-09-01 07:10:40.678204] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame type(glusterd mgmt), op(--(4)), xid = 0x804d, unique = 0, s ent = 2020-09-01 07:00:31.326774, timeout = 600 for 192.168.48.201:24007 [2020-09-01 07:10:40.678259] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors] 0-glusterd: Commit failed on ov-no1.ariadne-t.local. Please check log file for details. [2020-09-01 07:10:40.678395] E [rpcsvc.c:1577:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x2, Program: GlusterD svc cli, Pr ogVers: 2, Proc: 22) to rpc-transport (socket.management) [2020-09-01 07:10:40.678414] E [MSGID: 106430] [glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission failed [2020-09-01 08:02:31.553170] E [glusterd-op-sm.c:8132:glusterd_op_sm] (-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7efc4e68ea1e] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7efc4e6871be] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7efc 4e6ae06f] ) 0-management: Unable to get transaction opinfo for transaction ID :930a8a08-1044-41cf-b921-913b982e0c72 [2020-09-01 09:04:31.784847] W [MSGID: 106061] [glusterd-geo-rep.c:2388:glusterd_op_gsync_args_get] 0-management: master not found [2020-09-01 09:14:31.577944] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame type(glusterd mgmt), op(--(4)), xid = 0x82ac, unique = 0, s ent = 2020-09-01 09:04:31.786089, timeout = 600 for 192.168.48.201:24007 [2020-09-01 09:14:31.578010] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors] 0-glusterd: Commit failed on ov-no1.ariadne-t.local. Please check log file for details. [2020-09-01 09:14:37.578887] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame type(glusterd mgmt), op(--(4)), xid = 0x81d3, unique = 0, s ent = 2020-09-01 09:04:31.786116, timeout = 600 for 192.168.48.203:24007 [2020-09-01 09:14:37.578945] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors] 0-glusterd: Commit failed on ov-no3.ariadne-t.local. Please check log file for details. [2020-09-01 09:14:37.579189] E [rpcsvc.c:1577:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x2, Program: GlusterD svc cli, Pr ogVers: 2, Proc: 22) to rpc-transport (socket.management) [2020-09-01 09:14:37.579210] E [MSGID: 106430] [glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission failed [2020-09-01 10:06:32.014210] W [MSGID: 106061] [glusterd-geo-rep.c:2388:glusterd_op_gsync_args_get] 0-management: master not found [2020-09-01 10:16:32.031372] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame type(glusterd mgmt), op(--(4)), xid = 0x83d7, unique = 0, s ent = 2020-09-01 10:06:32.015520, timeout = 600 for 192.168.48.201:24007 [2020-09-01 10:16:32.031415] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors] 0-glusterd: Commit failed on ov-no1.ariadne-t.local. Please check log file for details. [2020-09-01 10:16:38.032229] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame type(glusterd mgmt), op(--(4)), xid = 0x82fe, unique = 0, s ent = 2020-09-01 10:06:32.015546, timeout = 600 for 192.168.48.203:24007 [2020-09-01 10:16:38.032281] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors] 0-glusterd: Commit failed on ov-no3.ariadne-t.local. Please check log file for details. [2020-09-01 10:16:38.032491] E [rpcsvc.c:1577:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x2, Program: GlusterD svc cli, Pr ogVers: 2, Proc: 22) to rpc-transport (socket.management) [2020-09-01 10:16:38.032519] E [MSGID: 106430] [glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission failed [root@ov-no2 glusterfs]# cat rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_engine.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no2 glusterfs]# cat rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_data.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no2 glusterfs]# cat rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_vmstore.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no2 glusterfs]# cat cli.log | egrep "( W | E )" |more [2020-09-01 08:49:38.289905] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 08:49:38.291840] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:04:39.528219] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 09:09:40.049281] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 09:09:40.318414] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:09:40.319842] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:14:40.557140] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 09:19:41.064083] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 09:19:41.338070] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:19:41.340016] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:24:41.860173] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:24:41.862096] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:29:42.110479] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 09:29:42.380124] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:29:42.381580] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:34:42.628085] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 09:44:43.895269] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:44:43.897282] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:54:44.902854] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:54:44.904765] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [root@ov-no2 glusterfs]# cat cmd_history.log | egrep "( W | E |fail)" |more [2020-09-01 04:04:39.342142] : volume geo-replication status : FAILED : Commit failed on ov-no3.ariadne-t.local. Please check log file for details. Commit failed on ov-no1.ariadne-t.local. Please check log file for details. [2020-09-01 05:06:39.783310] : volume geo-replication status : FAILED : Commit failed on ov-no3.ariadne-t.local. Please check log file for details. Commit failed on ov-no1.ariadne-t.local. Please check log file for details. [2020-09-01 07:10:40.678333] : volume geo-replication status : FAILED : Commit failed on ov-no3.ariadne-t.local. Please check log file for details. Commit failed on ov-no1.ariadne-t.local. Please check log file for details. [2020-09-01 09:14:37.579120] : volume geo-replication status : FAILED : Commit failed on ov-no1.ariadne-t.local. Please check log file for details. Commit failed on ov-no3.ariadne-t.local. Please check log file for details. [2020-09-01 10:16:38.032427] : volume geo-replication status : FAILED : Commit failed on ov-no1.ariadne-t.local. Please check log file for details. Commit failed on ov-no3.ariadne-t.local. Please check log file for details. NODE3: [root@ov-no3 glusterfs]# more bricks/gluster_bricks-engine-engine.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no3 glusterfs]# more bricks/gluster_bricks-vmstore-vmstore.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no3 glusterfs]# more bricks/gluster_bricks-data-data.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no3 glusterfs]# zcat cli.log |egrep "( W | E )" |more [2020-09-01 07:29:30.165436] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 07:29:30.166677] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 07:34:30.666721] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 07:34:30.668801] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 07:44:31.431679] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 07:49:31.936985] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 07:54:32.447922] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 08:09:34.242293] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 08:09:34.244149] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 08:29:36.002322] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 08:34:36.773092] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 08:34:36.775476] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 08:44:37.512700] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 08:54:38.528131] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 08:54:38.797499] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 08:54:38.798780] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 08:59:39.030389] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 08:59:39.293311] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 08:59:39.294612] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:14:40.823633] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:14:40.825547] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:34:42.890897] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:34:42.893012] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:39:43.129137] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 09:44:43.629375] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 09:49:44.395649] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:49:44.397886] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:59:45.148111] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 09:59:45.409361] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 09:59:45.410701] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 10:14:46.647785] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli: Cannot generate event for unknown type. [2020-09-01 10:14:46.908152] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op] 0-cli: Failed to get sub-cmd [2020-09-01 10:14:46.909401] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op] 0-cli: Failed to get sub-cmd [root@ov-no3 glusterfs]# more cmd_history.log | egrep "( W | E )" |more [root@ov-no3 glusterfs]# cat glusterd.log | egrep "( W | E )" |more [2020-09-01 08:02:31.550060] W [MSGID: 106061] [glusterd-geo-rep.c:2388:glusterd_op_gsync_args_get] 0-management: master not found [2020-09-01 08:12:31.582984] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame type(glusterd mgmt), op(--(4)), xid = 0x7fe0, unique = 0, s ent = 2020-09-01 08:02:31.551222, timeout = 600 for 192.168.48.202:24007 [2020-09-01 08:12:31.583040] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors] 0-glusterd: Commit failed on ov-no2.ariadne-t.local. Please check log file for details. [2020-09-01 08:12:31.583086] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame type(glusterd mgmt), op(--(4)), xid = 0x7fd9, unique = 0, s ent = 2020-09-01 08:02:31.551255, timeout = 600 for 192.168.48.201:24007 [2020-09-01 08:12:31.583100] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors] 0-glusterd: Commit failed on ov-no1.ariadne-t.local. Please check log file for details. [2020-09-01 08:12:31.583253] E [rpcsvc.c:1577:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x2, Program: GlusterD svc cli, Pr ogVers: 2, Proc: 22) to rpc-transport (socket.management) [2020-09-01 08:12:31.583269] E [MSGID: 106430] [glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission failed [2020-09-01 09:04:31.786592] E [glusterd-op-sm.c:8132:glusterd_op_sm] (-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f56fecfba1e] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f56fecf41be] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f56 fed1b06f] ) 0-management: Unable to get transaction opinfo for transaction ID :9942b579-5240-4fee-bb4c-78b9a1c98da8 [2020-09-01 10:06:32.014980] E [glusterd-op-sm.c:8132:glusterd_op_sm] (-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f56fecfba1e] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f56fecf41be] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f56 fed1b06f] ) 0-management: Unable to get transaction opinfo for transaction ID :324fc904-b147-4801-89d5-485120e1fb84 [2020-09-01 11:08:32.243189] E [glusterd-op-sm.c:8132:glusterd_op_sm] (-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f56fecfba1e] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f56fecf41be] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f56 fed1b06f] ) 0-management: Unable to get transaction opinfo for transaction ID :3b7b79b3-344c-4a67-b66e-c8fb8ddb2a2a [2020-09-01 12:10:32.473993] E [glusterd-op-sm.c:8132:glusterd_op_sm] (-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f56fecfba1e] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f56fecf41be] -->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f56 fed1b06f] ) 0-management: Unable to get transaction opinfo for transaction ID :2521f708-6bc4-4be4-b37f-41d67839e941 [2020-09-01 13:12:32.707217] W [MSGID: 106061] [glusterd-geo-rep.c:2388:glusterd_op_gsync_args_get] 0-management: master not found [2020-09-01 13:22:33.887995] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame type(glusterd mgmt), op(--(4)), xid = 0x866e, unique = 0, s ent = 2020-09-01 13:12:32.708475, timeout = 600 for 192.168.48.202:24007 [2020-09-01 13:22:33.888057] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors] 0-glusterd: Commit failed on ov-no2.ariadne-t.local. Please check log file for details. [2020-09-01 13:22:33.888095] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame type(glusterd mgmt), op(--(4)), xid = 0x8667, unique = 0, s ent = 2020-09-01 13:12:32.708500, timeout = 600 for 192.168.48.201:24007 [2020-09-01 13:22:33.888113] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors] 0-glusterd: Commit failed on ov-no1.ariadne-t.local. Please check log file for details. [2020-09-01 13:22:33.888234] E [rpcsvc.c:1577:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x2, Program: GlusterD svc cli, Pr ogVers: 2, Proc: 22) to rpc-transport (socket.management) [2020-09-01 13:22:33.888269] E [MSGID: 106430] [glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission failed [root@ov-no3 glusterfs]# cat events.log-20200828 | egrep "( W | E )" |more [nothing in the output] [root@ov-no3 glusterfs]# cat glfsheal-data.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no3 glusterfs]# cat glfsheal-vmstore.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no3 glusterfs]# cat glfsheal-engine.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no3 glusterfs]# cat glustershd.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no3 glusterfs]# cat rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_engine.log | egrep "( W | E )" |more [2020-08-31 14:49:13.805467] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2] 0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf) remote_fd is -1. EBADFD [File descriptor in bad state] [2020-08-31 14:51:13.882683] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2] 0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf) remote_fd is -1. EBADFD [File descriptor in bad state] [2020-08-31 14:53:12.691884] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2] 0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf) remote_fd is -1. EBADFD [File descriptor in bad state] [2020-08-31 14:55:12.713320] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2] 0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf) remote_fd is -1. EBADFD [File descriptor in bad state] [2020-08-31 14:57:13.146570] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2] 0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf) remote_fd is -1. EBADFD [File descriptor in bad state] [2020-08-31 14:59:13.502830] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2] 0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf) remote_fd is -1. EBADFD [File descriptor in bad state] [2020-08-31 15:01:13.039274] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2] 0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf) remote_fd is -1. EBADFD [File descriptor in bad state] [2020-08-31 15:03:13.401571] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2] 0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf) remote_fd is -1. EBADFD [File descriptor in bad state] [2020-08-31 15:05:13.764951] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2] 0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf) remote_fd is -1. EBADFD [File descriptor in bad state] [2020-08-31 15:07:14.140156] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2] 0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf) remote_fd is -1. EBADFD [File descriptor in bad state] [2020-08-31 15:09:13.061657] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2] 0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf) remote_fd is -1. EBADFD [File descriptor in bad state] [2020-08-31 15:11:14.616511] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2] 0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf) remote_fd is -1. EBADFD [File descriptor in bad state] [root@ov-no3 glusterfs]# cat rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_data.log | egrep "( W | E )" |more [nothing recent in the output] [root@ov-no3 glusterfs]# cat rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_vmstore.log | egrep "( W | E )" |more [nothing recent in the output] [root@ov-no3 glusterfs]# cat glustershd.log | egrep "( W | E )" |more [nothing in the output] [root@ov-no3 glusterfs]# cat events.log | egrep "( W | E )" |more [nothing in the output] Journalctl results on the nodes about the gluster service: NODE1: [root@ov-no1 ~]# journalctl --unit glusterd -l -- Logs begin at Thu 2020-08-27 19:30:14 EEST, end at Fri 2020-09-04 16:34:42 EEST. -- Aug 27 19:30:52 ov-no1.ariadne-t.local systemd[1]: Starting GlusterFS, a clustered file-system server... Aug 27 19:30:56 ov-no1.ariadne-t.local systemd[1]: Started GlusterFS, a clustered file-system server. Aug 27 19:30:56 ov-no1.ariadne-t.local glusterd[5591]: [2020-08-27 16:30:56.544861] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum Aug 27 19:30:56 ov-no1.ariadne-t.local glusterd[5591]: [2020-08-27 16:30:56.558812] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum Aug 27 19:30:56 ov-no1.ariadne-t.local glusterd[5591]: [2020-08-27 16:30:56.571473] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum NODE2: [root@ov-no2 ~]# journalctl --unit glusterd -l -- Logs begin at Wed 2019-09-11 17:40:13 EEST, end at Fri 2020-09-04 16:34:27 EEST. -- Aug 27 18:35:01 ov-no2.ariadne-t.local systemd[1]: Stopping GlusterFS, a clustered file-system server... Aug 27 18:35:01 ov-no2.ariadne-t.local systemd[1]: Stopped GlusterFS, a clustered file-system server. Aug 27 18:35:01 ov-no2.ariadne-t.local systemd[1]: Starting GlusterFS, a clustered file-system server... Aug 27 18:35:05 ov-no2.ariadne-t.local systemd[1]: Started GlusterFS, a clustered file-system server. Aug 27 18:35:07 ov-no2.ariadne-t.local glusterd[28722]: [2020-08-27 15:35:07.519155] C [MSGID: 106002] [glusterd-server-quorum.c:355:glusterd_do_volu Aug 27 18:35:08 ov-no2.ariadne-t.local glusterd[28722]: [2020-08-27 15:35:08.532018] C [MSGID: 106002] [glusterd-server-quorum.c:355:glusterd_do_volu Aug 27 18:35:09 ov-no2.ariadne-t.local glusterd[28722]: [2020-08-27 15:35:09.532867] C [MSGID: 106002] [glusterd-server-quorum.c:355:glusterd_do_volu Aug 27 18:35:10 ov-no2.ariadne-t.local glusterd[28722]: [2020-08-27 15:35:10.535976] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volu Aug 27 18:35:10 ov-no2.ariadne-t.local glusterd[28722]: [2020-08-27 15:35:10.550171] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volu Aug 27 18:35:10 ov-no2.ariadne-t.local glusterd[28722]: [2020-08-27 15:35:10.563481] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volu Aug 27 19:25:23 ov-no2.ariadne-t.local glustershd[22187]: [2020-08-27 16:25:23.640530] C [rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 0-data-cli NODE3: [root@ov-no3 glusterfs]# journalctl --unit glusterd -l -- Logs begin at Thu 2020-08-27 19:09:30 EEST, end at Fri 2020-09-04 16:34:02 EEST. -- Aug 27 19:10:08 ov-no3.ariadne-t.local systemd[1]: Starting GlusterFS, a clustered file-system server... Aug 27 19:10:13 ov-no3.ariadne-t.local systemd[1]: Started GlusterFS, a clustered file-system server. Aug 27 19:10:13 ov-no3.ariadne-t.local glusterd[5233]: [2020-08-27 16:10:13.274001] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum Aug 27 19:10:13 ov-no3.ariadne-t.local glusterd[5233]: [2020-08-27 16:10:13.288391] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum Aug 27 19:10:13 ov-no3.ariadne-t.local glusterd[5233]: [2020-08-27 16:10:13.300866] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum Aug 27 19:14:38 ov-no3.ariadne-t.local systemd[1]: Stopping GlusterFS, a clustered file-system server... Aug 27 19:14:38 ov-no3.ariadne-t.local systemd[1]: Stopped GlusterFS, a clustered file-system server. Aug 27 19:14:38 ov-no3.ariadne-t.local systemd[1]: Starting GlusterFS, a clustered file-system server... Aug 27 19:14:41 ov-no3.ariadne-t.local systemd[1]: Started GlusterFS, a clustered file-system server. Aug 27 19:14:48 ov-no3.ariadne-t.local glusterd[9843]: [2020-08-27 16:14:48.117530] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum Aug 27 19:14:48 ov-no3.ariadne-t.local glusterd[9843]: [2020-08-27 16:14:48.131054] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum Aug 27 19:14:48 ov-no3.ariadne-t.local glusterd[9843]: [2020-08-27 16:14:48.141323] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum Aug 27 19:25:25 ov-no3.ariadne-t.local glustershd[10081]: [2020-08-27 16:25:25.286189] C [rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 0-data-cli Thanks for your help, Maria Souvalioti _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/PZ7SUT7FY6PEY4...
Hi Maria, My system experiencing a similar issue as you have captured. Can you tell me where I can find the HostedEngine.xml to confirm the virtio disk associated with my hosted engine error?
participants (4)
-
souvaliotimaria＠mail.com -
Strahil Nikolov -
travisq＠gmail.com -
Yedidyah Bar David