7:14 p.m.
Hello everyone,
I have a replica 2 + arbiter installation and this morning the Hosted Engine gave the
following error on the UI and resumed on a different node (node3) than the one it was
originally running(node1). (The original node has more memory than the one it ended up,
but it had a better memory usage percentage at the time). Also, the only way I discovered
the migration had happened and there was an Error in Events, was because I logged in the
web interface of ovirt for a routine inspection. Βesides that, everything was working
properly and still is.
The error that popped is the following:
VM HostedEngine is down with error. Exit message: internal error: qemu unexpectedly closed
the monitor:
2020-09-01T06:49:20.749126Z qemu-kvm: warning: All CPU(s) up to maxcpus should be
described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and
will be removed in future
2020-09-01T06:49:20.927274Z qemu-kvm: -device
virtio-blk-pci,iothread=iothread1,scsi=off,bus=pci.0,addr=0x7,drive=drive-ua-d5de54b6-9f8e-4fba-819b-ebf6780757d2,id=ua-d5de54b6-9f8e-4fba-819b-ebf6780757d2,bootindex=1,write-cache=on:
Failed to get "write" lock
Is another process using the image?.
Which from what I could gather concerns the following snippet from the HostedEngine.xml
and it's the virtio disk of the Hosted Engine:
<disk type='file' device='disk' snapshot='no'>
<driver name='qemu' type='raw' cache='none'
error_policy='stop' io='threads' iothread='1'/>
<source
file='/var/run/vdsm/storage/80f6e393-9718-4738-a14a-64cf43c3d8c2/d5de54b6-9f8e-4fba-819b-ebf6780757d2/a48555f4-be23-4467-8a54-400ae7baf9d7'>
<seclabel model='dac' relabel='no'/>
</source>
<target dev='vda' bus='virtio'/>
<serial>d5de54b6-9f8e-4fba-819b-ebf6780757d2</serial>
<alias name='ua-d5de54b6-9f8e-4fba-819b-ebf6780757d2'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x07' function='0x0'/>
</disk>
I've tried looking into the logs and the sar command but I couldn't find anything
to relate with the above errors and determining the reason for it to happen. Is this a
Gluster or a QEMU problem?
The Hosted Engine was manually migrated five days before on node1.
Is there a standard practice I could follow to determine what happened and secure my
system?
Thank you very much for your time,
Maria Souvalioti
9:35 a.m.
On Tue, Sep 1, 2020 at 7:17 PM <souvaliotimaria(a)mail.com> wrote:
Hello everyone,
I have a replica 2 + arbiter installation and this morning the Hosted Engine gave the
following error on the UI and resumed on a different node (node3) than the one it was
originally running(node1). (The original node has more memory than the one it ended up,
but it had a better memory usage percentage at the time). Also, the only way I discovered
the migration had happened and there was an Error in Events, was because I logged in the
web interface of ovirt for a routine inspection. Βesides that, everything was working
properly and still is.
The error that popped is the following:
VM HostedEngine is down with error. Exit message: internal error: qemu unexpectedly
closed the monitor:
2020-09-01T06:49:20.749126Z qemu-kvm: warning: All CPU(s) up to maxcpus should be
described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and
will be removed in future
2020-09-01T06:49:20.927274Z qemu-kvm: -device
virtio-blk-pci,iothread=iothread1,scsi=off,bus=pci.0,addr=0x7,drive=drive-ua-d5de54b6-9f8e-4fba-819b-ebf6780757d2,id=ua-d5de54b6-9f8e-4fba-819b-ebf6780757d2,bootindex=1,write-cache=on:
Failed to get "write" lock
Is another process using the image?.
It's quite likely that this isn't the root cause.
Please check your logs from before that.
Above looks like something (ovirt-ha-agent?) tried to start the hosted
engine VM, but failed due to locking - most likely, because it was
already up elsewhere (on some other host?).
So you want to check when/where the VM was started before this error,
and then carefully any errors before it was started.
Also, check that the clocks on all your machines are in sync.
Which from what I could gather concerns the following snippet from the HostedEngine.xml
and it's the virtio disk of the Hosted Engine:
<disk type='file' device='disk' snapshot='no'>
<driver name='qemu' type='raw' cache='none'
error_policy='stop' io='threads' iothread='1'/>
<source
file='/var/run/vdsm/storage/80f6e393-9718-4738-a14a-64cf43c3d8c2/d5de54b6-9f8e-4fba-819b-ebf6780757d2/a48555f4-be23-4467-8a54-400ae7baf9d7'>
<seclabel model='dac' relabel='no'/>
</source>
<target dev='vda' bus='virtio'/>
<serial>d5de54b6-9f8e-4fba-819b-ebf6780757d2</serial>
<alias name='ua-d5de54b6-9f8e-4fba-819b-ebf6780757d2'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x07' function='0x0'/>
</disk>
I've tried looking into the logs and the sar command but I couldn't find anything
to relate with the above errors and determining the reason for it to happen. Is this a
Gluster or a QEMU problem?
Likely, but hard to tell without more information.
The Hosted Engine was manually migrated five days before on node1.
Is there a standard practice I could follow to determine what happened and secure my
system?
Nothing, other than checking the logs.
Check, on all of your hosts:
/var/log/messages
/var/log/vdsm/*
/var/log/ovirt-hosted-engine-ha/*
And on the engine (likely won't help in this case, but just in case):
/var/log/ovirt-engine/*
Thank you very much for your time,
Good luck and best regards,
--
Didi
8:46 p.m.
Thank you very much for your reply.
I checked the NTP and realized the service wasn't working properly on two of the three
nodes, but despite that the clocks seemed to have the correct time(date and hwdate). I
switched to chronyd and stopped the ntpd service and now it seems the servers' clocks
are synchronized.
The time in BIOS has different time than the systems. Does this affect the behaviour of
the overall performance?
This is what I could gather from the logs:
Node1:
[root@ov-no1 ~]# more /var/log/messages |egrep "(WARN|error)"|more
d
Aug 27 17:53:08 ov-no1 libvirtd: 2020-08-27 14:53:08.947+0000: 5613: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Aug 27 17:58:08 ov-no1 libvirtd: 2020-08-27 14:58:08.943+0000: 5613: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Aug 27 18:03:08 ov-no1 libvirtd: 2020-08-27 15:03:08.937+0000: 5614: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Aug 27 18:08:08 ov-no1 libvirtd: 2020-08-27 15:08:08.951+0000: 5617: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Aug 27 18:13:08 ov-no1 libvirtd: 2020-08-27 15:13:08.951+0000: 5616: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Aug 27 18:18:08 ov-no1 libvirtd: 2020-08-27 15:18:08.942+0000: 5618: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
.
.
[Around that time Node 3, who is the arbiter node, was placed in local Maintenance mode.
It was shut down for maintenance and when we boot it up again all seemed right. We removed
it from the maintenance mode and when the healing processes finished, Node 1 became
NonResponsive. Long story short, the VDSM agent sent Node1 a restart command. Node1
rebooted, HostedEngine was up on Node1 and the rest of the VMs that were hosted by Node1
had to be manually brought up. Since then everything seemed to be working as it should.
The HostedEngine VM shutdown with no apparent reason 5 days later, making us believe there
was no connection between the two incidents.]
.
.
.
.
.
Sep 1 05:53:30 ov-no1 vdsm[6706]: WARN Worker blocked: <Worker name=jsonrpc/3 running
<Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0',
'method':
u'GlusterVolume.geoRepSessionList', 'id':
u'e1f4db88-5d36-4ba1-8848-3855c1dbc6ee'} at 0x7f1ed7381650> timeout=60,
duration=60.00 at 0x7f1ed7381dd0> t
ask#=76268 at 0x7f1ebc0797d0>, traceback:#012File:
"/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012
self.__bootstrap_inner()#012Fil
e: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012
self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run
#012 self.__target(*self.__args, **self.__kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in
run#012 ret =
func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012
self._execute_task()#012File: "/usr/lib/p
ython2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012
task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 3
91, in __call__#012 self._callable()#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in
__call__#012 self._handler(sel
f._ctx, self._req)#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in
_serveRequest#012 response = self._handle_request
(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
line 345, in _handle_request#012 res = method(**params)#012File: "/usr
/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012
result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-package
s/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012
remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py",
l
ine 93, in wrapper#012 rv = func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in
volumeGeoRepSessionL
ist#012 remoteUserName,#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in
__call__#012 return callMethod()#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in
<lambda>#012 **kwargs)#012File: "<string>", line 2, in
glusterVolumeGeoRep
Status#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in
_callmethod#012 kind, result = conn.recv()
Sep 1 05:54:30 ov-no1 vdsm[6706]: WARN Worker blocked: <Worker name=jsonrpc/3 running
<Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0',
'method':
u'GlusterVolume.geoRepSessionList', 'id':
u'e1f4db88-5d36-4ba1-8848-3855c1dbc6ee'} at 0x7f1ed7381650> timeout=60,
duration=120.00 at 0x7f1ed7381dd0>
task#=76268 at 0x7f1ebc0797d0>, traceback:#012File:
"/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012
self.__bootstrap_inner()#012Fi
le: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012
self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in ru
n#012 self.__target(*self.__args, **self.__kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in
run#012 ret
= func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012
self._execute_task()#012File: "/usr/lib/
python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012
task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line
391, in __call__#012 self._callable()#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in
__call__#012 self._handler(se
lf._ctx, self._req)#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in
_serveRequest#012 response = self._handle_reques
t(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
line 345, in _handle_request#012 res = method(**params)#012File: "/us
r/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012
result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-packag
es/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012
remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py",
line 93, in wrapper#012 rv = func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in
volumeGeoRepSession
List#012 remoteUserName,#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in
__call__#012 return callMethod()#012File
: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in
<lambda>#012 **kwargs)#012File: "<string>", line 2, in
glusterVolumeGeoRe
pStatus#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759,
in _callmethod#012 kind, result = conn.recv()
[root@ov-no1 ~]# xzcat /var/log/vdsm/vdsm.log.57.xz |egrep "(WARN|error)" |more
2020-09-01 09:40:50,023+0300 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:40:50,023+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:41:00,061+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:41:00,061+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:41:10,078+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:41:10,078+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:41:20,086+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:41:20,086+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:41:30,107+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:41:30,107+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:41:38,710+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:41:38,711+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:41:40,115+0300 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:41:40,115+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.01 seconds (__init__:312)
2020-09-01 09:41:44,541+0300 WARN (monitor/e2e7282) [storage.PersistentDict] Could not
parse line `
2020-09-01 09:41:44,544+0300 WARN (monitor/0a5e552) [storage.PersistentDict] Could not
parse line `
[root@ov-no1 ~]# xzcat /var/log/vdsm/supervdsm.log.29.xz |egrep
"(WARN|error|ERR)" |more
MainProcess|jsonrpc/1::WARNING::2020-09-01
09:32:32,182::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-01
09:32:32,183::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-01
09:32:32,185::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-01
09:32:32,304::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-01
09:32:32,306::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-01
09:32:32,308::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,440::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,441::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,442::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,442::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,443::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,443::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,444::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,445::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,445::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,358::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,361::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,363::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,365::udev::53::blivet::(get_device) No
device at ''
[root@ov-no1 ~]# xzcat /var/log/vdsm/supervdsm.log.28.xz |egrep
"(WARN|error|ERR)" |more
MainProcess|jsonrpc/7::WARNING::2020-09-01
11:32:36,730::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/7::WARNING::2020-09-01
11:32:36,731::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/7::WARNING::2020-09-01
11:32:36,732::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/7::WARNING::2020-09-01
11:32:36,847::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/7::WARNING::2020-09-01
11:32:36,849::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/7::WARNING::2020-09-01
11:32:36,850::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,974::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,975::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,975::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,976::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,977::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,977::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,978::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,979::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,979::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/7::ERROR::2020-09-01 11:32:40,951::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/7::ERROR::2020-09-01 11:32:40,954::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/7::ERROR::2020-09-01 11:32:40,956::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/7::ERROR::2020-09-01 11:32:40,958::udev::53::blivet::(get_device) No
device at ''
[root@ov-no1 ~]# cat /var/log/ovirt-hosted-engine-ha/broker.log.2020-09-01 |egrep -i
"(WARN|ERR)" |more
[nothing in the output]
[root@ov-no1 ~]# gluster volume status
Status of volume: data
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ov-no1.ariadne-t.local:/gluster_brick
s/data/data 49152 0 Y 6070
Brick ov-no2.ariadne-t.local:/gluster_brick
s/data/data 49152 0 Y 28860
Brick ov-no3.ariadne-t.local:/gluster_brick
s/data/data 49152 0 Y 5752
Self-heal Daemon on localhost N/A N/A Y 6539
Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813
Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353
Task Status of Volume data
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: engine
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ov-no1.ariadne-t.local:/gluster_brick
s/engine/engine 49153 0 Y 6098
Brick ov-no2.ariadne-t.local:/gluster_brick
s/engine/engine 49156 0 Y 28867
Brick ov-no3.ariadne-t.local:/gluster_brick
s/engine/engine 49153 0 Y 5789
Self-heal Daemon on localhost N/A N/A Y 6539
Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813
Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353
Task Status of Volume engine
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: vmstore
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ov-no1.ariadne-t.local:/gluster_brick
s/vmstore/vmstore 49154 0 Y 6102
Brick ov-no2.ariadne-t.local:/gluster_brick
s/vmstore/vmstore 49157 0 Y 28878
Brick ov-no3.ariadne-t.local:/gluster_brick
s/vmstore/vmstore 49154 0 Y 5772
Self-heal Daemon on localhost N/A N/A Y 6539
Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813
Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353
Task Status of Volume vmstore
------------------------------------------------------------------------------
There are no active volume tasks
[root@ov-no1 ~]# gluster volume heal engine info
Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Number of entries: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Number of entries: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Number of entries: 0
[root@ov-no1 ~]# gluster volume heal engine info summary
Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
[root@ov-no1 ~]# gluster volume heal data info
Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Number of entries: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Number of entries: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Number of entries: 0
[root@ov-no1 ~]# gluster volume heal data info summary
Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
[root@ov-no1 ~]# gluster volume heal vmstore info
Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Number of entries: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Number of entries: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Number of entries: 0
[root@ov-no1 ~]# gluster volume heal vmstore info summary
Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
[root@ov-no1 ~]# ps -A|grep gluster
5522 ? 00:01:40 glustereventsd
5591 ? 00:05:04 glusterd
6070 ? 02:16:28 glusterfsd
6098 ? 05:44:14 glusterfsd
6102 ? 00:23:48 glusterfsd
6285 ? 00:00:38 glustereventsd
6539 ? 01:32:19 glusterfs
10470 ? 00:34:10 glusterfs
17763 ? 00:50:18 glusterfs
18101 ? 00:10:23 glusterfs
Node2:
[root@ov-no2 ~]# more /var/log/messages |egrep "(WARN|error)"|more
Sep 1 07:58:30 ov-no2 vdsm[13586]: WARN Worker blocked: <Worker name=jsonrpc/7 running
<Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0',
'method':
u'GlusterVolume.geoRepSessionList', 'id':
u'ec8af6cc-6109-4ddb-a6fb-ad9f4fbc1e7b'} at 0x7f44602a7c90> timeout=60,
duration=120.00 at 0x7f44602a7510>
task#=5773350 at 0x7f44c00c71d0>, traceback:#012File:
"/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012
self.__bootstrap_inner()#01
2File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012
self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in
run#012 self.__target(*self.__args, **self.__kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in
run#012 r
et = func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012
self._execute_task()#012File: "/usr/l
ib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012
task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", li
ne 391, in __call__#012 self._callable()#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in
__call__#012 self._handler
(self._ctx, self._req)#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in
_serveRequest#012 response = self._handle_req
uest(req, ctx)#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in
_handle_request#012 res = method(**params)#012File: "
/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012
result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-pac
kages/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012
remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py
", line 93, in wrapper#012 rv = func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in
volumeGeoRepSess
ionList#012 remoteUserName,#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in
__call__#012 return callMethod()#012F
ile: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in
<lambda>#012 **kwargs)#012File: "<string>", line 2, in
glusterVolumeGe
oRepStatus#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line
759, in _callmethod#012 kind, result = conn.recv()
Sep 1 09:48:02 ov-no2 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Unhandled monitoring loop
exception#012T
raceback (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 449, in start_moni
toring#012 self._monitoring_loop()#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 468, in _monit
oring_loop#012 for old_state, state, delay in self.fsm:#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py",
li
ne 127, in next#012 new_data = self.refresh(self._state.data)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/state_machi
ne.py", line 81, in refresh#012
stats.update(self.hosted_engine.collect_stats())#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_h
a/agent/hosted_engine.py", line 756, in collect_stats#012 all_stats =
self._broker.get_stats_from_storage()#012 File "/usr/lib/python2.7/site-pac
kages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 143, in
get_stats_from_storage#012 result = self._proxy.get_stats()#012 File
"/usr/lib64/py
thon2.7/xmlrpclib.py", line 1233, in __call__#012 return self.__send(self.__name,
args)#012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1591,
in __request#012 verbose=self.__verbose#012 File
"/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request#012 return
self.single_request(hos
t, handler, request_body, verbose)#012 File
"/usr/lib64/python2.7/xmlrpclib.py", line 1301, in single_request#012
self.send_content(h, request_bo
dy)#012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in
send_content#012 connection.endheaders(request_body)#012 File "/usr/lib64/python
2.7/httplib.py", line 1037, in endheaders#012 self._send_output(message_body)#012
File "/usr/lib64/python2.7/httplib.py", line 881, in _send_outp
ut#012 self.send(msg)#012 File "/usr/lib64/python2.7/httplib.py", line 843,
in send#012 self.connect()#012 File "/usr/lib/python2.7/site-pack
ages/ovirt_hosted_engine_ha/lib/unixrpc.py", line 52, in connect#012
self.sock.connect(base64.b16decode(self.host))#012 File "/usr/lib64/python2.
7/socket.py", line 224, in meth#012 return
getattr(self._sock,name)(*args)#012error: [Errno 2] No such file or directory
Sep 1 09:48:02 ov-no2 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent
ERROR Traceback (most recent call last):#012 File "/usr/lib/
python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in
_run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-p
ackages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012
return he.start_monitoring()#012 File "/usr/lib/python2.7/site-pack
ages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 456, in
start_monitoring#012 self.publish(stopped)#012 File "/usr/lib/python2.7/site-pa
ckages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 356, in publish#012
self._push_to_storage(blocks)#012 File "/usr/lib/python2.7/site-p
ackages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 727, in
_push_to_storage#012 self._broker.put_stats_on_storage(self.host_id, blocks)#
012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 113, in put_stats_on_storage#012 self._proxy.put_stats
(host_id, xmlrpclib.Binary(data))#012 File "/usr/lib64/python2.7/xmlrpclib.py",
line 1233, in __call__#012 return self.__send(self.__name, args)#
012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in __request#012
verbose=self.__verbose#012 File "/usr/lib64/python2.7/xmlrpclib.py", l
ine 1273, in request#012 return self.single_request(host, handler, request_body,
verbose)#012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1301
, in single_request#012 self.send_content(h, request_body)#012 File
"/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content#012 connec
tion.endheaders(request_body)#012 File "/usr/lib64/python2.7/httplib.py", line
1037, in endheaders#012 self._send_output(message_body)#012 File
"/usr/lib64/python2.7/httplib.py", line 881, in _send_output#012
self.send(msg)#012 File "/usr/lib64/python2.7/httplib.py", line 843, in
send#012
self.connect()#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py", line
52, in connect#012 self.sock.connect(b
ase64.b16decode(self.host))#012 File "/usr/lib64/python2.7/socket.py", line
224, in meth#012 return getattr(self._sock,name)(*args)#012error: [Er
rno 2] No such file or directory
Sep 1 09:48:15 ov-no2 journal: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to
host#012Trace
back (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py",
line 26, in send_email#012
timeout=float(cfg["smtp-timeout"]))#012 File
"/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) =
self.connect(host, po
rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012
self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l
ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return
socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so
cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No
route to host
Sep 1 09:48:25 ov-no2 journal: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to
host#012Trace
back (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py",
line 26, in send_email#012
timeout=float(cfg["smtp-timeout"]))#012 File
"/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) =
self.connect(host, po
rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012
self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l
ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return
socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so
cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No
route to host
Sep 1 09:49:15 ov-no2 journal: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to
host#012Trace
back (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py",
line 26, in send_email#012
timeout=float(cfg["smtp-timeout"]))#012 File
"/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) =
self.connect(host, po
rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012
self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l
ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return
socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so
cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No
route to host
Sep 1 09:49:16 ov-no2 journal: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to
host#012Trace
back (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py",
line 26, in send_email#012
timeout=float(cfg["smtp-timeout"]))#012 File
"/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) =
self.connect(host, po
rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012
self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l
ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return
socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so
cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No
route to host
Sep 1 09:49:17 ov-no2 vdsm[13586]: WARN Attempting to add an existing net user:
ovirtmgmt/baa3d4d1-6918-4447-80fa-64b86dbcb1f9
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables
--concurrent -t nat -D PREROUTING -i vnet0 -j libvirt-J-vnet0' f
ailed: Illegal target name 'libvirt-J-vnet0'.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables
--concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-P-vnet0'
failed: Illegal target name 'libvirt-P-vnet0'.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables
--concurrent -t nat -L libvirt-J-vnet0' failed: Chain 'libvirt-J
-vnet0' doesn't exist.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables
--concurrent -t nat -L libvirt-P-vnet0' failed: Chain 'libvirt-P
-vnet0' doesn't exist.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables
--concurrent -t nat -F libvirt-J-vnet0' failed: Chain 'libvirt-J
-vnet0' doesn't exist.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables
--concurrent -t nat -X libvirt-J-vnet0' failed: Chain 'libvirt-J
-vnet0' doesn't exist.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables
--concurrent -t nat -F libvirt-P-vnet0' failed: Chain 'libvirt-P
-vnet0' doesn't exist.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables
--concurrent -t nat -X libvirt-P-vnet0' failed: Chain 'libvirt-P
-vnet0' doesn't exist.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables
--concurrent -t nat -F J-vnet0-mac' failed: Chain 'J-vnet0-mac'
doesn't exist.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables
--concurrent -t nat -X J-vnet0-mac' failed: Chain 'J-vnet0-mac'
doesn't exist.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables
--concurrent -t nat -F J-vnet0-arp-mac' failed: Chain 'J-vnet0-a
rp-mac' doesn't exist.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables
--concurrent -t nat -X J-vnet0-arp-mac' failed: Chain 'J-vnet0-a
rp-mac' doesn't exist.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables
-w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-
out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a
chain#012#012Try `iptables -h' or 'iptables --help' for more information.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables
-w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0
' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try
`iptables -h' or 'iptables --help' for more information.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables
-w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0'
failed: iptables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `iptables
-h' or 'iptables --help' for more information.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables
-w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vn
et0' failed: iptables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try
`iptables -h' or 'iptables --help' for more information.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables
-w2 -w -F FO-vnet0' failed: iptables: No chain/target/match by t
hat name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables
-w2 -w -X FO-vnet0' failed: iptables: No chain/target/match by t
hat name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables
-w2 -w -F FI-vnet0' failed: iptables: No chain/target/match by t
hat name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables
-w2 -w -X FI-vnet0' failed: iptables: No chain/target/match by t
hat name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables
-w2 -w -F HI-vnet0' failed: iptables: No chain/target/match by t
hat name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables
-w2 -w -X HI-vnet0' failed: iptables: No chain/target/match by t
hat name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables
-w2 -w -E FP-vnet0 FO-vnet0' failed: iptables: No chain/target/m
atch by that name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables
-w2 -w -E FJ-vnet0 FI-vnet0' failed: iptables: No chain/target/m
atch by that name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables
-w2 -w -E HJ-vnet0 HI-vnet0' failed: iptables: No chain/target/m
atch by that name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables
-w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev
-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a
chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more informatio
n.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables
-w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet
0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try
`ip6tables -h' or 'ip6tables --help' for more information.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables
-w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0'
failed: ip6tables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `ip6tables
-h' or 'ip6tables --help' for more information.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables
-w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-v
net0' failed: ip6tables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try
`ip6tables -h' or 'ip6tables --help' for more information.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables
-w2 -w -F FO-vnet0' failed: ip6tables: No chain/target/match by
that name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables
-w2 -w -X FO-vnet0' failed: ip6tables: No chain/target/match by
that name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables
-w2 -w -F FI-vnet0' failed: ip6tables: No chain/target/match by
that name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables
-w2 -w -X FI-vnet0' failed: ip6tables: No chain/target/match by
that name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables
-w2 -w -F HI-vnet0' failed: ip6tables: No chain/target/match by
that name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables
-w2 -w -X HI-vnet0' failed: ip6tables: No chain/target/match by
that name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables
-w2 -w -E FP-vnet0 FO-vnet0' failed: ip6tables: No chain/target
/match by that name.
Sep 1 09:49:20 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables
-w2 -w -E FJ-vnet0 FI-vnet0' failed: ip6tables: No chain/target
/match by that name.
Sep 1 09:49:20 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables
-w2 -w -E HJ-vnet0 HI-vnet0' failed: ip6tables: No chain/target
/match by that name.
.
.
.
.
.
Sep 1 09:49:22 ov-no2 vdsm[13586]: WARN File:
/var/lib/libvirt/qemu/channels/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.ovirt-guest-agent.0
already remove
d
Sep 1 09:49:22 ov-no2 vdsm[13586]: WARN Attempting to remove a non existing net user:
ovirtmgmt/baa3d4d1-6918-4447-80fa-64b86dbcb1f9
Sep 1 09:49:22 ov-no2 vdsm[13586]: WARN File:
/var/lib/libvirt/qemu/channels/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.org.qemu.guest_agent.0
already rem
oved
Sep 1 09:49:22 ov-no2 vdsm[13586]: WARN File:
/var/run/ovirt-vmconsole-console/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.sock already
removed
Sep 1 09:49:27 ov-no2 journal: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to
host#012Trace
back (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py",
line 26, in send_email#012
timeout=float(cfg["smtp-timeout"]))#012 File
"/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) =
self.connect(host, po
rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012
self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l
ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return
socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so
cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No
route to host
Sep 1 09:49:27 ov-no2 journal: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to
host#012Trace
back (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py",
line 26, in send_email#012
timeout=float(cfg["smtp-timeout"]))#012 File
"/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) =
self.connect(host, po
rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012
self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l
ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return
socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so
cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No
route to host
Sep 1 09:49:37 ov-no2 journal: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to
host#012Trace
back (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py",
line 26, in send_email#012
timeout=float(cfg["smtp-timeout"]))#012 File
"/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) =
self.connect(host, po
rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012
self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l
ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return
socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so
cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No
route to host
Sep 1 10:01:31 ov-no2 vdsm[13586]: WARN Worker blocked: <Worker name=jsonrpc/4 running
<Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0',
'method':
u'GlusterVolume.geoRepSessionList', 'id':
u'7f800636-f107-4b98-a35f-2175ec72b7b8'} at 0x7f44805aa110> timeout=60,
duration=60.00 at 0x7f44805aad10>
task#=5775347 at 0x7f44d81c4110>, traceback:#012File:
"/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012
self.__bootstrap_inner()#012
File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012
self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in
run#012 self.__target(*self.__args, **self.__kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in
run#012 re
t = func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012
self._execute_task()#012File: "/usr/li
b/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012
task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", lin
e 391, in __call__#012 self._callable()#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in
__call__#012 self._handler(
self._ctx, self._req)#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in
_serveRequest#012 response = self._handle_requ
est(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
line 345, in _handle_request#012 res = method(**params)#012File: "/
usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012
result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-pack
ages/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012
remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py"
, line 93, in wrapper#012 rv = func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in
volumeGeoRepSessi
onList#012 remoteUserName,#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in
__call__#012 return callMethod()#012Fi
le: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in
<lambda>#012 **kwargs)#012File: "<string>", line 2, in
glusterVolumeGeo
RepStatus#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759,
in _callmethod#012 kind, result = conn.recv()
[root@ov-no2 ~]# xzcat /var/log/vdsm/vdsm.log.57.xz |egrep "(WARN|error)" |more
2020-09-01 09:24:06,889+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:24:16,910+0300 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:24:16,911+0300 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:24:26,932+0300 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:24:26,933+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.01 seconds (__init__:312)
2020-09-01 09:24:30,375+0300 WARN (monitor/0a5e552) [storage.PersistentDict] Could not
parse line `
2020-09-01 09:24:30,379+0300 WARN (monitor/e2e7282) [storage.PersistentDict] Could not
parse line `
.
.
.
.
2020-09-01 09:47:37,008+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:47:47,030+0300 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:47:47,030+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:48:01,712+0300 WARN (jsonrpc/6) [storage.PersistentDict] Could not parse
line `
2020-09-01 09:48:01,743+0300 WARN (jsonrpc/1) [storage.PersistentDict] Could not parse
line `
2020-09-01 09:48:02,220+0300 WARN (jsonrpc/0) [storage.PersistentDict] Could not parse
line `
2020-09-01 09:48:05,192+0300 WARN (periodic/22) [storage.PersistentDict] Could not parse
line `
2020-09-01 09:48:11,745+0300 WARN (monitor/e2e7282) [storage.PersistentDict] Could not
parse line `
2020-09-01 09:48:13,711+0300 INFO (jsonrpc/7) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:48:13,711+0300 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:48:23,591+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:48:23,591+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:48:23,731+0300 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:48:23,731+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:48:33,754+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:48:33,754+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:48:43,787+0300 INFO (jsonrpc/2) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:48:43,787+0300 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:48:53,808+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:48:53,809+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:49:03,830+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:49:03,831+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:49:13,852+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:49:13,853+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.01 seconds (__init__:312)
2020-09-01 09:49:16,042+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:49:16,042+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.01 seconds (__init__:312)
2020-09-01 09:49:16,405+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:49:16,405+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:49:16,797+0300 INFO (jsonrpc/7) [api.virt] START
create(vmParams={u'xml': u'<?xml version=\'1.0\'
encoding=\'UTF-8\'?>\n<domain xmlns:
ovirt-tune="http://ovirt.org/vm/tune/1.0"
xmlns:ovirt-vm="http://ovirt.org/vm/1.0"
type="kvm"><name>HostedEngine</name><uuid>baa3d4d1-6918-4447-80fa-
64b86dbcb1f9</uuid><memory>16777216</memory><currentMemory>16777216</currentMemory><iothreads>1</iothreads><maxMemory
slots="16">67108864</maxMemory>
<vcpu current="4">64</vcpu><sysinfo
type="smbios"><system><entry
name="manufacturer">oVirt</entry><entry
name="product">OS-NAME:</entry><entry name="
version">OS-VERSION:</entry><entry
name="serial">HOST-SERIAL:</entry><entry
name="uuid">baa3d4d1-6918-4447-80fa-64b86dbcb1f9</entry></system></sysinf
o><clock offset="variable" adjustment="0"><timer
name="rtc" tickpolicy="catchup"/><timer name="pit"
tickpolicy="delay"/><timer name="hpet" present="n
o"/></clock><features><acpi/></features><cpu
match="exact"><model>Westmere</model><feature
name="pcid" policy="require"/><feature
name="spec-ctrl" po
licy="require"/><feature name="ssbd"
policy="require"/><topology cores="4" threads="1"
sockets="16"/><numa><cell id="0" cpus="0,1,2,3"
memory="167772
16"/></numa></cpu><cputune/><devices><input
type="tablet" bus="usb"/><channel
type="unix"><target type="virtio"
name="ovirt-guest-agent.0"/><source m
ode="bind"
path="/var/lib/libvirt/qemu/channels/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.ovirt-guest-agent.0"/></channel><channel
type="unix"><target typ
e="virtio" name="org.qemu.guest_agent.0"/><source
mode="bind"
path="/var/lib/libvirt/qemu/channels/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.org.qemu.gues
t_agent.0"/></channel><controller type="usb"
model="piix3-uhci" index="0"><address bus="0x00"
domain="0x0000" function="0x2" slot="0x01"
type="pci"/>
</controller><console type="unix"><source
path="/var/run/ovirt-vmconsole-console/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.sock"
mode="bind"/><target type
="serial" port="0"/><alias
name="ua-228f9c56-c195-4f1e-8cd2-5f7ef60a78f1"/></console><memballoon
model="virtio"><stats period="5"/><alias name="ua-42
10968c-acf1-46cc-8a35-6e29e3c58b38"/><address bus="0x00"
domain="0x0000" function="0x0" slot="0x08"
type="pci"/></memballoon><controller type="scsi"
model="virtio-scsi" index="0"><driver
iothread="1"/><alias
name="ua-473cb0f7-09da-4077-a0e2-8eb5a5efdbe4"/><address
bus="0x00" domain="0x0000" functi
on="0x0" slot="0x05"
type="pci"/></controller><sound model="ich6"><alias
name="ua-496f9a72-5484-4dc0-83f4-caa8225100ae"/><address
bus="0x00" domain="
0x0000" function="0x0" slot="0x04"
type="pci"/></sound><graphics type="vnc"
port="-1" autoport="yes" passwd="*****"
passwdValidTo="1970-01-01T00:00:0
1" keymap="en-us"><listen type="network"
network="vdsm-ovirtmgmt"/></graphics><controller
type="ide" index="0"><address bus="0x00"
domain="0x0000" fu
nction="0x1" slot="0x01"
type="pci"/></controller><graphics type="spice"
port="-1" autoport="yes" passwd="*****"
passwdValidTo="1970-01-01T00:00:01"
tlsPort="-1"><channel name="main"
mode="secure"/><channel name="inputs"
mode="secure"/><channel name="cursor"
mode="secure"/><channel name="playback"
mode="secure"/><channel name="record"
mode="secure"/><channel name="display"
mode="secure"/><channel name="smartcard"
mode="secure"/><channel name="
usbredir" mode="secure"/><listen type="network"
network="vdsm-ovirtmgmt"/></graphics><rng
model="virtio"><backend
model="random">/dev/urandom</backen
d><alias
name="ua-9edfa822-3b11-424e-83f8-4846c6221a6e"/></rng><video><model
type="qxl" vram="32768" heads="1" ram="65536"
vgamem="16384"/><alias nam
e="ua-a4ad75db-c91d-45df-a767-33a71009587f"/><address bus="0x00"
domain="0x0000" function="0x0" slot="0x02"
type="pci"/></video><controller type="vir
tio-serial" index="0" ports="16"><alias
name="ua-f60f1742-07fe-464e-a160-1a16c94db61a"/><address
bus="0x00" domain="0x0000" function="0x0" slot="0x06
" type="pci"/></controller><serial
type="unix"><source
path="/var/run/ovirt-vmconsole-console/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.sock"
mode="bind"/
<target port="0"/></serial><channel
type="spicevmc"><target type="virtio"
name="com.redhat.spice.0"/></channel><interface
type="bridge"><model type=
"virtio"/><link
state="up"/><source bridge="ovirtmgmt"/><driver
queues="4" name="vhost"/><alias
name="ua-e97c94c0-ffc2-46dd-b5cd-9d9c123f4db0"/><addr
ess bus="0x00" domain="0x0000" function="0x0"
slot="0x03" type="pci"/><mac
address="00:16:3e:27:6e:ef"/><mtu size="1500"/><filterref
filter="vdsm-no-
mac-spoofing"/><bandwidth/></interface><disk type="file"
device="cdrom" snapshot="no"><driver name="qemu"
type="raw" error_policy="report"/><source f
ile="" startupPolicy="optional"><seclabel model="dac"
type="none" relabel="no"/></source><target
dev="hdc" bus="ide"/><readonly/><alias name="ua-9034
9138-d20e-4306-b0e2-271e6baaad4a"/><address bus="1"
controller="0" unit="0" type="drive"
target="0"/></disk><disk snapshot="no"
type="file" device="d
isk"><target dev="vda" bus="virtio"/><source
file="/rhev/data-center/00000000-0000-0000-0000-000000000000/80f6e393-9718-4738-a14a-64cf43c3d8c2/images
/d5de54b6-9f8e-4fba-819b-ebf6780757d2/a48555f4-be23-4467-8a54-400ae7baf9d7"><seclabel
model="dac" type="none"
relabel="no"/></source><driver name="qe
mu" iothread="1" io="threads" type="raw"
error_policy="stop" cache="none"/><alias
name="ua-d5de54b6-9f8e-4fba-819b-ebf6780757d2"/><address
bus="0x00"
domain="0x0000" function="0x0" slot="0x07"
type="pci"/><serial>d5de54b6-9f8e-4fba-819b-ebf6780757d2</serial></disk><lease><key>a48555f4-be23-4467-8a
54-400ae7baf9d7</key><lockspace>80f6e393-9718-4738-a14a-64cf43c3d8c2</lockspace><target
offset="LEASE-OFFSET:a48555f4-be23-4467-8a54-400ae7baf9d7:80f
6e393-9718-4738-a14a-64cf43c3d8c2"
path="LEASE-PATH:a48555f4-be23-4467-8a54-400ae7baf9d7:80f6e393-9718-4738-a14a-64cf43c3d8c2"/></lease></devices><pm
<suspend-to-disk enabled="no"/><suspend-to-mem
enabled="no"/></pm><os><type arch="x86_64"
machine="pc-i440fx-rhel7.6.0">hvm</type><smbios mode="sysi
nfo"/><bios
useserial="yes"/></os><metadata><ovirt-tune:qos/><ovirt-vm:vm><ovirt-vm:minGuaranteedMemoryMb
type="int">1024</ovirt-vm:minGuaranteedMemo
ryMb><ovirt-vm:clusterVersion>4.3</ovirt-vm:clusterVersion><ovirt-vm:custom/><ovirt-vm:device
mac_address="00:16:3e:27:6e:ef"><ovirt-vm:custom/></ovi
rt-vm:device><ovirt-vm:device devtype="disk"
name="vda"><ovirt-vm:poolID>00000000-0000-0000-0000-000000000000</ovirt-vm:poolID><ovirt-vm:volumeID>a48
555f4-be23-4467-8a54-400ae7baf9d7</ovirt-vm:volumeID><ovirt-vm:shared>exclusive</ovirt-vm:shared><ovirt-vm:imageID>d5de54b6-9f8e-4fba-819b-ebf6780757
d2</ovirt-vm:imageID><ovirt-vm:domainID>80f6e393-9718-4738-a14a-64cf43c3d8c2</ovirt-vm:domainID></ovirt-vm:device><ovirt-vm:launchPaused>false</ovirt
-vm:launchPaused><ovirt-vm:resumeBehavior>auto_resume</ovirt-vm:resumeBehavior></ovirt-vm:vm></metadata></domain>'})
from=::1,36826, vmId=baa3d4d1-69
18-4447-80fa-64b86dbcb1f9 (api:48)
[root@ov-no2 ~]# xzcat /var/log/vdsm/supervdsm.log.30.xz |egrep
"(WARN|error|ERR)" |more
MainProcess|jsonrpc/4::ERROR::2020-09-01
06:56:30,652::supervdsm_server::103::SuperVdsm.ServerCallback::(wrapper) Error in
volumeGeoRepStatus
MainProcess|jsonrpc/3::WARNING::2020-09-01
07:32:27,206::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/3::WARNING::2020-09-01
07:32:27,213::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/3::WARNING::2020-09-01
07:32:27,217::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/3::WARNING::2020-09-01
07:32:27,565::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/3::WARNING::2020-09-01
07:32:27,571::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/3::WARNING::2020-09-01
07:32:27,573::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,735::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,735::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,736::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,736::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,737::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,738::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,740::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,740::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,741::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/3::ERROR::2020-09-01 07:32:31,889::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/3::ERROR::2020-09-01 07:32:31,893::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/3::ERROR::2020-09-01 07:32:31,894::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/3::ERROR::2020-09-01 07:32:31,896::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/7::ERROR::2020-09-01
07:58:30,879::supervdsm_server::103::SuperVdsm.ServerCallback::(wrapper) Error in
volumeGeoRepStatus
[root@ov-no2 ~]# xzcat /var/log/vdsm/supervdsm.log.29.xz |egrep
"(WARN|error|ERR)" |more
MainProcess|jsonrpc/1::WARNING::2020-09-01
09:32:32,179::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-01
09:32:32,180::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-01
09:32:32,181::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-01
09:32:32,390::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-01
09:32:32,393::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-01
09:32:32,395::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,510::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,510::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,511::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,511::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,512::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,512::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,513::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,513::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,514::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,440::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,444::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,446::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,448::udev::53::blivet::(get_device) No
device at ''
[root@ov-no2 ~]# cat /var/log/vdsm/supervdsm.log |egrep "(WARN|error|ERR)"
|more
MainProcess|jsonrpc/1::WARNING::2020-09-03
19:34:55,056::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-03
19:34:55,057::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-03
19:34:55,059::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-03
19:34:55,177::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-03
19:34:55,179::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-03
19:34:55,180::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,292::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,292::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,293::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,294::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,294::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,295::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,295::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,296::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,296::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,217::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,220::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,222::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,223::udev::53::blivet::(get_device) No
device at ''
[root@ov-no2 ~]# cat /var/log/ovirt-hosted-engine-ha/agent.log.2020-09-01 |egrep -i
"(WARN|ERR)" |more
MainThread::ERROR::2020-09-01
09:48:02,766::hosted_engine::452::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Unhandled
monitoring loop exception
error: [Errno 2] No such file or directory
MainThread::ERROR::2020-09-01
09:48:02,786::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback
(most recent call last):
error: [Errno 2] No such file or directory
MainThread::ERROR::2020-09-01
09:48:02,786::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to
restart agent
MainThread::INFO::2020-09-01
09:49:16,925::hosted_engine::854::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm)
stderr: Co
mmand VM.getStats with args {'vmID':
'baa3d4d1-6918-4447-80fa-64b86dbcb1f9'} failed:
MainThread::INFO::2020-09-01
09:49:27,616::hosted_engine::931::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm)
stderr: Com
mand VM.destroy with args {'vmID': 'baa3d4d1-6918-4447-80fa-64b86dbcb1f9'}
failed:
MainThread::ERROR::2020-09-01
09:49:27,616::hosted_engine::936::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm)
Failed to
stop engine vm with /usr/sbin/hosted-engine --vm-poweroff: Command VM.destroy with args
{'vmID': 'baa3d4d1-6918-4447-80fa-64b86dbcb1f9'} failed:
MainThread::ERROR::2020-09-01
09:49:27,616::hosted_engine::942::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm)
Failed to
stop engine VM: Command VM.destroy with args {'vmID':
'baa3d4d1-6918-4447-80fa-64b86dbcb1f9'} failed:
[root@ov-no2 ~]# cat /var/log/ovirt-hosted-engine-ha/broker.log.2020-09-01 |egrep -i
"(WARN|ERR)" |more
StatusStorageThread::ERROR::2020-09-01
09:47:52,786::status_broker::90::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(run)
Failed
to update state.
raise ex.HostIdNotLockedError("Host id is not set")
HostIdNotLockedError: Host id is not set
StatusStorageThread::ERROR::2020-09-01
09:47:52,787::status_broker::70::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(trigger_res
tart) Trying to restart the broker
Listener::ERROR::2020-09-01
09:48:15,543::notifications::39::ovirt_hosted_engine_ha.broker.notifications.Notifications::(send_email)
[Errno 113] No r
oute to host
raise err
error: [Errno 113] No route to host
Listener::ERROR::2020-09-01
09:48:25,768::notifications::39::ovirt_hosted_engine_ha.broker.notifications.Notifications::(send_email)
[Errno 113] No r
oute to host
raise err
error: [Errno 113] No route to host
Listener::ERROR::2020-09-01
09:49:15,838::notifications::39::ovirt_hosted_engine_ha.broker.notifications.Notifications::(send_email)
[Errno 113] No r
oute to host
raise err
The Gluster status and info I could gather for the present are the following:
[root@ov-no2 ~]# gluster volume status
Status of volume: data
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ov-no1.ariadne-t.local:/gluster_brick
s/data/data 49152 0 Y 6070
Brick ov-no2.ariadne-t.local:/gluster_brick
s/data/data 49152 0 Y 28860
Brick ov-no3.ariadne-t.local:/gluster_brick
s/data/data 49152 0 Y 5752
Self-heal Daemon on localhost N/A N/A Y 813
Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539
Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353
Task Status of Volume data
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: engine
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ov-no1.ariadne-t.local:/gluster_brick
s/engine/engine 49153 0 Y 6098
Brick ov-no2.ariadne-t.local:/gluster_brick
s/engine/engine 49156 0 Y 28867
Brick ov-no3.ariadne-t.local:/gluster_brick
s/engine/engine 49153 0 Y 5789
Self-heal Daemon on localhost N/A N/A Y 813
Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539
Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353
Task Status of Volume engine
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: vmstore
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ov-no1.ariadne-t.local:/gluster_brick
s/vmstore/vmstore 49154 0 Y 6102
Brick ov-no2.ariadne-t.local:/gluster_brick
s/vmstore/vmstore 49157 0 Y 28878
Brick ov-no3.ariadne-t.local:/gluster_brick
s/vmstore/vmstore 49154 0 Y 5772
Self-heal Daemon on localhost N/A N/A Y 813
Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539
Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353
Task Status of Volume vmstore
------------------------------------------------------------------------------
There are no active volume tasks
[root@ov-no2 ~]# gluster volume heal engine info
Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Number of entries: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Number of entries: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Number of entries: 0
[root@ov-no2 ~]# gluster volume heal engine info summary
Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
[root@ov-no2 ~]# gluster volume heal data info
Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Number of entries: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Number of entries: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Number of entries: 0
[root@ov-no2 ~]# gluster volume heal data info summary
Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
[root@ov-no2 ~]# gluster volume heal vmstore info
Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Number of entries: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Number of entries: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Number of entries: 0
[root@ov-no2 ~]# gluster volume heal vmstore info summary
Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
[root@ov-no2 ~]# ps -A |grep gluster
813 ? 02:18:00 glusterfs
14183 ? 1-12:02:26 glusterfs
17944 ? 01:19:03 glustereventsd
17995 ? 00:31:50 glustereventsd
28722 ? 00:02:09 glusterd
28860 ? 02:19:48 glusterfsd
28867 ? 05:23:00 glusterfsd
28878 ? 00:20:15 glusterfsd
32187 ? 1-14:42:54 glusterfs
32309 ? 09:59:54 glusterfs
Node3:
[root@ov-no3 ~]# virsh qemu-agent-command HostedEngine
'{"execute":"guest-info"}'
Please enter your authentication name: root
Please enter your password:
{"return":{"version":"2.12.0","supported_commands":[{"enabled":true,"name":"guest-get-osinfo","success-response":true},{"enabled":true,"name":"guest-get-timezone","success-response":true},{"enabled":true,"name":"guest-get-users","success-response":true},{"enabled":true,"name":"guest-get-host-name","success-response":true},{"enabled":false,"name":"guest-exec","success-response":true},{"enabled":false,"name":"guest-exec-status","success-response":true},{"enabled":true,"name":"guest-get-memory-block-info","success-response":true},{"enabled":true,"name":"guest-set-memory-blocks","success-response":true},{"enabled":true,"name":"guest-get-memory-blocks","success-response":true},{"enabled":true,"name":"guest-set-user-password","success-response":true},{"enabled":true,"name":"guest-get-fsinfo","success-response":true},{"enabled":true,"name":"guest-set-vcpus","success-response":true},{"enabled":true,"name":"guest-get-vcpus","success-response":true},{"enabled":true,"name":"guest-network-get-in
terfaces","success-response":true},{"enabled":true,"name":"guest-suspend-hybrid","success-response":false},{"enabled":true,"name":"guest-suspend-ram","success-response":false},{"enabled":true,"name":"guest-suspend-disk","success-response":false},{"enabled":true,"name":"guest-fstrim","success-response":true},{"enabled":true,"name":"guest-fsfreeze-thaw","success-response":true},{"enabled":true,"name":"guest-fsfreeze-freeze-list","success-response":true},{"enabled":true,"name":"guest-fsfreeze-freeze","success-response":true},{"enabled":true,"name":"guest-fsfreeze-status","success-response":true},{"enabled":false,"name":"guest-file-flush","success-response":true},{"enabled":false,"name":"guest-file-seek","success-response":true},{"enabled":false,"name":"guest-file-write","success-response":true},{"enabled":false,"name":"guest-file-read","success-response":true},{"enabled":false,"name":"guest-file-close","success-response":true},{"enabled":false,"name":"guest-file-open","success-response
":true},{"enabled":true,"name":"guest-shutdown","success-response":false},{"enabled":true,"name":"guest-info","success-response":true},{"enabled":true,"name":"guest-set-time","success-response":true},{"enabled":true,"name":"guest-get-time","success-response":true},{"enabled":true,"name":"guest-ping","success-response":true},{"enabled":true,"name":"guest-sync","success-response":true},{"enabled":true,"name":"guest-sync-delimited","success-response":true}]}}
[root@ov-no3 ~]# cat /var/log/messages|egrep -i "(WARN|ERR)" |more
Sep 1 08:55:17 ov-no3 libvirtd: 2020-09-01 05:55:17.816+0000: 5284: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 08:59:30 ov-no3 vdsm[6112]: WARN Worker blocked: <Worker name=jsonrpc/3 running
<Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0',
'method':
u'GlusterVolume.geoRepSessionList', 'id':
u'8298ea63-eadc-47b8-b8b4-d54982200010'} at 0x7f3c14739d90> timeout=60,
duration=60.00 at 0x7f3c14739dd0> t
ask#=72655 at 0x7f3c34306550>, traceback:#012File:
"/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012
self.__bootstrap_inner()#012Fil
e: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012
self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run
#012 self.__target(*self.__args, **self.__kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in
run#012 ret =
func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012
self._execute_task()#012File: "/usr/lib/p
ython2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012
task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 3
91, in __call__#012 self._callable()#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in
__call__#012 self._handler(sel
f._ctx, self._req)#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in
_serveRequest#012 response = self._handle_request
(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
line 345, in _handle_request#012 res = method(**params)#012File: "/usr
/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012
result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-package
s/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012
remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py",
l
ine 93, in wrapper#012 rv = func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in
volumeGeoRepSessionL
ist#012 remoteUserName,#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in
__call__#012 return callMethod()#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in
<lambda>#012 **kwargs)#012File: "<string>", line 2, in
glusterVolumeGeoRep
Status#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in
_callmethod#012 kind, result = conn.recv()
Sep 1 09:00:17 ov-no3 libvirtd: 2020-09-01 06:00:17.817+0000: 5281: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 09:00:30 ov-no3 vdsm[6112]: WARN Worker blocked: <Worker name=jsonrpc/3 running
<Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0',
'method':
u'GlusterVolume.geoRepSessionList', 'id':
u'8298ea63-eadc-47b8-b8b4-d54982200010'} at 0x7f3c14739d90> timeout=60,
duration=120.00 at 0x7f3c14739dd0>
task#=72655 at 0x7f3c34306550>, traceback:#012File:
"/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012
self.__bootstrap_inner()#012Fi
le: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012
self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in ru
n#012 self.__target(*self.__args, **self.__kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in
run#012 ret
= func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012
self._execute_task()#012File: "/usr/lib/
python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012
task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line
391, in __call__#012 self._callable()#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in
__call__#012 self._handler(se
lf._ctx, self._req)#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in
_serveRequest#012 response = self._handle_reques
t(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
line 345, in _handle_request#012 res = method(**params)#012File: "/us
r/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012
result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-packag
es/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012
remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py",
line 93, in wrapper#012 rv = func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in
volumeGeoRepSession
List#012 remoteUserName,#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in
__call__#012 return callMethod()#012File
: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in
<lambda>#012 **kwargs)#012File: "<string>", line 2, in
glusterVolumeGeoRe
pStatus#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759,
in _callmethod#012 kind, result = conn.recv()
Sep 1 09:05:17 ov-no3 libvirtd: 2020-09-01 06:05:17.816+0000: 5283: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 09:10:17 ov-no3 libvirtd: 2020-09-01 06:10:17.823+0000: 5282: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 09:15:17 ov-no3 libvirtd: 2020-09-01 06:15:17.817+0000: 5281: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 09:20:17 ov-no3 libvirtd: 2020-09-01 06:20:17.820+0000: 5282: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 09:25:17 ov-no3 libvirtd: 2020-09-01 06:25:17.818+0000: 5280: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 09:30:17 ov-no3 libvirtd: 2020-09-01 06:30:17.820+0000: 5282: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 09:35:17 ov-no3 libvirtd: 2020-09-01 06:35:17.815+0000: 5281: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 09:40:17 ov-no3 libvirtd: 2020-09-01 06:40:17.823+0000: 5284: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 09:45:17 ov-no3 libvirtd: 2020-09-01 06:45:17.819+0000: 5280: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 09:50:17 ov-no3 libvirtd: 2020-09-01 06:50:17.824+0000: 5280: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 09:55:17 ov-no3 libvirtd: 2020-09-01 06:55:17.818+0000: 5284: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 10:00:17 ov-no3 libvirtd: 2020-09-01 07:00:17.822+0000: 5284: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 10:05:17 ov-no3 libvirtd: 2020-09-01 07:05:17.824+0000: 5282: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
.
.
.
.
[which continues as so 'till now]
[root@ov-no3 ~]# xzcat /var/log/vdsm/vdsm.log.57.xz |egrep "(WARN|error)" |more
2020-09-01 09:31:35,023+0300 WARN (monitor/0a5e552) [storage.PersistentDict] Could not
parse line ``. (persistent:241)
2020-09-01 09:31:35,026+0300 WARN (monitor/e2e7282) [storage.PersistentDict] Could not
parse line ``. (persistent:241)
2020-09-01 09:35:19,868+0300 WARN (monitor/80f6e39) [storage.PersistentDict] Could not
parse line ``. (persistent:241)
2020-09-01 09:35:22,210+0300 WARN (qgapoller/1) [virt.periodic.VmDispatcher] could not
run <function <lambda> at 0x7f3c680470c8> on
['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289)
2020-09-01 09:35:32,207+0300 WARN (qgapoller/3) [virt.periodic.VmDispatcher] could not
run <function <lambda> at 0x7f3c680470c8> on
['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289)
2020-09-01 09:35:42,213+0300 WARN (qgapoller/1) [virt.periodic.VmDispatcher] could not
run <function <lambda> at 0x7f3c680470c8> on
['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289)
2020-09-01 09:35:52,212+0300 WARN (qgapoller/2) [virt.periodic.VmDispatcher] could not
run <function <lambda> at 0x7f3c680470c8> on
['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289)
2020-09-01 09:36:02,207+0300 WARN (qgapoller/1) [virt.periodic.VmDispatcher] could not
run <function <lambda> at 0x7f3c680470c8> on
['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289)
2020-09-01 09:36:12,215+0300 WARN (qgapoller/2) [virt.periodic.VmDispatcher] could not
run <function <lambda> at 0x7f3c680470c8> on
['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289)
2020-09-01 09:36:35,276+0300 WARN (monitor/0a5e552) [storage.PersistentDict] Could not
parse line ``. (persistent:241)
2020-09-01 09:36:35,281+0300 WARN (monitor/e2e7282) [storage.PersistentDict] Could not
parse line ``. (persistent:241)
2020-09-01 09:40:18,494+0300 WARN (qgapoller/3) [virt.periodic.VmDispatcher] could not
run <function <lambda> at 0x7f3c6803ff50> on
['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289)
2020-09-01 09:40:18,534+0300 WARN (qgapoller/1) [virt.periodic.VmDispatcher] could not
run <function <lambda> at 0x7f3c6803fe60> on
['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289)
2020-09-01 09:40:20,244+0300 WARN (monitor/80f6e39) [storage.PersistentDict] Could not
parse line ``. (persistent:241)
2020-09-01 09:40:22,209+0300 WARN (qgapoller/3) [virt.periodic.VmDispatcher] could not
run <function <lambda> at 0x7f3c680470c8> on
['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289)
2020-09-01 09:40:32,207+0300 WARN (qgapoller/1) [virt.periodic.VmDispatcher] could not
run <function <lambda> at 0x7f3c680470c8> on
['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289)
2020-09-01 09:40:42,214+0300 WARN (qgapoller/3) [virt.periodic.VmDispatcher] could not
run <function <lambda> at 0x7f3c680470c8> on
['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289)
2020-09-01 09:40:52,215+0300 WARN (qgapoller/2) [virt.periodic.VmDispatcher] could not
run <function <lambda> at 0x7f3c680470c8> on
['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289)
[root@ov-no3 ~]# xzcat /var/log/vdsm/supervdsm.log.29.xz|egrep "(WARN|ERR)"
|more
MainProcess|jsonrpc/4::WARNING::2020-09-01
09:32:32,165::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/4::WARNING::2020-09-01
09:32:32,166::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/4::WARNING::2020-09-01
09:32:32,168::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/4::WARNING::2020-09-01
09:32:32,285::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/4::WARNING::2020-09-01
09:32:32,287::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/4::WARNING::2020-09-01
09:32:32,288::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,396::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,397::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,397::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,398::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,398::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,399::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,399::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,400::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,400::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/4::ERROR::2020-09-01 09:32:36,337::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/4::ERROR::2020-09-01 09:32:36,341::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/4::ERROR::2020-09-01 09:32:36,342::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/4::ERROR::2020-09-01 09:32:36,344::udev::53::blivet::(get_device) No
device at ''
[root@ov-no3 ~]# cat /var/log/vdsm/supervdsm.log|egrep "(WARN|ERR)" |more
MainProcess|jsonrpc/1::WARNING::2020-09-03
19:34:55,041::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-03
19:34:55,043::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-03
19:34:55,044::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-03
19:34:55,158::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-03
19:34:55,160::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-03
19:34:55,161::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,265::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,266::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,266::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,267::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,267::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,268::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,269::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,269::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,270::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,399::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,402::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,404::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,406::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/7::ERROR::2020-09-03
19:54:43,983::supervdsm_server::103::SuperVdsm.ServerCallback::(wrapper) Error in
volumeGeoRepStatus
[root@ov-no3 ~]# cat /var/log/ovirt-hosted-engine-ha/agent.log.2020-09-01 |egrep -i
"(WARN|ERR)" |more
[nothing in the output]
[root@ov-no3 ~]# cat /var/log/ovirt-hosted-engine-ha/broker.log.2020-09-01 |egrep -i
"(WARN|ERR)" |more
Thread-1::WARNING::2020-09-02 12:46:59,657::network::92::network.Network::(action) Failed
to verify network status, (4 out of 5)
[root@ov-no3 ~]# gluster volume status
Status of volume: data
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ov-no1.ariadne-t.local:/gluster_brick
s/data/data 49152 0 Y 6070
Brick ov-no2.ariadne-t.local:/gluster_brick
s/data/data 49152 0 Y 28860
Brick ov-no3.ariadne-t.local:/gluster_brick
s/data/data 49152 0 Y 5752
Self-heal Daemon on localhost N/A N/A Y 19353
Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539
Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813
Task Status of Volume data
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: engine
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ov-no1.ariadne-t.local:/gluster_brick
s/engine/engine 49153 0 Y 6098
Brick ov-no2.ariadne-t.local:/gluster_brick
s/engine/engine 49156 0 Y 28867
Brick ov-no3.ariadne-t.local:/gluster_brick
s/engine/engine 49153 0 Y 5789
Self-heal Daemon on localhost N/A N/A Y 19353
Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539
Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813
Task Status of Volume engine
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: vmstore
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ov-no1.ariadne-t.local:/gluster_brick
s/vmstore/vmstore 49154 0 Y 6102
Brick ov-no2.ariadne-t.local:/gluster_brick
s/vmstore/vmstore 49157 0 Y 28878
Brick ov-no3.ariadne-t.local:/gluster_brick
s/vmstore/vmstore 49154 0 Y 5772
Self-heal Daemon on localhost N/A N/A Y 19353
Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539
Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813
Task Status of Volume vmstore
------------------------------------------------------------------------------
There are no active volume tasks
[root@ov-no3 ~]# gluster volume heal engine info
Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Number of entries: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Number of entries: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Number of entries: 0
[root@ov-no3 ~]# gluster volume heal engine info summary
Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
[root@ov-no3 ~]# gluster volume heal data info
Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Number of entries: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Number of entries: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Number of entries: 0
[root@ov-no3 ~]# gluster volume heal data info summary
Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
[root@ov-no3 ~]# gluster volume heal vmstore info
Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Number of entries: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Number of entries: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Number of entries: 0
[root@ov-no3 ~]# gluster volume heal vmstore info summary
Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
[root@ov-no3 ~]# ps -A|grep gluster
5143 ? 00:01:31 glustereventsd
5752 ? 02:01:08 glusterfsd
5772 ? 00:17:26 glusterfsd
5789 ? 05:12:24 glusterfsd
5906 ? 00:00:38 glustereventsd
6382 ? 02:37:00 glusterfs
9843 ? 00:02:18 glusterd
10613 ? 00:22:08 glusterfs
10750 ? 00:13:24 glusterfs
19353 ? 01:20:36 glusterfs
Hosted Engine:
[root@ov-eng ~]# grep ERROR /var/log/ovirt-engine/engine.log |more
2020-09-02 03:36:35,199+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob]
(DefaultQuartzScheduler8) [1a87b58
d] VDS error Command execution failed: rc=1 out='Error : Request timed
out\ngeo-replication command failed\n' err=''
2020-09-02 04:38:35,429+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob]
(DefaultQuartzScheduler2) [15fadfa
2] VDS error Command execution failed: rc=1 out='Error : Request timed
out\ngeo-replication command failed\n' err=''
2020-09-02 05:40:35,655+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob]
(DefaultQuartzScheduler9) [50fa85f
0] VDS error Command execution failed: rc=1 out='Error : Request timed
out\ngeo-replication command failed\n' err=''
2020-09-02 06:42:35,885+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob]
(DefaultQuartzScheduler7) [3092fdc
c] VDS error Command execution failed: rc=1 out='Error : Request timed
out\ngeo-replication command failed\n' err=''
2020-09-02 07:44:36,119+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob]
(DefaultQuartzScheduler6) [696ffca
7] VDS error Command execution failed: rc=1 out='Error : Request timed
out\ngeo-replication command failed\n' err=''
2020-09-02 08:46:36,341+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob]
(DefaultQuartzScheduler9) [180bb50
] VDS error Command execution failed: rc=1 out='Error : Request timed
out\ngeo-replication command failed\n' err=''
2020-09-02 09:48:36,562+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob]
(DefaultQuartzScheduler6) [3a45e1b
3] VDS error Command execution failed: rc=1 out='Error : Request timed
out\ngeo-replication command failed\n' err=''
2020-09-02 10:50:36,788+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob]
(DefaultQuartzScheduler4) [7da1197
3] VDS error Command execution failed: rc=1 out='Error : Request timed
out\ngeo-replication command failed\n' err=''
2020-09-02 11:52:37,011+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob]
(DefaultQuartzScheduler6) [76b3cb6
5] VDS error Command execution failed: rc=1 out='Error : Request timed
out\ngeo-replication command failed\n' err=''
2020-09-02 12:54:37,234+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob]
(DefaultQuartzScheduler5) [6a0d904
9] VDS error Command execution failed: rc=1 out='Error : Request timed
out\ngeo-replication command failed\n' err=''
2020-09-02 13:56:37,459+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob]
(DefaultQuartzScheduler4) [1e25ecf
f] VDS error Command execution failed: rc=1 out='Error : Request timed
out\ngeo-replication command failed\n' err=''
2020-09-02 14:58:37,674+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob]
(DefaultQuartzScheduler1) [64144e3
4] VDS error Command execution failed: rc=1 out='Error : Request timed
out\ngeo-replication command failed\n' err=''
How can I find why the HE failover happened? I believe it's the gluster geo
replication, but I haven't configured geo replication in the enviroment. Can you help
me how to proceed?
Thank you all in advance,
Maria Souvalioti
9:26 p.m.
What about the gluster logs ?
Best Regards,
Strahil Nikolov
В четвъртък, 3 септември 2020 г., 20:57:05 Гринуич+3, souvaliotimaria(a)mail.com
<souvaliotimaria(a)mail.com> написа:
Thank you very much for your reply.
I checked the NTP and realized the service wasn't working properly on two of the three
nodes, but despite that the clocks seemed to have the correct time(date and hwdate). I
switched to chronyd and stopped the ntpd service and now it seems the servers' clocks
are synchronized.
The time in BIOS has different time than the systems. Does this affect the behaviour of
the overall performance?
This is what I could gather from the logs:
Node1:
[root@ov-no1 ~]# more /var/log/messages |egrep "(WARN|error)"|more
d
Aug 27 17:53:08 ov-no1 libvirtd: 2020-08-27 14:53:08.947+0000: 5613: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Aug 27 17:58:08 ov-no1 libvirtd: 2020-08-27 14:58:08.943+0000: 5613: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Aug 27 18:03:08 ov-no1 libvirtd: 2020-08-27 15:03:08.937+0000: 5614: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Aug 27 18:08:08 ov-no1 libvirtd: 2020-08-27 15:08:08.951+0000: 5617: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Aug 27 18:13:08 ov-no1 libvirtd: 2020-08-27 15:13:08.951+0000: 5616: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Aug 27 18:18:08 ov-no1 libvirtd: 2020-08-27 15:18:08.942+0000: 5618: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
.
.
[Around that time Node 3, who is the arbiter node, was placed in local Maintenance mode.
It was shut down for maintenance and when we boot it up again all seemed right. We removed
it from the maintenance mode and when the healing processes finished, Node 1 became
NonResponsive. Long story short, the VDSM agent sent Node1 a restart command. Node1
rebooted, HostedEngine was up on Node1 and the rest of the VMs that were hosted by Node1
had to be manually brought up. Since then everything seemed to be working as it should.
The HostedEngine VM shutdown with no apparent reason 5 days later, making us believe there
was no connection between the two incidents.]
.
.
.
.
.
Sep 1 05:53:30 ov-no1 vdsm[6706]: WARN Worker blocked: <Worker name=jsonrpc/3 running
<Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0',
'method':
u'GlusterVolume.geoRepSessionList', 'id':
u'e1f4db88-5d36-4ba1-8848-3855c1dbc6ee'} at 0x7f1ed7381650> timeout=60,
duration=60.00 at 0x7f1ed7381dd0> t
ask#=76268 at 0x7f1ebc0797d0>, traceback:#012File:
"/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012
self.__bootstrap_inner()#012Fil
e: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012
self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run
#012 self.__target(*self.__args, **self.__kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in
run#012 ret =
func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012
self._execute_task()#012File: "/usr/lib/p
ython2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012
task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 3
91, in __call__#012 self._callable()#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in
__call__#012 self._handler(sel
f._ctx, self._req)#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in
_serveRequest#012 response = self._handle_request
(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
line 345, in _handle_request#012 res = method(**params)#012File: "/usr
/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012
result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-package
s/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012
remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py",
l
ine 93, in wrapper#012 rv = func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in
volumeGeoRepSessionL
ist#012 remoteUserName,#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in
__call__#012 return callMethod()#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in
<lambda>#012 **kwargs)#012File: "<string>", line 2, in
glusterVolumeGeoRep
Status#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in
_callmethod#012 kind, result = conn.recv()
Sep 1 05:54:30 ov-no1 vdsm[6706]: WARN Worker blocked: <Worker name=jsonrpc/3 running
<Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0',
'method':
u'GlusterVolume.geoRepSessionList', 'id':
u'e1f4db88-5d36-4ba1-8848-3855c1dbc6ee'} at 0x7f1ed7381650> timeout=60,
duration=120.00 at 0x7f1ed7381dd0>
task#=76268 at 0x7f1ebc0797d0>, traceback:#012File:
"/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012
self.__bootstrap_inner()#012Fi
le: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012
self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in ru
n#012 self.__target(*self.__args, **self.__kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in
run#012 ret
= func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012
self._execute_task()#012File: "/usr/lib/
python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012
task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line
391, in __call__#012 self._callable()#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in
__call__#012 self._handler(se
lf._ctx, self._req)#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in
_serveRequest#012 response = self._handle_reques
t(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
line 345, in _handle_request#012 res = method(**params)#012File: "/us
r/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012
result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-packag
es/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012
remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py",
line 93, in wrapper#012 rv = func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in
volumeGeoRepSession
List#012 remoteUserName,#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in
__call__#012 return callMethod()#012File
: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in
<lambda>#012 **kwargs)#012File: "<string>", line 2, in
glusterVolumeGeoRe
pStatus#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759,
in _callmethod#012 kind, result = conn.recv()
[root@ov-no1 ~]# xzcat /var/log/vdsm/vdsm.log.57.xz |egrep "(WARN|error)" |more
2020-09-01 09:40:50,023+0300 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:40:50,023+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:41:00,061+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:41:00,061+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:41:10,078+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:41:10,078+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:41:20,086+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:41:20,086+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:41:30,107+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:41:30,107+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:41:38,710+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:41:38,711+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:41:40,115+0300 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:41:40,115+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.01 seconds (__init__:312)
2020-09-01 09:41:44,541+0300 WARN (monitor/e2e7282) [storage.PersistentDict] Could not
parse line `
2020-09-01 09:41:44,544+0300 WARN (monitor/0a5e552) [storage.PersistentDict] Could not
parse line `
[root@ov-no1 ~]# xzcat /var/log/vdsm/supervdsm.log.29.xz |egrep
"(WARN|error|ERR)" |more
MainProcess|jsonrpc/1::WARNING::2020-09-01
09:32:32,182::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-01
09:32:32,183::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-01
09:32:32,185::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-01
09:32:32,304::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-01
09:32:32,306::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-01
09:32:32,308::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,440::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,441::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,442::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,442::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,443::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,443::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,444::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,445::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,445::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,358::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,361::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,363::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,365::udev::53::blivet::(get_device) No
device at ''
[root@ov-no1 ~]# xzcat /var/log/vdsm/supervdsm.log.28.xz |egrep
"(WARN|error|ERR)" |more
MainProcess|jsonrpc/7::WARNING::2020-09-01
11:32:36,730::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/7::WARNING::2020-09-01
11:32:36,731::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/7::WARNING::2020-09-01
11:32:36,732::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/7::WARNING::2020-09-01
11:32:36,847::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/7::WARNING::2020-09-01
11:32:36,849::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/7::WARNING::2020-09-01
11:32:36,850::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,974::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,975::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,975::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,976::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,977::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,977::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,978::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,979::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/7::WARNING::2020-09-01 11:32:36,979::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/7::ERROR::2020-09-01 11:32:40,951::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/7::ERROR::2020-09-01 11:32:40,954::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/7::ERROR::2020-09-01 11:32:40,956::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/7::ERROR::2020-09-01 11:32:40,958::udev::53::blivet::(get_device) No
device at ''
[root@ov-no1 ~]# cat /var/log/ovirt-hosted-engine-ha/broker.log.2020-09-01 |egrep -i
"(WARN|ERR)" |more
[nothing in the output]
[root@ov-no1 ~]# gluster volume status
Status of volume: data
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ov-no1.ariadne-t.local:/gluster_brick
s/data/data 49152 0 Y 6070
Brick ov-no2.ariadne-t.local:/gluster_brick
s/data/data 49152 0 Y 28860
Brick ov-no3.ariadne-t.local:/gluster_brick
s/data/data 49152 0 Y 5752
Self-heal Daemon on localhost N/A N/A Y 6539
Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813
Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353
Task Status of Volume data
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: engine
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ov-no1.ariadne-t.local:/gluster_brick
s/engine/engine 49153 0 Y 6098
Brick ov-no2.ariadne-t.local:/gluster_brick
s/engine/engine 49156 0 Y 28867
Brick ov-no3.ariadne-t.local:/gluster_brick
s/engine/engine 49153 0 Y 5789
Self-heal Daemon on localhost N/A N/A Y 6539
Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813
Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353
Task Status of Volume engine
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: vmstore
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ov-no1.ariadne-t.local:/gluster_brick
s/vmstore/vmstore 49154 0 Y 6102
Brick ov-no2.ariadne-t.local:/gluster_brick
s/vmstore/vmstore 49157 0 Y 28878
Brick ov-no3.ariadne-t.local:/gluster_brick
s/vmstore/vmstore 49154 0 Y 5772
Self-heal Daemon on localhost N/A N/A Y 6539
Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813
Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353
Task Status of Volume vmstore
------------------------------------------------------------------------------
There are no active volume tasks
[root@ov-no1 ~]# gluster volume heal engine info
Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Number of entries: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Number of entries: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Number of entries: 0
[root@ov-no1 ~]# gluster volume heal engine info summary
Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
[root@ov-no1 ~]# gluster volume heal data info
Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Number of entries: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Number of entries: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Number of entries: 0
[root@ov-no1 ~]# gluster volume heal data info summary
Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
[root@ov-no1 ~]# gluster volume heal vmstore info
Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Number of entries: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Number of entries: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Number of entries: 0
[root@ov-no1 ~]# gluster volume heal vmstore info summary
Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
[root@ov-no1 ~]# ps -A|grep gluster
5522 ? 00:01:40 glustereventsd
5591 ? 00:05:04 glusterd
6070 ? 02:16:28 glusterfsd
6098 ? 05:44:14 glusterfsd
6102 ? 00:23:48 glusterfsd
6285 ? 00:00:38 glustereventsd
6539 ? 01:32:19 glusterfs
10470 ? 00:34:10 glusterfs
17763 ? 00:50:18 glusterfs
18101 ? 00:10:23 glusterfs
Node2:
[root@ov-no2 ~]# more /var/log/messages |egrep "(WARN|error)"|more
Sep 1 07:58:30 ov-no2 vdsm[13586]: WARN Worker blocked: <Worker name=jsonrpc/7 running
<Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0',
'method':
u'GlusterVolume.geoRepSessionList', 'id':
u'ec8af6cc-6109-4ddb-a6fb-ad9f4fbc1e7b'} at 0x7f44602a7c90> timeout=60,
duration=120.00 at 0x7f44602a7510>
task#=5773350 at 0x7f44c00c71d0>, traceback:#012File:
"/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012
self.__bootstrap_inner()#01
2File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012
self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in
run#012 self.__target(*self.__args, **self.__kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in
run#012 r
et = func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012
self._execute_task()#012File: "/usr/l
ib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012
task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", li
ne 391, in __call__#012 self._callable()#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in
__call__#012 self._handler
(self._ctx, self._req)#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in
_serveRequest#012 response = self._handle_req
uest(req, ctx)#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in
_handle_request#012 res = method(**params)#012File: "
/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in
_dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-pac
kages/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012
remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py
", line 93, in wrapper#012 rv = func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in
volumeGeoRepSess
ionList#012 remoteUserName,#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in
__call__#012 return callMethod()#012F
ile: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in
<lambda>#012 **kwargs)#012File: "<string>", line 2, in
glusterVolumeGe
oRepStatus#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line
759, in _callmethod#012 kind, result = conn.recv()
Sep 1 09:48:02 ov-no2 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Unhandled monitoring loop
exception#012T
raceback (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 449, in start_moni
toring#012 self._monitoring_loop()#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 468, in _monit
oring_loop#012 for old_state, state, delay in self.fsm:#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py",
li
ne 127, in next#012 new_data = self.refresh(self._state.data)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/state_machi
ne.py", line 81, in refresh#012
stats.update(self.hosted_engine.collect_stats())#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_h
a/agent/hosted_engine.py", line 756, in collect_stats#012 all_stats =
self._broker.get_stats_from_storage()#012 File "/usr/lib/python2.7/site-pac
kages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 143, in
get_stats_from_storage#012 result = self._proxy.get_stats()#012 File
"/usr/lib64/py
thon2.7/xmlrpclib.py", line 1233, in __call__#012 return self.__send(self.__name,
args)#012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1591,
in __request#012 verbose=self.__verbose#012 File
"/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request#012 return
self.single_request(hos
t, handler, request_body, verbose)#012 File
"/usr/lib64/python2.7/xmlrpclib.py", line 1301, in single_request#012
self.send_content(h, request_bo
dy)#012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in
send_content#012 connection.endheaders(request_body)#012 File "/usr/lib64/python
2.7/httplib.py", line 1037, in endheaders#012 self._send_output(message_body)#012
File "/usr/lib64/python2.7/httplib.py", line 881, in _send_outp
ut#012 self.send(msg)#012 File "/usr/lib64/python2.7/httplib.py", line 843,
in send#012 self.connect()#012 File "/usr/lib/python2.7/site-pack
ages/ovirt_hosted_engine_ha/lib/unixrpc.py", line 52, in connect#012
self.sock.connect(base64.b16decode(self.host))#012 File "/usr/lib64/python2.
7/socket.py", line 224, in meth#012 return
getattr(self._sock,name)(*args)#012error: [Errno 2] No such file or directory
Sep 1 09:48:02 ov-no2 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent
ERROR Traceback (most recent call last):#012 File "/usr/lib/
python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in
_run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-p
ackages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012
return he.start_monitoring()#012 File "/usr/lib/python2.7/site-pack
ages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 456, in
start_monitoring#012 self.publish(stopped)#012 File "/usr/lib/python2.7/site-pa
ckages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 356, in publish#012
self._push_to_storage(blocks)#012 File "/usr/lib/python2.7/site-p
ackages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 727, in
_push_to_storage#012 self._broker.put_stats_on_storage(self.host_id, blocks)#
012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 113, in put_stats_on_storage#012 self._proxy.put_stats
(host_id, xmlrpclib.Binary(data))#012 File "/usr/lib64/python2.7/xmlrpclib.py",
line 1233, in __call__#012 return self.__send(self.__name, args)#
012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in __request#012
verbose=self.__verbose#012 File "/usr/lib64/python2.7/xmlrpclib.py", l
ine 1273, in request#012 return self.single_request(host, handler, request_body,
verbose)#012 File "/usr/lib64/python2.7/xmlrpclib.py", line 1301
, in single_request#012 self.send_content(h, request_body)#012 File
"/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content#012 connec
tion.endheaders(request_body)#012 File "/usr/lib64/python2.7/httplib.py", line
1037, in endheaders#012 self._send_output(message_body)#012 File
"/usr/lib64/python2.7/httplib.py", line 881, in _send_output#012
self.send(msg)#012 File "/usr/lib64/python2.7/httplib.py", line 843, in
send#012
self.connect()#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py", line
52, in connect#012 self.sock.connect(b
ase64.b16decode(self.host))#012 File "/usr/lib64/python2.7/socket.py", line
224, in meth#012 return getattr(self._sock,name)(*args)#012error: [Er
rno 2] No such file or directory
Sep 1 09:48:15 ov-no2 journal: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to
host#012Trace
back (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py",
line 26, in send_email#012
timeout=float(cfg["smtp-timeout"]))#012 File
"/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) =
self.connect(host, po
rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012
self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l
ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return
socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so
cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No
route to host
Sep 1 09:48:25 ov-no2 journal: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to
host#012Trace
back (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py",
line 26, in send_email#012
timeout=float(cfg["smtp-timeout"]))#012 File
"/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) =
self.connect(host, po
rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012
self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l
ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return
socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so
cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No
route to host
Sep 1 09:49:15 ov-no2 journal: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to
host#012Trace
back (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py",
line 26, in send_email#012
timeout=float(cfg["smtp-timeout"]))#012 File
"/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) =
self.connect(host, po
rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012
self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l
ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return
socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so
cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No
route to host
Sep 1 09:49:16 ov-no2 journal: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to
host#012Trace
back (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py",
line 26, in send_email#012
timeout=float(cfg["smtp-timeout"]))#012 File
"/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) =
self.connect(host, po
rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012
self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l
ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return
socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so
cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No
route to host
Sep 1 09:49:17 ov-no2 vdsm[13586]: WARN Attempting to add an existing net user:
ovirtmgmt/baa3d4d1-6918-4447-80fa-64b86dbcb1f9
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables
--concurrent -t nat -D PREROUTING -i vnet0 -j libvirt-J-vnet0' f
ailed: Illegal target name 'libvirt-J-vnet0'.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables
--concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-P-vnet0'
failed: Illegal target name 'libvirt-P-vnet0'.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables
--concurrent -t nat -L libvirt-J-vnet0' failed: Chain 'libvirt-J
-vnet0' doesn't exist.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables
--concurrent -t nat -L libvirt-P-vnet0' failed: Chain 'libvirt-P
-vnet0' doesn't exist.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables
--concurrent -t nat -F libvirt-J-vnet0' failed: Chain 'libvirt-J
-vnet0' doesn't exist.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables
--concurrent -t nat -X libvirt-J-vnet0' failed: Chain 'libvirt-J
-vnet0' doesn't exist.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables
--concurrent -t nat -F libvirt-P-vnet0' failed: Chain 'libvirt-P
-vnet0' doesn't exist.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables
--concurrent -t nat -X libvirt-P-vnet0' failed: Chain 'libvirt-P
-vnet0' doesn't exist.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables
--concurrent -t nat -F J-vnet0-mac' failed: Chain 'J-vnet0-mac'
doesn't exist.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables
--concurrent -t nat -X J-vnet0-mac' failed: Chain 'J-vnet0-mac'
doesn't exist.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables
--concurrent -t nat -F J-vnet0-arp-mac' failed: Chain 'J-vnet0-a
rp-mac' doesn't exist.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables
--concurrent -t nat -X J-vnet0-arp-mac' failed: Chain 'J-vnet0-a
rp-mac' doesn't exist.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables
-w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-
out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a
chain#012#012Try `iptables -h' or 'iptables --help' for more information.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables
-w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0
' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try
`iptables -h' or 'iptables --help' for more information.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables
-w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0'
failed: iptables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `iptables
-h' or 'iptables --help' for more information.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables
-w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vn
et0' failed: iptables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try
`iptables -h' or 'iptables --help' for more information.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables
-w2 -w -F FO-vnet0' failed: iptables: No chain/target/match by t
hat name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables
-w2 -w -X FO-vnet0' failed: iptables: No chain/target/match by t
hat name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables
-w2 -w -F FI-vnet0' failed: iptables: No chain/target/match by t
hat name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables
-w2 -w -X FI-vnet0' failed: iptables: No chain/target/match by t
hat name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables
-w2 -w -F HI-vnet0' failed: iptables: No chain/target/match by t
hat name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables
-w2 -w -X HI-vnet0' failed: iptables: No chain/target/match by t
hat name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables
-w2 -w -E FP-vnet0 FO-vnet0' failed: iptables: No chain/target/m
atch by that name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables
-w2 -w -E FJ-vnet0 FI-vnet0' failed: iptables: No chain/target/m
atch by that name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables
-w2 -w -E HJ-vnet0 HI-vnet0' failed: iptables: No chain/target/m
atch by that name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables
-w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev
-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a
chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more informatio
n.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables
-w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet
0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try
`ip6tables -h' or 'ip6tables --help' for more information.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables
-w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0'
failed: ip6tables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `ip6tables
-h' or 'ip6tables --help' for more information.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables
-w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-v
net0' failed: ip6tables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try
`ip6tables -h' or 'ip6tables --help' for more information.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables
-w2 -w -F FO-vnet0' failed: ip6tables: No chain/target/match by
that name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables
-w2 -w -X FO-vnet0' failed: ip6tables: No chain/target/match by
that name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables
-w2 -w -F FI-vnet0' failed: ip6tables: No chain/target/match by
that name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables
-w2 -w -X FI-vnet0' failed: ip6tables: No chain/target/match by
that name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables
-w2 -w -F HI-vnet0' failed: ip6tables: No chain/target/match by
that name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables
-w2 -w -X HI-vnet0' failed: ip6tables: No chain/target/match by
that name.
Sep 1 09:49:19 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables
-w2 -w -E FP-vnet0 FO-vnet0' failed: ip6tables: No chain/target
/match by that name.
Sep 1 09:49:20 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables
-w2 -w -E FJ-vnet0 FI-vnet0' failed: ip6tables: No chain/target
/match by that name.
Sep 1 09:49:20 ov-no2 firewalld[2428]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables
-w2 -w -E HJ-vnet0 HI-vnet0' failed: ip6tables: No chain/target
/match by that name.
.
.
.
.
.
Sep 1 09:49:22 ov-no2 vdsm[13586]: WARN File:
/var/lib/libvirt/qemu/channels/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.ovirt-guest-agent.0
already remove
d
Sep 1 09:49:22 ov-no2 vdsm[13586]: WARN Attempting to remove a non existing net user:
ovirtmgmt/baa3d4d1-6918-4447-80fa-64b86dbcb1f9
Sep 1 09:49:22 ov-no2 vdsm[13586]: WARN File:
/var/lib/libvirt/qemu/channels/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.org.qemu.guest_agent.0
already rem
oved
Sep 1 09:49:22 ov-no2 vdsm[13586]: WARN File:
/var/run/ovirt-vmconsole-console/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.sock already
removed
Sep 1 09:49:27 ov-no2 journal: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to
host#012Trace
back (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py",
line 26, in send_email#012
timeout=float(cfg["smtp-timeout"]))#012 File
"/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) =
self.connect(host, po
rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012
self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l
ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return
socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so
cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No
route to host
Sep 1 09:49:27 ov-no2 journal: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to
host#012Trace
back (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py",
line 26, in send_email#012
timeout=float(cfg["smtp-timeout"]))#012 File
"/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) =
self.connect(host, po
rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012
self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l
ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return
socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so
cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No
route to host
Sep 1 09:49:37 ov-no2 journal: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.notifications.Notifications ERROR [Errno 113] No route to
host#012Trace
back (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/notifications.py",
line 26, in send_email#012
timeout=float(cfg["smtp-timeout"]))#012 File
"/usr/lib64/python2.7/smtplib.py", line 255, in __init__#012 (code, msg) =
self.connect(host, po
rt)#012 File "/usr/lib64/python2.7/smtplib.py", line 315, in connect#012
self.sock = self._get_socket(host, port, self.timeout)#012 File "/usr/l
ib64/python2.7/smtplib.py", line 290, in _get_socket#012 return
socket.create_connection((host, port), timeout)#012 File "/usr/lib64/python2.7/so
cket.py", line 571, in create_connection#012 raise err#012error: [Errno 113] No
route to host
Sep 1 10:01:31 ov-no2 vdsm[13586]: WARN Worker blocked: <Worker name=jsonrpc/4 running
<Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0',
'method':
u'GlusterVolume.geoRepSessionList', 'id':
u'7f800636-f107-4b98-a35f-2175ec72b7b8'} at 0x7f44805aa110> timeout=60,
duration=60.00 at 0x7f44805aad10>
task#=5775347 at 0x7f44d81c4110>, traceback:#012File:
"/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012
self.__bootstrap_inner()#012
File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012
self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in
run#012 self.__target(*self.__args, **self.__kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in
run#012 re
t = func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012
self._execute_task()#012File: "/usr/li
b/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012
task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", lin
e 391, in __call__#012 self._callable()#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in
__call__#012 self._handler(
self._ctx, self._req)#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in
_serveRequest#012 response = self._handle_requ
est(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
line 345, in _handle_request#012 res = method(**params)#012File: "/
usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012
result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-pack
ages/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012
remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py"
, line 93, in wrapper#012 rv = func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in
volumeGeoRepSessi
onList#012 remoteUserName,#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in
__call__#012 return callMethod()#012Fi
le: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in
<lambda>#012 **kwargs)#012File: "<string>", line 2, in
glusterVolumeGeo
RepStatus#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759,
in _callmethod#012 kind, result = conn.recv()
[root@ov-no2 ~]# xzcat /var/log/vdsm/vdsm.log.57.xz |egrep "(WARN|error)" |more
2020-09-01 09:24:06,889+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:24:16,910+0300 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:24:16,911+0300 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:24:26,932+0300 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:24:26,933+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.01 seconds (__init__:312)
2020-09-01 09:24:30,375+0300 WARN (monitor/0a5e552) [storage.PersistentDict] Could not
parse line `
2020-09-01 09:24:30,379+0300 WARN (monitor/e2e7282) [storage.PersistentDict] Could not
parse line `
.
.
.
.
2020-09-01 09:47:37,008+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:47:47,030+0300 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:47:47,030+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:48:01,712+0300 WARN (jsonrpc/6) [storage.PersistentDict] Could not parse
line `
2020-09-01 09:48:01,743+0300 WARN (jsonrpc/1) [storage.PersistentDict] Could not parse
line `
2020-09-01 09:48:02,220+0300 WARN (jsonrpc/0) [storage.PersistentDict] Could not parse
line `
2020-09-01 09:48:05,192+0300 WARN (periodic/22) [storage.PersistentDict] Could not parse
line `
2020-09-01 09:48:11,745+0300 WARN (monitor/e2e7282) [storage.PersistentDict] Could not
parse line `
2020-09-01 09:48:13,711+0300 INFO (jsonrpc/7) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:48:13,711+0300 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:48:23,591+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:48:23,591+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:48:23,731+0300 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:48:23,731+0300 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:48:33,754+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:48:33,754+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:48:43,787+0300 INFO (jsonrpc/2) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:48:43,787+0300 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:48:53,808+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:48:53,809+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:49:03,830+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:49:03,831+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:49:13,852+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:49:13,853+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.01 seconds (__init__:312)
2020-09-01 09:49:16,042+0300 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:49:16,042+0300 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.01 seconds (__init__:312)
2020-09-01 09:49:16,405+0300 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': u'baa3d4d1-6918-4447-80fa-64b86db
cb1f9'} (api:129)
2020-09-01 09:49:16,405+0300 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2020-09-01 09:49:16,797+0300 INFO (jsonrpc/7) [api.virt] START
create(vmParams={u'xml': u'<?xml version=\'1.0\'
encoding=\'UTF-8\'?>\n<domain xmlns:
ovirt-tune="http://ovirt.org/vm/tune/1.0"
xmlns:ovirt-vm="http://ovirt.org/vm/1.0"
type="kvm"><name>HostedEngine</name><uuid>baa3d4d1-6918-4447-80fa-
64b86dbcb1f9</uuid><memory>16777216</memory><currentMemory>16777216</currentMemory><iothreads>1</iothreads><maxMemory
slots="16">67108864</maxMemory>
<vcpu current="4">64</vcpu><sysinfo
type="smbios"><system><entry
name="manufacturer">oVirt</entry><entry
name="product">OS-NAME:</entry><entry name="
version">OS-VERSION:</entry><entry
name="serial">HOST-SERIAL:</entry><entry
name="uuid">baa3d4d1-6918-4447-80fa-64b86dbcb1f9</entry></system></sysinf
o><clock offset="variable" adjustment="0"><timer
name="rtc" tickpolicy="catchup"/><timer name="pit"
tickpolicy="delay"/><timer name="hpet" present="n
o"/></clock><features><acpi/></features><cpu
match="exact"><model>Westmere</model><feature
name="pcid" policy="require"/><feature
name="spec-ctrl" po
licy="require"/><feature name="ssbd"
policy="require"/><topology cores="4" threads="1"
sockets="16"/><numa><cell id="0" cpus="0,1,2,3"
memory="167772
16"/></numa></cpu><cputune/><devices><input
type="tablet" bus="usb"/><channel
type="unix"><target type="virtio"
name="ovirt-guest-agent.0"/><source m
ode="bind"
path="/var/lib/libvirt/qemu/channels/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.ovirt-guest-agent.0"/></channel><channel
type="unix"><target typ
e="virtio" name="org.qemu.guest_agent.0"/><source
mode="bind"
path="/var/lib/libvirt/qemu/channels/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.org.qemu.gues
t_agent.0"/></channel><controller type="usb"
model="piix3-uhci" index="0"><address bus="0x00"
domain="0x0000" function="0x2" slot="0x01"
type="pci"/>
</controller><console type="unix"><source
path="/var/run/ovirt-vmconsole-console/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.sock"
mode="bind"/><target type
="serial" port="0"/><alias
name="ua-228f9c56-c195-4f1e-8cd2-5f7ef60a78f1"/></console><memballoon
model="virtio"><stats period="5"/><alias name="ua-42
10968c-acf1-46cc-8a35-6e29e3c58b38"/><address bus="0x00"
domain="0x0000" function="0x0" slot="0x08"
type="pci"/></memballoon><controller type="scsi"
model="virtio-scsi" index="0"><driver
iothread="1"/><alias
name="ua-473cb0f7-09da-4077-a0e2-8eb5a5efdbe4"/><address
bus="0x00" domain="0x0000" functi
on="0x0" slot="0x05"
type="pci"/></controller><sound model="ich6"><alias
name="ua-496f9a72-5484-4dc0-83f4-caa8225100ae"/><address
bus="0x00" domain="
0x0000" function="0x0" slot="0x04"
type="pci"/></sound><graphics type="vnc"
port="-1" autoport="yes" passwd="*****"
passwdValidTo="1970-01-01T00:00:0
1" keymap="en-us"><listen type="network"
network="vdsm-ovirtmgmt"/></graphics><controller
type="ide" index="0"><address bus="0x00"
domain="0x0000" fu
nction="0x1" slot="0x01"
type="pci"/></controller><graphics type="spice"
port="-1" autoport="yes" passwd="*****"
passwdValidTo="1970-01-01T00:00:01"
tlsPort="-1"><channel name="main"
mode="secure"/><channel name="inputs"
mode="secure"/><channel name="cursor"
mode="secure"/><channel name="playback"
mode="secure"/><channel name="record"
mode="secure"/><channel name="display"
mode="secure"/><channel name="smartcard"
mode="secure"/><channel name="
usbredir" mode="secure"/><listen type="network"
network="vdsm-ovirtmgmt"/></graphics><rng
model="virtio"><backend
model="random">/dev/urandom</backen
d><alias
name="ua-9edfa822-3b11-424e-83f8-4846c6221a6e"/></rng><video><model
type="qxl" vram="32768" heads="1" ram="65536"
vgamem="16384"/><alias nam
e="ua-a4ad75db-c91d-45df-a767-33a71009587f"/><address bus="0x00"
domain="0x0000" function="0x0" slot="0x02"
type="pci"/></video><controller type="vir
tio-serial" index="0" ports="16"><alias
name="ua-f60f1742-07fe-464e-a160-1a16c94db61a"/><address
bus="0x00" domain="0x0000" function="0x0" slot="0x06
" type="pci"/></controller><serial
type="unix"><source
path="/var/run/ovirt-vmconsole-console/baa3d4d1-6918-4447-80fa-64b86dbcb1f9.sock"
mode="bind"/
<target port="0"/></serial><channel
type="spicevmc"><target type="virtio"
name="com.redhat.spice.0"/></channel><interface
type="bridge"><model type=
"virtio"/><link
state="up"/><source bridge="ovirtmgmt"/><driver
queues="4" name="vhost"/><alias
name="ua-e97c94c0-ffc2-46dd-b5cd-9d9c123f4db0"/><addr
ess bus="0x00" domain="0x0000" function="0x0"
slot="0x03" type="pci"/><mac
address="00:16:3e:27:6e:ef"/><mtu size="1500"/><filterref
filter="vdsm-no-
mac-spoofing"/><bandwidth/></interface><disk type="file"
device="cdrom" snapshot="no"><driver name="qemu"
type="raw" error_policy="report"/><source f
ile="" startupPolicy="optional"><seclabel model="dac"
type="none" relabel="no"/></source><target
dev="hdc" bus="ide"/><readonly/><alias name="ua-9034
9138-d20e-4306-b0e2-271e6baaad4a"/><address bus="1"
controller="0" unit="0" type="drive"
target="0"/></disk><disk snapshot="no"
type="file" device="d
isk"><target dev="vda" bus="virtio"/><source
file="/rhev/data-center/00000000-0000-0000-0000-000000000000/80f6e393-9718-4738-a14a-64cf43c3d8c2/images
/d5de54b6-9f8e-4fba-819b-ebf6780757d2/a48555f4-be23-4467-8a54-400ae7baf9d7"><seclabel
model="dac" type="none"
relabel="no"/></source><driver name="qe
mu" iothread="1" io="threads" type="raw"
error_policy="stop" cache="none"/><alias
name="ua-d5de54b6-9f8e-4fba-819b-ebf6780757d2"/><address
bus="0x00"
domain="0x0000" function="0x0" slot="0x07"
type="pci"/><serial>d5de54b6-9f8e-4fba-819b-ebf6780757d2</serial></disk><lease><key>a48555f4-be23-4467-8a
54-400ae7baf9d7</key><lockspace>80f6e393-9718-4738-a14a-64cf43c3d8c2</lockspace><target
offset="LEASE-OFFSET:a48555f4-be23-4467-8a54-400ae7baf9d7:80f
6e393-9718-4738-a14a-64cf43c3d8c2"
path="LEASE-PATH:a48555f4-be23-4467-8a54-400ae7baf9d7:80f6e393-9718-4738-a14a-64cf43c3d8c2"/></lease></devices><pm
<suspend-to-disk enabled="no"/><suspend-to-mem
enabled="no"/></pm><os><type arch="x86_64"
machine="pc-i440fx-rhel7.6.0">hvm</type><smbios mode="sysi
nfo"/><bios
useserial="yes"/></os><metadata><ovirt-tune:qos/><ovirt-vm:vm><ovirt-vm:minGuaranteedMemoryMb
type="int">1024</ovirt-vm:minGuaranteedMemo
ryMb><ovirt-vm:clusterVersion>4.3</ovirt-vm:clusterVersion><ovirt-vm:custom/><ovirt-vm:device
mac_address="00:16:3e:27:6e:ef"><ovirt-vm:custom/></ovi
rt-vm:device><ovirt-vm:device devtype="disk"
name="vda"><ovirt-vm:poolID>00000000-0000-0000-0000-000000000000</ovirt-vm:poolID><ovirt-vm:volumeID>a48
555f4-be23-4467-8a54-400ae7baf9d7</ovirt-vm:volumeID><ovirt-vm:shared>exclusive</ovirt-vm:shared><ovirt-vm:imageID>d5de54b6-9f8e-4fba-819b-ebf6780757
d2</ovirt-vm:imageID><ovirt-vm:domainID>80f6e393-9718-4738-a14a-64cf43c3d8c2</ovirt-vm:domainID></ovirt-vm:device><ovirt-vm:launchPaused>false</ovirt
-vm:launchPaused><ovirt-vm:resumeBehavior>auto_resume</ovirt-vm:resumeBehavior></ovirt-vm:vm></metadata></domain>'})
from=::1,36826, vmId=baa3d4d1-69
18-4447-80fa-64b86dbcb1f9 (api:48)
[root@ov-no2 ~]# xzcat /var/log/vdsm/supervdsm.log.30.xz |egrep
"(WARN|error|ERR)" |more
MainProcess|jsonrpc/4::ERROR::2020-09-01
06:56:30,652::supervdsm_server::103::SuperVdsm.ServerCallback::(wrapper) Error in
volumeGeoRepStatus
MainProcess|jsonrpc/3::WARNING::2020-09-01
07:32:27,206::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/3::WARNING::2020-09-01
07:32:27,213::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/3::WARNING::2020-09-01
07:32:27,217::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/3::WARNING::2020-09-01
07:32:27,565::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/3::WARNING::2020-09-01
07:32:27,571::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/3::WARNING::2020-09-01
07:32:27,573::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,735::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,735::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,736::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,736::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,737::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,738::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,740::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,740::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/3::WARNING::2020-09-01 07:32:27,741::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/3::ERROR::2020-09-01 07:32:31,889::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/3::ERROR::2020-09-01 07:32:31,893::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/3::ERROR::2020-09-01 07:32:31,894::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/3::ERROR::2020-09-01 07:32:31,896::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/7::ERROR::2020-09-01
07:58:30,879::supervdsm_server::103::SuperVdsm.ServerCallback::(wrapper) Error in
volumeGeoRepStatus
[root@ov-no2 ~]# xzcat /var/log/vdsm/supervdsm.log.29.xz |egrep
"(WARN|error|ERR)" |more
MainProcess|jsonrpc/1::WARNING::2020-09-01
09:32:32,179::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-01
09:32:32,180::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-01
09:32:32,181::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-01
09:32:32,390::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-01
09:32:32,393::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-01
09:32:32,395::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,510::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,510::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,511::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,511::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,512::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,512::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,513::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,513::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/1::WARNING::2020-09-01 09:32:32,514::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,440::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,444::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,446::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/1::ERROR::2020-09-01 09:32:36,448::udev::53::blivet::(get_device) No
device at ''
[root@ov-no2 ~]# cat /var/log/vdsm/supervdsm.log |egrep "(WARN|error|ERR)"
|more
MainProcess|jsonrpc/1::WARNING::2020-09-03
19:34:55,056::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-03
19:34:55,057::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-03
19:34:55,059::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-03
19:34:55,177::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-03
19:34:55,179::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-03
19:34:55,180::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,292::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,292::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,293::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,294::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,294::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,295::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,295::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,296::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,296::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,217::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,220::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,222::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,223::udev::53::blivet::(get_device) No
device at ''
[root@ov-no2 ~]# cat /var/log/ovirt-hosted-engine-ha/agent.log.2020-09-01 |egrep -i
"(WARN|ERR)" |more
MainThread::ERROR::2020-09-01
09:48:02,766::hosted_engine::452::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Unhandled
monitoring loop exception
error: [Errno 2] No such file or directory
MainThread::ERROR::2020-09-01
09:48:02,786::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback
(most recent call last):
error: [Errno 2] No such file or directory
MainThread::ERROR::2020-09-01
09:48:02,786::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to
restart agent
MainThread::INFO::2020-09-01
09:49:16,925::hosted_engine::854::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm)
stderr: Co
mmand VM.getStats with args {'vmID':
'baa3d4d1-6918-4447-80fa-64b86dbcb1f9'} failed:
MainThread::INFO::2020-09-01
09:49:27,616::hosted_engine::931::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm)
stderr: Com
mand VM.destroy with args {'vmID': 'baa3d4d1-6918-4447-80fa-64b86dbcb1f9'}
failed:
MainThread::ERROR::2020-09-01
09:49:27,616::hosted_engine::936::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm)
Failed to
stop engine vm with /usr/sbin/hosted-engine --vm-poweroff: Command VM.destroy with args
{'vmID': 'baa3d4d1-6918-4447-80fa-64b86dbcb1f9'} failed:
MainThread::ERROR::2020-09-01
09:49:27,616::hosted_engine::942::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm)
Failed to
stop engine VM: Command VM.destroy with args {'vmID':
'baa3d4d1-6918-4447-80fa-64b86dbcb1f9'} failed:
[root@ov-no2 ~]# cat /var/log/ovirt-hosted-engine-ha/broker.log.2020-09-01 |egrep -i
"(WARN|ERR)" |more
StatusStorageThread::ERROR::2020-09-01
09:47:52,786::status_broker::90::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(run)
Failed
to update state.
raise ex.HostIdNotLockedError("Host id is not set")
HostIdNotLockedError: Host id is not set
StatusStorageThread::ERROR::2020-09-01
09:47:52,787::status_broker::70::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(trigger_res
tart) Trying to restart the broker
Listener::ERROR::2020-09-01
09:48:15,543::notifications::39::ovirt_hosted_engine_ha.broker.notifications.Notifications::(send_email)
[Errno 113] No r
oute to host
raise err
error: [Errno 113] No route to host
Listener::ERROR::2020-09-01
09:48:25,768::notifications::39::ovirt_hosted_engine_ha.broker.notifications.Notifications::(send_email)
[Errno 113] No r
oute to host
raise err
error: [Errno 113] No route to host
Listener::ERROR::2020-09-01
09:49:15,838::notifications::39::ovirt_hosted_engine_ha.broker.notifications.Notifications::(send_email)
[Errno 113] No r
oute to host
raise err
The Gluster status and info I could gather for the present are the following:
[root@ov-no2 ~]# gluster volume status
Status of volume: data
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ov-no1.ariadne-t.local:/gluster_brick
s/data/data 49152 0 Y 6070
Brick ov-no2.ariadne-t.local:/gluster_brick
s/data/data 49152 0 Y 28860
Brick ov-no3.ariadne-t.local:/gluster_brick
s/data/data 49152 0 Y 5752
Self-heal Daemon on localhost N/A N/A Y 813
Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539
Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353
Task Status of Volume data
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: engine
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ov-no1.ariadne-t.local:/gluster_brick
s/engine/engine 49153 0 Y 6098
Brick ov-no2.ariadne-t.local:/gluster_brick
s/engine/engine 49156 0 Y 28867
Brick ov-no3.ariadne-t.local:/gluster_brick
s/engine/engine 49153 0 Y 5789
Self-heal Daemon on localhost N/A N/A Y 813
Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539
Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353
Task Status of Volume engine
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: vmstore
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ov-no1.ariadne-t.local:/gluster_brick
s/vmstore/vmstore 49154 0 Y 6102
Brick ov-no2.ariadne-t.local:/gluster_brick
s/vmstore/vmstore 49157 0 Y 28878
Brick ov-no3.ariadne-t.local:/gluster_brick
s/vmstore/vmstore 49154 0 Y 5772
Self-heal Daemon on localhost N/A N/A Y 813
Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539
Self-heal Daemon on ov-no3.ariadne-t.local N/A N/A Y 19353
Task Status of Volume vmstore
------------------------------------------------------------------------------
There are no active volume tasks
[root@ov-no2 ~]# gluster volume heal engine info
Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Number of entries: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Number of entries: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Number of entries: 0
[root@ov-no2 ~]# gluster volume heal engine info summary
Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
[root@ov-no2 ~]# gluster volume heal data info
Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Number of entries: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Number of entries: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Number of entries: 0
[root@ov-no2 ~]# gluster volume heal data info summary
Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
[root@ov-no2 ~]# gluster volume heal vmstore info
Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Number of entries: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Number of entries: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Number of entries: 0
[root@ov-no2 ~]# gluster volume heal vmstore info summary
Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
[root@ov-no2 ~]# ps -A |grep gluster
813 ? 02:18:00 glusterfs
14183 ? 1-12:02:26 glusterfs
17944 ? 01:19:03 glustereventsd
17995 ? 00:31:50 glustereventsd
28722 ? 00:02:09 glusterd
28860 ? 02:19:48 glusterfsd
28867 ? 05:23:00 glusterfsd
28878 ? 00:20:15 glusterfsd
32187 ? 1-14:42:54 glusterfs
32309 ? 09:59:54 glusterfs
Node3:
[root@ov-no3 ~]# virsh qemu-agent-command HostedEngine
'{"execute":"guest-info"}'
Please enter your authentication name: root
Please enter your password:
{"return":{"version":"2.12.0","supported_commands":[{"enabled":true,"name":"guest-get-osinfo","success-response":true},{"enabled":true,"name":"guest-get-timezone","success-response":true},{"enabled":true,"name":"guest-get-users","success-response":true},{"enabled":true,"name":"guest-get-host-name","success-response":true},{"enabled":false,"name":"guest-exec","success-response":true},{"enabled":false,"name":"guest-exec-status","success-response":true},{"enabled":true,"name":"guest-get-memory-block-info","success-response":true},{"enabled":true,"name":"guest-set-memory-blocks","success-response":true},{"enabled":true,"name":"guest-get-memory-blocks","success-response":true},{"enabled":true,"name":"guest-set-user-password","success-response":true},{"enabled":true,"name":"guest-get-fsinfo","success-response":true},{"enabled":true,"name":"guest-set-vcpus","success-response":true},{"enabled":true,"name":"guest-get-vcpus","success-response":true},{"enabled":true,"name":"guest-network-get-in
terfaces","success-response":true},{"enabled":true,"name":"guest-suspend-hybrid","success-response":false},{"enabled":true,"name":"guest-suspend-ram","success-response":false},{"enabled":true,"name":"guest-suspend-disk","success-response":false},{"enabled":true,"name":"guest-fstrim","success-response":true},{"enabled":true,"name":"guest-fsfreeze-thaw","success-response":true},{"enabled":true,"name":"guest-fsfreeze-freeze-list","success-response":true},{"enabled":true,"name":"guest-fsfreeze-freeze","success-response":true},{"enabled":true,"name":"guest-fsfreeze-status","success-response":true},{"enabled":false,"name":"guest-file-flush","success-response":true},{"enabled":false,"name":"guest-file-seek","success-response":true},{"enabled":false,"name":"guest-file-write","success-response":true},{"enabled":false,"name":"guest-file-read","success-response":true},{"enabled":false,"name":"guest-file-close","success-response":true},{"enabled":false,"name":"guest-file-open","success-response
":true},{"enabled":true,"name":"guest-shutdown","success-response":false},{"enabled":true,"name":"guest-info","success-response":true},{"enabled":true,"name":"guest-set-time","success-response":true},{"enabled":true,"name":"guest-get-time","success-response":true},{"enabled":true,"name":"guest-ping","success-response":true},{"enabled":true,"name":"guest-sync","success-response":true},{"enabled":true,"name":"guest-sync-delimited","success-response":true}]}}
[root@ov-no3 ~]# cat /var/log/messages|egrep -i "(WARN|ERR)" |more
Sep 1 08:55:17 ov-no3 libvirtd: 2020-09-01 05:55:17.816+0000: 5284: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 08:59:30 ov-no3 vdsm[6112]: WARN Worker blocked: <Worker name=jsonrpc/3 running
<Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0',
'method':
u'GlusterVolume.geoRepSessionList', 'id':
u'8298ea63-eadc-47b8-b8b4-d54982200010'} at 0x7f3c14739d90> timeout=60,
duration=60.00 at 0x7f3c14739dd0> t
ask#=72655 at 0x7f3c34306550>, traceback:#012File:
"/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012
self.__bootstrap_inner()#012Fil
e: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012
self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run
#012 self.__target(*self.__args, **self.__kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in
run#012 ret =
func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012
self._execute_task()#012File: "/usr/lib/p
ython2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012
task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 3
91, in __call__#012 self._callable()#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in
__call__#012 self._handler(sel
f._ctx, self._req)#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in
_serveRequest#012 response = self._handle_request
(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
line 345, in _handle_request#012 res = method(**params)#012File: "/usr
/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012
result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-package
s/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012
remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py",
l
ine 93, in wrapper#012 rv = func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in
volumeGeoRepSessionL
ist#012 remoteUserName,#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in
__call__#012 return callMethod()#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in
<lambda>#012 **kwargs)#012File: "<string>", line 2, in
glusterVolumeGeoRep
Status#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in
_callmethod#012 kind, result = conn.recv()
Sep 1 09:00:17 ov-no3 libvirtd: 2020-09-01 06:00:17.817+0000: 5281: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 09:00:30 ov-no3 vdsm[6112]: WARN Worker blocked: <Worker name=jsonrpc/3 running
<Task <JsonRpcTask {'params': {}, 'jsonrpc': '2.0',
'method':
u'GlusterVolume.geoRepSessionList', 'id':
u'8298ea63-eadc-47b8-b8b4-d54982200010'} at 0x7f3c14739d90> timeout=60,
duration=120.00 at 0x7f3c14739dd0>
task#=72655 at 0x7f3c34306550>, traceback:#012File:
"/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012
self.__bootstrap_inner()#012Fi
le: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012
self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in ru
n#012 self.__target(*self.__args, **self.__kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in
run#012 ret
= func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012
self._execute_task()#012File: "/usr/lib/
python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012
task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line
391, in __call__#012 self._callable()#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in
__call__#012 self._handler(se
lf._ctx, self._req)#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in
_serveRequest#012 response = self._handle_reques
t(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
line 345, in _handle_request#012 res = method(**params)#012File: "/us
r/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod#012
result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-packag
es/vdsm/gluster/apiwrapper.py", line 237, in geoRepSessionList#012
remoteUserName)#012File: "/usr/lib/python2.7/site-packages/vdsm/gluster/api.py",
line 93, in wrapper#012 rv = func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/gluster/api.py", line 551, in
volumeGeoRepSession
List#012 remoteUserName,#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in
__call__#012 return callMethod()#012File
: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in
<lambda>#012 **kwargs)#012File: "<string>", line 2, in
glusterVolumeGeoRe
pStatus#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759,
in _callmethod#012 kind, result = conn.recv()
Sep 1 09:05:17 ov-no3 libvirtd: 2020-09-01 06:05:17.816+0000: 5283: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 09:10:17 ov-no3 libvirtd: 2020-09-01 06:10:17.823+0000: 5282: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 09:15:17 ov-no3 libvirtd: 2020-09-01 06:15:17.817+0000: 5281: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 09:20:17 ov-no3 libvirtd: 2020-09-01 06:20:17.820+0000: 5282: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 09:25:17 ov-no3 libvirtd: 2020-09-01 06:25:17.818+0000: 5280: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 09:30:17 ov-no3 libvirtd: 2020-09-01 06:30:17.820+0000: 5282: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 09:35:17 ov-no3 libvirtd: 2020-09-01 06:35:17.815+0000: 5281: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 09:40:17 ov-no3 libvirtd: 2020-09-01 06:40:17.823+0000: 5284: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 09:45:17 ov-no3 libvirtd: 2020-09-01 06:45:17.819+0000: 5280: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 09:50:17 ov-no3 libvirtd: 2020-09-01 06:50:17.824+0000: 5280: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 09:55:17 ov-no3 libvirtd: 2020-09-01 06:55:17.818+0000: 5284: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 10:00:17 ov-no3 libvirtd: 2020-09-01 07:00:17.822+0000: 5284: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
Sep 1 10:05:17 ov-no3 libvirtd: 2020-09-01 07:05:17.824+0000: 5282: error :
qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU gues
t agent is not connected
.
.
.
.
[which continues as so 'till now]
[root@ov-no3 ~]# xzcat /var/log/vdsm/vdsm.log.57.xz |egrep "(WARN|error)" |more
2020-09-01 09:31:35,023+0300 WARN (monitor/0a5e552) [storage.PersistentDict] Could not
parse line ``. (persistent:241)
2020-09-01 09:31:35,026+0300 WARN (monitor/e2e7282) [storage.PersistentDict] Could not
parse line ``. (persistent:241)
2020-09-01 09:35:19,868+0300 WARN (monitor/80f6e39) [storage.PersistentDict] Could not
parse line ``. (persistent:241)
2020-09-01 09:35:22,210+0300 WARN (qgapoller/1) [virt.periodic.VmDispatcher] could not
run <function <lambda> at 0x7f3c680470c8> on
['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289)
2020-09-01 09:35:32,207+0300 WARN (qgapoller/3) [virt.periodic.VmDispatcher] could not
run <function <lambda> at 0x7f3c680470c8> on
['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289)
2020-09-01 09:35:42,213+0300 WARN (qgapoller/1) [virt.periodic.VmDispatcher] could not
run <function <lambda> at 0x7f3c680470c8> on
['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289)
2020-09-01 09:35:52,212+0300 WARN (qgapoller/2) [virt.periodic.VmDispatcher] could not
run <function <lambda> at 0x7f3c680470c8> on
['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289)
2020-09-01 09:36:02,207+0300 WARN (qgapoller/1) [virt.periodic.VmDispatcher] could not
run <function <lambda> at 0x7f3c680470c8> on
['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289)
2020-09-01 09:36:12,215+0300 WARN (qgapoller/2) [virt.periodic.VmDispatcher] could not
run <function <lambda> at 0x7f3c680470c8> on
['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289)
2020-09-01 09:36:35,276+0300 WARN (monitor/0a5e552) [storage.PersistentDict] Could not
parse line ``. (persistent:241)
2020-09-01 09:36:35,281+0300 WARN (monitor/e2e7282) [storage.PersistentDict] Could not
parse line ``. (persistent:241)
2020-09-01 09:40:18,494+0300 WARN (qgapoller/3) [virt.periodic.VmDispatcher] could not
run <function <lambda> at 0x7f3c6803ff50> on
['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289)
2020-09-01 09:40:18,534+0300 WARN (qgapoller/1) [virt.periodic.VmDispatcher] could not
run <function <lambda> at 0x7f3c6803fe60> on
['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289)
2020-09-01 09:40:20,244+0300 WARN (monitor/80f6e39) [storage.PersistentDict] Could not
parse line ``. (persistent:241)
2020-09-01 09:40:22,209+0300 WARN (qgapoller/3) [virt.periodic.VmDispatcher] could not
run <function <lambda> at 0x7f3c680470c8> on
['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289)
2020-09-01 09:40:32,207+0300 WARN (qgapoller/1) [virt.periodic.VmDispatcher] could not
run <function <lambda> at 0x7f3c680470c8> on
['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289)
2020-09-01 09:40:42,214+0300 WARN (qgapoller/3) [virt.periodic.VmDispatcher] could not
run <function <lambda> at 0x7f3c680470c8> on
['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289)
2020-09-01 09:40:52,215+0300 WARN (qgapoller/2) [virt.periodic.VmDispatcher] could not
run <function <lambda> at 0x7f3c680470c8> on
['5836fc52-b585-46f8-8780-2cc5accd5847'] (periodic:289)
[root@ov-no3 ~]# xzcat /var/log/vdsm/supervdsm.log.29.xz|egrep "(WARN|ERR)"
|more
MainProcess|jsonrpc/4::WARNING::2020-09-01
09:32:32,165::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/4::WARNING::2020-09-01
09:32:32,166::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/4::WARNING::2020-09-01
09:32:32,168::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/4::WARNING::2020-09-01
09:32:32,285::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/4::WARNING::2020-09-01
09:32:32,287::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/4::WARNING::2020-09-01
09:32:32,288::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,396::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,397::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,397::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,398::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,398::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,399::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,399::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,400::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/4::WARNING::2020-09-01 09:32:32,400::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/4::ERROR::2020-09-01 09:32:36,337::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/4::ERROR::2020-09-01 09:32:36,341::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/4::ERROR::2020-09-01 09:32:36,342::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/4::ERROR::2020-09-01 09:32:36,344::udev::53::blivet::(get_device) No
device at ''
[root@ov-no3 ~]# cat /var/log/vdsm/supervdsm.log|egrep "(WARN|ERR)" |more
MainProcess|jsonrpc/1::WARNING::2020-09-03
19:34:55,041::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-03
19:34:55,043::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-03
19:34:55,044::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-03
19:34:55,158::util::196::blivet::(get_sysfs_attr) queue/logical_block_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-03
19:34:55,160::util::196::blivet::(get_sysfs_attr) queue/optimal_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-03
19:34:55,161::util::196::blivet::(get_sysfs_attr) queue/minimum_io_size is not a valid
attribute
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,265::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,266::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,266::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,267::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,267::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,268::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,269::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Read-only locking type set. Write loc
ks are prohibited.
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,269::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Recovery of standalone physical volum
es failed.
MainProcess|jsonrpc/1::WARNING::2020-09-03 19:34:55,270::lvm::360::blivet::(pvinfo)
ignoring pvs output line: Cannot process standalone physical vo
lumes
MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,399::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,402::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,404::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/1::ERROR::2020-09-03 19:34:59,406::udev::53::blivet::(get_device) No
device at ''
MainProcess|jsonrpc/7::ERROR::2020-09-03
19:54:43,983::supervdsm_server::103::SuperVdsm.ServerCallback::(wrapper) Error in
volumeGeoRepStatus
[root@ov-no3 ~]# cat /var/log/ovirt-hosted-engine-ha/agent.log.2020-09-01 |egrep -i
"(WARN|ERR)" |more
[nothing in the output]
[root@ov-no3 ~]# cat /var/log/ovirt-hosted-engine-ha/broker.log.2020-09-01 |egrep -i
"(WARN|ERR)" |more
Thread-1::WARNING::2020-09-02 12:46:59,657::network::92::network.Network::(action) Failed
to verify network status, (4 out of 5)
[root@ov-no3 ~]# gluster volume status
Status of volume: data
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ov-no1.ariadne-t.local:/gluster_brick
s/data/data 49152 0 Y 6070
Brick ov-no2.ariadne-t.local:/gluster_brick
s/data/data 49152 0 Y 28860
Brick ov-no3.ariadne-t.local:/gluster_brick
s/data/data 49152 0 Y 5752
Self-heal Daemon on localhost N/A N/A Y 19353
Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539
Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813
Task Status of Volume data
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: engine
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ov-no1.ariadne-t.local:/gluster_brick
s/engine/engine 49153 0 Y 6098
Brick ov-no2.ariadne-t.local:/gluster_brick
s/engine/engine 49156 0 Y 28867
Brick ov-no3.ariadne-t.local:/gluster_brick
s/engine/engine 49153 0 Y 5789
Self-heal Daemon on localhost N/A N/A Y 19353
Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539
Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813
Task Status of Volume engine
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: vmstore
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ov-no1.ariadne-t.local:/gluster_brick
s/vmstore/vmstore 49154 0 Y 6102
Brick ov-no2.ariadne-t.local:/gluster_brick
s/vmstore/vmstore 49157 0 Y 28878
Brick ov-no3.ariadne-t.local:/gluster_brick
s/vmstore/vmstore 49154 0 Y 5772
Self-heal Daemon on localhost N/A N/A Y 19353
Self-heal Daemon on ov-no1.ariadne-t.local N/A N/A Y 6539
Self-heal Daemon on ov-no2.ariadne-t.local N/A N/A Y 813
Task Status of Volume vmstore
------------------------------------------------------------------------------
There are no active volume tasks
[root@ov-no3 ~]# gluster volume heal engine info
Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Number of entries: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Number of entries: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Number of entries: 0
[root@ov-no3 ~]# gluster volume heal engine info summary
Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
[root@ov-no3 ~]# gluster volume heal data info
Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Number of entries: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Number of entries: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Number of entries: 0
[root@ov-no3 ~]# gluster volume heal data info summary
Brick ov-no1.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/data/data
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
[root@ov-no3 ~]# gluster volume heal vmstore info
Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Number of entries: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Number of entries: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Number of entries: 0
[root@ov-no3 ~]# gluster volume heal vmstore info summary
Brick ov-no1.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no2.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick ov-no3.ariadne-t.local:/gluster_bricks/vmstore/vmstore
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
[root@ov-no3 ~]# ps -A|grep gluster
5143 ? 00:01:31 glustereventsd
5752 ? 02:01:08 glusterfsd
5772 ? 00:17:26 glusterfsd
5789 ? 05:12:24 glusterfsd
5906 ? 00:00:38 glustereventsd
6382 ? 02:37:00 glusterfs
9843 ? 00:02:18 glusterd
10613 ? 00:22:08 glusterfs
10750 ? 00:13:24 glusterfs
19353 ? 01:20:36 glusterfs
Hosted Engine:
[root@ov-eng ~]# grep ERROR /var/log/ovirt-engine/engine.log |more
2020-09-02 03:36:35,199+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob]
(DefaultQuartzScheduler8) [1a87b58
d] VDS error Command execution failed: rc=1 out='Error : Request timed
out\ngeo-replication command failed\n' err=''
2020-09-02 04:38:35,429+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob]
(DefaultQuartzScheduler2) [15fadfa
2] VDS error Command execution failed: rc=1 out='Error : Request timed
out\ngeo-replication command failed\n' err=''
2020-09-02 05:40:35,655+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob]
(DefaultQuartzScheduler9) [50fa85f
0] VDS error Command execution failed: rc=1 out='Error : Request timed
out\ngeo-replication command failed\n' err=''
2020-09-02 06:42:35,885+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob]
(DefaultQuartzScheduler7) [3092fdc
c] VDS error Command execution failed: rc=1 out='Error : Request timed
out\ngeo-replication command failed\n' err=''
2020-09-02 07:44:36,119+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob]
(DefaultQuartzScheduler6) [696ffca
7] VDS error Command execution failed: rc=1 out='Error : Request timed
out\ngeo-replication command failed\n' err=''
2020-09-02 08:46:36,341+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob]
(DefaultQuartzScheduler9) [180bb50
] VDS error Command execution failed: rc=1 out='Error : Request timed
out\ngeo-replication command failed\n' err=''
2020-09-02 09:48:36,562+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob]
(DefaultQuartzScheduler6) [3a45e1b
3] VDS error Command execution failed: rc=1 out='Error : Request timed
out\ngeo-replication command failed\n' err=''
2020-09-02 10:50:36,788+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob]
(DefaultQuartzScheduler4) [7da1197
3] VDS error Command execution failed: rc=1 out='Error : Request timed
out\ngeo-replication command failed\n' err=''
2020-09-02 11:52:37,011+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob]
(DefaultQuartzScheduler6) [76b3cb6
5] VDS error Command execution failed: rc=1 out='Error : Request timed
out\ngeo-replication command failed\n' err=''
2020-09-02 12:54:37,234+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob]
(DefaultQuartzScheduler5) [6a0d904
9] VDS error Command execution failed: rc=1 out='Error : Request timed
out\ngeo-replication command failed\n' err=''
2020-09-02 13:56:37,459+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob]
(DefaultQuartzScheduler4) [1e25ecf
f] VDS error Command execution failed: rc=1 out='Error : Request timed
out\ngeo-replication command failed\n' err=''
2020-09-02 14:58:37,674+03 ERROR [org.ovirt.engine.core.bll.gluster.GlusterGeoRepSyncJob]
(DefaultQuartzScheduler1) [64144e3
4] VDS error Command execution failed: rc=1 out='Error : Request timed
out\ngeo-replication command failed\n' err=''
How can I find why the HE failover happened? I believe it's the gluster geo
replication, but I haven't configured geo replication in the enviroment. Can you help
me how to proceed?
Thank you all in advance,
Maria Souvalioti
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/IZOXEF6MGTZ...
7:48 p.m.
Hello,
This is what I could gather from the gluster logs around the time frame of the HE
shutdown.
NODE1:
[root@ov-no1 glusterfs]# more bricks/gluster_bricks-vmstore-vmstore.log-20200830 |egrep
"( W | E )"|more
[2020-08-27 15:35:03.090477] W [glusterfsd.c:1570:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dd5) [0x7fa6e04a3dd5] -->/usr/sbin/glusterfsd(glus
terfs_sigwaiter+0xe5) [0x55a40138d1b5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b)
[0x55a40138d01b] ) 0-: received signum (15), shutting down
[2020-08-27 15:35:14.926794] E [MSGID: 100018]
[glusterfsd.c:2333:glusterfs_pidfile_update] 0-glusterfsd: pidfile
/var/run/gluster/vols/vmstore/ov-no
1.ariadne-t.local-gluster_bricks-vmstore-vmstore.pid lock failed [Resource temporarily
unavailable]
[root@ov-no1 glusterfs]# more bricks/gluster_bricks-data-data.log-20200830 |egrep "(
W | E )"|more
[2020-08-27 15:35:01.087875] W [glusterfsd.c:1570:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dd5) [0x7fc3cbf69dd5] -->/usr/sbin/glusterfsd(glus
terfs_sigwaiter+0xe5) [0x555e313711b5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b)
[0x555e3137101b] ) 0-: received signum (15), shutting down
[2020-08-27 15:35:14.890471] E [MSGID: 100018]
[glusterfsd.c:2333:glusterfs_pidfile_update] 0-glusterfsd: pidfile
/var/run/gluster/vols/data/ov-no1.a
riadne-t.local-gluster_bricks-data-data.pid lock failed [Resource temporarily
unavailable]
[root@ov-no1 glusterfs]# more bricks/gluster_bricks-engine-engine.log-20200830 |egrep
"( W | E )"|more
[2020-08-27 15:35:02.088732] W [glusterfsd.c:1570:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dd5) [0x7f70b99cbdd5] -->/usr/sbin/glusterfsd(glus
terfs_sigwaiter+0xe5) [0x55ebd132b1b5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b)
[0x55ebd132b01b] ) 0-: received signum (15), shutting down
[2020-08-27 15:35:14.907603] E [MSGID: 100018]
[glusterfsd.c:2333:glusterfs_pidfile_update] 0-glusterfsd: pidfile
/var/run/gluster/vols/engine/ov-no1
.ariadne-t.local-gluster_bricks-engine-engine.pid lock failed [Resource temporarily
unavailable]
[root@ov-no1 glusterfs]# more bricks/gluster_bricks-vmstore-vmstore.log |egrep "( W |
E )"|more
[nothing in the output]
[root@ov-no1 glusterfs]# more bricks/gluster_bricks-data-data.log |egrep "( W | E
)"|more
[nothing in the output]
[root@ov-no1 glusterfs]# more bricks/gluster_bricks-engine-engine.log |egrep "( W | E
)"|more
[nothing in the output]
[root@ov-no1 glusterfs]# more cmd_history.log | egrep "(WARN|error|fail)" |more
[2020-09-01 02:00:38.685251] : volume geo-replication status : FAILED : Commit failed on
ov-no2.ariadne-t.local. Please check log file for details.
Commit failed on ov-no3.ariadne-t.local. Please check log file for details.
[2020-09-01 03:02:39.094984] : volume geo-replication status : FAILED : Commit failed on
ov-no2.ariadne-t.local. Please check log file for details.
Commit failed on ov-no3.ariadne-t.local. Please check log file for details.
[2020-09-01 11:18:32.510224] : volume geo-replication status : FAILED : Commit failed on
ov-no2.ariadne-t.local. Please check log file for details.
Commit failed on ov-no3.ariadne-t.local. Please check log file for details.
[2020-09-01 14:24:33.778942] : volume geo-replication status : FAILED : Commit failed on
ov-no2.ariadne-t.local. Please check log file for details.
Commit failed on ov-no3.ariadne-t.local. Please check log file for details.
[root@ov-no1 glusterfs]# cat glusterd.log | egrep "( W | E )" |more
[2020-09-01 07:00:31.326169] E [glusterd-op-sm.c:8132:glusterd_op_sm]
(-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f23d8ac8a1e]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f23d8ac11be]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f23
d8ae806f] ) 0-management: Unable to get transaction opinfo for transaction ID
:435d3780-aa0c-4a64-bc28-56ae394159d0
[2020-09-01 08:02:31.551563] E [glusterd-op-sm.c:8132:glusterd_op_sm]
(-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f23d8ac8a1e]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f23d8ac11be]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f23
d8ae806f] ) 0-management: Unable to get transaction opinfo for transaction ID
:930a8a08-1044-41cf-b921-913b982e0c72
[2020-09-01 09:04:31.786157] E [glusterd-op-sm.c:8132:glusterd_op_sm]
(-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f23d8ac8a1e]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f23d8ac11be]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f23
d8ae806f] ) 0-management: Unable to get transaction opinfo for transaction ID
:9942b579-5240-4fee-bb4c-78b9a1c98da8
[2020-09-01 10:06:32.014362] E [glusterd-op-sm.c:8132:glusterd_op_sm]
(-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f23d8ac8a1e]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f23d8ac11be]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f23
d8ae806f] ) 0-management: Unable to get transaction opinfo for transaction ID
:324fc904-b147-4801-89d5-485120e1fb84
[2020-09-01 11:08:32.240906] W [MSGID: 106061]
[glusterd-geo-rep.c:2388:glusterd_op_gsync_args_get] 0-management: master not found
[2020-09-01 11:18:32.510041] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(glusterd mgmt), op(--(4)), xid = 0x812c, unique = 0, s
ent = 2020-09-01 11:08:32.242199, timeout = 600 for 192.168.48.202:24007
[root@ov-no1 glusterfs]# cat events.log-20200828 | egrep "( W | E )" |more
[nothing in the output]
[root@ov-no1 glusterfs]# cat events.log | egrep "( W | E )" |more
[nothing in the output]
[root@ov-no1 glusterfs]# cat glfsheal-data.log | egrep "( W | E )" |more
[nothing in the output]
[root@ov-no1 glusterfs]# cat glfsheal-vmstore.log | egrep "( W | E )" |more
[nothing in the output]
[root@ov-no1 glusterfs]# cat glfsheal-engine.log | egrep "( W | E )" |more
[nothing in the output]
[root@ov-no1 glusterfs]# cat glfsheal-data.log-20200830 | egrep "( W | E )"
|more
[nothing in the output]
[root@ov-no1 glusterfs]# cat glfsheal-vmstore.log-20200830 | egrep "( W | E )"
|more
[nothing in the output]
[root@ov-no1 glusterfs]# cat glfsheal-engine.log-20200830 | egrep "( W | E )"
|more
[nothing in the output]
[root@ov-no1 glusterfs]# cat glustershd.log | egrep "( W | E )" |more
[nothing in the output]
[root@ov-no1 glusterfs]# cat
rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_engine.log | egrep "( W | E
)" |more
[nothing in the output]
[root@ov-no1 glusterfs]# cat
rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_data.log | egrep "( W | E
)" |more
[nothing in the output]
[root@ov-no1 glusterfs]# cat
rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_vmstore.log | egrep "( W | E
)" |more
[nothing in the output]
[root@ov-no1 glusterfs]# cat
rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_data.log
[2020-08-31 00:41:02.169424] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-08-31 00:41:02.170418] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-08-31 08:59:07.694673] I [MSGID: 133022] [shard.c:3673:shard_delete_shards]
0-data-shard: Deleted shards of gfid=d25c9bb6-4a25-4025-8652-686be0a4bdf1 from backend
[2020-08-31 08:59:07.890175] I [MSGID: 133017] [shard.c:6696:shard_seek] 0-data-shard:
seek called on 0e4c3fb8-7617-4e43-879a-cc784198216d. [Operation not supported]
[2020-08-31 08:59:07.892803] I [MSGID: 133017] [shard.c:6696:shard_seek] 0-data-shard:
seek called on 17c220e1-fa2c-42d7-a948-6ea2bf5a93d7. [Operation not supported]
[2020-08-31 08:59:07.897929] I [MSGID: 133017] [shard.c:6696:shard_seek] 0-data-shard:
seek called on f9315204-ec16-4c46-847e-d84f0169ea8f. [Operation not supported]
[2020-08-31 08:59:07.928488] I [MSGID: 133022] [shard.c:3673:shard_delete_shards]
0-data-shard: Deleted shards of gfid=17c220e1-fa2c-42d7-a948-6ea2bf5a93d7 from backend
[2020-09-01 00:36:01.667677] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-09-01 00:36:01.668986] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-09-02 00:48:02.278423] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-09-02 00:48:02.279637] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-09-04 00:36:01.546102] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-09-04 00:36:01.547101] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[root@ov-no1 glusterfs]# cat
rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_vmstore.log
[2020-08-28 00:12:02.424709] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-08-29 00:29:02.375537] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-08-29 00:29:02.377088] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-08-30 00:13:03.449758] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-08-30 00:13:03.451232] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-08-31 00:41:02.169556] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-08-31 00:41:02.171040] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-09-01 00:36:01.667768] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-09-01 00:36:01.669572] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-09-02 00:48:02.278490] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-09-02 00:48:02.280253] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-09-04 00:36:01.546134] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-09-04 00:36:01.547694] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[root@ov-no1 glusterfs]#
[root@ov-no1 glusterfs]# cat
rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_engine.log
[2020-08-30 00:13:03.447882] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-08-30 00:13:03.449756] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-08-31 00:41:02.167497] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-08-31 00:41:02.169892] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-09-01 00:36:01.666356] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-09-01 00:36:01.668276] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-09-02 00:48:02.277024] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-09-02 00:48:02.279273] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-09-04 00:36:01.543799] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-09-04 00:36:01.545779] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[root@ov-no1 glusterfs]# cat glustershd.log | egrep "( W | E )" |more
[root@ov-no1 glusterfs]# cat glustershd.log
[2020-08-30 20:31:26.884546] I [MSGID: 108026]
[afr-self-heal-common.c:1741:afr_log_selfheal] 0-engine-replicate-0: Completed data
selfheal on 3fafabf3-d0cd-4b9a-8dd7-43145451f7cf. sources=[1] 2 sinks=0
.
.
.
[every ten minutes the same log appears]
.
.
[2020-08-31 15:03:57.893224] I [MSGID: 108026]
[afr-self-heal-common.c:1741:afr_log_selfheal] 0-engine-replicate-0: Completed data
selfheal on 3fafabf3-d0cd-4b9a-8dd7-43145451f7cf. sources=[1] 2 sinks=0
The message "I [MSGID: 108026] [afr-self-heal-common.c:1741:afr_log_selfheal]
0-engine-replicate-0: Completed data selfheal on 3fafabf3-d0cd-4b9a-8dd7-43145451f7cf.
sources=[1] 2 sinks=0 " repeated 15 times between [2020-08-31 15:03:57.893224] and
[2020-08-31 15:04:42.605183]
[2020-08-31 15:14:46.171044] I [MSGID: 108026]
[afr-self-heal-common.c:1741:afr_log_selfheal] 0-engine-replicate-0: Completed data
selfheal on 3fafabf3-d0cd-4b9a-8dd7-43145451f7cf. sources=[1] 2 sinks=0
[2020-09-01 00:36:01.663267] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-09-01 00:36:01.665109] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-09-02 00:48:02.274383] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-09-02 00:48:02.275562] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[root@ov-no1 glusterfs]# zcat cli.log-20200902.gz |egrep "( W | E )" |more
[2020-09-01 09:04:39.799900] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:04:39.801917] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:24:41.581140] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 09:39:43.395156] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:39:43.397147] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:49:44.128540] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 09:54:44.629071] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 10:04:45.642437] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 10:04:45.904006] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 10:04:45.905279] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 10:39:49.183655] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 10:44:49.968219] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 10:44:49.970487] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 10:49:50.206005] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 10:59:51.489242] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 10:59:51.491277] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[root@ov-no1 glusterfs]# cat cli.log |grep "( W | E )" |more
[nothing in the output]
[root@ov-no1 glusterfs]# cat cmd_history.log|egrep "( W| E|fail)" |more
[2020-09-01 00:58:38.265743] : volume geo-replication status : FAILED : Commit failed on
ov-no2.ariadne-t.local. Please check log file for details.
Commit failed on ov-no3.ariadne-t.local. Please check log file for details.
[2020-09-01 02:00:38.685251] : volume geo-replication status : FAILED : Commit failed on
ov-no2.ariadne-t.local. Please check log file for details.
Commit failed on ov-no3.ariadne-t.local. Please check log file for details.
[2020-09-01 03:02:39.094984] : volume geo-replication status : FAILED : Commit failed on
ov-no2.ariadne-t.local. Please check log file for details.
Commit failed on ov-no3.ariadne-t.local. Please check log file for details.
[2020-09-01 11:18:32.510224] : volume geo-replication status : FAILED : Commit failed on
ov-no2.ariadne-t.local. Please check log file for details.
Commit failed on ov-no3.ariadne-t.local. Please check log file for details.
[2020-09-01 14:24:33.778942] : volume geo-replication status : FAILED : Commit failed on
ov-no2.ariadne-t.local. Please check log file for details.
Commit failed on ov-no3.ariadne-t.local. Please check log file for details.
[2020-09-01 16:28:34.610703] : volume geo-replication status : FAILED : Commit failed on
ov-no2.ariadne-t.local. Please check log file for details.
Commit failed on ov-no3.ariadne-t.local. Please check log file for details.
NODE2:
[root@ov-no2 ~]# cd /var/log/glusterfs/
[root@ov-no2 glusterfs]# more bricks/gluster_bricks-data-data.log-20200830 |egrep "(
W | E )"|more
[2020-08-27 15:35:07.532064] W [glusterfsd.c:1570:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dd5) [0x7f01ef674dd5] -->/usr/sbin/glusterfsd(glus
terfs_sigwaiter+0xe5) [0x55d96c9791b5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b)
[0x55d96c97901b] ) 0-: received signum (15), shutting down
[2020-08-27 15:35:07.545460] W [rpc-clnt.c:1704:rpc_clnt_submit] 0-glusterfs: failed to
submit rpc-request (unique: 0, XID: 0x33 Program: Gluster Por
tmap, ProgVers: 1, Proc: 5) to rpc-transport (glusterfs)
[2020-08-27 15:35:07.545626] E [timer.c:34:gf_timer_call_after]
(-->/usr/sbin/glusterfsd(mgmt_submit_request+0x13d) [0x55d96c97ecdd]
-->/lib64/libgfr
pc.so.0(rpc_clnt_submit+0x72d) [0x7f01f05dfc1d]
-->/lib64/libglusterfs.so.0(gf_timer_call_after+0x229) [0x7f01f0844349] ) 0-timer:
Either ctx is NULL
or ctx cleanup started [Invalid argument]
[2020-08-27 15:35:07.545724] E [timer.c:34:gf_timer_call_after]
(-->/usr/sbin/glusterfsd(mgmt_submit_request+0x13d) [0x55d96c97ecdd]
-->/lib64/libgfr
pc.so.0(rpc_clnt_submit+0x72d) [0x7f01f05dfc1d]
-->/lib64/libglusterfs.so.0(gf_timer_call_after+0x229) [0x7f01f0844349] ) 0-timer:
Either ctx is NULL
or ctx cleanup started [Invalid argument]
[2020-08-27 16:25:42.954545] W [inodelk.c:609:pl_inodelk_log_cleanup] 0-data-server:
releasing lock on 03848819-f37b-4fa7-81d9-3f5dc3eef2da held by {
client=0x7fd84400f350, pid=7725 lk-owner=e8100a64447f0000}
[root@ov-no2 glusterfs]#
[root@ov-no2 glusterfs]#
[root@ov-no2 glusterfs]#
[root@ov-no2 glusterfs]#
[root@ov-no2 glusterfs]# more bricks/gluster_bricks-engine-engine.log-20200830 |egrep
"( W | E )"|more
[2020-08-27 15:35:08.532812] W [glusterfsd.c:1570:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dd5) [0x7ff67fa1edd5] -->/usr/sbin/glusterfsd(glus
terfs_sigwaiter+0xe5) [0x55d4f15951b5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b)
[0x55d4f159501b] ) 0-: received signum (15), shutting down
[2020-08-27 15:35:08.536715] W [rpc-clnt.c:1704:rpc_clnt_submit] 0-glusterfs: failed to
submit rpc-request (unique: 0, XID: 0x33 Program: Gluster Por
tmap, ProgVers: 1, Proc: 5) to rpc-transport (glusterfs)
[2020-08-27 15:35:08.536903] E [timer.c:34:gf_timer_call_after]
(-->/usr/sbin/glusterfsd(mgmt_submit_request+0x13d) [0x55d4f159acdd]
-->/lib64/libgfr
pc.so.0(rpc_clnt_submit+0x72d) [0x7ff680989c1d]
-->/lib64/libglusterfs.so.0(gf_timer_call_after+0x229) [0x7ff680bee349] ) 0-timer:
Either ctx is NULL
or ctx cleanup started [Invalid argument]
[2020-08-27 15:35:08.537018] E [timer.c:34:gf_timer_call_after]
(-->/usr/sbin/glusterfsd(mgmt_submit_request+0x13d) [0x55d4f159acdd]
-->/lib64/libgfr
pc.so.0(rpc_clnt_submit+0x72d) [0x7ff680989c1d]
-->/lib64/libglusterfs.so.0(gf_timer_call_after+0x229) [0x7ff680bee349] ) 0-timer:
Either ctx is NULL
or ctx cleanup started [Invalid argument]
[root@ov-no2 glusterfs]#
[root@ov-no2 glusterfs]#
[root@ov-no2 glusterfs]#
[root@ov-no2 glusterfs]#
[root@ov-no2 glusterfs]#
[root@ov-no2 glusterfs]# more bricks/gluster_bricks-vmstore-vmstore.log-20200830 |egrep
"( W | E )"|more
[2020-08-27 15:35:09.533389] W [glusterfsd.c:1570:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dd5) [0x7f2d8f5f8dd5] -->/usr/sbin/glusterfsd(glus
terfs_sigwaiter+0xe5) [0x563cbb1081b5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b)
[0x563cbb10801b] ) 0-: received signum (15), shutting down
[2020-08-27 15:35:09.540492] W [rpc-clnt.c:1704:rpc_clnt_submit] 0-glusterfs: failed to
submit rpc-request (unique: 0, XID: 0x33 Program: Gluster Por
tmap, ProgVers: 1, Proc: 5) to rpc-transport (glusterfs)
[2020-08-27 15:35:09.540657] E [timer.c:34:gf_timer_call_after]
(-->/usr/sbin/glusterfsd(mgmt_submit_request+0x13d) [0x563cbb10dcdd]
-->/lib64/libgfr
pc.so.0(rpc_clnt_submit+0x72d) [0x7f2d90563c1d]
-->/lib64/libglusterfs.so.0(gf_timer_call_after+0x229) [0x7f2d907c8349] ) 0-timer:
Either ctx is NULL
or ctx cleanup started [Invalid argument]
[2020-08-27 15:35:09.540762] E [timer.c:34:gf_timer_call_after]
(-->/usr/sbin/glusterfsd(mgmt_submit_request+0x13d) [0x563cbb10dcdd]
-->/lib64/libgfr
pc.so.0(rpc_clnt_submit+0x72d) [0x7f2d90563c1d]
-->/lib64/libglusterfs.so.0(gf_timer_call_after+0x229) [0x7f2d907c8349] ) 0-timer:
Either ctx is NULL
or ctx cleanup started [Invalid argument]
[root@ov-no2 glusterfs]# more bricks/gluster_bricks-data-data.log |egrep "( W | E
)"|more
[root@ov-no2 glusterfs]# more bricks/gluster_bricks-vmstore-vmstore.log |egrep "( W |
E )"|more
[root@ov-no2 glusterfs]# more bricks/gluster_bricks-engine-engine.log |egrep "( W | E
)"|more
[root@ov-no2 glusterfs]# more cmd_history.log | egrep "( W | E )" |more
[nothing in the output]
[root@ov-no2 glusterfs]# cat glusterd.log | egrep "( W | E )" |more
[2020-09-01 07:00:31.325533] W [MSGID: 106061]
[glusterd-geo-rep.c:2388:glusterd_op_gsync_args_get] 0-management: master not found
[2020-09-01 07:10:36.677633] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(glusterd mgmt), op(--(4)), xid = 0x7f74, unique = 0, s
ent = 2020-09-01 07:00:31.326798, timeout = 600 for 192.168.48.203:24007
[2020-09-01 07:10:36.677697] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors]
0-glusterd: Commit failed on ov-no3.ariadne-t.local. Please
check log file for details.
[2020-09-01 07:10:40.678204] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(glusterd mgmt), op(--(4)), xid = 0x804d, unique = 0, s
ent = 2020-09-01 07:00:31.326774, timeout = 600 for 192.168.48.201:24007
[2020-09-01 07:10:40.678259] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors]
0-glusterd: Commit failed on ov-no1.ariadne-t.local. Please
check log file for details.
[2020-09-01 07:10:40.678395] E [rpcsvc.c:1577:rpcsvc_submit_generic] 0-rpc-service: failed
to submit message (XID: 0x2, Program: GlusterD svc cli, Pr
ogVers: 2, Proc: 22) to rpc-transport (socket.management)
[2020-09-01 07:10:40.678414] E [MSGID: 106430]
[glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission failed
[2020-09-01 08:02:31.553170] E [glusterd-op-sm.c:8132:glusterd_op_sm]
(-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7efc4e68ea1e]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7efc4e6871be]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7efc
4e6ae06f] ) 0-management: Unable to get transaction opinfo for transaction ID
:930a8a08-1044-41cf-b921-913b982e0c72
[2020-09-01 09:04:31.784847] W [MSGID: 106061]
[glusterd-geo-rep.c:2388:glusterd_op_gsync_args_get] 0-management: master not found
[2020-09-01 09:14:31.577944] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(glusterd mgmt), op(--(4)), xid = 0x82ac, unique = 0, s
ent = 2020-09-01 09:04:31.786089, timeout = 600 for 192.168.48.201:24007
[2020-09-01 09:14:31.578010] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors]
0-glusterd: Commit failed on ov-no1.ariadne-t.local. Please
check log file for details.
[2020-09-01 09:14:37.578887] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(glusterd mgmt), op(--(4)), xid = 0x81d3, unique = 0, s
ent = 2020-09-01 09:04:31.786116, timeout = 600 for 192.168.48.203:24007
[2020-09-01 09:14:37.578945] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors]
0-glusterd: Commit failed on ov-no3.ariadne-t.local. Please
check log file for details.
[2020-09-01 09:14:37.579189] E [rpcsvc.c:1577:rpcsvc_submit_generic] 0-rpc-service: failed
to submit message (XID: 0x2, Program: GlusterD svc cli, Pr
ogVers: 2, Proc: 22) to rpc-transport (socket.management)
[2020-09-01 09:14:37.579210] E [MSGID: 106430]
[glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission failed
[2020-09-01 10:06:32.014210] W [MSGID: 106061]
[glusterd-geo-rep.c:2388:glusterd_op_gsync_args_get] 0-management: master not found
[2020-09-01 10:16:32.031372] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(glusterd mgmt), op(--(4)), xid = 0x83d7, unique = 0, s
ent = 2020-09-01 10:06:32.015520, timeout = 600 for 192.168.48.201:24007
[2020-09-01 10:16:32.031415] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors]
0-glusterd: Commit failed on ov-no1.ariadne-t.local. Please
check log file for details.
[2020-09-01 10:16:38.032229] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(glusterd mgmt), op(--(4)), xid = 0x82fe, unique = 0, s
ent = 2020-09-01 10:06:32.015546, timeout = 600 for 192.168.48.203:24007
[2020-09-01 10:16:38.032281] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors]
0-glusterd: Commit failed on ov-no3.ariadne-t.local. Please
check log file for details.
[2020-09-01 10:16:38.032491] E [rpcsvc.c:1577:rpcsvc_submit_generic] 0-rpc-service: failed
to submit message (XID: 0x2, Program: GlusterD svc cli, Pr
ogVers: 2, Proc: 22) to rpc-transport (socket.management)
[2020-09-01 10:16:38.032519] E [MSGID: 106430]
[glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission failed
[root@ov-no2 glusterfs]# cat
rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_engine.log | egrep "( W | E
)" |more
[nothing in the output]
[root@ov-no2 glusterfs]# cat
rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_data.log | egrep "( W | E
)" |more
[nothing in the output]
[root@ov-no2 glusterfs]# cat
rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_vmstore.log | egrep "( W | E
)" |more
[nothing in the output]
[root@ov-no2 glusterfs]# cat cli.log | egrep "( W | E )" |more
[2020-09-01 08:49:38.289905] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 08:49:38.291840] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:04:39.528219] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 09:09:40.049281] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 09:09:40.318414] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:09:40.319842] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:14:40.557140] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 09:19:41.064083] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 09:19:41.338070] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:19:41.340016] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:24:41.860173] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:24:41.862096] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:29:42.110479] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 09:29:42.380124] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:29:42.381580] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:34:42.628085] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 09:44:43.895269] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:44:43.897282] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:54:44.902854] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:54:44.904765] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[root@ov-no2 glusterfs]# cat cmd_history.log | egrep "( W | E |fail)" |more
[2020-09-01 04:04:39.342142] : volume geo-replication status : FAILED : Commit failed on
ov-no3.ariadne-t.local. Please check log file for details.
Commit failed on ov-no1.ariadne-t.local. Please check log file for details.
[2020-09-01 05:06:39.783310] : volume geo-replication status : FAILED : Commit failed on
ov-no3.ariadne-t.local. Please check log file for details.
Commit failed on ov-no1.ariadne-t.local. Please check log file for details.
[2020-09-01 07:10:40.678333] : volume geo-replication status : FAILED : Commit failed on
ov-no3.ariadne-t.local. Please check log file for details.
Commit failed on ov-no1.ariadne-t.local. Please check log file for details.
[2020-09-01 09:14:37.579120] : volume geo-replication status : FAILED : Commit failed on
ov-no1.ariadne-t.local. Please check log file for details.
Commit failed on ov-no3.ariadne-t.local. Please check log file for details.
[2020-09-01 10:16:38.032427] : volume geo-replication status : FAILED : Commit failed on
ov-no1.ariadne-t.local. Please check log file for details.
Commit failed on ov-no3.ariadne-t.local. Please check log file for details.
NODE3:
[root@ov-no3 glusterfs]# more bricks/gluster_bricks-engine-engine.log | egrep "( W |
E )" |more
[nothing in the output]
[root@ov-no3 glusterfs]# more bricks/gluster_bricks-vmstore-vmstore.log | egrep "( W
| E )" |more
[nothing in the output]
[root@ov-no3 glusterfs]# more bricks/gluster_bricks-data-data.log | egrep "( W | E
)" |more
[nothing in the output]
[root@ov-no3 glusterfs]# zcat cli.log |egrep "( W | E )" |more
[2020-09-01 07:29:30.165436] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 07:29:30.166677] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 07:34:30.666721] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 07:34:30.668801] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 07:44:31.431679] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 07:49:31.936985] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 07:54:32.447922] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 08:09:34.242293] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 08:09:34.244149] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 08:29:36.002322] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 08:34:36.773092] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 08:34:36.775476] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 08:44:37.512700] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 08:54:38.528131] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 08:54:38.797499] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 08:54:38.798780] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 08:59:39.030389] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 08:59:39.293311] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 08:59:39.294612] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:14:40.823633] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:14:40.825547] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:34:42.890897] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:34:42.893012] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:39:43.129137] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 09:44:43.629375] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 09:49:44.395649] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:49:44.397886] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:59:45.148111] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 09:59:45.409361] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:59:45.410701] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 10:14:46.647785] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 10:14:46.908152] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 10:14:46.909401] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[root@ov-no3 glusterfs]# more cmd_history.log | egrep "( W | E )" |more
[root@ov-no3 glusterfs]# cat glusterd.log | egrep "( W | E )" |more
[2020-09-01 08:02:31.550060] W [MSGID: 106061]
[glusterd-geo-rep.c:2388:glusterd_op_gsync_args_get] 0-management: master not found
[2020-09-01 08:12:31.582984] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(glusterd mgmt), op(--(4)), xid = 0x7fe0, unique = 0, s
ent = 2020-09-01 08:02:31.551222, timeout = 600 for 192.168.48.202:24007
[2020-09-01 08:12:31.583040] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors]
0-glusterd: Commit failed on ov-no2.ariadne-t.local. Please
check log file for details.
[2020-09-01 08:12:31.583086] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(glusterd mgmt), op(--(4)), xid = 0x7fd9, unique = 0, s
ent = 2020-09-01 08:02:31.551255, timeout = 600 for 192.168.48.201:24007
[2020-09-01 08:12:31.583100] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors]
0-glusterd: Commit failed on ov-no1.ariadne-t.local. Please
check log file for details.
[2020-09-01 08:12:31.583253] E [rpcsvc.c:1577:rpcsvc_submit_generic] 0-rpc-service: failed
to submit message (XID: 0x2, Program: GlusterD svc cli, Pr
ogVers: 2, Proc: 22) to rpc-transport (socket.management)
[2020-09-01 08:12:31.583269] E [MSGID: 106430]
[glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission failed
[2020-09-01 09:04:31.786592] E [glusterd-op-sm.c:8132:glusterd_op_sm]
(-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f56fecfba1e]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f56fecf41be]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f56
fed1b06f] ) 0-management: Unable to get transaction opinfo for transaction ID
:9942b579-5240-4fee-bb4c-78b9a1c98da8
[2020-09-01 10:06:32.014980] E [glusterd-op-sm.c:8132:glusterd_op_sm]
(-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f56fecfba1e]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f56fecf41be]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f56
fed1b06f] ) 0-management: Unable to get transaction opinfo for transaction ID
:324fc904-b147-4801-89d5-485120e1fb84
[2020-09-01 11:08:32.243189] E [glusterd-op-sm.c:8132:glusterd_op_sm]
(-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f56fecfba1e]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f56fecf41be]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f56
fed1b06f] ) 0-management: Unable to get transaction opinfo for transaction ID
:3b7b79b3-344c-4a67-b66e-c8fb8ddb2a2a
[2020-09-01 12:10:32.473993] E [glusterd-op-sm.c:8132:glusterd_op_sm]
(-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f56fecfba1e]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f56fecf41be]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f56
fed1b06f] ) 0-management: Unable to get transaction opinfo for transaction ID
:2521f708-6bc4-4be4-b37f-41d67839e941
[2020-09-01 13:12:32.707217] W [MSGID: 106061]
[glusterd-geo-rep.c:2388:glusterd_op_gsync_args_get] 0-management: master not found
[2020-09-01 13:22:33.887995] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(glusterd mgmt), op(--(4)), xid = 0x866e, unique = 0, s
ent = 2020-09-01 13:12:32.708475, timeout = 600 for 192.168.48.202:24007
[2020-09-01 13:22:33.888057] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors]
0-glusterd: Commit failed on ov-no2.ariadne-t.local. Please
check log file for details.
[2020-09-01 13:22:33.888095] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(glusterd mgmt), op(--(4)), xid = 0x8667, unique = 0, s
ent = 2020-09-01 13:12:32.708500, timeout = 600 for 192.168.48.201:24007
[2020-09-01 13:22:33.888113] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors]
0-glusterd: Commit failed on ov-no1.ariadne-t.local. Please
check log file for details.
[2020-09-01 13:22:33.888234] E [rpcsvc.c:1577:rpcsvc_submit_generic] 0-rpc-service: failed
to submit message (XID: 0x2, Program: GlusterD svc cli, Pr
ogVers: 2, Proc: 22) to rpc-transport (socket.management)
[2020-09-01 13:22:33.888269] E [MSGID: 106430]
[glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission failed
[root@ov-no3 glusterfs]# cat events.log-20200828 | egrep "( W | E )" |more
[nothing in the output]
[root@ov-no3 glusterfs]# cat glfsheal-data.log | egrep "( W | E )" |more
[nothing in the output]
[root@ov-no3 glusterfs]# cat glfsheal-vmstore.log | egrep "( W | E )" |more
[nothing in the output]
[root@ov-no3 glusterfs]# cat glfsheal-engine.log | egrep "( W | E )" |more
[nothing in the output]
[root@ov-no3 glusterfs]# cat glustershd.log | egrep "( W | E )" |more
[nothing in the output]
[root@ov-no3 glusterfs]# cat
rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_engine.log | egrep "( W | E
)" |more
[2020-08-31 14:49:13.805467] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2]
0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf)
remote_fd is -1. EBADFD [File descriptor in bad state]
[2020-08-31 14:51:13.882683] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2]
0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf)
remote_fd is -1. EBADFD [File descriptor in bad state]
[2020-08-31 14:53:12.691884] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2]
0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf)
remote_fd is -1. EBADFD [File descriptor in bad state]
[2020-08-31 14:55:12.713320] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2]
0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf)
remote_fd is -1. EBADFD [File descriptor in bad state]
[2020-08-31 14:57:13.146570] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2]
0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf)
remote_fd is -1. EBADFD [File descriptor in bad state]
[2020-08-31 14:59:13.502830] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2]
0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf)
remote_fd is -1. EBADFD [File descriptor in bad state]
[2020-08-31 15:01:13.039274] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2]
0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf)
remote_fd is -1. EBADFD [File descriptor in bad state]
[2020-08-31 15:03:13.401571] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2]
0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf)
remote_fd is -1. EBADFD [File descriptor in bad state]
[2020-08-31 15:05:13.764951] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2]
0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf)
remote_fd is -1. EBADFD [File descriptor in bad state]
[2020-08-31 15:07:14.140156] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2]
0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf)
remote_fd is -1. EBADFD [File descriptor in bad state]
[2020-08-31 15:09:13.061657] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2]
0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf)
remote_fd is -1. EBADFD [File descriptor in bad state]
[2020-08-31 15:11:14.616511] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2]
0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf)
remote_fd is -1. EBADFD [File descriptor in bad state]
[root@ov-no3 glusterfs]# cat
rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_data.log | egrep "( W | E
)" |more
[nothing recent in the output]
[root@ov-no3 glusterfs]# cat
rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_vmstore.log | egrep "( W | E
)" |more
[nothing recent in the output]
[root@ov-no3 glusterfs]# cat glustershd.log | egrep "( W | E )" |more
[nothing in the output]
[root@ov-no3 glusterfs]# cat events.log | egrep "( W | E )" |more
[nothing in the output]
Journalctl results on the nodes about the gluster service:
NODE1:
[root@ov-no1 ~]# journalctl --unit glusterd -l
-- Logs begin at Thu 2020-08-27 19:30:14 EEST, end at Fri 2020-09-04 16:34:42 EEST. --
Aug 27 19:30:52 ov-no1.ariadne-t.local systemd[1]: Starting GlusterFS, a clustered
file-system server...
Aug 27 19:30:56 ov-no1.ariadne-t.local systemd[1]: Started GlusterFS, a clustered
file-system server.
Aug 27 19:30:56 ov-no1.ariadne-t.local glusterd[5591]: [2020-08-27 16:30:56.544861] C
[MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum
Aug 27 19:30:56 ov-no1.ariadne-t.local glusterd[5591]: [2020-08-27 16:30:56.558812] C
[MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum
Aug 27 19:30:56 ov-no1.ariadne-t.local glusterd[5591]: [2020-08-27 16:30:56.571473] C
[MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum
NODE2:
[root@ov-no2 ~]# journalctl --unit glusterd -l
-- Logs begin at Wed 2019-09-11 17:40:13 EEST, end at Fri 2020-09-04 16:34:27 EEST. --
Aug 27 18:35:01 ov-no2.ariadne-t.local systemd[1]: Stopping GlusterFS, a clustered
file-system server...
Aug 27 18:35:01 ov-no2.ariadne-t.local systemd[1]: Stopped GlusterFS, a clustered
file-system server.
Aug 27 18:35:01 ov-no2.ariadne-t.local systemd[1]: Starting GlusterFS, a clustered
file-system server...
Aug 27 18:35:05 ov-no2.ariadne-t.local systemd[1]: Started GlusterFS, a clustered
file-system server.
Aug 27 18:35:07 ov-no2.ariadne-t.local glusterd[28722]: [2020-08-27 15:35:07.519155] C
[MSGID: 106002] [glusterd-server-quorum.c:355:glusterd_do_volu
Aug 27 18:35:08 ov-no2.ariadne-t.local glusterd[28722]: [2020-08-27 15:35:08.532018] C
[MSGID: 106002] [glusterd-server-quorum.c:355:glusterd_do_volu
Aug 27 18:35:09 ov-no2.ariadne-t.local glusterd[28722]: [2020-08-27 15:35:09.532867] C
[MSGID: 106002] [glusterd-server-quorum.c:355:glusterd_do_volu
Aug 27 18:35:10 ov-no2.ariadne-t.local glusterd[28722]: [2020-08-27 15:35:10.535976] C
[MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volu
Aug 27 18:35:10 ov-no2.ariadne-t.local glusterd[28722]: [2020-08-27 15:35:10.550171] C
[MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volu
Aug 27 18:35:10 ov-no2.ariadne-t.local glusterd[28722]: [2020-08-27 15:35:10.563481] C
[MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volu
Aug 27 19:25:23 ov-no2.ariadne-t.local glustershd[22187]: [2020-08-27 16:25:23.640530] C
[rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 0-data-cli
NODE3:
[root@ov-no3 glusterfs]# journalctl --unit glusterd -l
-- Logs begin at Thu 2020-08-27 19:09:30 EEST, end at Fri 2020-09-04 16:34:02 EEST. --
Aug 27 19:10:08 ov-no3.ariadne-t.local systemd[1]: Starting GlusterFS, a clustered
file-system server...
Aug 27 19:10:13 ov-no3.ariadne-t.local systemd[1]: Started GlusterFS, a clustered
file-system server.
Aug 27 19:10:13 ov-no3.ariadne-t.local glusterd[5233]: [2020-08-27 16:10:13.274001] C
[MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum
Aug 27 19:10:13 ov-no3.ariadne-t.local glusterd[5233]: [2020-08-27 16:10:13.288391] C
[MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum
Aug 27 19:10:13 ov-no3.ariadne-t.local glusterd[5233]: [2020-08-27 16:10:13.300866] C
[MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum
Aug 27 19:14:38 ov-no3.ariadne-t.local systemd[1]: Stopping GlusterFS, a clustered
file-system server...
Aug 27 19:14:38 ov-no3.ariadne-t.local systemd[1]: Stopped GlusterFS, a clustered
file-system server.
Aug 27 19:14:38 ov-no3.ariadne-t.local systemd[1]: Starting GlusterFS, a clustered
file-system server...
Aug 27 19:14:41 ov-no3.ariadne-t.local systemd[1]: Started GlusterFS, a clustered
file-system server.
Aug 27 19:14:48 ov-no3.ariadne-t.local glusterd[9843]: [2020-08-27 16:14:48.117530] C
[MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum
Aug 27 19:14:48 ov-no3.ariadne-t.local glusterd[9843]: [2020-08-27 16:14:48.131054] C
[MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum
Aug 27 19:14:48 ov-no3.ariadne-t.local glusterd[9843]: [2020-08-27 16:14:48.141323] C
[MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum
Aug 27 19:25:25 ov-no3.ariadne-t.local glustershd[10081]: [2020-08-27 16:25:25.286189] C
[rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 0-data-cli
Thanks for your help,
Maria Souvalioti
9:09 p.m.
Hi Maria,
I am quite puzzled about:
/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x555e3137101b] ) 0-:
received signum (15), shutting down
[2020-08-27 15:35:14.890471] E [MSGID: 100018]
[glusterfsd.c:2333:glusterfs_pidfile_update] 0-glusterfsd: pidfile
/var/run/gluster/vols/data/ov-no1.a
riadne-t.local-gluster_bricks-data-data.pid lock failed [Resource temporarily
unavailable]
That doesn't make sense.
Can you share the logs in a separate thread at gluster-users(a)gluster.org ?
Best Regards,
Strahil Nikolov
В петък, 4 септември 2020 г., 19:55:24 Гринуич+3, souvaliotimaria(a)mail.com
<souvaliotimaria(a)mail.com> написа:
Hello,
This is what I could gather from the gluster logs around the time frame of the HE
shutdown.
NODE1:
[root@ov-no1 glusterfs]# more bricks/gluster_bricks-vmstore-vmstore.log-20200830 |egrep
"( W | E )"|more
[2020-08-27 15:35:03.090477] W [glusterfsd.c:1570:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dd5) [0x7fa6e04a3dd5] -->/usr/sbin/glusterfsd(glus
terfs_sigwaiter+0xe5) [0x55a40138d1b5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b)
[0x55a40138d01b] ) 0-: received signum (15), shutting down
[2020-08-27 15:35:14.926794] E [MSGID: 100018]
[glusterfsd.c:2333:glusterfs_pidfile_update] 0-glusterfsd: pidfile
/var/run/gluster/vols/vmstore/ov-no
1.ariadne-t.local-gluster_bricks-vmstore-vmstore.pid lock failed [Resource temporarily
unavailable]
[root@ov-no1 glusterfs]# more bricks/gluster_bricks-data-data.log-20200830 |egrep "(
W | E )"|more
[2020-08-27 15:35:01.087875] W [glusterfsd.c:1570:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dd5) [0x7fc3cbf69dd5] -->/usr/sbin/glusterfsd(glus
terfs_sigwaiter+0xe5) [0x555e313711b5] --/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x555e3137101b] ) 0-:
received signum (15), shutting down
[2020-08-27 15:35:14.890471] E [MSGID: 100018]
[glusterfsd.c:2333:glusterfs_pidfile_update] 0-glusterfsd: pidfile
/var/run/gluster/vols/data/ov-no1.a
riadne-t.local-gluster_bricks-data-data.pid lock failed [Resource temporarily
unavailable]
[root@ov-no1 glusterfs]# more bricks/gluster_bricks-engine-engine.log-20200830 |egrep
"( W | E )"|more
[2020-08-27 15:35:02.088732] W [glusterfsd.c:1570:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dd5) [0x7f70b99cbdd5] -->/usr/sbin/glusterfsd(glus
terfs_sigwaiter+0xe5) [0x55ebd132b1b5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b)
[0x55ebd132b01b] ) 0-: received signum (15), shutting down
[2020-08-27 15:35:14.907603] E [MSGID: 100018]
[glusterfsd.c:2333:glusterfs_pidfile_update] 0-glusterfsd: pidfile
/var/run/gluster/vols/engine/ov-no1
.ariadne-t.local-gluster_bricks-engine-engine.pid lock failed [Resource temporarily
unavailable]
[root@ov-no1 glusterfs]# more bricks/gluster_bricks-vmstore-vmstore.log |egrep "( W |
E )"|more
[nothing in the output]
[root@ov-no1 glusterfs]# more bricks/gluster_bricks-data-data.log |egrep "( W | E
)"|more
[nothing in the output]
[root@ov-no1 glusterfs]# more bricks/gluster_bricks-engine-engine.log |egrep "( W | E
)"|more
[nothing in the output]
[root@ov-no1 glusterfs]# more cmd_history.log | egrep "(WARN|error|fail)" |more
[2020-09-01 02:00:38.685251] : volume geo-replication status : FAILED : Commit failed on
ov-no2.ariadne-t.local. Please check log file for details.
Commit failed on ov-no3.ariadne-t.local. Please check log file for details.
[2020-09-01 03:02:39.094984] : volume geo-replication status : FAILED : Commit failed on
ov-no2.ariadne-t.local. Please check log file for details.
Commit failed on ov-no3.ariadne-t.local. Please check log file for details.
[2020-09-01 11:18:32.510224] : volume geo-replication status : FAILED : Commit failed on
ov-no2.ariadne-t.local. Please check log file for details.
Commit failed on ov-no3.ariadne-t.local. Please check log file for details.
[2020-09-01 14:24:33.778942] : volume geo-replication status : FAILED : Commit failed on
ov-no2.ariadne-t.local. Please check log file for details.
Commit failed on ov-no3.ariadne-t.local. Please check log file for details.
[root@ov-no1 glusterfs]# cat glusterd.log | egrep "( W | E )" |more
[2020-09-01 07:00:31.326169] E [glusterd-op-sm.c:8132:glusterd_op_sm]
(-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f23d8ac8a1e]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f23d8ac11be]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f23
d8ae806f] ) 0-management: Unable to get transaction opinfo for transaction ID
:435d3780-aa0c-4a64-bc28-56ae394159d0
[2020-09-01 08:02:31.551563] E [glusterd-op-sm.c:8132:glusterd_op_sm]
(-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f23d8ac8a1e]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f23d8ac11be]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f23
d8ae806f] ) 0-management: Unable to get transaction opinfo for transaction ID
:930a8a08-1044-41cf-b921-913b982e0c72
[2020-09-01 09:04:31.786157] E [glusterd-op-sm.c:8132:glusterd_op_sm]
(-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f23d8ac8a1e]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f23d8ac11be]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f23
d8ae806f] ) 0-management: Unable to get transaction opinfo for transaction ID
:9942b579-5240-4fee-bb4c-78b9a1c98da8
[2020-09-01 10:06:32.014362] E [glusterd-op-sm.c:8132:glusterd_op_sm]
(-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f23d8ac8a1e]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f23d8ac11be]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f23
d8ae806f] ) 0-management: Unable to get transaction opinfo for transaction ID
:324fc904-b147-4801-89d5-485120e1fb84
[2020-09-01 11:08:32.240906] W [MSGID: 106061]
[glusterd-geo-rep.c:2388:glusterd_op_gsync_args_get] 0-management: master not found
[2020-09-01 11:18:32.510041] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(glusterd mgmt), op(--(4)), xid = 0x812c, unique = 0, s
ent = 2020-09-01 11:08:32.242199, timeout = 600 for 192.168.48.202:24007
[root@ov-no1 glusterfs]# cat events.log-20200828 | egrep "( W | E )" |more
[nothing in the output]
[root@ov-no1 glusterfs]# cat events.log | egrep "( W | E )" |more
[nothing in the output]
[root@ov-no1 glusterfs]# cat glfsheal-data.log | egrep "( W | E )" |more
[nothing in the output]
[root@ov-no1 glusterfs]# cat glfsheal-vmstore.log | egrep "( W | E )" |more
[nothing in the output]
[root@ov-no1 glusterfs]# cat glfsheal-engine.log | egrep "( W | E )" |more
[nothing in the output]
[root@ov-no1 glusterfs]# cat glfsheal-data.log-20200830 | egrep "( W | E )"
|more
[nothing in the output]
[root@ov-no1 glusterfs]# cat glfsheal-vmstore.log-20200830 | egrep "( W | E )"
|more
[nothing in the output]
[root@ov-no1 glusterfs]# cat glfsheal-engine.log-20200830 | egrep "( W | E )"
|more
[nothing in the output]
[root@ov-no1 glusterfs]# cat glustershd.log | egrep "( W | E )" |more
[nothing in the output]
[root@ov-no1 glusterfs]# cat
rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_engine.log | egrep "( W | E
)" |more
[nothing in the output]
[root@ov-no1 glusterfs]# cat
rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_data.log | egrep "( W | E
)" |more
[nothing in the output]
[root@ov-no1 glusterfs]# cat
rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_vmstore.log | egrep "( W | E
)" |more
[nothing in the output]
[root@ov-no1 glusterfs]# cat
rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_data.log
[2020-08-31 00:41:02.169424] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-08-31 00:41:02.170418] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-08-31 08:59:07.694673] I [MSGID: 133022] [shard.c:3673:shard_delete_shards]
0-data-shard: Deleted shards of gfid=d25c9bb6-4a25-4025-8652-686be0a4bdf1 from backend
[2020-08-31 08:59:07.890175] I [MSGID: 133017] [shard.c:6696:shard_seek] 0-data-shard:
seek called on 0e4c3fb8-7617-4e43-879a-cc784198216d. [Operation not supported]
[2020-08-31 08:59:07.892803] I [MSGID: 133017] [shard.c:6696:shard_seek] 0-data-shard:
seek called on 17c220e1-fa2c-42d7-a948-6ea2bf5a93d7. [Operation not supported]
[2020-08-31 08:59:07.897929] I [MSGID: 133017] [shard.c:6696:shard_seek] 0-data-shard:
seek called on f9315204-ec16-4c46-847e-d84f0169ea8f. [Operation not supported]
[2020-08-31 08:59:07.928488] I [MSGID: 133022] [shard.c:3673:shard_delete_shards]
0-data-shard: Deleted shards of gfid=17c220e1-fa2c-42d7-a948-6ea2bf5a93d7 from backend
[2020-09-01 00:36:01.667677] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-09-01 00:36:01.668986] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-09-02 00:48:02.278423] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-09-02 00:48:02.279637] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-09-04 00:36:01.546102] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-09-04 00:36:01.547101] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[root@ov-no1 glusterfs]# cat
rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_vmstore.log
[2020-08-28 00:12:02.424709] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-08-29 00:29:02.375537] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-08-29 00:29:02.377088] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-08-30 00:13:03.449758] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-08-30 00:13:03.451232] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-08-31 00:41:02.169556] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-08-31 00:41:02.171040] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-09-01 00:36:01.667768] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-09-01 00:36:01.669572] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-09-02 00:48:02.278490] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-09-02 00:48:02.280253] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-09-04 00:36:01.546134] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-09-04 00:36:01.547694] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[root@ov-no1 glusterfs]#
[root@ov-no1 glusterfs]# cat
rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_engine.log
[2020-08-30 00:13:03.447882] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-08-30 00:13:03.449756] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-08-31 00:41:02.167497] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-08-31 00:41:02.169892] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-09-01 00:36:01.666356] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-09-01 00:36:01.668276] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-09-02 00:48:02.277024] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-09-02 00:48:02.279273] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-09-04 00:36:01.543799] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-09-04 00:36:01.545779] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[root@ov-no1 glusterfs]# cat glustershd.log | egrep "( W | E )" |more
[root@ov-no1 glusterfs]# cat glustershd.log
[2020-08-30 20:31:26.884546] I [MSGID: 108026]
[afr-self-heal-common.c:1741:afr_log_selfheal] 0-engine-replicate-0: Completed data
selfheal on 3fafabf3-d0cd-4b9a-8dd7-43145451f7cf. sources=[1] 2 sinks=0
.
.
.
[every ten minutes the same log appears]
.
.
[2020-08-31 15:03:57.893224] I [MSGID: 108026]
[afr-self-heal-common.c:1741:afr_log_selfheal] 0-engine-replicate-0: Completed data
selfheal on 3fafabf3-d0cd-4b9a-8dd7-43145451f7cf. sources=[1] 2 sinks=0
The message "I [MSGID: 108026] [afr-self-heal-common.c:1741:afr_log_selfheal]
0-engine-replicate-0: Completed data selfheal on 3fafabf3-d0cd-4b9a-8dd7-43145451f7cf.
sources=[1] 2 sinks=0 " repeated 15 times between [2020-08-31 15:03:57.893224] and
[2020-08-31 15:04:42.605183]
[2020-08-31 15:14:46.171044] I [MSGID: 108026]
[afr-self-heal-common.c:1741:afr_log_selfheal] 0-engine-replicate-0: Completed data
selfheal on 3fafabf3-d0cd-4b9a-8dd7-43145451f7cf. sources=[1] 2 sinks=0
[2020-09-01 00:36:01.663267] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-09-01 00:36:01.665109] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[2020-09-02 00:48:02.274383] I [MSGID: 100011] [glusterfsd.c:1641:reincarnate]
0-glusterfsd: Fetching the volume file from server...
[2020-09-02 00:48:02.275562] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No
change in volfile,continuing
[root@ov-no1 glusterfs]# zcat cli.log-20200902.gz |egrep "( W | E )" |more
[2020-09-01 09:04:39.799900] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:04:39.801917] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:24:41.581140] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 09:39:43.395156] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:39:43.397147] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:49:44.128540] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 09:54:44.629071] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 10:04:45.642437] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 10:04:45.904006] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 10:04:45.905279] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 10:39:49.183655] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 10:44:49.968219] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 10:44:49.970487] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 10:49:50.206005] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 10:59:51.489242] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 10:59:51.491277] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[root@ov-no1 glusterfs]# cat cli.log |grep "( W | E )" |more
[nothing in the output]
[root@ov-no1 glusterfs]# cat cmd_history.log|egrep "( W| E|fail)" |more
[2020-09-01 00:58:38.265743] : volume geo-replication status : FAILED : Commit failed on
ov-no2.ariadne-t.local. Please check log file for details.
Commit failed on ov-no3.ariadne-t.local. Please check log file for details.
[2020-09-01 02:00:38.685251] : volume geo-replication status : FAILED : Commit failed on
ov-no2.ariadne-t.local. Please check log file for details.
Commit failed on ov-no3.ariadne-t.local. Please check log file for details.
[2020-09-01 03:02:39.094984] : volume geo-replication status : FAILED : Commit failed on
ov-no2.ariadne-t.local. Please check log file for details.
Commit failed on ov-no3.ariadne-t.local. Please check log file for details.
[2020-09-01 11:18:32.510224] : volume geo-replication status : FAILED : Commit failed on
ov-no2.ariadne-t.local. Please check log file for details.
Commit failed on ov-no3.ariadne-t.local. Please check log file for details.
[2020-09-01 14:24:33.778942] : volume geo-replication status : FAILED : Commit failed on
ov-no2.ariadne-t.local. Please check log file for details.
Commit failed on ov-no3.ariadne-t.local. Please check log file for details.
[2020-09-01 16:28:34.610703] : volume geo-replication status : FAILED : Commit failed on
ov-no2.ariadne-t.local. Please check log file for details.
Commit failed on ov-no3.ariadne-t.local. Please check log file for details.
NODE2:
[root@ov-no2 ~]# cd /var/log/glusterfs/
[root@ov-no2 glusterfs]# more bricks/gluster_bricks-data-data.log-20200830 |egrep "(
W | E )"|more
[2020-08-27 15:35:07.532064] W [glusterfsd.c:1570:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dd5) [0x7f01ef674dd5] -->/usr/sbin/glusterfsd(glus
terfs_sigwaiter+0xe5) [0x55d96c9791b5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b)
[0x55d96c97901b] ) 0-: received signum (15), shutting down
[2020-08-27 15:35:07.545460] W [rpc-clnt.c:1704:rpc_clnt_submit] 0-glusterfs: failed to
submit rpc-request (unique: 0, XID: 0x33 Program: Gluster Por
tmap, ProgVers: 1, Proc: 5) to rpc-transport (glusterfs)
[2020-08-27 15:35:07.545626] E [timer.c:34:gf_timer_call_after]
(-->/usr/sbin/glusterfsd(mgmt_submit_request+0x13d) [0x55d96c97ecdd]
-->/lib64/libgfr
pc.so.0(rpc_clnt_submit+0x72d) [0x7f01f05dfc1d]
-->/lib64/libglusterfs.so.0(gf_timer_call_after+0x229) [0x7f01f0844349] ) 0-timer:
Either ctx is NULL
or ctx cleanup started [Invalid argument]
[2020-08-27 15:35:07.545724] E [timer.c:34:gf_timer_call_after]
(-->/usr/sbin/glusterfsd(mgmt_submit_request+0x13d) [0x55d96c97ecdd]
-->/lib64/libgfr
pc.so.0(rpc_clnt_submit+0x72d) [0x7f01f05dfc1d]
-->/lib64/libglusterfs.so.0(gf_timer_call_after+0x229) [0x7f01f0844349] ) 0-timer:
Either ctx is NULL
or ctx cleanup started [Invalid argument]
[2020-08-27 16:25:42.954545] W [inodelk.c:609:pl_inodelk_log_cleanup] 0-data-server:
releasing lock on 03848819-f37b-4fa7-81d9-3f5dc3eef2da held by {
client=0x7fd84400f350, pid=7725 lk-owner=e8100a64447f0000}
[root@ov-no2 glusterfs]#
[root@ov-no2 glusterfs]#
[root@ov-no2 glusterfs]#
[root@ov-no2 glusterfs]#
[root@ov-no2 glusterfs]# more bricks/gluster_bricks-engine-engine.log-20200830 |egrep
"( W | E )"|more
[2020-08-27 15:35:08.532812] W [glusterfsd.c:1570:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dd5) [0x7ff67fa1edd5] -->/usr/sbin/glusterfsd(glus
terfs_sigwaiter+0xe5) [0x55d4f15951b5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b)
[0x55d4f159501b] ) 0-: received signum (15), shutting down
[2020-08-27 15:35:08.536715] W [rpc-clnt.c:1704:rpc_clnt_submit] 0-glusterfs: failed to
submit rpc-request (unique: 0, XID: 0x33 Program: Gluster Por
tmap, ProgVers: 1, Proc: 5) to rpc-transport (glusterfs)
[2020-08-27 15:35:08.536903] E [timer.c:34:gf_timer_call_after]
(-->/usr/sbin/glusterfsd(mgmt_submit_request+0x13d) [0x55d4f159acdd]
-->/lib64/libgfr
pc.so.0(rpc_clnt_submit+0x72d) [0x7ff680989c1d]
-->/lib64/libglusterfs.so.0(gf_timer_call_after+0x229) [0x7ff680bee349] ) 0-timer:
Either ctx is NULL
or ctx cleanup started [Invalid argument]
[2020-08-27 15:35:08.537018] E [timer.c:34:gf_timer_call_after]
(-->/usr/sbin/glusterfsd(mgmt_submit_request+0x13d) [0x55d4f159acdd]
-->/lib64/libgfr
pc.so.0(rpc_clnt_submit+0x72d) [0x7ff680989c1d]
-->/lib64/libglusterfs.so.0(gf_timer_call_after+0x229) [0x7ff680bee349] ) 0-timer:
Either ctx is NULL
or ctx cleanup started [Invalid argument]
[root@ov-no2 glusterfs]#
[root@ov-no2 glusterfs]#
[root@ov-no2 glusterfs]#
[root@ov-no2 glusterfs]#
[root@ov-no2 glusterfs]#
[root@ov-no2 glusterfs]# more bricks/gluster_bricks-vmstore-vmstore.log-20200830 |egrep
"( W | E )"|more
[2020-08-27 15:35:09.533389] W [glusterfsd.c:1570:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dd5) [0x7f2d8f5f8dd5] -->/usr/sbin/glusterfsd(glus
terfs_sigwaiter+0xe5) [0x563cbb1081b5] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b)
[0x563cbb10801b] ) 0-: received signum (15), shutting down
[2020-08-27 15:35:09.540492] W [rpc-clnt.c:1704:rpc_clnt_submit] 0-glusterfs: failed to
submit rpc-request (unique: 0, XID: 0x33 Program: Gluster Por
tmap, ProgVers: 1, Proc: 5) to rpc-transport (glusterfs)
[2020-08-27 15:35:09.540657] E [timer.c:34:gf_timer_call_after]
(-->/usr/sbin/glusterfsd(mgmt_submit_request+0x13d) [0x563cbb10dcdd]
-->/lib64/libgfr
pc.so.0(rpc_clnt_submit+0x72d) [0x7f2d90563c1d]
-->/lib64/libglusterfs.so.0(gf_timer_call_after+0x229) [0x7f2d907c8349] ) 0-timer:
Either ctx is NULL
or ctx cleanup started [Invalid argument]
[2020-08-27 15:35:09.540762] E [timer.c:34:gf_timer_call_after]
(-->/usr/sbin/glusterfsd(mgmt_submit_request+0x13d) [0x563cbb10dcdd]
-->/lib64/libgfr
pc.so.0(rpc_clnt_submit+0x72d) [0x7f2d90563c1d]
-->/lib64/libglusterfs.so.0(gf_timer_call_after+0x229) [0x7f2d907c8349] ) 0-timer:
Either ctx is NULL
or ctx cleanup started [Invalid argument]
[root@ov-no2 glusterfs]# more bricks/gluster_bricks-data-data.log |egrep "( W | E
)"|more
[root@ov-no2 glusterfs]# more bricks/gluster_bricks-vmstore-vmstore.log |egrep "( W |
E )"|more
[root@ov-no2 glusterfs]# more bricks/gluster_bricks-engine-engine.log |egrep "( W | E
)"|more
[root@ov-no2 glusterfs]# more cmd_history.log | egrep "( W | E )" |more
[nothing in the output]
[root@ov-no2 glusterfs]# cat glusterd.log | egrep "( W | E )" |more
[2020-09-01 07:00:31.325533] W [MSGID: 106061]
[glusterd-geo-rep.c:2388:glusterd_op_gsync_args_get] 0-management: master not found
[2020-09-01 07:10:36.677633] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(glusterd mgmt), op(--(4)), xid = 0x7f74, unique = 0, s
ent = 2020-09-01 07:00:31.326798, timeout = 600 for 192.168.48.203:24007
[2020-09-01 07:10:36.677697] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors]
0-glusterd: Commit failed on ov-no3.ariadne-t.local. Please
check log file for details.
[2020-09-01 07:10:40.678204] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(glusterd mgmt), op(--(4)), xid = 0x804d, unique = 0, s
ent = 2020-09-01 07:00:31.326774, timeout = 600 for 192.168.48.201:24007
[2020-09-01 07:10:40.678259] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors]
0-glusterd: Commit failed on ov-no1.ariadne-t.local. Please
check log file for details.
[2020-09-01 07:10:40.678395] E [rpcsvc.c:1577:rpcsvc_submit_generic] 0-rpc-service: failed
to submit message (XID: 0x2, Program: GlusterD svc cli, Pr
ogVers: 2, Proc: 22) to rpc-transport (socket.management)
[2020-09-01 07:10:40.678414] E [MSGID: 106430]
[glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission failed
[2020-09-01 08:02:31.553170] E [glusterd-op-sm.c:8132:glusterd_op_sm]
(-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7efc4e68ea1e]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7efc4e6871be]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7efc
4e6ae06f] ) 0-management: Unable to get transaction opinfo for transaction ID
:930a8a08-1044-41cf-b921-913b982e0c72
[2020-09-01 09:04:31.784847] W [MSGID: 106061]
[glusterd-geo-rep.c:2388:glusterd_op_gsync_args_get] 0-management: master not found
[2020-09-01 09:14:31.577944] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(glusterd mgmt), op(--(4)), xid = 0x82ac, unique = 0, s
ent = 2020-09-01 09:04:31.786089, timeout = 600 for 192.168.48.201:24007
[2020-09-01 09:14:31.578010] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors]
0-glusterd: Commit failed on ov-no1.ariadne-t.local. Please
check log file for details.
[2020-09-01 09:14:37.578887] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(glusterd mgmt), op(--(4)), xid = 0x81d3, unique = 0, s
ent = 2020-09-01 09:04:31.786116, timeout = 600 for 192.168.48.203:24007
[2020-09-01 09:14:37.578945] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors]
0-glusterd: Commit failed on ov-no3.ariadne-t.local. Please
check log file for details.
[2020-09-01 09:14:37.579189] E [rpcsvc.c:1577:rpcsvc_submit_generic] 0-rpc-service: failed
to submit message (XID: 0x2, Program: GlusterD svc cli, Pr
ogVers: 2, Proc: 22) to rpc-transport (socket.management)
[2020-09-01 09:14:37.579210] E [MSGID: 106430]
[glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission failed
[2020-09-01 10:06:32.014210] W [MSGID: 106061]
[glusterd-geo-rep.c:2388:glusterd_op_gsync_args_get] 0-management: master not found
[2020-09-01 10:16:32.031372] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(glusterd mgmt), op(--(4)), xid = 0x83d7, unique = 0, s
ent = 2020-09-01 10:06:32.015520, timeout = 600 for 192.168.48.201:24007
[2020-09-01 10:16:32.031415] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors]
0-glusterd: Commit failed on ov-no1.ariadne-t.local. Please
check log file for details.
[2020-09-01 10:16:38.032229] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(glusterd mgmt), op(--(4)), xid = 0x82fe, unique = 0, s
ent = 2020-09-01 10:06:32.015546, timeout = 600 for 192.168.48.203:24007
[2020-09-01 10:16:38.032281] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors]
0-glusterd: Commit failed on ov-no3.ariadne-t.local. Please
check log file for details.
[2020-09-01 10:16:38.032491] E [rpcsvc.c:1577:rpcsvc_submit_generic] 0-rpc-service: failed
to submit message (XID: 0x2, Program: GlusterD svc cli, Pr
ogVers: 2, Proc: 22) to rpc-transport (socket.management)
[2020-09-01 10:16:38.032519] E [MSGID: 106430]
[glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission failed
[root@ov-no2 glusterfs]# cat
rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_engine.log | egrep "( W | E
)" |more
[nothing in the output]
[root@ov-no2 glusterfs]# cat
rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_data.log | egrep "( W | E
)" |more
[nothing in the output]
[root@ov-no2 glusterfs]# cat
rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_vmstore.log | egrep "( W | E
)" |more
[nothing in the output]
[root@ov-no2 glusterfs]# cat cli.log | egrep "( W | E )" |more
[2020-09-01 08:49:38.289905] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 08:49:38.291840] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:04:39.528219] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 09:09:40.049281] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 09:09:40.318414] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:09:40.319842] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:14:40.557140] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 09:19:41.064083] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 09:19:41.338070] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:19:41.340016] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:24:41.860173] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:24:41.862096] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:29:42.110479] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 09:29:42.380124] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:29:42.381580] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:34:42.628085] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 09:44:43.895269] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:44:43.897282] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:54:44.902854] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:54:44.904765] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[root@ov-no2 glusterfs]# cat cmd_history.log | egrep "( W | E |fail)" |more
[2020-09-01 04:04:39.342142] : volume geo-replication status : FAILED : Commit failed on
ov-no3.ariadne-t.local. Please check log file for details.
Commit failed on ov-no1.ariadne-t.local. Please check log file for details.
[2020-09-01 05:06:39.783310] : volume geo-replication status : FAILED : Commit failed on
ov-no3.ariadne-t.local. Please check log file for details.
Commit failed on ov-no1.ariadne-t.local. Please check log file for details.
[2020-09-01 07:10:40.678333] : volume geo-replication status : FAILED : Commit failed on
ov-no3.ariadne-t.local. Please check log file for details.
Commit failed on ov-no1.ariadne-t.local. Please check log file for details.
[2020-09-01 09:14:37.579120] : volume geo-replication status : FAILED : Commit failed on
ov-no1.ariadne-t.local. Please check log file for details.
Commit failed on ov-no3.ariadne-t.local. Please check log file for details.
[2020-09-01 10:16:38.032427] : volume geo-replication status : FAILED : Commit failed on
ov-no1.ariadne-t.local. Please check log file for details.
Commit failed on ov-no3.ariadne-t.local. Please check log file for details.
NODE3:
[root@ov-no3 glusterfs]# more bricks/gluster_bricks-engine-engine.log | egrep "( W |
E )" |more
[nothing in the output]
[root@ov-no3 glusterfs]# more bricks/gluster_bricks-vmstore-vmstore.log | egrep "( W
| E )" |more
[nothing in the output]
[root@ov-no3 glusterfs]# more bricks/gluster_bricks-data-data.log | egrep "( W | E
)" |more
[nothing in the output]
[root@ov-no3 glusterfs]# zcat cli.log |egrep "( W | E )" |more
[2020-09-01 07:29:30.165436] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 07:29:30.166677] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 07:34:30.666721] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 07:34:30.668801] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 07:44:31.431679] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 07:49:31.936985] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 07:54:32.447922] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 08:09:34.242293] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 08:09:34.244149] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 08:29:36.002322] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 08:34:36.773092] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 08:34:36.775476] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 08:44:37.512700] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 08:54:38.528131] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 08:54:38.797499] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 08:54:38.798780] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 08:59:39.030389] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 08:59:39.293311] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 08:59:39.294612] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:14:40.823633] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:14:40.825547] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:34:42.890897] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:34:42.893012] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:39:43.129137] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 09:44:43.629375] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 09:49:44.395649] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:49:44.397886] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:59:45.148111] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 09:59:45.409361] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 09:59:45.410701] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 10:14:46.647785] W [cli-rpc-ops.c:10753:gf_cli_generate_snapshot_event] 0-cli:
Cannot generate event for unknown type.
[2020-09-01 10:14:46.908152] E [cli-xml-output.c:5838:cli_xml_snapshot_begin_composite_op]
0-cli: Failed to get sub-cmd
[2020-09-01 10:14:46.909401] E [cli-xml-output.c:5884:cli_xml_snapshot_end_composite_op]
0-cli: Failed to get sub-cmd
[root@ov-no3 glusterfs]# more cmd_history.log | egrep "( W | E )" |more
[root@ov-no3 glusterfs]# cat glusterd.log | egrep "( W | E )" |more
[2020-09-01 08:02:31.550060] W [MSGID: 106061]
[glusterd-geo-rep.c:2388:glusterd_op_gsync_args_get] 0-management: master not found
[2020-09-01 08:12:31.582984] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(glusterd mgmt), op(--(4)), xid = 0x7fe0, unique = 0, s
ent = 2020-09-01 08:02:31.551222, timeout = 600 for 192.168.48.202:24007
[2020-09-01 08:12:31.583040] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors]
0-glusterd: Commit failed on ov-no2.ariadne-t.local. Please
check log file for details.
[2020-09-01 08:12:31.583086] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(glusterd mgmt), op(--(4)), xid = 0x7fd9, unique = 0, s
ent = 2020-09-01 08:02:31.551255, timeout = 600 for 192.168.48.201:24007
[2020-09-01 08:12:31.583100] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors]
0-glusterd: Commit failed on ov-no1.ariadne-t.local. Please
check log file for details.
[2020-09-01 08:12:31.583253] E [rpcsvc.c:1577:rpcsvc_submit_generic] 0-rpc-service: failed
to submit message (XID: 0x2, Program: GlusterD svc cli, Pr
ogVers: 2, Proc: 22) to rpc-transport (socket.management)
[2020-09-01 08:12:31.583269] E [MSGID: 106430]
[glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission failed
[2020-09-01 09:04:31.786592] E [glusterd-op-sm.c:8132:glusterd_op_sm]
(-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f56fecfba1e]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f56fecf41be]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f56
fed1b06f] ) 0-management: Unable to get transaction opinfo for transaction ID
:9942b579-5240-4fee-bb4c-78b9a1c98da8
[2020-09-01 10:06:32.014980] E [glusterd-op-sm.c:8132:glusterd_op_sm]
(-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f56fecfba1e]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f56fecf41be]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f56
fed1b06f] ) 0-management: Unable to get transaction opinfo for transaction ID
:324fc904-b147-4801-89d5-485120e1fb84
[2020-09-01 11:08:32.243189] E [glusterd-op-sm.c:8132:glusterd_op_sm]
(-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f56fecfba1e]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f56fecf41be]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f56
fed1b06f] ) 0-management: Unable to get transaction opinfo for transaction ID
:3b7b79b3-344c-4a67-b66e-c8fb8ddb2a2a
[2020-09-01 12:10:32.473993] E [glusterd-op-sm.c:8132:glusterd_op_sm]
(-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x23a1e) [0x7f56fecfba1e]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x1c1be) [0x7f56fecf41be]
-->/usr/lib64/glusterfs/6.4/xlator/mgmt/glusterd.so(+0x4306f) [0x7f56
fed1b06f] ) 0-management: Unable to get transaction opinfo for transaction ID
:2521f708-6bc4-4be4-b37f-41d67839e941
[2020-09-01 13:12:32.707217] W [MSGID: 106061]
[glusterd-geo-rep.c:2388:glusterd_op_gsync_args_get] 0-management: master not found
[2020-09-01 13:22:33.887995] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(glusterd mgmt), op(--(4)), xid = 0x866e, unique = 0, s
ent = 2020-09-01 13:12:32.708475, timeout = 600 for 192.168.48.202:24007
[2020-09-01 13:22:33.888057] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors]
0-glusterd: Commit failed on ov-no2.ariadne-t.local. Please
check log file for details.
[2020-09-01 13:22:33.888095] E [rpc-clnt.c:183:call_bail] 0-management: bailing out frame
type(glusterd mgmt), op(--(4)), xid = 0x8667, unique = 0, s
ent = 2020-09-01 13:12:32.708500, timeout = 600 for 192.168.48.201:24007
[2020-09-01 13:22:33.888113] E [MSGID: 106152] [glusterd-syncop.c:104:gd_collate_errors]
0-glusterd: Commit failed on ov-no1.ariadne-t.local. Please
check log file for details.
[2020-09-01 13:22:33.888234] E [rpcsvc.c:1577:rpcsvc_submit_generic] 0-rpc-service: failed
to submit message (XID: 0x2, Program: GlusterD svc cli, Pr
ogVers: 2, Proc: 22) to rpc-transport (socket.management)
[2020-09-01 13:22:33.888269] E [MSGID: 106430]
[glusterd-utils.c:558:glusterd_submit_reply] 0-glusterd: Reply submission failed
[root@ov-no3 glusterfs]# cat events.log-20200828 | egrep "( W | E )" |more
[nothing in the output]
[root@ov-no3 glusterfs]# cat glfsheal-data.log | egrep "( W | E )" |more
[nothing in the output]
[root@ov-no3 glusterfs]# cat glfsheal-vmstore.log | egrep "( W | E )" |more
[nothing in the output]
[root@ov-no3 glusterfs]# cat glfsheal-engine.log | egrep "( W | E )" |more
[nothing in the output]
[root@ov-no3 glusterfs]# cat glustershd.log | egrep "( W | E )" |more
[nothing in the output]
[root@ov-no3 glusterfs]# cat
rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_engine.log | egrep "( W | E
)" |more
[2020-08-31 14:49:13.805467] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2]
0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf)
remote_fd is -1. EBADFD [File descriptor in bad state]
[2020-08-31 14:51:13.882683] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2]
0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf)
remote_fd is -1. EBADFD [File descriptor in bad state]
[2020-08-31 14:53:12.691884] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2]
0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf)
remote_fd is -1. EBADFD [File descriptor in bad state]
[2020-08-31 14:55:12.713320] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2]
0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf)
remote_fd is -1. EBADFD [File descriptor in bad state]
[2020-08-31 14:57:13.146570] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2]
0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf)
remote_fd is -1. EBADFD [File descriptor in bad state]
[2020-08-31 14:59:13.502830] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2]
0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf)
remote_fd is -1. EBADFD [File descriptor in bad state]
[2020-08-31 15:01:13.039274] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2]
0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf)
remote_fd is -1. EBADFD [File descriptor in bad state]
[2020-08-31 15:03:13.401571] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2]
0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf)
remote_fd is -1. EBADFD [File descriptor in bad state]
[2020-08-31 15:05:13.764951] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2]
0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf)
remote_fd is -1. EBADFD [File descriptor in bad state]
[2020-08-31 15:07:14.140156] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2]
0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf)
remote_fd is -1. EBADFD [File descriptor in bad state]
[2020-08-31 15:09:13.061657] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2]
0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf)
remote_fd is -1. EBADFD [File descriptor in bad state]
[2020-08-31 15:11:14.616511] W [MSGID: 114061] [client-common.c:2644:client_pre_fsync_v2]
0-engine-client-0: (3fafabf3-d0cd-4b9a-8dd7-43145451f7cf)
remote_fd is -1. EBADFD [File descriptor in bad state]
[root@ov-no3 glusterfs]# cat
rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_data.log | egrep "( W | E
)" |more
[nothing recent in the output]
[root@ov-no3 glusterfs]# cat
rhev-data-center-mnt-glusterSD-ov-no1.ariadne-t.local\:_vmstore.log | egrep "( W | E
)" |more
[nothing recent in the output]
[root@ov-no3 glusterfs]# cat glustershd.log | egrep "( W | E )" |more
[nothing in the output]
[root@ov-no3 glusterfs]# cat events.log | egrep "( W | E )" |more
[nothing in the output]
Journalctl results on the nodes about the gluster service:
NODE1:
[root@ov-no1 ~]# journalctl --unit glusterd -l
-- Logs begin at Thu 2020-08-27 19:30:14 EEST, end at Fri 2020-09-04 16:34:42 EEST. --
Aug 27 19:30:52 ov-no1.ariadne-t.local systemd[1]: Starting GlusterFS, a clustered
file-system server...
Aug 27 19:30:56 ov-no1.ariadne-t.local systemd[1]: Started GlusterFS, a clustered
file-system server.
Aug 27 19:30:56 ov-no1.ariadne-t.local glusterd[5591]: [2020-08-27 16:30:56.544861] C
[MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum
Aug 27 19:30:56 ov-no1.ariadne-t.local glusterd[5591]: [2020-08-27 16:30:56.558812] C
[MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum
Aug 27 19:30:56 ov-no1.ariadne-t.local glusterd[5591]: [2020-08-27 16:30:56.571473] C
[MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum
NODE2:
[root@ov-no2 ~]# journalctl --unit glusterd -l
-- Logs begin at Wed 2019-09-11 17:40:13 EEST, end at Fri 2020-09-04 16:34:27 EEST. --
Aug 27 18:35:01 ov-no2.ariadne-t.local systemd[1]: Stopping GlusterFS, a clustered
file-system server...
Aug 27 18:35:01 ov-no2.ariadne-t.local systemd[1]: Stopped GlusterFS, a clustered
file-system server.
Aug 27 18:35:01 ov-no2.ariadne-t.local systemd[1]: Starting GlusterFS, a clustered
file-system server...
Aug 27 18:35:05 ov-no2.ariadne-t.local systemd[1]: Started GlusterFS, a clustered
file-system server.
Aug 27 18:35:07 ov-no2.ariadne-t.local glusterd[28722]: [2020-08-27 15:35:07.519155] C
[MSGID: 106002] [glusterd-server-quorum.c:355:glusterd_do_volu
Aug 27 18:35:08 ov-no2.ariadne-t.local glusterd[28722]: [2020-08-27 15:35:08.532018] C
[MSGID: 106002] [glusterd-server-quorum.c:355:glusterd_do_volu
Aug 27 18:35:09 ov-no2.ariadne-t.local glusterd[28722]: [2020-08-27 15:35:09.532867] C
[MSGID: 106002] [glusterd-server-quorum.c:355:glusterd_do_volu
Aug 27 18:35:10 ov-no2.ariadne-t.local glusterd[28722]: [2020-08-27 15:35:10.535976] C
[MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volu
Aug 27 18:35:10 ov-no2.ariadne-t.local glusterd[28722]: [2020-08-27 15:35:10.550171] C
[MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volu
Aug 27 18:35:10 ov-no2.ariadne-t.local glusterd[28722]: [2020-08-27 15:35:10.563481] C
[MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volu
Aug 27 19:25:23 ov-no2.ariadne-t.local glustershd[22187]: [2020-08-27 16:25:23.640530] C
[rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 0-data-cli
NODE3:
[root@ov-no3 glusterfs]# journalctl --unit glusterd -l
-- Logs begin at Thu 2020-08-27 19:09:30 EEST, end at Fri 2020-09-04 16:34:02 EEST. --
Aug 27 19:10:08 ov-no3.ariadne-t.local systemd[1]: Starting GlusterFS, a clustered
file-system server...
Aug 27 19:10:13 ov-no3.ariadne-t.local systemd[1]: Started GlusterFS, a clustered
file-system server.
Aug 27 19:10:13 ov-no3.ariadne-t.local glusterd[5233]: [2020-08-27 16:10:13.274001] C
[MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum
Aug 27 19:10:13 ov-no3.ariadne-t.local glusterd[5233]: [2020-08-27 16:10:13.288391] C
[MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum
Aug 27 19:10:13 ov-no3.ariadne-t.local glusterd[5233]: [2020-08-27 16:10:13.300866] C
[MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum
Aug 27 19:14:38 ov-no3.ariadne-t.local systemd[1]: Stopping GlusterFS, a clustered
file-system server...
Aug 27 19:14:38 ov-no3.ariadne-t.local systemd[1]: Stopped GlusterFS, a clustered
file-system server.
Aug 27 19:14:38 ov-no3.ariadne-t.local systemd[1]: Starting GlusterFS, a clustered
file-system server...
Aug 27 19:14:41 ov-no3.ariadne-t.local systemd[1]: Started GlusterFS, a clustered
file-system server.
Aug 27 19:14:48 ov-no3.ariadne-t.local glusterd[9843]: [2020-08-27 16:14:48.117530] C
[MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum
Aug 27 19:14:48 ov-no3.ariadne-t.local glusterd[9843]: [2020-08-27 16:14:48.131054] C
[MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum
Aug 27 19:14:48 ov-no3.ariadne-t.local glusterd[9843]: [2020-08-27 16:14:48.141323] C
[MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volum
Aug 27 19:25:25 ov-no3.ariadne-t.local glustershd[10081]: [2020-08-27 16:25:25.286189] C
[rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 0-data-cli
Thanks for your help,
Maria Souvalioti
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/PZ7SUT7FY6P...
3:53 p.m.
Hi Maria, My system experiencing a similar issue as you have captured. Can you tell me
where I can find the HostedEngine.xml to confirm the virtio disk associated with my hosted
engine error?
910
days inactive
1542
days old
6 comments
4 participants
participants (4)
-
souvaliotimaria＠mail.com -
Strahil Nikolov -
travisq＠gmail.com -
Yedidyah Bar David