1:33 p.m.
I have a 4 node setup (centos 7) with hosted engine on glusterfs (replica 3 arbiter1).
Gluster fs is like this
ohost01 104G brick (real data)
ohost02 104g brick (real data)
ohost04 104g brick (arbiter)
ohost05 104g partition used as nfs-storage.
hosted engine is on gluster. I also have an fc domain of 3,6 TB
mount of gluster is like this
storage=172.16.224.10:/engine
mnt_options=backup-volfile-servers=172.16.224.11:172.16.224.13
172.16.224.10 is ohost01 storage network
172.16.224.12 is ohost02 storage network
172.16.224.13 is ohost04 storage network
Today I upgraded all nodes. I did it like this:
hosted-engine was running on ohost05 at the time
Put ohost04 (arbiter) on maintenance and did upgrade (ok)
Same with ohost02
Ohost01 was spm so I did ohost05 spm then put ohost01 on maintenance and then upgraded it.
I notticed that engine VM paused during the process (which usualy does not happen) as I
have backup-volfile-servers mount option. But today I notticed that this option is
ignored. On the hosts I also noted that mount is like this
172.16.224.10:/engine on /rhev/data-center/mnt/glusterSD/172.16.224.10:_engine type
fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
so reduduncy is gone from gluster and I cannot figure out why.
If I restart ohost01 (after maintenance) hosted engine get paused until ohost01 comes back
up.
How can I solve this issue
2:42 p.m.
it seems that reduduncy of glusterfs is working. It doesn't show on mount options but
it is there in the processes. This must be something else that caused the engine to pause.
So ignore this. Is there a way to debug why the hosted engine paused?
2:49 p.m.
Can you provide the engine mount logs under
/var/log/glusterfs/rhev_data-center*engine.log and also the
ovirt-ha/agent.log?
On Mon, Jun 18, 2018 at 8:42 AM, <g.vasilopoulos(a)uoc.gr> wrote:
it seems that reduduncy of glusterfs is working. It doesn't show
on mount
options but it is there in the processes. This must be something else that
caused the engine to pause. So ignore this. Is there a way to debug why the
hosted engine paused?
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-
guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/
message/7ATGYEUZG3YZR2BMYTEMCZZ267QQ4LCW/
4:39 p.m.
from glusterfs/rhev_datacenter
[2018-06-18 12:32:50.854668] W [socket.c:593:__socket_rwv] 0-glusterfs: readv on
172.16.224.10:24007 failed (No data available)
[2018-06-18 12:33:38.194322] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired]
0-engine-client-0: server 172.16.224.10:49152 has not responded in the last 30 seconds,
disconnecting.
[2018-06-18 12:33:38.195797] E [rpc-clnt.c:350:saved_frames_unwind] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710] (-->
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] ))))) 0-engine-client-0:
forced unwinding frame type(GlusterFS 3.3) op(STATFS(14)) called at 2018-06-18
12:33:07.800145 (xid=0x21be)
[2018-06-18 12:33:38.195824] W [MSGID: 114031]
[client-rpc-fops.c:777:client3_3_statfs_cbk] 0-engine-client-0: remote operation failed
[Transport endpoint is not connected]
[2018-06-18 12:33:38.196036] E [rpc-clnt.c:350:saved_frames_unwind] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710] (-->
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] ))))) 0-engine-client-0:
forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2018-06-18 12:33:07.800158
(xid=0x21bf)
[2018-06-18 12:33:38.196057] W [rpc-clnt-ping.c:223:rpc_clnt_ping_cbk] 0-engine-client-0:
socket disconnected
[2018-06-18 12:33:38.196224] E [rpc-clnt.c:350:saved_frames_unwind] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710] (-->
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] ))))) 0-engine-client-0:
forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-06-18
12:33:08.307804 (xid=0x21c0)
[2018-06-18 12:33:38.196243] W [MSGID: 114031]
[client-rpc-fops.c:2922:client3_3_readv_cbk] 0-engine-client-0: remote operation failed
[Transport endpoint is not connected]
[2018-06-18 12:33:38.196279] W [MSGID: 114061] [client-common.c:704:client_pre_fstat]
0-engine-client-0: (6f81ab66-8991-45a2-9576-5c7defdec302) remote_fd is -1. EBADFD [File
descriptor in bad state]
[2018-06-18 12:33:38.196545] E [rpc-clnt.c:350:saved_frames_unwind] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710] (-->
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] ))))) 0-engine-client-0:
forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-06-18
12:33:08.307812 (xid=0x21c4)
last part get repeated a lot of times.
2018-06-18 12:33:38.199480] W [MSGID: 114061] [client-common.c:704:client_pre_fstat]
0-engine-client-0: (2bddf734-3baa-439f-90e9-93b483260e01) remote_fd is -1. EBADFD [File
descriptor in bad state]
[2018-06-18 12:33:38.199721] E [rpc-clnt.c:350:saved_frames_unwind] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710] (-->
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] ))))) 0-engine-client-0:
forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-06-18
12:33:16.836613 (xid=0x21cb)
this part also get repeated sometimes
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710] (-->
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] ))))) 0-engine-client-0:
forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-06-18
12:33:37.337567 (xid=0x21db)
The message "W [MSGID: 114031]
[client-rpc-fops.c:2922:client3_3_readv_cbk] 0-engine-client-0: remote operation failed
[Transport endpoint is not connected]" repeated 8 times between [2018-06-18
12:33:38.202531] and [2018-06-18 12:33:38.205070]
[2018-06-18 12:33:38.205097] W [MSGID: 114061] [client-common.c:704:client_pre_fstat]
0-engine-client-0: (2bddf734-3baa-439f-90e9-93b483260e01) remote_fd is -1. EBADFD [File
descriptor in bad state]
[2018-06-18 12:33:38.205461] E [rpc-clnt.c:350:saved_frames_unwind] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710] (-->
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] ))))) 0-engine-client-0:
forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-06-18
12:33:37.337596 (xid=0x21dc)
[2018-06-18 12:33:38.208554] W [MSGID: 114031]
[client-rpc-fops.c:2922:client3_3_readv_cbk] 0-engine-client-0: remote operation failed
[Transport endpoint is not connected]
[2018-06-18 12:33:38.208590] W [MSGID: 114061] [client-common.c:704:client_pre_fstat]
0-engine-client-0: (2bddf734-3baa-439f-90e9-93b483260e01) remote_fd is -1. EBADFD [File
descriptor in bad state]
[2018-06-18 12:34:07.234501] E [socket.c:2369:socket_connect_finish] 0-engine-client-0:
connection to 172.16.224.10:24007 failed (No route to host); disconnecting socket
[2018-06-18 12:39:25.760451] E [MSGID: 114058]
[client-handshake.c:1565:client_query_portmap_cbk] 0-engine-client-0: failed to get the
port number for remote subvolume. Please run 'gluster volume status' on server to
see if brick process is running.
----------------------------------------
that's what's on gluster log on the host that engine was running . (Do you use
some external site to embeed logs ? sorry I'm new on the list)
agent.log is prety big so I 'll try to find some pastebin like site to link it
10:44 a.m.
Adding Krutika to look at gluster logs
On Mon, Jun 18, 2018 at 10:39 AM, <g.vasilopoulos(a)uoc.gr> wrote:
from glusterfs/rhev_datacenter
[2018-06-18 12:32:50.854668] W [socket.c:593:__socket_rwv] 0-glusterfs:
readv on 172.16.224.10:24007 failed (No data available)
[2018-06-18 12:33:38.194322] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired]
0-engine-client-0: server 172.16.224.10:49152 has not responded in the
last 30 seconds, disconnecting.
[2018-06-18 12:33:38.195797] E [rpc-clnt.c:350:saved_frames_unwind] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] )))))
0-engine-client-0: forced unwinding frame type(GlusterFS 3.3)
op(STATFS(14)) called at 2018-06-18 12:33:07.800145 (xid=0x21be)
[2018-06-18 12:33:38.195824] W [MSGID: 114031]
[client-rpc-fops.c:777:client3_3_statfs_cbk]
0-engine-client-0: remote operation failed [Transport endpoint is not
connected]
[2018-06-18 12:33:38.196036] E [rpc-clnt.c:350:saved_frames_unwind] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] )))))
0-engine-client-0: forced unwinding frame type(GF-DUMP) op(NULL(2)) called
at 2018-06-18 12:33:07.800158 (xid=0x21bf)
[2018-06-18 12:33:38.196057] W [rpc-clnt-ping.c:223:rpc_clnt_ping_cbk]
0-engine-client-0: socket disconnected
[2018-06-18 12:33:38.196224] E [rpc-clnt.c:350:saved_frames_unwind] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] )))))
0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12))
called at 2018-06-18 12:33:08.307804 (xid=0x21c0)
[2018-06-18 12:33:38.196243] W [MSGID: 114031]
[client-rpc-fops.c:2922:client3_3_readv_cbk]
0-engine-client-0: remote operation failed [Transport endpoint is not
connected]
[2018-06-18 12:33:38.196279] W [MSGID: 114061] [client-common.c:704:client_pre_fstat]
0-engine-client-0: (6f81ab66-8991-45a2-9576-5c7defdec302) remote_fd is
-1. EBADFD [File descriptor in bad state]
[2018-06-18 12:33:38.196545] E [rpc-clnt.c:350:saved_frames_unwind] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] )))))
0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12))
called at 2018-06-18 12:33:08.307812 (xid=0x21c4)
last part get repeated a lot of times.
2018-06-18 12:33:38.199480] W [MSGID: 114061] [client-common.c:704:client_pre_fstat]
0-engine-client-0: (2bddf734-3baa-439f-90e9-93b483260e01) remote_fd is
-1. EBADFD [File descriptor in bad state]
[2018-06-18 12:33:38.199721] E [rpc-clnt.c:350:saved_frames_unwind] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] )))))
0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12))
called at 2018-06-18 12:33:16.836613 (xid=0x21cb)
this part also get repeated sometimes
> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] )))))
0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12))
called at 2018-06-18 12:33:37.337567 (xid=0x21db)
The message "W [MSGID: 114031] [client-rpc-fops.c:2922:client3_3_readv_cbk]
0-engine-client-0: remote operation failed [Transport endpoint is not
connected]" repeated 8 times between [2018-06-18 12:33:38.202531] and
[2018-06-18 12:33:38.205070]
[2018-06-18 12:33:38.205097] W [MSGID: 114061] [client-common.c:704:client_pre_fstat]
0-engine-client-0: (2bddf734-3baa-439f-90e9-93b483260e01) remote_fd is
-1. EBADFD [File descriptor in bad state]
[2018-06-18 12:33:38.205461] E [rpc-clnt.c:350:saved_frames_unwind] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] )))))
0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12))
called at 2018-06-18 12:33:37.337596 (xid=0x21dc)
[2018-06-18 12:33:38.208554] W [MSGID: 114031]
[client-rpc-fops.c:2922:client3_3_readv_cbk]
0-engine-client-0: remote operation failed [Transport endpoint is not
connected]
[2018-06-18 12:33:38.208590] W [MSGID: 114061] [client-common.c:704:client_pre_fstat]
0-engine-client-0: (2bddf734-3baa-439f-90e9-93b483260e01) remote_fd is
-1. EBADFD [File descriptor in bad state]
[2018-06-18 12:34:07.234501] E [socket.c:2369:socket_connect_finish]
0-engine-client-0: connection to 172.16.224.10:24007 failed (No route to
host); disconnecting socket
[2018-06-18 12:39:25.760451] E [MSGID: 114058]
[client-handshake.c:1565:client_query_portmap_cbk]
0-engine-client-0: failed to get the port number for remote subvolume.
Please run 'gluster volume status' on server to see if brick process is
running.
----------------------------------------
that's what's on gluster log on the host that engine was running . (Do you
use some external site to embeed logs ? sorry I'm new on the list)
agent.log is prety big so I 'll try to find some pastebin like site to
link it
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-
guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/
message/QKNGD7TZT2UVAFG6H6HCPDIJQ6ROAX74/
2159
days inactive
2160
days old
6 comments
2 participants
participants (2)
-
g.vasilopoulos＠uoc.gr -
Sahina Bose