[ovirt-users] Re: glusterfs backup-volfile-servers lost

from glusterfs/rhev_datacenter [2018-06-18 12:32:50.854668] W [socket.c:593:__socket_rwv] 0-glusterfs: readv on 172.16.224.10:24007 failed (No data available) [2018-06-18 12:33:38.194322] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-engine-client-0: server 172.16.224.10:49152 has not responded in the last 30 seconds, disconnecting. [2018-06-18 12:33:38.195797] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] ))))) 0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(STATFS(14)) called at 2018-06-18 12:33:07.800145 (xid=0x21be) [2018-06-18 12:33:38.195824] W [MSGID: 114031] [client-rpc-fops.c:777:client3_3_statfs_cbk] 0-engine-client-0: remote operation failed [Transport endpoint is not connected] [2018-06-18 12:33:38.196036] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] ))))) 0-engine-client-0: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2018-06-18 12:33:07.800158 (xid=0x21bf) [2018-06-18 12:33:38.196057] W [rpc-clnt-ping.c:223:rpc_clnt_ping_cbk] 0-engine-client-0: socket disconnected [2018-06-18 12:33:38.196224] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] ))))) 0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-06-18 12:33:08.307804 (xid=0x21c0) [2018-06-18 12:33:38.196243] W [MSGID: 114031] [client-rpc-fops.c:2922:client3_3_readv_cbk] 0-engine-client-0: remote operation failed [Transport endpoint is not connected] [2018-06-18 12:33:38.196279] W [MSGID: 114061] [client-common.c:704:client_pre_fstat] 0-engine-client-0: (6f81ab66-8991-45a2-9576-5c7defdec302) remote_fd is -1. EBADFD [File descriptor in bad state] [2018-06-18 12:33:38.196545] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] ))))) 0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-06-18 12:33:08.307812 (xid=0x21c4) last part get repeated a lot of times. 2018-06-18 12:33:38.199480] W [MSGID: 114061] [client-common.c:704:client_pre_fstat] 0-engine-client-0: (2bddf734-3baa-439f-90e9-93b483260e01) remote_fd is -1. EBADFD [File descriptor in bad state] [2018-06-18 12:33:38.199721] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] ))))) 0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-06-18 12:33:16.836613 (xid=0x21cb) this part also get repeated sometimes The message "W [MSGID: 114031] [client-rpc-fops.c:2922:client3_3_readv_cbk] 0-engine-client-0: remote operation failed [Transport endpoint is not connected]" repeated 8 times between [2018-06-18 12:33:38.202531] and [2018-06-18 12:33:38.205070] [2018-06-18 12:33:38.205097] W [MSGID: 114061] [client-common.c:704:client_pre_fstat] 0-engine-client-0: (2bddf734-3baa-439f-90e9-93b483260e01) remote_fd is -1. EBADFD [File descriptor in bad state] [2018-06-18 12:33:38.205461] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] ))))) 0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-06-18 12:33:37.337596 (xid=0x21dc) [2018-06-18 12:33:38.208554] W [MSGID: 114031] [client-rpc-fops.c:2922:client3_3_readv_cbk] 0-engine-client-0: remote operation failed [Transport endpoint is not connected] [2018-06-18 12:33:38.208590] W [MSGID: 114061] [client-common.c:704:client_pre_fstat] 0-engine-client-0: (2bddf734-3baa-439f-90e9-93b483260e01) remote_fd is -1. EBADFD [File descriptor in bad state] [2018-06-18 12:34:07.234501] E [socket.c:2369:socket_connect_finish] 0-engine-client-0: connection to 172.16.224.10:24007 failed (No route to host); disconnecting socket [2018-06-18 12:39:25.760451] E [MSGID: 114058] [client-handshake.c:1565:client_query_portmap_cbk] 0-engine-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. ---------------------------------------- that's what's on gluster log on the host that engine was running . (Do you use some external site to embeed logs ? sorry I'm new on the list) agent.log is prety big so I 'll try to find some pastebin like site to link it