Adding Krutika to look at gluster logs

On Mon, Jun 18, 2018 at 10:39 AM, <g.vasilopoulos@uoc.gr> wrote:
from glusterfs/rhev_datacenter
[2018-06-18 12:32:50.854668] W [socket.c:593:__socket_rwv] 0-glusterfs: readv on 172.16.224.10:24007 failed (No data available)
[2018-06-18 12:33:38.194322] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-engine-client-0: server 172.16.224.10:49152 has not responded in the last 30 seconds, disconnecting.
[2018-06-18 12:33:38.195797] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] ))))) 0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(STATFS(14)) called at 2018-06-18 12:33:07.800145 (xid=0x21be)
[2018-06-18 12:33:38.195824] W [MSGID: 114031] [client-rpc-fops.c:777:client3_3_statfs_cbk] 0-engine-client-0: remote operation failed [Transport endpoint is not connected]
[2018-06-18 12:33:38.196036] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] ))))) 0-engine-client-0: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2018-06-18 12:33:07.800158 (xid=0x21bf)
[2018-06-18 12:33:38.196057] W [rpc-clnt-ping.c:223:rpc_clnt_ping_cbk] 0-engine-client-0: socket disconnected
[2018-06-18 12:33:38.196224] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] ))))) 0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-06-18 12:33:08.307804 (xid=0x21c0)
[2018-06-18 12:33:38.196243] W [MSGID: 114031] [client-rpc-fops.c:2922:client3_3_readv_cbk] 0-engine-client-0: remote operation failed [Transport endpoint is not connected]
[2018-06-18 12:33:38.196279] W [MSGID: 114061] [client-common.c:704:client_pre_fstat] 0-engine-client-0:  (6f81ab66-8991-45a2-9576-5c7defdec302) remote_fd is -1. EBADFD [File descriptor in bad state]
[2018-06-18 12:33:38.196545] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] ))))) 0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-06-18 12:33:08.307812 (xid=0x21c4)

last part get repeated a lot of times.

2018-06-18 12:33:38.199480] W [MSGID: 114061] [client-common.c:704:client_pre_fstat] 0-engine-client-0:  (2bddf734-3baa-439f-90e9-93b483260e01) remote_fd is -1. EBADFD [File descriptor in bad state]
[2018-06-18 12:33:38.199721] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] ))))) 0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-06-18 12:33:16.836613 (xid=0x21cb)

this part also get repeated sometimes

> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] ))))) 0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-06-18 12:33:37.337567 (xid=0x21db)
The message "W [MSGID: 114031] [client-rpc-fops.c:2922:client3_3_readv_cbk] 0-engine-client-0: remote operation failed [Transport endpoint is not connected]" repeated 8 times between [2018-06-18 12:33:38.202531] and [2018-06-18 12:33:38.205070]
[2018-06-18 12:33:38.205097] W [MSGID: 114061] [client-common.c:704:client_pre_fstat] 0-engine-client-0:  (2bddf734-3baa-439f-90e9-93b483260e01) remote_fd is -1. EBADFD [File descriptor in bad state]
[2018-06-18 12:33:38.205461] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] ))))) 0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-06-18 12:33:37.337596 (xid=0x21dc)

[2018-06-18 12:33:38.208554] W [MSGID: 114031] [client-rpc-fops.c:2922:client3_3_readv_cbk] 0-engine-client-0: remote operation failed [Transport endpoint is not connected]
[2018-06-18 12:33:38.208590] W [MSGID: 114061] [client-common.c:704:client_pre_fstat] 0-engine-client-0:  (2bddf734-3baa-439f-90e9-93b483260e01) remote_fd is -1. EBADFD [File descriptor in bad state]
[2018-06-18 12:34:07.234501] E [socket.c:2369:socket_connect_finish] 0-engine-client-0: connection to 172.16.224.10:24007 failed (No route to host); disconnecting socket
[2018-06-18 12:39:25.760451] E [MSGID: 114058] [client-handshake.c:1565:client_query_portmap_cbk] 0-engine-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
----------------------------------------
that's what's on gluster log on the host that engine was running . (Do you use some external site to embeed logs ? sorry I'm new on the list)

agent.log is prety big so I 'll try to find some pastebin like site to link it
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QKNGD7TZT2UVAFG6H6HCPDIJQ6ROAX74/