from glusterfs/rhev_datacenter
[2018-06-18 12:32:50.854668] W [socket.c:593:__socket_rwv] 0-glusterfs: readv on 172.16.224.10:24007 failed (No data available)
[2018-06-18 12:33:38.194322] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-engine-client-0: server 172.16.224.10:49152 has not responded in the last 30 seconds, disconnecting.
[2018-06-18 12:33:38.195797] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_ log_callingfn+0x13b)[ 0x7fb89f2c0efb] (--> /lib64/libgfrpc.so.0(saved_ frames_unwind+0x1de)[ 0x7fb89f085e6e] (--> /lib64/libgfrpc.so.0(saved_ frames_destroy+0xe)[ 0x7fb89f085f8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_ connection_cleanup+0x90)[ 0x7fb89f087710] (--> /lib64/libgfrpc.so.0(rpc_clnt_ notify+0x2a0)[0x7fb89f088200] ))))) 0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(STATFS(14)) called at 2018-06-18 12:33:07.800145 (xid=0x21be)
[2018-06-18 12:33:38.195824] W [MSGID: 114031] [client-rpc-fops.c:777:client3_3_statfs_cbk] 0-engine-client-0: remote operation failed [Transport endpoint is not connected]
[2018-06-18 12:33:38.196036] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_ log_callingfn+0x13b)[ 0x7fb89f2c0efb] (--> /lib64/libgfrpc.so.0(saved_ frames_unwind+0x1de)[ 0x7fb89f085e6e] (--> /lib64/libgfrpc.so.0(saved_ frames_destroy+0xe)[ 0x7fb89f085f8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_ connection_cleanup+0x90)[ 0x7fb89f087710] (--> /lib64/libgfrpc.so.0(rpc_clnt_ notify+0x2a0)[0x7fb89f088200] ))))) 0-engine-client-0: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2018-06-18 12:33:07.800158 (xid=0x21bf)
[2018-06-18 12:33:38.196057] W [rpc-clnt-ping.c:223:rpc_clnt_ping_cbk] 0-engine-client-0: socket disconnected
[2018-06-18 12:33:38.196224] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_ log_callingfn+0x13b)[ 0x7fb89f2c0efb] (--> /lib64/libgfrpc.so.0(saved_ frames_unwind+0x1de)[ 0x7fb89f085e6e] (--> /lib64/libgfrpc.so.0(saved_ frames_destroy+0xe)[ 0x7fb89f085f8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_ connection_cleanup+0x90)[ 0x7fb89f087710] (--> /lib64/libgfrpc.so.0(rpc_clnt_ notify+0x2a0)[0x7fb89f088200] ))))) 0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-06-18 12:33:08.307804 (xid=0x21c0)
[2018-06-18 12:33:38.196243] W [MSGID: 114031] [client-rpc-fops.c:2922:client3_3_readv_cbk] 0-engine-client-0: remote operation failed [Transport endpoint is not connected]
[2018-06-18 12:33:38.196279] W [MSGID: 114061] [client-common.c:704:client_pre_fstat] 0-engine-client-0: (6f81ab66-8991-45a2-9576- 5c7defdec302) remote_fd is -1. EBADFD [File descriptor in bad state]
[2018-06-18 12:33:38.196545] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_ log_callingfn+0x13b)[ 0x7fb89f2c0efb] (--> /lib64/libgfrpc.so.0(saved_ frames_unwind+0x1de)[ 0x7fb89f085e6e] (--> /lib64/libgfrpc.so.0(saved_ frames_destroy+0xe)[ 0x7fb89f085f8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_ connection_cleanup+0x90)[ 0x7fb89f087710] (--> /lib64/libgfrpc.so.0(rpc_clnt_ notify+0x2a0)[0x7fb89f088200] ))))) 0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-06-18 12:33:08.307812 (xid=0x21c4)
last part get repeated a lot of times.
2018-06-18 12:33:38.199480] W [MSGID: 114061] [client-common.c:704:client_pre_fstat] 0-engine-client-0: (2bddf734-3baa-439f-90e9- 93b483260e01) remote_fd is -1. EBADFD [File descriptor in bad state]
[2018-06-18 12:33:38.199721] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_ log_callingfn+0x13b)[ 0x7fb89f2c0efb] (--> /lib64/libgfrpc.so.0(saved_ frames_unwind+0x1de)[ 0x7fb89f085e6e] (--> /lib64/libgfrpc.so.0(saved_ frames_destroy+0xe)[ 0x7fb89f085f8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_ connection_cleanup+0x90)[ 0x7fb89f087710] (--> /lib64/libgfrpc.so.0(rpc_clnt_ notify+0x2a0)[0x7fb89f088200] ))))) 0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-06-18 12:33:16.836613 (xid=0x21cb)
this part also get repeated sometimes
> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[ 0x7fb89f085f8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_ connection_cleanup+0x90)[ 0x7fb89f087710] (--> /lib64/libgfrpc.so.0(rpc_clnt_ notify+0x2a0)[0x7fb89f088200] ))))) 0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-06-18 12:33:37.337567 (xid=0x21db)
The message "W [MSGID: 114031] [client-rpc-fops.c:2922:client3_3_readv_cbk] 0-engine-client-0: remote operation failed [Transport endpoint is not connected]" repeated 8 times between [2018-06-18 12:33:38.202531] and [2018-06-18 12:33:38.205070]
[2018-06-18 12:33:38.205097] W [MSGID: 114061] [client-common.c:704:client_pre_fstat] 0-engine-client-0: (2bddf734-3baa-439f-90e9- 93b483260e01) remote_fd is -1. EBADFD [File descriptor in bad state]
[2018-06-18 12:33:38.205461] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_ log_callingfn+0x13b)[ 0x7fb89f2c0efb] (--> /lib64/libgfrpc.so.0(saved_ frames_unwind+0x1de)[ 0x7fb89f085e6e] (--> /lib64/libgfrpc.so.0(saved_ frames_destroy+0xe)[ 0x7fb89f085f8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_ connection_cleanup+0x90)[ 0x7fb89f087710] (--> /lib64/libgfrpc.so.0(rpc_clnt_ notify+0x2a0)[0x7fb89f088200] ))))) 0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-06-18 12:33:37.337596 (xid=0x21dc)
[2018-06-18 12:33:38.208554] W [MSGID: 114031] [client-rpc-fops.c:2922:client3_3_readv_cbk] 0-engine-client-0: remote operation failed [Transport endpoint is not connected]
[2018-06-18 12:33:38.208590] W [MSGID: 114061] [client-common.c:704:client_pre_fstat] 0-engine-client-0: (2bddf734-3baa-439f-90e9- 93b483260e01) remote_fd is -1. EBADFD [File descriptor in bad state]
[2018-06-18 12:34:07.234501] E [socket.c:2369:socket_connect_finish] 0-engine-client-0: connection to 172.16.224.10:24007 failed (No route to host); disconnecting socket
[2018-06-18 12:39:25.760451] E [MSGID: 114058] [client-handshake.c:1565:client_query_portmap_cbk] 0-engine-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
----------------------------------------
that's what's on gluster log on the host that engine was running . (Do you use some external site to embeed logs ? sorry I'm new on the list)
agent.log is prety big so I 'll try to find some pastebin like site to link it
_______________________________________________ List Archives: https://lists.ovirt.org/
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community- guidelines/
archives/list/users@ovirt.org/ message/ QKNGD7TZT2UVAFG6H6HCPDIJQ6ROAX 74/