10:44 a.m.
Adding Krutika to look at gluster logs
On Mon, Jun 18, 2018 at 10:39 AM, <g.vasilopoulos(a)uoc.gr> wrote:
from glusterfs/rhev_datacenter
[2018-06-18 12:32:50.854668] W [socket.c:593:__socket_rwv] 0-glusterfs:
readv on 172.16.224.10:24007 failed (No data available)
[2018-06-18 12:33:38.194322] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired]
0-engine-client-0: server 172.16.224.10:49152 has not responded in the
last 30 seconds, disconnecting.
[2018-06-18 12:33:38.195797] E [rpc-clnt.c:350:saved_frames_unwind] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] )))))
0-engine-client-0: forced unwinding frame type(GlusterFS 3.3)
op(STATFS(14)) called at 2018-06-18 12:33:07.800145 (xid=0x21be)
[2018-06-18 12:33:38.195824] W [MSGID: 114031]
[client-rpc-fops.c:777:client3_3_statfs_cbk]
0-engine-client-0: remote operation failed [Transport endpoint is not
connected]
[2018-06-18 12:33:38.196036] E [rpc-clnt.c:350:saved_frames_unwind] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] )))))
0-engine-client-0: forced unwinding frame type(GF-DUMP) op(NULL(2)) called
at 2018-06-18 12:33:07.800158 (xid=0x21bf)
[2018-06-18 12:33:38.196057] W [rpc-clnt-ping.c:223:rpc_clnt_ping_cbk]
0-engine-client-0: socket disconnected
[2018-06-18 12:33:38.196224] E [rpc-clnt.c:350:saved_frames_unwind] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] )))))
0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12))
called at 2018-06-18 12:33:08.307804 (xid=0x21c0)
[2018-06-18 12:33:38.196243] W [MSGID: 114031]
[client-rpc-fops.c:2922:client3_3_readv_cbk]
0-engine-client-0: remote operation failed [Transport endpoint is not
connected]
[2018-06-18 12:33:38.196279] W [MSGID: 114061] [client-common.c:704:client_pre_fstat]
0-engine-client-0: (6f81ab66-8991-45a2-9576-5c7defdec302) remote_fd is
-1. EBADFD [File descriptor in bad state]
[2018-06-18 12:33:38.196545] E [rpc-clnt.c:350:saved_frames_unwind] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] )))))
0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12))
called at 2018-06-18 12:33:08.307812 (xid=0x21c4)
last part get repeated a lot of times.
2018-06-18 12:33:38.199480] W [MSGID: 114061] [client-common.c:704:client_pre_fstat]
0-engine-client-0: (2bddf734-3baa-439f-90e9-93b483260e01) remote_fd is
-1. EBADFD [File descriptor in bad state]
[2018-06-18 12:33:38.199721] E [rpc-clnt.c:350:saved_frames_unwind] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] )))))
0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12))
called at 2018-06-18 12:33:16.836613 (xid=0x21cb)
this part also get repeated sometimes
> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] )))))
0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12))
called at 2018-06-18 12:33:37.337567 (xid=0x21db)
The message "W [MSGID: 114031] [client-rpc-fops.c:2922:client3_3_readv_cbk]
0-engine-client-0: remote operation failed [Transport endpoint is not
connected]" repeated 8 times between [2018-06-18 12:33:38.202531] and
[2018-06-18 12:33:38.205070]
[2018-06-18 12:33:38.205097] W [MSGID: 114061] [client-common.c:704:client_pre_fstat]
0-engine-client-0: (2bddf734-3baa-439f-90e9-93b483260e01) remote_fd is
-1. EBADFD [File descriptor in bad state]
[2018-06-18 12:33:38.205461] E [rpc-clnt.c:350:saved_frames_unwind] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] )))))
0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12))
called at 2018-06-18 12:33:37.337596 (xid=0x21dc)
[2018-06-18 12:33:38.208554] W [MSGID: 114031]
[client-rpc-fops.c:2922:client3_3_readv_cbk]
0-engine-client-0: remote operation failed [Transport endpoint is not
connected]
[2018-06-18 12:33:38.208590] W [MSGID: 114061] [client-common.c:704:client_pre_fstat]
0-engine-client-0: (2bddf734-3baa-439f-90e9-93b483260e01) remote_fd is
-1. EBADFD [File descriptor in bad state]
[2018-06-18 12:34:07.234501] E [socket.c:2369:socket_connect_finish]
0-engine-client-0: connection to 172.16.224.10:24007 failed (No route to
host); disconnecting socket
[2018-06-18 12:39:25.760451] E [MSGID: 114058]
[client-handshake.c:1565:client_query_portmap_cbk]
0-engine-client-0: failed to get the port number for remote subvolume.
Please run 'gluster volume status' on server to see if brick process is
running.
----------------------------------------
that's what's on gluster log on the host that engine was running . (Do you
use some external site to embeed logs ? sorry I'm new on the list)
agent.log is prety big so I 'll try to find some pastebin like site to
link it
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-
guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/
message/QKNGD7TZT2UVAFG6H6HCPDIJQ6ROAX74/