3:43 p.m.
It seems that a vm with 3 disks boot in domain engine another disk in domain vol1 and a
third in domain v3 became non responsive when one gluster host went down.
To explain a bit the situation I have 3 glusterfs hosts with 3 volumes
hosts are g1,g2,g3 each have 3 bricks
g1 has vol1,vol2 and vol3 arbiter
g2 has vol1, vol2arbiter and vol3
g3 has vol1arb vol2 and vol3
libgfapi is enabled . I put a host in maintenance to update the bios and the vm who had
disks in two domain became unresponsive..
is this normal? qemu logs showing that it tries Domain configuration shows host1 as
primary for vol1 and host2 as primary for vol3 with the other two as backup-volfile
servers..
it seems it always try to connect to the server that is down and not to one of the
alternative hosts...
is this libgapi/libvirt problem ?
Here are some libvirt logs showing what it tries to do..
[2018-09-10 19:43:42.876114] T [socket.c:3133:socket_connect] 0-vol1-client-2: connecting
0x55ed673525c0, state=2 gen=0 sock=-1
[2018-09-10 19:43:42.876124] T [name.c:243:af_inet_client_get_remote_sockaddr]
0-vol1-client-2: option remote-port missing in volume vol1-client-2. Defaulting to 24007
[2018-09-10 19:43:42.878566] D [socket.c:3051:socket_fix_ssl_opts] 0-vol1-client-2:
disabling SSL for portmapper connection
[2018-09-10 19:43:42.878770] T [socket.c:834:__socket_nodelay] 0-vol1-client-2: NODELAY
enabled for socket 30
[2018-09-10 19:43:42.878780] T [socket.c:920:__socket_keepalive] 0-vol1-client-2:
Keep-alive enabled for socket: 30, (idle: 20, interval: 2, max-probes: 9, timeout: 0)
[2018-09-10 19:43:42.878830] T [rpc-clnt.c:406:rpc_clnt_reconnect] 0-vol3-client-1:
attempting reconnect
[2018-09-10 19:43:42.878846] T [socket.c:3133:socket_connect] 0-vol3-client-1: connecting
0x55ed673546c0, state=2 gen=0 sock=-1
[2018-09-10 19:43:42.878856] T [name.c:243:af_inet_client_get_remote_sockaddr]
0-vol3-client-1: option remote-port missing in volume vol3-client-1. Defaulting to 24007
[2018-09-10 19:43:42.881229] D [socket.c:3051:socket_fix_ssl_opts] 0-vol3-client-1:
disabling SSL for portmapper connection
[2018-09-10 19:43:42.881255] T [socket.c:834:__socket_nodelay] 0-vol3-client-1: NODELAY
enabled for socket 38
[2018-09-10 19:43:42.881264] T [socket.c:920:__socket_keepalive] 0-vol3-client-1:
Keep-alive enabled for socket: 38, (idle: 20, interval: 2, max-probes: 9, timeout: 0)
[2018-09-10 19:43:45.569298] T [socket.c:724:__socket_disconnect] 0-vol3-client-1:
disconnecting 0x55ed673546c0, state=2 gen=0 sock=38
[2018-09-10 19:43:45.569308] T [socket.c:724:__socket_disconnect] 0-vol1-client-2:
disconnecting 0x55ed673525c0, state=2 gen=0 sock=30
[2018-09-10 19:43:45.570000] T [socket.c:728:__socket_disconnect] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fddbadade9b] (-->
/usr/lib64/glusterfs/3.12.13/rpc-t
ransport/socket.so(+0x4ea0)[0x7fdda7bbfea0] (-->
/usr/lib64/glusterfs/3.12.13/rpc-transport/socket.so(+0x530a)[0x7fdda7bc030a] (-->
/usr/lib64/glusterfs/3.12.13/rpc-transport/s
ocket.so(+0x9a08)[0x7fdda7bc4a08] (-->
/lib64/libglusterfs.so.0(+0x883c4)[0x7fddbae093c4] ))))) 0-vol3-client-1: tearing down
socket connection
[2018-09-10 19:43:45.570020] D [socket.c:686:__socket_shutdown] 0-vol3-client-1:
shutdown() returned -1. Transport endpoint is not connected
[2018-09-10 19:43:45.570038] D [socket.c:733:__socket_disconnect] 0-vol3-client-1:
__socket_teardown_connection () failed: Transport endpoint is not connected
[2018-09-10 19:43:45.570043] D [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR
- disconnecting now
[2018-09-10 19:43:45.570907] T [socket.c:728:__socket_disconnect] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fddbadade9b] (-->
/usr/lib64/glusterfs/3.12.13/rpc-t
ransport/socket.so(+0x4ea0)[0x7fdda7bbfea0] (-->
/usr/lib64/glusterfs/3.12.13/rpc-transport/socket.so(+0x530a)[0x7fdda7bc030a] (-->
/usr/lib64/glusterfs/3.12.13/rpc-transport/s
ocket.so(+0x9a08)[0x7fdda7bc4a08] (-->
/lib64/libglusterfs.so.0(+0x883c4)[0x7fddbae093c4] ))))) 0-vol1-client-2: tearing down
socket connection
[2018-09-10 19:43:45.570928] D [socket.c:686:__socket_shutdown] 0-vol1-client-2:
shutdown() returned -1. Transport endpoint is not connected
[2018-09-10 19:43:45.570936] D [socket.c:733:__socket_disconnect] 0-vol1-client-2:
__socket_teardown_connection () failed: Transport endpoint is not connected
[2018-09-10 19:43:45.570940] D [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR
- disconnecting now
[2018-09-10 19:43:45.570960] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fddbadade9b] (-->
/lib64/libgfrp
c.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fddbab7828b] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x5f)[0x7fddbab7460f] (-->
/lib64/libgfrpc.so.0(rpc_clnt_no
tify+0x2a0)[0x7fddbab75130] (-->
/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fddbab70ea3] ))))) 0-:
10.xxx.xxx.130:24007: ping timer event already removed
[2018-09-10 19:43:45.571098] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fddbadade9b] (-->
/lib64/libgfrp
c.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fddbab7828b] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x5f)[0x7fddbab7460f] (-->
/lib64/libgfrpc.so.0(rpc_clnt_no
tify+0x2a0)[0x7fddbab75130] (-->
/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fddbab70ea3] ))))) 0-:
10.xxx.xxx.130:24007: ping timer event already removed
[2018-09-10 19:43:45.878885] T [rpc-clnt.c:406:rpc_clnt_reconnect] 0-vol1-client-2:
attempting reconnect
[2018-09-10 19:43:45.881546] T [socket.c:834:__socket_nodelay] 0-vol1-client-2: NODELAY
enabled for socket 38
[2018-09-10 19:43:45.881555] T [socket.c:920:__socket_keepalive] 0-vol1-client-2:
Keep-alive enabled for socket: 38, (idle: 20, interval: 2, max-probes: 9, timeout: 0)
[2018-09-10 19:43:45.883839] D [socket.c:3051:socket_fix_ssl_opts] 0-vol3-client-1:
disabling SSL for portmapper connection
[2018-09-10 19:43:45.883878] T [socket.c:834:__socket_nodelay] 0-vol3-client-1: NODELAY
enabled for socket 30
[2018-09-10 19:43:45.883886] T [socket.c:920:__socket_keepalive] 0-vol3-client-1:
Keep-alive enabled for socket: 30, (idle: 20, interval: 2, max-probes: 9, timeout: 0)
[2018-09-10 19:43:48.575316] T [socket.c:724:__socket_disconnect] 0-vol3-client-1:
disconnecting 0x55ed673546c0, state=2 gen=0 sock=30
[2018-09-10 19:43:48.575329] T [socket.c:724:__socket_disconnect] 0-vol1-client-2:
disconnecting 0x55ed673525c0, state=2 gen=0 sock=38
[2018-09-10 19:43:48.576022] T [socket.c:728:__socket_disconnect] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fddbadade9b] (-->
/usr/lib64/glusterfs/3.12.13/rpc-transport/socket.so(+0x4ea0)[0x7fdda7bbfea0] (-->
/usr/lib64/glusterfs/3.12.13/rpc-transport/socket.so(+0x530a)[0x7fdda7bc030a] (-->
/usr/lib64/glusterfs/3.12.13/rpc-transport/socket.so(+0x9a08)[0x7fdda7bc4a08] (-->
/lib64/libglusterfs.so.0(+0x883c4)[0x7fddbae093c4] ))))) 0-vol3-client-1: tearing down
socket connection
[2018-09-10 19:43:48.576045] D [socket.c:686:__socket_shutdown] 0-vol3-client-1:
shutdown() returned -1. Transport endpoint is not connected
[2018-09-10 19:43:48.576054] D [socket.c:733:__socket_disconnect] 0-vol3-client-1:
__socket_teardown_connection () failed: Transport endpoint is not connected
[2018-09-10 19:43:48.576059] D [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR
- disconnecting now
[2018-09-10 19:43:48.576079] T [socket.c:728:__socket_disconnect] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fddbadade9b] (-->
/usr/lib64/glusterfs/3.12.13/rpc-transport/socket.so(+0x4ea0)[0x7fdda7bbfea0] (-->
/usr/lib64/glusterfs/3.12.13/rpc-transport/socket.so(+0x530a)[0x7fdda7bc030a] (-->
/usr/lib64/glusterfs/3.12.13/rpc-transport/socket.so(+0x9a08)[0x7fdda7bc4a08] (-->
/lib64/libglusterfs.so.0(+0x883c4)[0x7fddbae093c4] ))))) 0-vol1-client-2: tearing down
socket connection
[2018-09-10 19:43:48.576099] D [socket.c:686:__socket_shutdown] 0-vol1-client-2:
shutdown() returned -1. Transport endpoint is not connected
[2018-09-10 19:43:48.576106] D [socket.c:733:__socket_disconnect] 0-vol1-client-2:
__socket_teardown_connection () failed: Transport endpoint is not connected
[2018-09-10 19:43:48.576111] D [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR
- disconnecting now
[2018-09-10 19:43:48.576879] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fddbadade9b] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fddbab7828b] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x5f)[0x7fddbab7460f] (-->
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fddbab75130] (-->
/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fddbab70ea3] ))))) 0-:
10.xxx.xxx.130:24007: ping timer event already removed
[2018-09-10 19:43:48.576958] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fddbadade9b] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fddbab7828b] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x5f)[0x7fddbab7460f] (-->
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fddbab75130] (-->
/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fddbab70ea3] ))))) 0-:
10.xxx.xxx.130:24007: ping timer event already removed
[2018-09-10 19:43:48.881651] T [rpc-clnt.c:406:rpc_clnt_reconnect] 0-vol1-client-2:
attempting reconnect
[2018-09-10 19:43:48.881667] T [socket.c:3133:socket_connect] 0-vol1-client-2: connecting
0x55ed673525c0, state=2 gen=0 sock=-1
[2018-09-10 19:43:48.881689] T [name.c:243:af_inet_client_get_remote_sockaddr]
0-vol1-client-2: option remote-port missing in volume vol1-client-2. Defaulting to 24007
[2018-09-10 19:43:48.884056] T [rpc-clnt.c:406:rpc_clnt_reconnect] 0-vol3-client-1:
attempting reconnect
[2018-09-10 19:43:48.884072] T [socket.c:3133:socket_connect] 0-vol3-client-1: connecting
0x55ed673546c0, state=2 gen=0 sock=-1
[2018-09-10 19:43:48.884084] T [name.c:243:af_inet_client_get_remote_sockaddr]
0-vol3-client-1: option remote-port missing in volume vol3-client-1. Defaulting to 24007
[2018-09-10 19:43:48.884190] D [socket.c:3051:socket_fix_ssl_opts] 0-vol1-client-2:
disabling SSL for portmapper connection
[2018-09-10 19:43:48.886524] T [socket.c:834:__socket_nodelay] 0-vol3-client-1: NODELAY
enabled for socket 30
[2018-09-10 19:43:48.886532] T [socket.c:920:__socket_keepalive] 0-vol3-client-1:
Keep-alive enabled for socket: 30, (idle: 20, interval: 2, max-probes: 9, timeout: 0)
[2018-09-10 19:43:51.581293] T [socket.c:724:__socket_disconnect] 0-vol3-client-1:
disconnecting 0x55ed673546c0, state=2 gen=0 sock=30
[2018-09-10 19:43:51.581293] T [socket.c:724:__socket_disconnect] 0-vol1-client-2:
disconnecting 0x55ed673525c0, state=2 gen=0 sock=38
[2018-09-10 19:43:51.582009] T [socket.c:728:__socket_disconnect] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fddbadade9b] (-->
/usr/lib64/glusterfs/3.12.13/rpc-transport/socket.so(+0x4ea0)[0x7fdda7bbfea0] (-->
/usr/lib64/glusterfs/3.12.13/rpc-transport/socket.so(+0x530a)[0x7fdda7bc030a] (-->
/usr/lib64/glusterfs/3.12.13/rpc-transport/socket.so(+0x9a08)[0x7fdda7bc4a08] (-->
/lib64/libglusterfs.so.0(+0x883c4)[0x7fddbae093c4] ))))) 0-vol1-client-2: tearing down
socket connection
[2018-09-10 19:43:51.582030] D [socket.c:686:__socket_shutdown] 0-vol1-client-2:
shutdown() returned -1. Transport endpoint is not connected
[2018-09-10 19:43:51.582036] D [socket.c:733:__socket_disconnect] 0-vol1-client-2:
__socket_teardown_connection () failed: Transport endpoint is not connected
[2018-09-10 19:43:51.582040] D [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR
- disconnecting now
[2018-09-10 19:43:51.582084] T [socket.c:728:__socket_disconnect] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fddbadade9b] (-->
/usr/lib64/glusterfs/3.12.13/rpc-transport/socket.so(+0x4ea0)[0x7fdda7bbfea0] (-->
/usr/lib64/glusterfs/3.12.13/rpc-transport/socket.so(+0x530a)[0x7fdda7bc030a] (-->
/usr/lib64/glusterfs/3.12.13/rpc-transport/socket.so(+0x9a08)[0x7fdda7bc4a08] (-->
/lib64/libglusterfs.so.0(+0x883c4)[0x7fddbae093c4] ))))) 0-vol3-client-1: tearing down
socket connection
[2018-09-10 19:43:51.582105] D [socket.c:686:__socket_shutdown] 0-vol3-client-1:
shutdown() returned -1. Transport endpoint is not connected
[2018-09-10 19:43:51.582111] D [socket.c:733:__socket_disconnect] 0-vol3-client-1:
__socket_teardown_connection () failed: Transport endpoint is not connected
[2018-09-10 19:43:51.582116] D [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR
- disconnecting now
[2018-09-10 19:43:51.582812] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fddbadade9b] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fddbab7828b] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x5f)[0x7fddbab7460f] (-->
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fddbab75130] (-->
/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fddbab70ea3] ))))) 0-:
10.xxx.xxx.130:24007: ping timer event already removed
[2018-09-10 19:43:51.582865] D [rpc-clnt-ping.c:99:rpc_clnt_remove_ping_timer_locked]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fddbadade9b] (-->
/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8b)[0x7fddbab7828b] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x5f)[0x7fddbab7460f] (-->
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fddbab75130] (-->
/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fddbab70ea3] ))))) 0-:
10.xxx.xxx.130:24007: ping timer event already removed
[2018-09-10 19:43:51.884349] T [rpc-clnt.c:406:rpc_clnt_reconnect] 0-vol1-client-2:
attempting reconnect
[2018-09-10 19:43:51.884367] T [socket.c:3133:socket_connect] 0-vol1-client-2: connecting
0x55ed673525c0, state=2 gen=0 sock=-1
[2018-09-10 19:43:51.884376] T [name.c:243:af_inet_client_get_remote_sockaddr]
0-vol1-client-2: option remote-port missing in volume vol1-client-2. Defaulting to 24007
[2018-09-10 19:43:51.886644] T [rpc-clnt.c:406:rpc_clnt_reconnect] 0-vol3-client-1:
attempting reconnect
[2018-09-10 19:43:51.886659] T [socket.c:3133:socket_connect] 0-vol3-client-1: connecting
0x55ed673546c0, state=2 gen=0 sock=-1
[2018-09-10 19:43:51.886669] T [name.c:243:af_inet_client_get_remote_sockaddr]
0-vol3-client-1: option remote-port missing in volume vol3-client-1. Defaulting to 24007
[2018-09-10 19:43:51.887251] D [socket.c:3051:socket_fix_ssl_opts] 0-vol1-client-2:
disabling SSL for portmapper connection
[2018-09-10 19:43:51.887281] T [socket.c:834:__socket_nodelay] 0-vol1-client-2: NODELAY
enabled for socket 38
[2018-09-10 19:43:51.887290] T [socket.c:920:__socket_keepalive] 0-vol1-client-2:
Keep-alive enabled for socket: 38, (idle: 20, interval: 2, max-probes: 9, timeout: 0)
[2018-09-10 19:43:51.889141] D [socket.c:3051:socket_fix_ssl_opts] 0-vol3-client-1:
disabling SSL for portmapper connection
: