[ovirt-users] Replica2 stripe2 hang on write to VM disk

Mon May 25 16:02:53 UTC 2015

Hello,
can anybody help me with hanging  replica2 stripe2 datastore on 4 nodes 
cluster ??

oVirt -  ovirt-engine-lib-3.5.2.1-1.el7.centos.noarch
gluster - glusterfs-server-3.7.0-2.el7.x86_64
VM - Centos 7.1

If I use any bigger write to VM disk ( eg 2-5GB ) storage hosted virtual 
disk will hang = I/O error
created by :
gluster volume create 12KVM12SC4 replica 2 stripe 2 
16.0.0.161:/STORAGES/SlowClass/p4/GFS1 
16.0.0.162:/STORAGES/SlowClass/p4/GFS1 
16.0.0.163:/STORAGES/SlowClass/p4/GFS1 
16.0.0.164:/STORAGES/SlowClass/p4/GFS1

rhev-data-center-mnt-glusterSD-localhost:_12KVM12SC4.log
-----------------------------------------------------------------------------------
[2015-05-25 14:47:24.205609] I [rpc-clnt.c:1807:rpc_clnt_reconfig] 
0-12KVM12SC4-client-3: changing port to 49158 (from 0)
[2015-05-25 14:47:24.210824] I 
[client-handshake.c:1405:select_server_supported_programs] 
0-12KVM12SC4-client-3: Using Program GlusterFS 3.3, Num (1298437), 
Version (330)
[2015-05-25 14:47:24.211204] I 
[client-handshake.c:1193:client_setvolume_cbk] 0-12KVM12SC4-client-3: 
Connected to 12KVM12SC4-client-3, attached to remote volume 
'/STORAGES/SlowClass/p4/GFS1'.
[2015-05-25 14:47:24.211225] I 
[client-handshake.c:1203:client_setvolume_cbk] 0-12KVM12SC4-client-3: 
Server and Client lk-version numbers are not same, reopening the fds
[2015-05-25 14:47:24.211275] I [MSGID: 108005] 
[afr-common.c:3880:afr_notify] 0-12KVM12SC4-replicate-1: Subvolume 
'12KVM12SC4-client-3' came back up; going online.
[2015-05-25 14:47:24.216465] I [fuse-bridge.c:5077:fuse_graph_setup] 
0-fuse: switched to graph 0
[2015-05-25 14:47:24.216556] I 
[client-handshake.c:187:client_set_lk_version_cbk] 
0-12KVM12SC4-client-3: Server lk version = 1
[2015-05-25 14:47:24.216643] I [fuse-bridge.c:4007:fuse_init] 
0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 
kernel 7.22
[2015-05-25 14:47:24.217998] I 
[afr-common.c:1673:afr_local_discovery_cbk] 0-12KVM12SC4-replicate-0: 
selecting local read_child 12KVM12SC4-client-0
[2015-05-25 14:47:29.737732] W [fuse-bridge.c:1080:fuse_setattr_cbk] 
0-glusterfs-fuse: 40: SETATTR() /__DIRECT_IO_TEST__ => -1 (Read-only 
file system)
[2015-05-25 14:49:18.266212] E 
[client-handshake.c:1488:client_query_portmap_cbk] 
0-12KVM12SC4-client-2: failed to get the port number for remote 
subvolume. Please run 'gluster volume status' on server to see if brick 
process is running.
[2015-05-25 14:49:18.266274] I [client.c:2086:client_rpc_notify] 
0-12KVM12SC4-client-2: disconnected from 12KVM12SC4-client-2. Client 
process will keep trying to connect to glusterd until brick's port is 
available
[2015-05-25 14:49:19.346555] I [rpc-clnt.c:1807:rpc_clnt_reconfig] 
0-12KVM12SC4-client-2: changing port to 49158 (from 0)
[2015-05-25 14:49:19.351812] I 
[client-handshake.c:1405:select_server_supported_programs] 
0-12KVM12SC4-client-2: Using Program GlusterFS 3.3, Num (1298437), 
Version (330)
[2015-05-25 14:49:19.352169] I 
[client-handshake.c:1193:client_setvolume_cbk] 0-12KVM12SC4-client-2: 
Connected to 12KVM12SC4-client-2, attached to remote volume 
'/STORAGES/SlowClass/p4/GFS1'.
[2015-05-25 14:49:19.352191] I 
[client-handshake.c:1203:client_setvolume_cbk] 0-12KVM12SC4-client-2: 
Server and Client lk-version numbers are not same, reopening the fds
[2015-05-25 14:49:19.352242] I [MSGID: 108002] 
[afr-common.c:3959:afr_notify] 0-12KVM12SC4-replicate-1: Client-quorum 
is met
[2015-05-25 14:49:19.352353] I 
[client-handshake.c:187:client_set_lk_version_cbk] 
0-12KVM12SC4-client-2: Server lk version = 1
[2015-05-25 14:49:27.843616] W [fuse-bridge.c:1263:fuse_err_cbk] 
0-glusterfs-fuse: 151: REMOVEXATTR() /__DIRECT_IO_TEST__ => -1 (No data 
available)
[2015-05-25 14:49:58.356900] W [fuse-bridge.c:1263:fuse_err_cbk] 
0-glusterfs-fuse: 327: REMOVEXATTR() /__DIRECT_IO_TEST__ => -1 (No data 
available)

# gluster volume status
Status of volume: 12KVM12SC4
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 16.0.0.161:/STORAGES/SlowClass/p4/GFS
1                                           49173 0          Y       17678
Brick 16.0.0.162:/STORAGES/SlowClass/p4/GFS
1                                           49158 0          Y       19184
Brick 16.0.0.163:/STORAGES/SlowClass/p4/GFS
1                                           49158 0          Y       9784
Brick 16.0.0.164:/STORAGES/SlowClass/p4/GFS
1                                           49158 0          Y       9327
NFS Server on localhost                     2049 0          Y       17697
Self-heal Daemon on localhost               N/A N/A        Y       17708
NFS Server on 16.0.0.162                    2049 0          Y       19205
Self-heal Daemon on 16.0.0.162              N/A N/A        Y       19215
NFS Server on 16.0.0.163                    2049 0          Y       9806
Self-heal Daemon on 16.0.0.163              N/A N/A        Y       9813
NFS Server on 16.0.0.164                    2049 0          Y       9347
Self-heal Daemon on 16.0.0.164              N/A N/A        Y       9359

Task Status of Volume 12KVM12SC4
------------------------------------------------------------------------------
There are no active volume tasks

any idea ??
regs.
Pavel