On Nov 6, 2016, at 9:50 PM, Sahina Bose <sabose@redhat.com> wrote: =20 However your volume configuration seems suspect -"stripe 2 replica 2". = Can you provide gluster volume info of your second storage domain = gluster volume? The mount logs of the volume (under = /var/log/glusterfs/rhev-datacenter..<volname>.log) from the host where =
--Apple-Mail=_74966C91-4D7B-4631-992A-7104BF8281C2 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 [root@fai-kvm-1-gfs admin]# gluster volume status data2 Status of volume: data2 Gluster process TCP Port RDMA Port Online = Pid = --------------------------------------------------------------------------= ---- Brick fai-kvm-1-vmn.ravnalaska.net:/kvm2/gl uster/data2/brick 49156 0 Y = 3484 Brick fai-kvm-2-vmn.ravnalaska.net:/kvm2/gl uster/data2/brick 49156 0 Y = 34791 Brick fai-kvm-3-vmn.ravnalaska.net:/kvm2/gl uster/data2/brick 49156 0 Y = 177340 Brick fai-kvm-4-vmn.ravnalaska.net:/kvm2/gl uster/data2/brick 49152 0 Y = 146038 NFS Server on localhost 2049 0 Y = 40844 Self-heal Daemon on localhost N/A N/A Y = 40865 NFS Server on fai-kvm-2-gfs.ravnalaska.net 2049 0 Y = 99905 Self-heal Daemon on fai-kvm-2-gfs.ravnalask a.net N/A N/A Y = 99915 NFS Server on fai-kvm-4-gfs.ravnalaska.net 2049 0 Y = 176305 Self-heal Daemon on fai-kvm-4-gfs.ravnalask a.net N/A N/A Y = 176326 NFS Server on fai-kvm-3-gfs.ravnalaska.net 2049 0 Y = 226271 Self-heal Daemon on fai-kvm-3-gfs.ravnalask a.net N/A N/A Y = 226287 Task Status of Volume data2 = --------------------------------------------------------------------------= ---- There are no active volume tasks [root@fai-kvm-1-gfs admin]# gluster volume info data2 Volume Name: data2 Type: Striped-Replicate Volume ID: 20f85c9a-541b-4df4-9dba-44c5179bbfb0 Status: Started Number of Bricks: 1 x 2 x 2 =3D 4 Transport-type: tcp Bricks: Brick1: fai-kvm-1-vmn.ravnalaska.net:/kvm2/gluster/data2/brick Brick2: fai-kvm-2-vmn.ravnalaska.net:/kvm2/gluster/data2/brick Brick3: fai-kvm-3-vmn.ravnalaska.net:/kvm2/gluster/data2/brick Brick4: fai-kvm-4-vmn.ravnalaska.net:/kvm2/gluster/data2/brick Options Reconfigured: performance.readdir-ahead: on performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.stat-prefetch: off cluster.eager-lock: enable network.remote-dio: enable cluster.quorum-type: auto cluster.server-quorum-type: server storage.owner-uid: 36 storage.owner-gid: 36 See attached file for the mount log. Gary ------------------------------------------------------------------------ Gary Pedretty gary@ravnalaska.net = <mailto:gary@eraalaska.net> Systems Manager www.flyravn.com = <http://www.flyravn.com/> Ravn Alaska /\ 907-450-7251 5245 Airport Industrial Road / \/\ 907-450-7238 fax Fairbanks, Alaska 99709 /\ / \ \ Second greatest commandment Serving All of Alaska / \/ /\ \ \/\ =E2=80=9CLove your = neighbor as Really loving the record green up date! Summmer!! yourself=E2=80=9D = Matt 22:39 ------------------------------------------------------------------------ the volume is being mounted will also help.
=20
<div class=3D"">Brick2: <a href=3D"http://fai-kvm-2-vmn.ravnalaska.net" = class=3D"">fai-kvm-2-vmn.ravnalaska.net</a>:/kvm2/gluster/data2/brick</div= <div class=3D"">Brick3: <a href=3D"http://fai-kvm-3-vmn.ravnalaska.net" = class=3D"">fai-kvm-3-vmn.ravnalaska.net</a>:/kvm2/gluster/data2/brick</div= <div class=3D"">Brick4: <a href=3D"http://fai-kvm-4-vmn.ravnalaska.net" = class=3D"">fai-kvm-4-vmn.ravnalaska.net</a>:/kvm2/gluster/data2/brick</div= <div class=3D"">Options Reconfigured:</div><div = class=3D"">performance.readdir-ahead: on</div><div = class=3D"">performance.quick-read: off</div><div = class=3D"">performance.read-ahead: off</div><div class=3D""><a =
--Apple-Mail=_74966C91-4D7B-4631-992A-7104BF8281C2 Content-Type: multipart/mixed; boundary="Apple-Mail=_D3985462-55E4-4FE5-A078-30EC039EC062" --Apple-Mail=_D3985462-55E4-4FE5-A078-30EC039EC062 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div class=3D"">[root@fai-kvm-1-gfs admin]# gluster volume = status data2</div><div class=3D"">Status of volume: data2</div><div = class=3D"">Gluster process = TCP Port = RDMA Port Online Pid</div><div = class=3D"">---------------------------------------------------------------= ---------------</div><div class=3D"">Brick <a = href=3D"http://fai-kvm-1-vmn.ravnalaska.net" = class=3D"">fai-kvm-1-vmn.ravnalaska.net</a>:/kvm2/gl</div><div = class=3D"">uster/data2/brick = 49156 0 = Y 3484</div><div = class=3D"">Brick <a href=3D"http://fai-kvm-2-vmn.ravnalaska.net" = class=3D"">fai-kvm-2-vmn.ravnalaska.net</a>:/kvm2/gl</div><div = class=3D"">uster/data2/brick = 49156 0 = Y 34791</div><div = class=3D"">Brick <a href=3D"http://fai-kvm-3-vmn.ravnalaska.net" = class=3D"">fai-kvm-3-vmn.ravnalaska.net</a>:/kvm2/gl</div><div = class=3D"">uster/data2/brick = 49156 0 = Y = 177340</div><div class=3D"">Brick <a = href=3D"http://fai-kvm-4-vmn.ravnalaska.net" = class=3D"">fai-kvm-4-vmn.ravnalaska.net</a>:/kvm2/gl</div><div = class=3D"">uster/data2/brick = 49152 0 = Y = 146038</div><div class=3D"">NFS Server on localhost = 2049 = 0 Y = 40844</div><div class=3D"">Self-heal Daemon on localhost = N/A N/A = Y 40865</div><div class=3D"">NFS = Server on <a href=3D"http://fai-kvm-2-gfs.ravnalaska.net" = class=3D"">fai-kvm-2-gfs.ravnalaska.net</a> 2049 = 0 Y = 99905</div><div class=3D"">Self-heal Daemon on = fai-kvm-2-gfs.ravnalask</div><div class=3D""><a href=3D"http://a.net" = class=3D"">a.net</a> = = N/A N/A Y = 99915</div><div class=3D"">NFS Server on <a = href=3D"http://fai-kvm-4-gfs.ravnalaska.net" = class=3D"">fai-kvm-4-gfs.ravnalaska.net</a> 2049 = 0 Y = 176305</div><div class=3D"">Self-heal Daemon on = fai-kvm-4-gfs.ravnalask</div><div class=3D""><a href=3D"http://a.net" = class=3D"">a.net</a> = = N/A N/A Y = 176326</div><div class=3D"">NFS Server on <a = href=3D"http://fai-kvm-3-gfs.ravnalaska.net" = class=3D"">fai-kvm-3-gfs.ravnalaska.net</a> 2049 = 0 Y = 226271</div><div class=3D"">Self-heal Daemon on = fai-kvm-3-gfs.ravnalask</div><div class=3D""><a href=3D"http://a.net" = class=3D"">a.net</a> = = N/A N/A Y = 226287</div><div class=3D""><br class=3D""></div><div= class=3D"">Task Status of Volume data2</div><div = class=3D"">---------------------------------------------------------------= ---------------</div><div class=3D"">There are no active volume = tasks</div><div class=3D""><br class=3D""></div><div class=3D""><br = class=3D""></div><div class=3D""><div class=3D"">[root@fai-kvm-1-gfs = admin]# gluster volume info data2</div><div class=3D""><br = class=3D""></div><div class=3D"">Volume Name: data2</div><div = class=3D"">Type: Striped-Replicate</div><div class=3D"">Volume ID: = 20f85c9a-541b-4df4-9dba-44c5179bbfb0</div><div class=3D"">Status: = Started</div><div class=3D"">Number of Bricks: 1 x 2 x 2 =3D 4</div><div = class=3D"">Transport-type: tcp</div><div class=3D"">Bricks:</div><div = class=3D"">Brick1: <a href=3D"http://fai-kvm-1-vmn.ravnalaska.net" = class=3D"">fai-kvm-1-vmn.ravnalaska.net</a>:/kvm2/gluster/data2/brick</div= href=3D"http://performance.io" class=3D"">performance.io</a>-cache: = off</div><div class=3D"">performance.stat-prefetch: off</div><div = class=3D"">cluster.eager-lock: enable</div><div = class=3D"">network.remote-dio: enable</div><div = class=3D"">cluster.quorum-type: auto</div><div = class=3D"">cluster.server-quorum-type: server</div><div = class=3D"">storage.owner-uid: 36</div><div class=3D"">storage.owner-gid: = 36</div></div><div class=3D""><br class=3D""></div><div class=3D""><br = class=3D""></div><div class=3D"">See attached file for the mount = log.</div><div class=3D""><br class=3D""></div><div = class=3D"">Gary</div><div class=3D""><br class=3D""></div><div = class=3D""></div></body></html>= --Apple-Mail=_D3985462-55E4-4FE5-A078-30EC039EC062 Content-Disposition: attachment; filename=glusterlog.txt Content-Type: text/plain; x-unix-mode=0644; name="glusterlog.txt" Content-Transfer-Encoding: quoted-printable more rhev-data-center-mnt-glusterSD-glustermount2:data2.log-20161106 [2016-11-04 22:54:32.118587] I [MSGID: 100030] [glusterfsd.c:2338:main] = 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version = 3.7.16 (args: /usr/sbin/glusterfs --volfile-server=3Dglustermou nt2 --volfile-server=3Dfai-kvm-1-vmn.ravnalaska.net = --volfile-server=3Dfai-kvm-2-vmn.ravnalaska.net = --volfile-server=3Dfai-kvm-3-vmn.ravnalaska.net = --volfile-server=3Dfai-kvm-4-vmn.ravnalaska.net --volfile-id =3Ddata2 /rhev/data-center/mnt/glusterSD/glustermount2:data2) [2016-11-04 22:54:32.128807] I [MSGID: 101190] = [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread = with index 1 [2016-11-04 22:54:32.138959] I [MSGID: 101190] = [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread = with index 2 [2016-11-04 22:54:32.139878] I [MSGID: 114020] [client.c:2113:notify] = 0-data2-client-0: parent translators are ready, attempting connect on = transport [2016-11-04 22:54:32.142057] I [MSGID: 114020] [client.c:2113:notify] = 0-data2-client-1: parent translators are ready, attempting connect on = transport [2016-11-04 22:54:32.142331] I [rpc-clnt.c:1960:rpc_clnt_reconfig] = 0-data2-client-0: changing port to 49156 (from 0) [2016-11-04 22:54:32.143961] I [MSGID: 114020] [client.c:2113:notify] = 0-data2-client-2: parent translators are ready, attempting connect on = transport [2016-11-04 22:54:32.146123] I [MSGID: 114057] = [client-handshake.c:1437:select_server_supported_programs] = 0-data2-client-0: Using Program GlusterFS 3.3, Num (1298437), Version = (330) [2016-11-04 22:54:32.146308] I [rpc-clnt.c:1960:rpc_clnt_reconfig] = 0-data2-client-1: changing port to 49156 (from 0) [2016-11-04 22:54:32.147618] I [MSGID: 114020] [client.c:2113:notify] = 0-data2-client-3: parent translators are ready, attempting connect on = transport [2016-11-04 22:54:32.149874] I [MSGID: 114046] = [client-handshake.c:1213:client_setvolume_cbk] 0-data2-client-0: = Connected to data2-client-0, attached to remote volume = '/kvm2/gluster/data2/brick'. [2016-11-04 22:54:32.149891] I [MSGID: 114047] = [client-handshake.c:1224:client_setvolume_cbk] 0-data2-client-0: Server = and Client lk-version numbers are not same, reopening the fds [2016-11-04 22:54:32.149941] I [MSGID: 108005] = [afr-common.c:4299:afr_notify] 0-data2-replicate-0: Subvolume = 'data2-client-0' came back up; going online. [2016-11-04 22:54:32.150000] I [MSGID: 114035] = [client-handshake.c:193:client_set_lk_version_cbk] 0-data2-client-0: = Server lk version =3D 1 [2016-11-04 22:54:32.150080] I [MSGID: 114057] = [client-handshake.c:1437:select_server_supported_programs] = 0-data2-client-1: Using Program GlusterFS 3.3, Num (1298437), Version = (330) [2016-11-04 22:54:32.150274] I [rpc-clnt.c:1960:rpc_clnt_reconfig] = 0-data2-client-2: changing port to 49156 (from 0) Final graph: = +-------------------------------------------------------------------------= -----+ 1: volume data2-client-0 2: type protocol/client 3: option clnt-lk-version 1 4: option volfile-checksum 0 5: option volfile-key data2 6: option client-version 3.7.16 7: option process-uuid = fai-kvm-1-gfs.ravnalaska.net-105264-2016/11/04-22:54:32:114962-data2-clien= t-0-0-0 8: option fops-version 1298437 9: option ping-timeout 42 10: option remote-host fai-kvm-1-vmn.ravnalaska.net 11: option remote-subvolume /kvm2/gluster/data2/brick 12: option transport-type socket 13: option username 0b7a59b1-4aab-4041-a271-bd8a07e5cba5 14: option password 6dca1e9d-f6a8-490b-bb60-babc56eb131a 15: option filter-O_DIRECT enable 16: option send-gids true 17: end-volume 18: 19: volume data2-client-1 20: type protocol/client 21: option clnt-lk-version 1 22: option volfile-checksum 0 23: option volfile-key data2 24: option client-version 3.7.16 25: option process-uuid = fai-kvm-1-gfs.ravnalaska.net-105264-2016/11/04-22:54:32:114962-data2-clien= t-1-0-0 26: option fops-version 1298437 27: option ping-timeout 42 28: option remote-host fai-kvm-2-vmn.ravnalaska.net 29: option remote-subvolume /kvm2/gluster/data2/brick 30: option transport-type socket 31: option username 0b7a59b1-4aab-4041-a271-bd8a07e5cba5 32: option password 6dca1e9d-f6a8-490b-bb60-babc56eb131a 33: option filter-O_DIRECT enable 34: option send-gids true 35: end-volume 36: 37: volume data2-replicate-0 38: type cluster/replicate 39: option eager-lock enable 40: option quorum-type auto 41: subvolumes data2-client-0 data2-client-1 42: end-volume 43: 44: volume data2-client-2 45: type protocol/client 46: option ping-timeout 42 47: option remote-host fai-kvm-3-vmn.ravnalaska.net 48: option remote-subvolume /kvm2/gluster/data2/brick 49: option transport-type socket 50: option username 0b7a59b1-4aab-4041-a271-bd8a07e5cba5 51: option password 6dca1e9d-f6a8-490b-bb60-babc56eb131a 52: option filter-O_DIRECT enable 53: option send-gids true 54: end-volume 55: 56: volume data2-client-3 57: type protocol/client 58: option ping-timeout 42 59: option remote-host fai-kvm-4-vmn.ravnalaska.net 60: option remote-subvolume /kvm2/gluster/data2/brick 61: option transport-type socket 62: option username 0b7a59b1-4aab-4041-a271-bd8a07e5cba5 63: option password 6dca1e9d-f6a8-490b-bb60-babc56eb131a 64: option filter-O_DIRECT enable 65: option send-gids true 66: end-volume 67: 68: volume data2-replicate-1 69: type cluster/replicate 70: option eager-lock enable 71: option quorum-type auto 72: subvolumes data2-client-2 data2-client-3 73: end-volume 74: 75: volume data2-stripe-0 76: type cluster/stripe 77: subvolumes data2-replicate-0 data2-replicate-1 78: end-volume 79: 80: volume data2-dht 81: type cluster/distribute 82: subvolumes data2-stripe-0 83: end-volume 84: 85: volume data2-write-behind 86: type performance/write-behind 87: subvolumes data2-dht 88: end-volume 89: 90: volume data2-readdir-ahead 91: type performance/readdir-ahead 92: subvolumes data2-write-behind 93: end-volume 94: 95: volume data2-open-behind 96: type performance/open-behind 97: subvolumes data2-readdir-ahead 98: end-volume 99: 100: volume data2 101: type debug/io-stats 102: option log-level INFO 103: option latency-measurement off 104: option count-fop-hits off 105: subvolumes data2-open-behind 106: end-volume 107: 108: volume meta-autoload 109: type meta 110: subvolumes data2 111: end-volume 112: = +-------------------------------------------------------------------------= -----+ [2016-11-04 22:54:32.151927] I [MSGID: 114046] = [client-handshake.c:1213:client_setvolume_cbk] 0-data2-client-1: = Connected to data2-client-1, attached to remote volume = '/kvm2/gluster/data2/brick'. [2016-11-04 22:54:32.151939] I [MSGID: 114047] = [client-handshake.c:1224:client_setvolume_cbk] 0-data2-client-1: Server = and Client lk-version numbers are not same, reopening the fds [2016-11-04 22:54:32.152083] I [MSGID: 114035] = [client-handshake.c:193:client_set_lk_version_cbk] 0-data2-client-1: = Server lk version =3D 1 [2016-11-04 22:54:32.152341] I [rpc-clnt.c:1960:rpc_clnt_reconfig] = 0-data2-client-3: changing port to 49152 (from 0) [2016-11-04 22:54:32.154296] I [MSGID: 114057] = [client-handshake.c:1437:select_server_supported_programs] = 0-data2-client-2: Using Program GlusterFS 3.3, Num (1298437), Version = (330) [2016-11-04 22:54:32.154774] I [MSGID: 114046] = [client-handshake.c:1213:client_setvolume_cbk] 0-data2-client-2: = Connected to data2-client-2, attached to remote volume = '/kvm2/gluster/data2/brick'. [2016-11-04 22:54:32.154789] I [MSGID: 114047] = [client-handshake.c:1224:client_setvolume_cbk] 0-data2-client-2: Server = and Client lk-version numbers are not same, reopening the fds [2016-11-04 22:54:32.154821] I [MSGID: 108005] = [afr-common.c:4299:afr_notify] 0-data2-replicate-1: Subvolume = 'data2-client-2' came back up; going online. [2016-11-04 22:54:32.154916] I [MSGID: 114035] = [client-handshake.c:193:client_set_lk_version_cbk] 0-data2-client-2: = Server lk version =3D 1 [2016-11-04 22:54:32.156348] I [MSGID: 114057] = [client-handshake.c:1437:select_server_supported_programs] = 0-data2-client-3: Using Program GlusterFS 3.3, Num (1298437), Version = (330) [2016-11-04 22:54:32.156860] I [MSGID: 114046] = [client-handshake.c:1213:client_setvolume_cbk] 0-data2-client-3: = Connected to data2-client-3, attached to remote volume = '/kvm2/gluster/data2/brick'. [2016-11-04 22:54:32.156874] I [MSGID: 114047] = [client-handshake.c:1224:client_setvolume_cbk] 0-data2-client-3: Server = and Client lk-version numbers are not same, reopening the fds [2016-11-04 22:54:32.160382] I [fuse-bridge.c:5171:fuse_graph_setup] = 0-fuse: switched to graph 0 [2016-11-04 22:54:32.160451] I [MSGID: 114035] = [client-handshake.c:193:client_set_lk_version_cbk] 0-data2-client-3: = Server lk version =3D 1 [2016-11-04 22:54:32.160524] I [fuse-bridge.c:4083:fuse_init] = 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 = kernel 7.22 [2016-11-04 22:54:32.161171] I [MSGID: 108031] = [afr-common.c:2071:afr_local_discovery_cbk] 0-data2-replicate-0: = selecting local read_child data2-client-0 [2016-11-05 00:05:37.796730] W [glusterfsd.c:1251:cleanup_and_exit] = (-->/lib64/libpthread.so.0(+0x7dc5) [0x7fe90527adc5] = -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fe9068f5915] = -->/usr/sbin/ glusterfs(cleanup_and_exit+0x6b) [0x7fe9068f578b] ) 0-: received signum = (15), shutting down [2016-11-05 00:05:37.796770] I [fuse-bridge.c:5719:fini] 0-fuse: = Unmounting '/rhev/data-center/mnt/glusterSD/glustermount2:data2'. [2016-11-05 00:13:04.440431] I [MSGID: 100030] [glusterfsd.c:2338:main] = 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version = 3.7.16 (args: /usr/sbin/glusterfs --volfile-server=3Dglustermou nt2 --volfile-server=3Dfai-kvm-1-vmn.ravnalaska.net = --volfile-server=3Dfai-kvm-2-vmn.ravnalaska.net = --volfile-server=3Dfai-kvm-3-vmn.ravnalaska.net = --volfile-server=3Dfai-kvm-4-vmn.ravnalaska.net --volfile-id =3Ddata2 /rhev/data-center/mnt/glusterSD/glustermount2:data2) [2016-11-05 00:13:04.452151] I [MSGID: 101190] = [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread = with index 1 [2016-11-05 00:13:04.463593] I [MSGID: 101190] = [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread = with index 2 [2016-11-05 00:13:04.464394] I [MSGID: 114020] [client.c:2113:notify] = 0-data2-client-0: parent translators are ready, attempting connect on = transport [2016-11-05 00:13:04.466698] I [MSGID: 114020] [client.c:2113:notify] = 0-data2-client-1: parent translators are ready, attempting connect on = transport [2016-11-05 00:13:04.466885] I [rpc-clnt.c:1960:rpc_clnt_reconfig] = 0-data2-client-0: changing port to 49156 (from 0) [2016-11-05 00:13:04.468672] I [MSGID: 114020] [client.c:2113:notify] = 0-data2-client-2: parent translators are ready, attempting connect on = transport [2016-11-05 00:13:04.470686] I [MSGID: 114057] = [client-handshake.c:1437:select_server_supported_programs] = 0-data2-client-0: Using Program GlusterFS 3.3, Num (1298437), Version = (330) [2016-11-05 00:13:04.470849] I [rpc-clnt.c:1960:rpc_clnt_reconfig] = 0-data2-client-1: changing port to 49156 (from 0) [2016-11-05 00:13:04.472375] I [MSGID: 114020] [client.c:2113:notify] = 0-data2-client-3: parent translators are ready, attempting connect on = transport [2016-11-05 00:13:04.475429] I [MSGID: 114046] = [client-handshake.c:1213:client_setvolume_cbk] 0-data2-client-0: = Connected to data2-client-0, attached to remote volume = '/kvm2/gluster/data2/brick'. [2016-11-05 00:13:04.475448] I [MSGID: 114047] = [client-handshake.c:1224:client_setvolume_cbk] 0-data2-client-0: Server = and Client lk-version numbers are not same, reopening the fds [2016-11-05 00:13:04.475503] I [MSGID: 108005] = [afr-common.c:4299:afr_notify] 0-data2-replicate-0: Subvolume = 'data2-client-0' came back up; going online. [2016-11-05 00:13:04.475615] I [MSGID: 114035] = [client-handshake.c:193:client_set_lk_version_cbk] 0-data2-client-0: = Server lk version =3D 1 [2016-11-05 00:13:04.475687] I [MSGID: 114057] = [client-handshake.c:1437:select_server_supported_programs] = 0-data2-client-1: Using Program GlusterFS 3.3, Num (1298437), Version = (330) [2016-11-05 00:13:04.475857] I [rpc-clnt.c:1960:rpc_clnt_reconfig] = 0-data2-client-2: changing port to 49156 (from 0) Final graph: = +-------------------------------------------------------------------------= -----+ 1: volume data2-client-0 2: type protocol/client 3: option clnt-lk-version 1 4: option volfile-checksum 0 5: option volfile-key data2 6: option client-version 3.7.16 7: option process-uuid = fai-kvm-1-gfs.ravnalaska.net-13785-2016/11/05-00:13:04:437247-data2-client= -0-0-0 8: option fops-version 1298437 9: option ping-timeout 42 10: option remote-host fai-kvm-1-vmn.ravnalaska.net 11: option remote-subvolume /kvm2/gluster/data2/brick 12: option transport-type socket 13: option username 0b7a59b1-4aab-4041-a271-bd8a07e5cba5 14: option password 6dca1e9d-f6a8-490b-bb60-babc56eb131a 15: option filter-O_DIRECT enable 16: option send-gids true 17: end-volume 18: 19: volume data2-client-1 20: type protocol/client 21: option clnt-lk-version 1 22: option volfile-checksum 0 23: option volfile-key data2 24: option client-version 3.7.16 25: option process-uuid = fai-kvm-1-gfs.ravnalaska.net-13785-2016/11/05-00:13:04:437247-data2-client= -1-0-0 26: option fops-version 1298437 27: option ping-timeout 42 28: option remote-host fai-kvm-2-vmn.ravnalaska.net 29: option remote-subvolume /kvm2/gluster/data2/brick 30: option transport-type socket 31: option username 0b7a59b1-4aab-4041-a271-bd8a07e5cba5 32: option password 6dca1e9d-f6a8-490b-bb60-babc56eb131a 33: option filter-O_DIRECT enable 34: option send-gids true 35: end-volume 36: 37: volume data2-replicate-0 38: type cluster/replicate 39: option eager-lock enable 40: option quorum-type auto 41: subvolumes data2-client-0 data2-client-1 42: end-volume 43: 44: volume data2-client-2 45: type protocol/client 46: option ping-timeout 42 47: option remote-host fai-kvm-3-vmn.ravnalaska.net 48: option remote-subvolume /kvm2/gluster/data2/brick 49: option transport-type socket 50: option username 0b7a59b1-4aab-4041-a271-bd8a07e5cba5 51: option password 6dca1e9d-f6a8-490b-bb60-babc56eb131a 52: option filter-O_DIRECT enable 53: option send-gids true 54: end-volume 55: 56: volume data2-client-3 57: type protocol/client 58: option ping-timeout 42 59: option remote-host fai-kvm-4-vmn.ravnalaska.net 60: option remote-subvolume /kvm2/gluster/data2/brick 61: option transport-type socket 62: option username 0b7a59b1-4aab-4041-a271-bd8a07e5cba5 63: option password 6dca1e9d-f6a8-490b-bb60-babc56eb131a 64: option filter-O_DIRECT enable 65: option send-gids true 66: end-volume 67: 68: volume data2-replicate-1 69: type cluster/replicate 70: option eager-lock enable 71: option quorum-type auto 72: subvolumes data2-client-2 data2-client-3 73: end-volume 74: 75: volume data2-stripe-0 76: type cluster/stripe 77: subvolumes data2-replicate-0 data2-replicate-1 78: end-volume 79: 80: volume data2-dht 81: type cluster/distribute 82: subvolumes data2-stripe-0 83: end-volume 84: 85: volume data2-write-behind 86: type performance/write-behind 87: subvolumes data2-dht 88: end-volume 89: 90: volume data2-readdir-ahead 91: type performance/readdir-ahead 92: subvolumes data2-write-behind 93: end-volume 94: 95: volume data2-open-behind 96: type performance/open-behind 97: subvolumes data2-readdir-ahead 98: end-volume 99: 100: volume data2 101: type debug/io-stats 102: option log-level INFO 103: option latency-measurement off 104: option count-fop-hits off 105: subvolumes data2-open-behind 106: end-volume 107: 108: volume meta-autoload 109: type meta 110: subvolumes data2 111: end-volume 112: = +-------------------------------------------------------------------------= -----+ [2016-11-05 00:13:04.477682] I [MSGID: 114046] = [client-handshake.c:1213:client_setvolume_cbk] 0-data2-client-1: = Connected to data2-client-1, attached to remote volume = '/kvm2/gluster/data2/brick'. [2016-11-05 00:13:04.477702] I [MSGID: 114047] = [client-handshake.c:1224:client_setvolume_cbk] 0-data2-client-1: Server = and Client lk-version numbers are not same, reopening the fds [2016-11-05 00:13:04.477834] I [MSGID: 114035] = [client-handshake.c:193:client_set_lk_version_cbk] 0-data2-client-1: = Server lk version =3D 1 [2016-11-05 00:13:04.478162] I [rpc-clnt.c:1960:rpc_clnt_reconfig] = 0-data2-client-3: changing port to 49152 (from 0) [2016-11-05 00:13:04.480593] I [MSGID: 114057] = [client-handshake.c:1437:select_server_supported_programs] = 0-data2-client-2: Using Program GlusterFS 3.3, Num (1298437), Version = (330) [2016-11-05 00:13:04.480961] I [MSGID: 114046] = [client-handshake.c:1213:client_setvolume_cbk] 0-data2-client-2: = Connected to data2-client-2, attached to remote volume = '/kvm2/gluster/data2/brick'. [2016-11-05 00:13:04.480982] I [MSGID: 114047] = [client-handshake.c:1224:client_setvolume_cbk] 0-data2-client-2: Server = and Client lk-version numbers are not same, reopening the fds [2016-11-05 00:13:04.481020] I [MSGID: 108005] = [afr-common.c:4299:afr_notify] 0-data2-replicate-1: Subvolume = 'data2-client-2' came back up; going online. [2016-11-05 00:13:04.481120] I [MSGID: 114035] = [client-handshake.c:193:client_set_lk_version_cbk] 0-data2-client-2: = Server lk version =3D 1 [2016-11-05 00:13:04.483500] I [MSGID: 114057] = [client-handshake.c:1437:select_server_supported_programs] = 0-data2-client-3: Using Program GlusterFS 3.3, Num (1298437), Version = (330) [2016-11-05 00:13:04.483956] I [MSGID: 114046] = [client-handshake.c:1213:client_setvolume_cbk] 0-data2-client-3: = Connected to data2-client-3, attached to remote volume = '/kvm2/gluster/data2/brick'. [2016-11-05 00:13:04.483973] I [MSGID: 114047] = [client-handshake.c:1224:client_setvolume_cbk] 0-data2-client-3: Server = and Client lk-version numbers are not same, reopening the fds [2016-11-05 00:13:04.488332] I [fuse-bridge.c:5171:fuse_graph_setup] = 0-fuse: switched to graph 0 [2016-11-05 00:13:04.488889] I [MSGID: 114035] = [client-handshake.c:193:client_set_lk_version_cbk] 0-data2-client-3: = Server lk version =3D 1 [2016-11-05 00:13:04.488933] I [fuse-bridge.c:4083:fuse_init] = 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 = kernel 7.22 [2016-11-05 00:13:04.489627] I [MSGID: 108031] = [afr-common.c:2071:afr_local_discovery_cbk] 0-data2-replicate-0: = selecting local read_child data2-client-0 [2016-11-05 00:52:07.549997] I [fuse-bridge.c:5012:fuse_thread_proc] = 0-fuse: unmounting /rhev/data-center/mnt/glusterSD/glustermount2:data2 [2016-11-05 00:52:07.550544] W [glusterfsd.c:1251:cleanup_and_exit] = (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f40b1c24dc5] = -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f40b329f915] = -->/usr/sbin/ glusterfs(cleanup_and_exit+0x6b) [0x7f40b329f78b] ) 0-: received signum = (15), shutting down [2016-11-05 00:52:07.550583] I [fuse-bridge.c:5719:fini] 0-fuse: = Unmounting '/rhev/data-center/mnt/glusterSD/glustermount2:data2'. [2016-11-05 00:52:39.557750] I [MSGID: 100030] [glusterfsd.c:2338:main] = 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version = 3.7.16 (args: /usr/sbin/glusterfs --volfile-server=3Dglustermou nt2 --volfile-server=3Dfai-kvm-1-vmn.ravnalaska.net = --volfile-server=3Dfai-kvm-2-vmn.ravnalaska.net = --volfile-server=3Dfai-kvm-3-vmn.ravnalaska.net = --volfile-server=3Dfai-kvm-4-vmn.ravnalaska.net --volfile-id =3Ddata2 /rhev/data-center/mnt/glusterSD/glustermount2:data2) [2016-11-05 00:52:39.567614] I [MSGID: 101190] = [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread = with index 1 [2016-11-05 00:52:39.578148] I [MSGID: 101190] = [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread = with index 2 [2016-11-05 00:52:39.579039] I [MSGID: 114020] [client.c:2113:notify] = 0-data2-client-0: parent translators are ready, attempting connect on = transport [2016-11-05 00:52:39.581159] I [MSGID: 114020] [client.c:2113:notify] = 0-data2-client-1: parent translators are ready, attempting connect on = transport [2016-11-05 00:52:39.581483] I [rpc-clnt.c:1960:rpc_clnt_reconfig] = 0-data2-client-0: changing port to 49156 (from 0) [2016-11-05 00:52:39.583109] I [MSGID: 114020] [client.c:2113:notify] = 0-data2-client-2: parent translators are ready, attempting connect on = transport [2016-11-05 00:52:39.585673] I [MSGID: 114057] = [client-handshake.c:1437:select_server_supported_programs] = 0-data2-client-0: Using Program GlusterFS 3.3, Num (1298437), Version = (330) [2016-11-05 00:52:39.585849] I [rpc-clnt.c:1960:rpc_clnt_reconfig] = 0-data2-client-1: changing port to 49156 (from 0) [2016-11-05 00:52:39.587044] I [MSGID: 114020] [client.c:2113:notify] = 0-data2-client-3: parent translators are ready, attempting connect on = transport [2016-11-05 00:52:39.589467] I [MSGID: 114046] = [client-handshake.c:1213:client_setvolume_cbk] 0-data2-client-0: = Connected to data2-client-0, attached to remote volume = '/kvm2/gluster/data2/brick'. [2016-11-05 00:52:39.589499] I [MSGID: 114047] = [client-handshake.c:1224:client_setvolume_cbk] 0-data2-client-0: Server = and Client lk-version numbers are not same, reopening the fds [2016-11-05 00:52:39.589571] I [MSGID: 108005] = [afr-common.c:4299:afr_notify] 0-data2-replicate-0: Subvolume = 'data2-client-0' came back up; going online. [2016-11-05 00:52:39.589654] I [MSGID: 114035] = [client-handshake.c:193:client_set_lk_version_cbk] 0-data2-client-0: = Server lk version =3D 1 [2016-11-05 00:52:39.589753] I [MSGID: 114057] = [client-handshake.c:1437:select_server_supported_programs] = 0-data2-client-1: Using Program GlusterFS 3.3, Num (1298437), Version = (330) [2016-11-05 00:52:39.590026] I [rpc-clnt.c:1960:rpc_clnt_reconfig] = 0-data2-client-2: changing port to 49156 (from 0) [2016-11-05 00:52:39.590074] I [MSGID: 114046] = [client-handshake.c:1213:client_setvolume_cbk] 0-data2-client-1: = Connected to data2-client-1, attached to remote volume = '/kvm2/gluster/data2/brick'. [2016-11-05 00:52:39.590088] I [MSGID: 114047] = [client-handshake.c:1224:client_setvolume_cbk] 0-data2-client-1: Server = and Client lk-version numbers are not same, reopening the fds Final graph: = +-------------------------------------------------------------------------= -----+ 1: volume data2-client-0 2: type protocol/client 3: option clnt-lk-version 1 4: option volfile-checksum 0 5: option volfile-key data2 6: option client-version 3.7.16 7: option process-uuid = fai-kvm-1-gfs.ravnalaska.net-30520-2016/11/05-00:52:39:553646-data2-client= -0-0-0 8: option fops-version 1298437 9: option ping-timeout 42 10: option remote-host fai-kvm-1-vmn.ravnalaska.net 11: option remote-subvolume /kvm2/gluster/data2/brick 12: option transport-type socket 13: option username 0b7a59b1-4aab-4041-a271-bd8a07e5cba5 14: option password 6dca1e9d-f6a8-490b-bb60-babc56eb131a 15: option filter-O_DIRECT enable 16: option send-gids true 17: end-volume 18: 19: volume data2-client-1 20: type protocol/client 21: option clnt-lk-version 1 22: option volfile-checksum 0 23: option volfile-key data2 24: option client-version 3.7.16 25: option process-uuid = fai-kvm-1-gfs.ravnalaska.net-30520-2016/11/05-00:52:39:553646-data2-client= -1-0-0 26: option fops-version 1298437 27: option ping-timeout 42 28: option remote-host fai-kvm-2-vmn.ravnalaska.net 29: option remote-subvolume /kvm2/gluster/data2/brick 30: option transport-type socket 31: option username 0b7a59b1-4aab-4041-a271-bd8a07e5cba5 32: option password 6dca1e9d-f6a8-490b-bb60-babc56eb131a 33: option filter-O_DIRECT enable 34: option send-gids true 35: end-volume 36: 37: volume data2-replicate-0 38: type cluster/replicate 39: option eager-lock enable 40: option quorum-type auto 41: subvolumes data2-client-0 data2-client-1 42: end-volume 43: 44: volume data2-client-2 45: type protocol/client 46: option ping-timeout 42 47: option remote-host fai-kvm-3-vmn.ravnalaska.net 48: option remote-subvolume /kvm2/gluster/data2/brick 49: option transport-type socket 50: option username 0b7a59b1-4aab-4041-a271-bd8a07e5cba5 51: option password 6dca1e9d-f6a8-490b-bb60-babc56eb131a 52: option filter-O_DIRECT enable 53: option send-gids true 54: end-volume 55: 56: volume data2-client-3 57: type protocol/client 58: option ping-timeout 42 59: option remote-host fai-kvm-4-vmn.ravnalaska.net 60: option remote-subvolume /kvm2/gluster/data2/brick 61: option transport-type socket 62: option username 0b7a59b1-4aab-4041-a271-bd8a07e5cba5 63: option password 6dca1e9d-f6a8-490b-bb60-babc56eb131a 64: option filter-O_DIRECT enable 65: option send-gids true 66: end-volume 67: 68: volume data2-replicate-1 69: type cluster/replicate 70: option eager-lock enable 71: option quorum-type auto 72: subvolumes data2-client-2 data2-client-3 73: end-volume 74: 75: volume data2-stripe-0 76: type cluster/stripe 77: subvolumes data2-replicate-0 data2-replicate-1 78: end-volume 79: 80: volume data2-dht 81: type cluster/distribute 82: subvolumes data2-stripe-0 83: end-volume 84: 85: volume data2-write-behind 86: type performance/write-behind 87: subvolumes data2-dht 88: end-volume 89: 90: volume data2-readdir-ahead 91: type performance/readdir-ahead 92: subvolumes data2-write-behind 93: end-volume 94: 95: volume data2-open-behind 96: type performance/open-behind 97: subvolumes data2-readdir-ahead 98: end-volume 99: 100: volume data2 101: type debug/io-stats 102: option log-level INFO 103: option latency-measurement off 104: option count-fop-hits off 105: subvolumes data2-open-behind 106: end-volume 107: 108: volume meta-autoload 109: type meta 110: subvolumes data2 111: end-volume 112: = +-------------------------------------------------------------------------= -----+ [2016-11-05 00:52:39.591502] I [MSGID: 114035] = [client-handshake.c:193:client_set_lk_version_cbk] 0-data2-client-1: = Server lk version =3D 1 [2016-11-05 00:52:39.591817] I [rpc-clnt.c:1960:rpc_clnt_reconfig] = 0-data2-client-3: changing port to 49152 (from 0) [2016-11-05 00:52:39.593714] I [MSGID: 114057] = [client-handshake.c:1437:select_server_supported_programs] = 0-data2-client-2: Using Program GlusterFS 3.3, Num (1298437), Version = (330) [2016-11-05 00:52:39.594285] I [MSGID: 114046] = [client-handshake.c:1213:client_setvolume_cbk] 0-data2-client-2: = Connected to data2-client-2, attached to remote volume = '/kvm2/gluster/data2/brick'. [2016-11-05 00:52:39.594308] I [MSGID: 114047] = [client-handshake.c:1224:client_setvolume_cbk] 0-data2-client-2: Server = and Client lk-version numbers are not same, reopening the fds [2016-11-05 00:52:39.594365] I [MSGID: 108005] = [afr-common.c:4299:afr_notify] 0-data2-replicate-1: Subvolume = 'data2-client-2' came back up; going online. [2016-11-05 00:52:39.594552] I [MSGID: 114035] = [client-handshake.c:193:client_set_lk_version_cbk] 0-data2-client-2: = Server lk version =3D 1 [2016-11-05 00:52:39.595616] I [MSGID: 114057] = [client-handshake.c:1437:select_server_supported_programs] = 0-data2-client-3: Using Program GlusterFS 3.3, Num (1298437), Version = (330) [2016-11-05 00:52:39.596115] I [MSGID: 114046] = [client-handshake.c:1213:client_setvolume_cbk] 0-data2-client-3: = Connected to data2-client-3, attached to remote volume = '/kvm2/gluster/data2/brick'. [2016-11-05 00:52:39.596130] I [MSGID: 114047] = [client-handshake.c:1224:client_setvolume_cbk] 0-data2-client-3: Server = and Client lk-version numbers are not same, reopening the fds [2016-11-05 00:52:39.600510] I [fuse-bridge.c:5171:fuse_graph_setup] = 0-fuse: switched to graph 0 [2016-11-05 00:52:39.600578] I [MSGID: 114035] = [client-handshake.c:193:client_set_lk_version_cbk] 0-data2-client-3: = Server lk version =3D 1 [2016-11-05 00:52:39.600654] I [fuse-bridge.c:4083:fuse_init] = 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 = kernel 7.22 [2016-11-05 00:52:39.601452] I [MSGID: 108031] = [afr-common.c:2071:afr_local_discovery_cbk] 0-data2-replicate-0: = selecting local read_child data2-client-0 [2016-11-05 00:52:41.619852] I [fuse-bridge.c:5012:fuse_thread_proc] = 0-fuse: unmounting /rhev/data-center/mnt/glusterSD/glustermount2:data2 [2016-11-05 00:52:41.620095] W [glusterfsd.c:1251:cleanup_and_exit] = (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f3444227dc5] = -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f34458a2915] = -->/usr/sbin/ glusterfs(cleanup_and_exit+0x6b) [0x7f34458a278b] ) 0-: received signum = (15), shutting down [2016-11-05 00:52:41.620110] I [fuse-bridge.c:5719:fini] 0-fuse: = Unmounting '/rhev/data-center/mnt/glusterSD/glustermount2:data2'. [root@fai-kvm-1-gfs glusterfs]#= --Apple-Mail=_D3985462-55E4-4FE5-A078-30EC039EC062 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div class=3D""></div><div class=3D""> <div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; orphans: = auto; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"orphans: auto; text-align: start; text-indent: = 0px; widows: auto; word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;" class=3D""><div style=3D"orphans: = auto; text-align: start; text-indent: 0px; widows: auto; word-wrap: = break-word; -webkit-nbsp-mode: space; -webkit-line-break: = after-white-space;" class=3D""><div style=3D"orphans: auto; text-align: = start; text-indent: 0px; widows: auto; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"orphans: auto; text-align: start; text-indent: = 0px; widows: auto; word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;" class=3D""><div style=3D"orphans: = auto; text-align: start; text-indent: 0px; widows: auto; word-wrap: = break-word; -webkit-nbsp-mode: space; -webkit-line-break: = after-white-space;" class=3D""><div style=3D"orphans: auto; text-align: = start; text-indent: 0px; widows: auto; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"orphans: auto; text-align: start; text-indent: = 0px; widows: auto; word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;" class=3D""><font face=3D"Menlo" = style=3D"color: rgb(0, 0, 0); font-size: 12px; letter-spacing: normal; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px;" class=3D""><div = class=3D"">---------------------------------------------------------------= ---------</div><div class=3D"">Gary Pedretty = = <a = href=3D"mailto:gary@eraalaska.net" = class=3D"">gary@ravnalaska.net</a></div><div class=3D"">Systems Manager = = = <a href=3D"http://www.flyravn.com" = class=3D"">www.flyravn.com</a></div><div class=3D"">Ravn Alaska = = /\ = 907-450-7251</div><div class=3D"">5245 Airport Industrial = Road / \/\ = 907-450-7238 fax</div><div class=3D"">Fairbanks, Alaska = 99709 /\ / \ \ = Second greatest commandment</div></font><font face=3D"Monaco" = class=3D""><span style=3D"font-size: 12px;" class=3D"">Serving All of = Alaska / \/ /\ \ \/\ = =E2=80=9CLove your neighbor as</span></font><br = style=3D"font-family: Monaco;" class=3D""><font face=3D"Menlo" = style=3D"color: rgb(0, 0, 0); letter-spacing: normal; text-transform: = none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: = 0px;" class=3D""><span style=3D"font-size: 12px;" class=3D"">Really = loving the record green up date! Summmer!! yourself=E2=80=9D M= att 22:39</span></font><div style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; font-family: Menlo;" = class=3D""></div><font face=3D"Menlo" style=3D"font-size: 12px;" = class=3D""></font><span style=3D"color: rgb(0, 0, 0); letter-spacing: = normal; text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; font-size: 12px;" class=3D""><font = face=3D"Menlo" class=3D""><div = class=3D"">---------------------------------------------------------------= ---------</div></font></span><div style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><font = face=3D"Menlo" style=3D"font-size: 12px;" class=3D""><br = class=3D""></font></div></div><span style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; font-size: 12px;" = class=3D""><br class=3D"Apple-interchange-newline"></span></div><span = style=3D"color: rgb(0, 0, 0); letter-spacing: normal; text-transform: = none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: = 0px; font-size: 12px;" class=3D""><br = class=3D"Apple-interchange-newline"></span></div><span style=3D"color: = rgb(0, 0, 0); letter-spacing: normal; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; font-size: = 12px;" class=3D""><br class=3D"Apple-interchange-newline"></span></div><br= class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"><br = class=3D"Apple-interchange-newline"> </div> <br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On Nov 6, 2016, at 9:50 PM, Sahina Bose <<a = href=3D"mailto:sabose@redhat.com" class=3D"">sabose@redhat.com</a>> = wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><span = style=3D"font-family: LucidaGrande; font-size: 13px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px; float: none; display: inline = !important;" class=3D"">However your volume configuration seems suspect = -"stripe 2 replica 2". Can you provide gluster volume info of your = second storage domain gluster volume? The mount logs of the volume = (under /var/log/glusterfs/rhev-datacenter..<volname>.log) from the = host where the volume is being mounted will also help.</span><br = style=3D"font-family: LucidaGrande; font-size: 13px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px;" class=3D""><span style=3D"font-family: = LucidaGrande; font-size: 13px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; = float: none; display: inline !important;" = class=3D""> </span></div></blockquote></div><br = class=3D""></body></html>= --Apple-Mail=_D3985462-55E4-4FE5-A078-30EC039EC062-- --Apple-Mail=_74966C91-4D7B-4631-992A-7104BF8281C2--